Wednesday, July 18, 2012

A second look at probability theory

I'm studying Principles of Communication Engineering by Wozencraft and Jacobs and the second chapter goes over probability theory, except it uses measure theoretic probability theory.

Motivation

Example: Toss a coin, have it come up heads, and if it comes up
tails, pick a random number in the range [0,1]. How do you come up
with a PDF/CDF?
Further reading: Look at the excerpt from A User's Guide to Measure Theoretic Probability.
Related video: Why use measure theory for probability?
So, measure-theoretic probability unified the discrete and continuous cases.

First approach

Pick a set of all possible outcomes Ω (capital omega). When
flipping coins, this set is {H,T}, and when rolling dice this set
is {1,2,3,4,5,6}.
Give a function $$P$$ that assigns a probability to arbitrary
subsets of Ω. You can write this as: $$P : 2^{\Omega} \to [0,1]$$.
The $$2^X$$ notation denotes the power set of $$X$$.
We'd like this function to behave nicely and obey some intuitive laws:
1. $$P(\Omega)=1$$
2. For every event $$A \in \Omega$$, $$P(A) \ge 0$$
3. For any two disjoint events $$A,B$$, $$P(A \cup B)=P(A)+P(B)$$
All scenarios we consider will be some subset of Ω. For example,
$$P(\text{roll even number}) = P(\{2,4,6\})$$. If $$P$$ assigned
a number to all subsets, then we'd be happy.
However, for the continuous case, something like $$2^{[0,1]}$$ leads us to paradoxes, so the question is: $$P : ? \to [0,1]$$.

Sigma-algebras

Choose some subset of the power set and define the domain of $$P$$ using it:
$$P : \Sigma \to [0,1]$$ $$\emptyset \le \Sigma \le \Omega$$
What we'd like Σ to contain:
• The empty set ∅, so that we can talk about the
probability of something not happening.
• The set of all possible outcomes Ω, so we can talk about the
possibility of anything happening.
Actually, this gives you the trivial σ-algebra:
$$\Sigma = \{ \emptyset, \Omega \}$$
There are certain rules a σ-algebra has to obey:
1. $$\Omega \in \Sigma$$: As above
2. $$A \in \Sigma$$ implies that the complement $$A^C \in \Sigma$$: if I can talk
about some event A happening, then I'd like to talk about some
event A not happening
3. $$A,B \in \Sigma$$ implies that $$A \cup B \in \Sigma$$
Note: I'm not sure why we talk about countable unions with reference to (3) above.
Note: To resolve the paradox mentioned above, see the Borel σ-algebra.
In the usual parsimonious fashion, the above three rules along with
the definition of $$P$$ in the previous section, are used to derive
the rest.