## Wednesday, July 18, 2012

### A second look at probability theory

I'm studying Principles of Communication Engineering by Wozencraft and Jacobs and the second chapter goes over probability theory, except it uses measure theoretic probability theory.

### Motivation

Example: Toss a coin, have it come up heads, and if it comes up
tails, pick a random number in the range [0,1]. How do you come up
with a PDF/CDF?
Further reading: Look at the excerpt from A User's Guide to Measure Theoretic Probability.
Related video: Why use measure theory for probability?
So, measure-theoretic probability unified the discrete and continuous cases.

### First approach

Pick a set of all possible outcomes Ω (capital omega). When
flipping coins, this set is {H,T}, and when rolling dice this set
is {1,2,3,4,5,6}.
Give a function $$P$$ that assigns a probability to arbitrary
subsets of Ω. You can write this as: $$P : 2^{\Omega} \to [0,1]$$.
The $$2^X$$ notation denotes the power set of $$X$$.
We'd like this function to behave nicely and obey some intuitive laws:
1. $$P(\Omega)=1$$
2. For every event $$A \in \Omega$$, $$P(A) \ge 0$$
3. For any two disjoint events $$A,B$$, $$P(A \cup B)=P(A)+P(B)$$
All scenarios we consider will be some subset of Ω. For example,
$$P(\text{roll even number}) = P(\{2,4,6\})$$. If $$P$$ assigned
a number to all subsets, then we'd be happy.
However, for the continuous case, something like $$2^{[0,1]}$$ leads us to paradoxes, so the question is: $$P : ? \to [0,1]$$.

### Sigma-algebras

Choose some subset of the power set and define the domain of $$P$$ using it:
$$P : \Sigma \to [0,1]$$ $$\emptyset \le \Sigma \le \Omega$$
What we'd like Σ to contain:
• The empty set ∅, so that we can talk about the
probability of something not happening.
• The set of all possible outcomes Ω, so we can talk about the
possibility of anything happening.
Actually, this gives you the trivial σ-algebra:
$$\Sigma = \{ \emptyset, \Omega \}$$
There are certain rules a σ-algebra has to obey:
1. $$\Omega \in \Sigma$$: As above
2. $$A \in \Sigma$$ implies that the complement $$A^C \in \Sigma$$: if I can talk
about some event A happening, then I'd like to talk about some
event A not happening
3. $$A,B \in \Sigma$$ implies that $$A \cup B \in \Sigma$$
Note: I'm not sure why we talk about countable unions with reference to (3) above.
Note: To resolve the paradox mentioned above, see the Borel σ-algebra.
In the usual parsimonious fashion, the above three rules along with
the definition of $$P$$ in the previous section, are used to derive
the rest.