Conditional Probability and Bayes Rule

3 Conditional probability

Definition 6 Conditional Probability: Let $S$ be a sample space and let $P$ be a probability over $S$. Let $A$ and $B$ be events such that $P(B) > 0$. The conditional probability of $A$ given $B$ is given by:

$$P(A|B) = \frac{P(A ∩ B)}{P(B)}$$

We can think of conditional probability as a restriction of the set of possible outcomes from the sample space $S$ to $B$.

Example 2 If we roll two fair dice and observe that the first die is a four, what is the probability that the sum of the two dice equals a six given that the first die is a four?

Given that the first roll is a 4, the restricted sample space is $\{(4, 1),(4, 2),(4, 3),(4, 4),(4, 5),(4, 6)\}$. The conditional probability of this event is 1/6. Another way to calculate it is to use the formula 1/36/6/36 = 1/6.

4 Marginals and Total Probability Rule

A useful way to organize the probability space of two events A and B is to divide the sample space into four mutually exclusive events (note: the logical motivation here is similar to the concept of a proof by cases: we attack each independent case separately, and put them all together at the end of present a single solution).

The probability in the margins are called marginals and are calculated by summing across the rows and the columns. The probability of two events $A \cap B$ is called joint distribution. Sometimes we denote is as $A, B$ or $AB$.

From the marginals and the definition of conditional probability, we have:

$$P(B) = P(A \cap B) + P(\bar{A} \cap B)$$

Definition 7 Total Probability Rule:

$$P(B) = P(A \cap B) + P(\bar{A} \cap B)$$

It is called Total because $A$ and $\bar{A}$ orm the totality of the sample space.

Using conditional probability we can write:

$$P(B) = P(B|A)P(A) + P(B|\bar{A} )P(\bar{A})$$

5 Bayes rule

We can derive a useful rule, called Bayes Rule (after the English philosopher Thomas Bayes) by using the definition of conditional probabilities:

$$P(A \cap B) = P(B \cap A)$$

$$P(A|B)P(B) = P(B|A)P(A)$$

$$P(A|B) = \frac{P(B|A) \times P(A)}{ P(B)}$$

$P(A|B)$ is called posterior (posterior distribution on A given B.)

$P(A)$ is called prior.

$P(B)$ is called evidence.

$P(B|A)$ is called likelihood.

This formula known as Bayes rule.

Using the table above, we can write $P(A|B)$ as follows:

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)} = \frac{P(B|A)P(A)}{P(A \cap B) + P(\bar{A} \cap B)}$$

$$P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|\bar{A})P(\bar{A})}$$

A common use of Bayes rule is when we want to know the probability of an unobserved event given an observed event.