Posterior probability

Part of a series on Statistics
Bayesian statistics
Theory
Admissible decision rule Bayesian efficiency Bayesian probability Probability interpretations Bayes' theorem Bayes factor Bayesian inference Bayesian network Prior Posterior Likelihood Conjugate prior Posterior predictive Hyperparameter Hyperprior Principle of indifference Principle of maximum entropy Empirical Bayes method Cromwell's rule Bernstein–von Mises theorem Schwarz criterion Credible interval Maximum a posteriori estimation Radical probabilism
Techniques
Bayesian linear regression Bayesian estimator Approximate Bayesian computation Markov chain Monte Carlo
Statistics portal

In Bayesian statistics, the posterior probability of a random event or an uncertain proposition is the conditional probability that is assigned after the relevant evidence or background is taken into account. Similarly, the posterior probability distribution is the probability distribution of an unknown quantity, treated as a random variable, conditional on the evidence obtained from an experiment or survey. "Posterior", in this context, means after taking into account the relevant evidence related to the particular case being examined. For instance, there is a ("non-posterior") probability of a person finding buried treasure if they dig in a random spot, and a posterior probability of finding buried treasure if they dig in a spot where their metal detector rings.

Definition

The posterior probability is the probability of the parameters $\theta$ given the evidence $X$: $p(\theta |X)$.

It contrasts with the likelihood function, which is the probability of the evidence given the parameters: $p(X|\theta )$.

The two are related as follows:

Let us have a prior belief that the probability distribution function is $p(\theta )$ and observations $x$ with the likelihood $p(x|\theta )$, then the posterior probability is defined as

$p(\theta |x)={\frac {p(x|\theta )p(\theta )}{p(x)}}.$

The posterior probability can be written in the memorable form as

${\text{Posterior probability}}\propto {\text{Likelihood}}\times {\text{Prior probability}}$.

Example

Suppose there is a mixed school having 60% boys and 40% girls as students. The girls wear trousers or skirts in equal numbers; all boys wear trousers. An observer sees a (random) student from a distance; all the observer can see is that this student is wearing trousers. What is the probability this student is a girl? The correct answer can be computed using Bayes' theorem.

The event $G$ is that the student observed is a girl, and the event $T$ is that the student observed is wearing trousers. To compute the posterior probability $P(G|T)$, we first need to know:

• $P(G)$, or the probability that the student is a girl regardless of any other information. Since the observer sees a random student, meaning that all students have the same probability of being observed, and the percentage of girls among the students is 40%, this probability equals 0.4.
• $P(B)$, or the probability that the student is not a girl (i.e. a boy) regardless of any other information ($B$ is the complementary event to $G$). This is 60%, or 0.6.
• $P(T|G)$, or the probability of the student wearing trousers given that the student is a girl. As they are as likely to wear skirts as trousers, this is 0.5.
• $P(T|B)$, or the probability of the student wearing trousers given that the student is a boy. This is given as 1.
• $P(T)$, or the probability of a (randomly selected) student wearing trousers regardless of any other information. Since $P(T)=P(T|G)P(G)+P(T|B)P(B)$ (via the law of total probability), this is $P(T)=0.5\times 0.4+1\times 0.6=0.8$.

Given all this information, the posterior probability of the observer having spotted a girl given that the observed student is wearing trousers can be computed by substituting these values in the formula:

$P(G|T)={\frac {P(T|G)P(G)}{P(T)}}={\frac {0.5\times 0.4}{0.8}}=0.25.$

The intuition of this result is that out of every hundred students (60 boys and 40 girls), since we observe trousers the student is one of the 80 students who wear these (60 boys and 20 girls); since 20/80 = 1/4 of these are girls, the probability that the student in trousers is a girl is 1/4.

Calculation

The posterior probability distribution of one random variable given the value of another can be calculated with Bayes' theorem by multiplying the prior probability distribution by the likelihood function, and then dividing by the normalizing constant, as follows:

$f_{X\mid Y=y}(x)={f_{X}(x)L_{X\mid Y=y}(x) \over {\int _{-\infty }^{\infty }f_{X}(u)L_{X\mid Y=y}(u)\,du}}$

gives the posterior probability density function for a random variable $X$ given the data $Y=y$, where

• $f_{X}(x)$ is the prior density of $X$,
• $L_{X\mid Y=y}(x)=f_{Y\mid X=x}(y)$ is the likelihood function as a function of $x$,
• $\int _{-\infty }^{\infty }f_{X}(u)L_{X\mid Y=y}(u)\,du$ is the normalizing constant, and
• $f_{X\mid Y=y}(x)$ is the posterior density of $X$ given the data $Y=y$.

Credible interval

Posterior probability is a conditional probability conditioned on randomly observed data. Hence it is a random variable. For a random variable, it is important to summarize its amount of uncertainty. One way to achieve this goal is to provide a credible interval of the posterior probability.

Classification

In classification, posterior probabilities reflect the uncertainty of assessing an observation to particular class, see also Class membership probabilities. While statistical classification methods by definition generate posterior probabilities, Machine Learners usually supply membership values which do not induce any probabilistic confidence. It is desirable to transform or re-scale membership values to class membership probabilities, since they are comparable and additionally more easily applicable for post-processing.

See also

• Prediction interval
• Bernstein–von Mises theorem
• Monty Hall Problem
• Three Prisoners Problem
• Bertrand's box paradox
• Spike and slab variable selection
• Bayesian structural time series
• Probability of success

References

• Peter M. Lee (2004). Bayesian Statistics, an introduction (3rd ed.). Wiley. ISBN 978-0-340-81405-5.