The Nature of Bayesian Posterior Distributions: Proper vs. Improper

In Bayesian statistics, the posterior distribution does not always need to be a proper distribution. While it is often desired for practical purposes, there are scenarios where an improper posterior arises. This article explores the definitions, situations, and implications of proper and improper posterior distributions in Bayesian inference.

Definitions: Proper and Improper Distributions

A proper distribution is one where the integral over its entire support equals 1, representing a valid probability distribution. Conversely, an improper distribution is one that does not integrate to 1, often occurring when the normalization constant is infinite.

Situations: Proper and Improper Posterior Distributions

Improper Prior

An improper prior arises when the prior distribution is not proper, such as a uniform prior over all real numbers. In some cases, if the likelihood is sufficiently informative, the posterior can still be proper. However, if the prior and likelihood do not provide enough information, the posterior may also become improper. This can lead to difficulties in interpretation and inference.

Improper Posterior

An improper posterior occurs when the prior is improper and the data does not provide enough information to yield a proper distribution. In such cases, the posterior does not provide valid probability measures. This can lead to challenges in making probabilistic statements and performing credible interval calculations.

Practical Implications

Inference and Modeling Choices

For valid Bayesian inference, it is generally preferred that the posterior is a proper distribution. This ensures that probabilistic statements can be made and credible interval calculations performed accurately. However, when using improper priors, it is crucial to assess whether the posterior is proper based on the likelihood and the data.

Examples and Intuition

To illustrate the concept, consider a scenario where (theta sim text{Exp}(lambda)), representing an exponentially distributed random variable, and (x sim N(0, theta)), a zero-mean normal random variable with standard deviation (theta). If we observe (x 0), the posterior estimate (p(theta | x)) can become a Dirac spike, where (theta) is precisely zero. This scenario, while unlikely, highlights the importance of the almost everywhere well-defined nature of Bayes' theorem.

Another example involves a positive random variable (theta geq 0) with an improper prior (p(theta) propto 1), and a mean-zero normally distributed random variable with standard deviation (theta). The evidence integral (p(x) int_{mathbb{R}} p(x|theta) p(theta) dtheta) can fail to converge for all values of (x in mathbb{R}), rendering Bayes' theorem meaningless. In such cases, the posterior is also improper, leading to an unchanged prior.

Conclusion

While a Bayesian posterior can be technically improper, ensuring that the prior leads to a proper posterior is generally advisable for meaningful statistical inference. This article provides insights into the nuances of proper and improper distributions in Bayesian statistics, emphasizing the importance of understanding and addressing these challenges.