The Gaussian Cheatsheet
This is a collection of key derivations involving Gaussian distributions
which commonly arise almost everywhere in Machine Learning.
A Gaussian distribution with mean and variance is given by
To derive the normalizing constant for this density, consider the following integral
First using change of variables and , we have
Transforming to polar coordinates and
using the standard change of variables, we require the Jacobian determinant
Solving this final integral requires another change of variable. Let ,
This implies and hence the complete distribution is now written as
As a Maximum Entropy Distribution
Interestingly, the Gaussian distribution also turns out to be the maximum entropy
distribution on the infinite support for a finite second moment .
The differential entropy is defined as the expected information
of a random variable .
To find the maximum entropy distribution, we formally write the constrained optimization
problem stated before
The constraints correspond to the normalization of probability distributions,
finite mean (first moment) and finite variance (finite second moment). This
can be converted into an unconstrained optimization problem using Lagrange multipliers .
The complete objective becomes
Setting the functional derivative , we get
To recover the precise values of the Lagrange multipliers, we substitute them back
into the constraints. The derivation is involved but straightforward. We first manipulate the exponent by
completing the squares which will allow us to re-use results from the previous section. We also further always make use of the subsitution .
Putting back these new definitions the normalization constraint is,
Using another change of variable , we reduce this integral to a familiar form and can evaluate using the polar coordinate transformation trick as earlier.
Similarly, we put this into the finite first moment constraint and re-apply
the substitution . The first term is an integral
of an odd function over the full domain and is nullified.
Combining (1) with (2), we get . Substituting and
as defined earlier, we get
With similar approaches and substitutions, we substitute values in the integral for finite
second moment constraint.
Focusing on the remaining term, we first apply the change of variable and note
that this is an even function. This allows us to use the next change of variables.
To allow a change of variable further, we first note that this is an even function and symmetric around .
Using this knowledge, we can change the limits of integration to positive values and use
where we utilize the fact that and . Plugging
everything back and using , we get
Using this, we get
Substituting back in (1), we have
Substituting back into gives us the form for .
- Bishop, C.M., 2006. Pattern recognition and machine learning, springer.
- Boyd, S., Boyd, S.P. & Vandenberghe, L., 2004. Convex optimization, Cambridge university press.
Created: Wed Apr 22 2020