1
$\begingroup$

I'm looking for a beginner-friendly explanation on how this Probability Density function works when dealing with mutliple features and what the variables and terms mean in detail.

I'm seriously furstrated with myself because I've gone through this probability topic to this point perfectly and now nothing makes sense. It doesnt help that tutorials, videos, examples, everything just don't link up together or make sense to me.

E.g. First set of notes in this new topic after I covered probability density functions, etc

Multivariate normal distribution

L-dimensional pdf:
p(x;μ,Σ) = (CANT PASTE FORMULA)
– mean μ is a column vector with L elements
– Σ is an L × L covariance matrix
-|Σ| is the determinant and Σ−1 is the inverse
– denoted as N(μ,Σ)

And then an explanation soon after on the parameter estimation:

Parameter Estimation

X is a set of random samples drawn from pdf p(x; θ)
X = {x1, x2,.... xN}

p(x; θ) is short hand for p(x|wi; θ)

Now suddenly I'm dealing with entirely new stuff and after - no kidding - 5 hours of tutorials online and reading through different views and other university's notes - nothing still made sense. I still don't understand this beginning part of this topic, in fact this first couple notes I have pasted for you.

So what exactly is it I don't get? I was dealing with probability density functions p(x) = something, where I would integrate the function from certain points to find the probability of certain things.

Now that was just p(x), and now there is new notation like "p(x; θ)". Now there is a note at the end as you can see mentioning it is short hand for "p(x|wi; θ)". So I understand usually if something is "p(x|y)" that is the probability of x given y, but what does this "p(x|wi; θ)" mean then? And how does it translate into "p(x; θ)"?

I wish someone would explain this notation and also a very very specific explanation on how this distributions I'm dealing with is now "multivariate", how p(x;μ,Σ) relates to a simple normal distribution I was doing before and what is the new features that I have to learn.

I simple can't transit to this new content from the pdf stuff I learned before and its killing me. Any help is greatly appreciated!

EDIT: If this question is very confusing its because I'm extremely confused! You can then help me immensely this way so I don't fail my course (sigh): Knowing that I only know univariate (or standard) normal distributions, how they look on a graph, how to integrate and find the probability, etc, how would you explain to me the additional features to understand MULTIVARIATE normal distributions? What are the notations and what additional parameters are there?

$\endgroup$
2
  • $\begingroup$ what does wi stand for? $\endgroup$
    – Kamster
    Aug 23, 2014 at 17:39
  • $\begingroup$ "w" is a variable and i is a counter for it, e.g. like "x" is a variable and you could say "xi" where i=1 all the way to a certain N. I have no idea why its a w thats why I need help, but yeah the "i" is supposed to be a little i and "w" is a variable $\endgroup$ Aug 23, 2014 at 17:44

1 Answer 1

0
$\begingroup$

It might be helpful to start with the univariate standard normal distribution, and build things up from there, step by step. For a single standard normal random variable $x$ the probability density is

$f(x) = \frac{1}{\sqrt{2 \pi}}e^{-x^2/2} = (2 \pi)^{-1/2}e^{-x^2/2}$

For $L$ independent standard normal variates $x_1$, $x_2$, . . . , $x_L$ we then have

$f(x_1, x_2, \ldots , x_L) = f(\mathbf{x}) = \prod_{i=1}^L f(x_i) = (2 \pi)^{-L/2}e^{-\mathbf{x}^{\prime} \mathbf{x}/2}$

This is the multivariate normal distribution in the case $E[x_i] = 0$, $V[x_i] = 1$, and $E[x_i x_j] = 0$ for $i \ne j$. In this case, the covariance matrix is the identity matrix, $\mathbf{\Sigma} = \mathbf{I}$, and we can write the above equivalently as

$f(\mathbf{x}) = (2 \pi)^{-L/2}|\mathbf{I}|^{-1/2}e^{(-1/2)\mathbf{x}^{\prime} \mathbf{I}^{-1}\mathbf{x}}$

This is a useful way to write it because we can then easily generalise to the case $E[x_i] = \mu_i$ and $E[(x_i-\mu_i) (x_j-\mu_j)] = \sigma_{ij}$ by replacing $\mathbf{I}$ with the general covariance matrix $\mathbf{\Sigma}$. In this case the covariance matrix is $\mathbf{\Sigma}$ with $\mathbf{\Sigma}_{ij} = \sigma_{ij}$ and replacing $\mathbf{I}$ by $\mathbf{\Sigma}$ in the above we get

$f(\mathbf{x};\mathbf{\mu}, \mathbf{\Sigma}) = (2 \pi)^{-L/2}|\mathbf{\Sigma}|^{-1/2}e^{(-1/2)\mathbf{(x-\mu)}^{\prime} \mathbf{\Sigma}^{-1}\mathbf{(x-\mu)}}$

where $\mathbf{\mu}$ is the mean vector for $\mathbf{x}$. This is the general multivariate normal distribution. In the case $L = 1$, it reduces to the usual univariate normal distribution

$f(x; \mu, \sigma^2) = (2 \pi)^{-1/2}(\sigma^2)^{-1/2}e^{(-1/2)(x-\mu)^2/\sigma^2}$

$= \frac{1}{\sigma\sqrt{2 \pi}}e^{-(x-\mu)^2/2\sigma^2}$

where $\mu$ here is just the mean of $x$. When talking about parameter estimation, you will sometimes see something like $f(x|y ; \theta)$. This is specifying the density of x conditional on some data y and given the parameters $\theta$. For example, using the multivariate normal distribution above you might assume $\mathbf{\mu} = \mathbf{y}'\mathbf{\beta}$ where $\mathbf{y}$ is some observed data vector, and then write as the distributional assumption

$f(\mathbf{x}|\mathbf{y};\mathbf{\beta}, \mathbf{\Sigma}) = (2 \pi)^{-L/2}|\mathbf{\Sigma}|^{-1/2}e^{(-1/2)\mathbf{(x-y'\beta)}^{\prime} \mathbf{\Sigma}^{-1}\mathbf{(x-y'\beta)}}$

The notation $f(\mathbf{x}|\mathbf{y};\mathbf{\beta}, \mathbf{\Sigma})$ simply means 'the conditional density of $\mathbf{x}$ given the data $\mathbf{y}$ and the parameters $\beta$ and $\mathbf{\Sigma}$'. I think that is the situation you are referring to in your question.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .