1
$\begingroup$

I'm working with the multivariate generalized error distribution to model some data. (The parameterization that I am working with follows Graham Giller's work here: https://www.researchgate.net/publication/255626258_A_Generalized_Error_Distribution) I'd like to prove that the Hessian matrix with respect to the mean vector is negative semi-definite for all $\kappa$. To be fair, I don't have a definitive source that says this is indeed true but I have plotted the log likelihood functions for increasing values of $\kappa$ and have yet to encounter a case where it is not concave, with it gradually flattening as $\kappa\rightarrow\inf$, as one might expect from inspection of the pdf. Given that, and the distribution's close relation to other elliptically symmetric distributions with negative-definite Hessians, I feel confident the GED's Hessian should be negative definite w.r.t. the mean vector but the results I am getting suggest that is not the case for $\kappa>1$.

The log likelihood function for $x: \mathbb{R}^n, \mu: \mathbb{R}^n, \Sigma: \mathbb{R}^{n\times n}, \kappa: \mathbb{R}\in(0,\inf)$ is of the form:

$$ \ln(L(x|\mu,\Sigma,\kappa)) = - \left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^\frac{1}{2\kappa} - \frac{1}{2}\ln\left(\left|\Sigma^{-1} \right|\right) - \frac{n}{2} \ln\left(\frac{\pi\Gamma(\kappa)}{\Gamma(3\kappa)}\right) - \ln\left(\frac{\Gamma(1+n\kappa)}{\Gamma(1+\frac{n}{2})}\right) $$

I've calculated the gradient of $\ln(L(x))$ w.r.t. $\mu$ as follows:

$$ \begin{eqnarray*} \triangledown_{\mu}\ln(L(x)) &=& \frac{\partial}{\partial\mu}\left[- \left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^\frac{1}{2\kappa}\right]\\ &=&-\left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}\right]^{\frac{1}{2\kappa}}\frac{\partial}{\partial\mu}\left[z^{\frac{1}{2\kappa}}\right] \text{, where $z=(x-\mu)^T\Sigma^{-1}(x-\mu)$}\\ &=& -\left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}\right]^{\frac{1}{2\kappa}}\frac{1}{2\kappa}z^{\frac{1}{2\kappa}}\frac{\partial z}{\partial \mu} \text{, where $\frac{\partial z}{\partial \mu}=-2\Sigma^{-1}(x-\mu)$}\\ &=& \left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}\right]^{\frac{1}{2\kappa}}\frac{1}{ \kappa}\Sigma^{-1}(x-\mu)\left[(x-\mu)^T\Sigma{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1}\\ \end{eqnarray*} $$

Ignoring the leading constants $\left[\frac{\Gamma(3\kappa)}{\Gamma(\kappa)}\right]^{\frac{1}{2\kappa}}\frac{1}{ \kappa}$, I've calculated the Hessian as follows:

$$ \begin{eqnarray*} H &=& \frac{\partial}{\partial\mu}\left[\Sigma^{-1}(x-\mu)\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1}\right]\\ &=& \left[\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1}\right] J_\mu\left[\Sigma^{-1}(x-\mu)\right] + \Sigma^{-1}(x-\mu)\triangledown^T\left[\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1}\right]\\ &=& -\Sigma^{-1}\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1} + \left(\frac{1}{2\kappa}-1\right)\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-2}(-2)\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1}\\ &=& \left(2-\frac{1}{\kappa}\right)\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-2}\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1} - \Sigma^{-1}\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1} \end{eqnarray*} $$

Factoring out $\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-2}$ from both terms, we're left with:

$$ H\varpropto \left(2-\frac{1}{\kappa}\right)\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1} - \Sigma^{-1}(x-\mu)^T\Sigma^{-1}(x-\mu) $$

Recognizing that $\Sigma^{-1}$ is positive semi-definite by construction, it is easy to see that the Hessian is negative semi-definite for at least all $\kappa \leq 0.5$. It is for $\kappa > 0.5$ that I am unable to prove that the Hessian must be negative (semi-)definite. In fact, if I assume $d=1$, it would seem to be that the Hessian cannot be negative definite for any $\kappa > 1$. Assuming $d=1$:

$$ (x-\mu)(x-\mu)^T\Sigma^{-1} = (x-\mu)^T\Sigma^{-1}(x-\mu) $$

As a result,

$$ H\varpropto \left(2-\frac{1}{\kappa}\right)\Sigma^{-1} - \Sigma^{-1} $$

which must be positive definite when $(2-\frac{1}{\kappa}) > 1$, i.e., when $\kappa > 1$. This does not seem to agree with numerical calculations of the $\ln(L(x))$ across a range of $\mu$ values when $\kappa > 1$, which show a function that is clearly concave w.r.t $\mu$. I'd also note that setting $\kappa = 0.5$ is equivalent to having a multivariate Gaussian and, in that case, the $H$ I derived for the GED evaluates to $-\Sigma^{-1}$, which is the expected result for a Gaussian.

Did I make a mistake in my derivation of the Hessian matrix? I am a self-taught novice when it comes to matrix calculus, so it wouldn't be a shock if I did. Can it be proven that the Hessian is negative (semi-)definite for all $\kappa$? If not, what are the implications for parameter estimation via numerical optimization of $\ln(L(x))$? Any help here would be greatly appreciated.

$\endgroup$

2 Answers 2

0
$\begingroup$

Start with the gradient formulated in simplified terms: $$ \mathbf{g} = -2\beta a^{\beta-1} \mathbf{\Lambda} (\mathbf{m-x}) $$ with $\phi=a^\beta$ and the scalar $a=(\mathbf{m-x})^T \mathbf{\Lambda} (\mathbf{m-x}) > 0$.

Now using differentials, \begin{eqnarray} d\mathbf{g} &=& -2\beta a^{\beta-1} \mathbf{\Lambda} (d\mathbf{m}) - 4\beta (\beta-1) a^{\beta-2} \mathbf{\Lambda} (\mathbf{m-x}) (\mathbf{m-x})^T \mathbf{\Lambda} (d\mathbf{m}) \end{eqnarray} Thus \begin{equation} \mathbf{H} = 2 \beta a^{\beta-2} \left[ 2(1-\beta) \mathbf{u} \mathbf{u}^T - a \mathbf{\Lambda} \right] \end{equation} with $\mathbf{u} = \mathbf{\Lambda} (\mathbf{m-x})$. This is a rank-one update of the precision matrix. When $\beta>1$, the matrix is clearly negative (semi)definite.

$\endgroup$
4
  • $\begingroup$ Thanks for answering. Other than a difference in a +/- sign between our work, it looks like we are deriving the same Hessian with the exception of a missing factor of 2 in your second term. Can you help me understand in greater detail how you derived that second term? $\endgroup$
    – Taylor
    Apr 29 at 22:02
  • $\begingroup$ I have updated my answer with now a gradient with a minus sign and changed the subsequent formulas. I simply use product rule to derive the second term. $\endgroup$
    – Steph
    Apr 30 at 7:50
  • $\begingroup$ @Taylor, so it seems that your Hessian computation is correct. It remains to deal with $\beta$ in the range ]0,1] or [1/2,+inf[ in your case with $\kappa$... $\endgroup$
    – Steph
    Apr 30 at 8:09
  • $\begingroup$ Yes, I am in agreement with you. Since reading your answer, I've figured out how to prove negative (semi-)definiteness with $\kappa$ in the range of 0.5 to 1 and will demonstrate that when I have the time to type it up. I've also created a set of graphs that visually demonstrate the likelihood flips to convex for $\kappa > 1$, so one of my big issues was a false assumption around what result I should expect. The ultimate conclusion is the Hessian w.r.t. $\mu$ is negative (semi-)definite for $\kappa$ in the range of $[0,1]$ and is positive definite for $\kappa > 1$. $\endgroup$
    – Taylor
    May 1 at 3:46
0
$\begingroup$

Picking up from:

$$ H \propto \left(2-\frac{1}{\kappa}\right)\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-2}\Sigma^{-1}(x-\mu)(x-\mu)^T\Sigma^{-1} - \Sigma^{-1}\left[(x-\mu)^T\Sigma^{-1}(x-\mu)\right]^{\frac{1}{2\kappa}-1} $$

we can define $\Sigma^{-1} = L^TL$ via a Cholesky decomposition and define $c = (x-\mu)L$:

$$ \begin{eqnarray*} H &\propto& \left[(x-\mu)^TL^TL(x-\mu)\right]^{\frac{1}{2\kappa}-1} \left(\frac{\left(2-\frac{1}{\kappa}\right)}{(x-\mu)^TL^TL(x-\,u)}L^TL(x-\mu)(x-\mu)^TL^TL - \Sigma^{-1}\right)\\ H &\propto& (c^Tc)^{\frac{1}{2\kappa}-1}\left(\frac{\left(2-\frac{1}{\kappa}\right)}{c^Tc}L^Tcc^TL - \Sigma^{-1}\right)\\ H &\propto& \langle c,c\rangle^{\frac{1}{2\kappa}-1} \left(\left(2-\frac{1}{\kappa}\right) L^T \frac{cc^T}{\langle c,c\rangle} L - \Sigma^{-1}\right) \end{eqnarray*} $$

$\frac{cc^T}{\langle c,c\rangle} = \frac{cc^T}{\lVert{c}\rVert \lVert{c}\rVert} $, which is what's known as a projection matrix. A projection matrix, $P$, has the property that $AP = A$. Therefore, the relevant portion of our Hessian becomes:

$$ \begin{eqnarray*} H &\propto& \left(2-\frac{1}{\kappa}\right)L^TL - \Sigma^{-1} \\ H &\propto& \left(2-\frac{1}{\kappa}\right)\Sigma^{-1} - \Sigma^{-1}\\ H &\propto& \left(1-\frac{1}{\kappa}\right)\Sigma^{-1} \end{eqnarray*} $$

From this, it is readily apparent that the Hessian w.r.t. $\mu$ is negative (semi-)definite for $0 < \kappa \leq 1$ and positive definite for $\kappa > 1$.

$\endgroup$
2
  • $\begingroup$ Are you sure about the projection matrix ? because preliminary numerical tests do not verify your assumption $AP=A$ Maybe I missed something. $\endgroup$
    – Steph
    May 3 at 7:05
  • $\begingroup$ No, I am not. And I've had doubts since typing it out. The source I learned about projection matrices from seems to have been imprecisely written and I now believe that for my projection matrix, I can only say $cP = c$. i.e., the formula $AP = A$ only necessarily holds when the $A$ matrix is the same matrix used to construct $P$. Don't quote me on that - projection matrices are an entirely new concept to me. Back to the drawing board, I suppose... $\endgroup$
    – Taylor
    May 4 at 16:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .