So I'm trying to show the fact that the Hessian of log-likelihood function for Logistic Regression is NSD using matrix calculus.
I've come across an issue in which the direction from which a scalar multiplies the vector matters. Here is what I did:
The log-likelihood is given by:
$l(\theta)=\sum_{i=1}^{m}{\left[y^{(i)}log\left(h\left(x^{(i)}\right)\right) + (1-y^{(i)})\cdot log\left(1- h\left(x^{(i)}\right)\right) \right]}$
$h(x)=\frac{1}{1+e^{-\theta^T\cdot x}}$
I've shown that the gradient equals: $\frac{\partial l(\theta)}{\partial \theta}=\sum_{i=1}^{m}{\left(y^{(i)}-h\left(x^{(i)}\right)\right)\cdot x^{(i)}}$
When trying to differentiate the gradient I've gotten the following:
$H=\frac{\partial l(\theta)}{\partial \theta \partial \theta^T}=\frac{\partial}{\partial \theta}\left[\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(y^{(i)}-h\left(x^{(i)}\right)\right)\cdot x^{(i)}}\right]^T= \frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-h\left(x^{(i)}\right)\cdot x^{(i)^T}\right)}$
Here is the issue. $h\left(x^{(i)}\right)$ is a scalar while $x^{(i)^T}$ is a row vector.
If we continue the derivation as is, we'll get: $\sum_{i=1}^m\left[-h\left(x^{(i)}\right)\left(1-h\left(x^{(i)}\right)\right)\cdot x^{(i)}\cdot x^{(i)^T} \right]$
However, if we change $\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-h\left(x^{(i)}\right)\cdot x^{(i)^T}\right)}$ to $\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-x^{(i)^T}\cdot h\left(x^{(i)}\right) \right)}$ (i.e. change the order of the scalar multipying the vector), we get: $\sum_{i=1}^m\left[- x^{(i)^T}\cdot h\left(x^{(i)}\right)\left(1-h\left(x^{(i)}\right)\right)\cdot x^{(i)}\right]$
i.e. - In the 2nd formulation we get a scalar instead of a matrix.
What am I missing?