Issue while deriving Hessian for Logistic Regression loss function with matrix calculus

Question

So I'm trying to show the fact that the Hessian of log-likelihood function for Logistic Regression is NSD using matrix calculus.

I've come across an issue in which the direction from which a scalar multiplies the vector matters. Here is what I did:

The log-likelihood is given by:

$l(\theta)=\sum_{i=1}^{m}{\left[y^{(i)}log\left(h\left(x^{(i)}\right)\right) + (1-y^{(i)})\cdot log\left(1- h\left(x^{(i)}\right)\right) \right]}$

$h(x)=\frac{1}{1+e^{-\theta^T\cdot x}}$

I've shown that the gradient equals: $\frac{\partial l(\theta)}{\partial \theta}=\sum_{i=1}^{m}{\left(y^{(i)}-h\left(x^{(i)}\right)\right)\cdot x^{(i)}}$

When trying to differentiate the gradient I've gotten the following:

$H=\frac{\partial l(\theta)}{\partial \theta \partial \theta^T}=\frac{\partial}{\partial \theta}\left[\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(y^{(i)}-h\left(x^{(i)}\right)\right)\cdot x^{(i)}}\right]^T= \frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-h\left(x^{(i)}\right)\cdot x^{(i)^T}\right)}$

Here is the issue. $h\left(x^{(i)}\right)$ is a scalar while $x^{(i)^T}$ is a row vector.

If we continue the derivation as is, we'll get: $\sum_{i=1}^m\left[-h\left(x^{(i)}\right)\left(1-h\left(x^{(i)}\right)\right)\cdot x^{(i)}\cdot x^{(i)^T} \right]$

However, if we change $\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-h\left(x^{(i)}\right)\cdot x^{(i)^T}\right)}$ to $\frac{\partial}{\partial \theta}\sum_{i=1}^{m}{\left(-x^{(i)^T}\cdot h\left(x^{(i)}\right) \right)}$ (i.e. change the order of the scalar multipying the vector), we get: $\sum_{i=1}^m\left[- x^{(i)^T}\cdot h\left(x^{(i)}\right)\left(1-h\left(x^{(i)}\right)\right)\cdot x^{(i)}\right]$

i.e. - In the 2nd formulation we get a scalar instead of a matrix.

What am I missing?

Answer 1

For ease of typing, let's use $\,\,w=\theta.\,$ To make things less cluttered, let's also drop the $(i)$ superscripts and simply treat $\,\{h,w,y\}\,$ as vectors.

Let's define the matrix $\,D={\rm Diag}(h)\,$ with the vector $h$ along its main diagonal, and the matrix $X\,$ whose columns are $\,\,[x_1 \,x_2 \,... \,x_n]$.

Write your gradient expression in terms of these matrices and vectors $$g=\frac{\partial\ell}{\partial w}=X(y-h)$$ and calculate its differential and hessian $$\eqalign{ dg &= -X\,dh = X\,(D^2-D)X^Tdw \cr H = \frac{\partial g}{\partial w} &= X(D^2-D)X^T \cr }$$