What is the Hessian matrix of $L(\beta)=\sum_{i=1}^n[y_ix_i^T\beta-\exp(x_i^T\beta)]$? Given a covariate vector $x_i=[x_{i1}, x_{i2}, \dots, x_{ip}]^T \in R^{p\times 1}$ and associated labels $y_i\in R$. We have a parameter $\beta=[\beta_1, \beta_2, \dots, \beta_p]^T$.
For
$$L(\beta)=\sum_{i=1}^n[y_ix_i^T\beta-\exp(x_i^T\beta)]$$
I want to get the Hessian matrix of $L(\beta)$.
The first derivative of $L$ is given by
$$\frac{\partial L}{\partial \beta}=\sum_{i=1}^n[y_ix_i-\exp(x_i^T\beta)x_i]=\sum_{i=1}^n[y_i-\exp(x_i^T\beta)]x_i\in R^{p\times 1}$$
However, I am confused about the result of second order  derivative of $L$:
$$\frac{\partial^2 L}{\partial \beta \partial \beta^T}=\sum_{i=1}^n[-\exp(x_i^T\beta)x_i^T]x_i\in R^1
$$
which is not of size $R^{p\times p}$?
 A: $
\def\bbR#1{{\mathbb R}^{#1}}
\def\b{\beta}\def\l{\lambda}\def\o{{\tt1}}\def\p{\partial}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\diag#1{\operatorname{diag}\LR{#1}}
\def\Diag#1{\operatorname{Diag}\LR{#1}}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\qiq{\quad\implies\quad}
\def\grad#1#2{\frac{\p #1}{\p #2}}
\def\hess#1#2#3{\frac{\p^2 #1}{\p #2\,\p #3}}
\def\c#1{\color{red}{#1}}
$The
elementwise/Hadamard product $(\odot)$ with a vector can be replaced by
ordinary matrix multiplication with a diagonal matrix created from the
vector, e.g.
$$\eqalign{
A = \Diag{a} &\qiq Ax=a\odot x \\
I = \Diag{\o} &\qiq Ix=\o\odot x = x \\
}$$
The trace/Frobenius product $(:)$ is a concise notation for the trace
$$\eqalign{
A:B &= \sum_{i=1}^n\sum_{j=1}^p A_{ij}B_{ij} \;=\; \trace{A^TB} \\
A:A &= \big\|A\big\|^2_F \\
}$$
This is also called the double-dot or double contraction product.
When applied to vectors $(p=\o)$ it reduces to the standard dot product.
The properties of the underlying trace function allow the terms in a
Frobenius product to be rearranged in many different but equivalent ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
C:AB &= CB^T:A = A^TC:B \\
}$$
Together the Hadamard and Frobenius product form a triple scalar product for matrices
$$\eqalign{
A:\LR{B\odot C}
 &= \sum_{i=1}^n\sum_{j=1}^p A_{ij}B_{ij}C_{ij}
 \;=\; \LR{A\odot B}:C \\
}$$

Define the variables
$$\eqalign{
X &= [x_1\;x_2\;x_3\;&\ldots\;x_n]\; \in\bbR{p\times n} \\
w &= X^T\b &\qiq dw = X^Td\b \;\in\bbR{n\times 1} \\ 
E &= \Diag{e^w} &\qiq de^w = e^w\odot dw \;=\; E\,dw \\
}$$
where the exponential function is applied elementwise.
Write the function using the above notation, then calculate its differential and gradient.
$$\eqalign{
L &= y:w - \o:e^w \\
dL &= y:dw - \o:\LR{e^w\odot dw} \\
 &= \LR{y-e^w}:dw \\
 &= \LR{y-e^w}:X^Td\b \\
 &= X\LR{y-e^w}:d\b \\
\grad{L}{\b}
 &= X\LR{y-e^w}
 \;\doteq\; g &\big({\rm gradient\;vector}\big) \\
}$$
Now calculate the differential and gradient of $g$.
$$\eqalign{
dg &= -X\,de^w = -X\LR{e^w\odot dw} = -X\LR{EX^Td\b} \\
\grad{g}{\b} &= -XEX^T
 \;\doteq\; H \qquad\qquad\big({\rm hessian\;matrix}\big) \\\\
}$$
Note that $\,g\in\bbR{p\times\o}\,$ and $\,H\in\bbR{p\times p}\,$ as expected.
