Having the multinomial logistic loss defined as:
$$ L(z; y=j) = -\log[\operatorname{softmax}(z)]_j $$ with, $$[\operatorname{softmax}(z)]_j = \frac{\exp(z_j)}{\sum^K_{k=1} \exp(z_k)}$$
How can I compute the gradient and the Hessian of L with respect to z?
Right now I have the following for the gradient of $L$ with respect to $z$:
$$ \begin{array}{ll} s_i & i \neq j \\ (s_j-1) & i = j \end{array} $$
And so, from my understanding, computing the Hessian will give me the following:
$$ \begin{array}{ll} 0 & i \neq j \\ 1 & i = j \end{array} $$
Leaving me with the Identity matrix.
What I need help with, is to understand if these values are correct or not.
They don't look correct, to say the least.
Thank you.