Jacobian matrix and Hessian matrix identity I am trying understand the following identity in two dimensions:
$$ H_{XY} = J^TH_{xy}J$$
Here $x,y$ and $X,Y$ are different coordinates for $\mathbb R^2$, the $J$ is the Jacobian matrix and the $H$ are the Hessian matrices in the coordinates. 
My thoughts are the following: the $J$ is locally a coordinate transformation from $XY$ to $xy$ coordinates. At least is how I thought of Jacobians until today. The problem is that then $J^T$ should be $J^{-1}$. I checked and Jacobians are not necessarily unitary. Now I wonder: how do I understand the equation above and in general the Jacobian matrix? In particular why is it $J^T$ and not $J^{-1}$?
 A: Concerning the hessian, there is no simple formula for the change of variable, EXCEPT when we are on a critical point. Let denote by $\phi$ the function "change of variable" $X\rightarrow x$. First, we have a look on the case $n=1$. $(f\circ \phi)'(X)=f'(x)\phi'(X)$ and $(f\circ \phi)''(X)=f''(x)\phi'^2(X)+f'(x)\phi''(X)$. The previous expression has an interesting form only if $f'(x)=0$ ($x$ is a critical point and $X$ also since $(f\circ \phi)'(X)=0$). Now , in the general case, it is more complicated because we must consider the first derivative (linear) and the second derivative (bilinear and then, associated to a symmetric matrix). $D(f\circ \phi)_X:H\rightarrow Df_x(D\phi_X(H))$ and $D^2(f\circ \phi)_X:(H,K)\rightarrow D^2f_x(D\phi_X(H),D\phi_X(K))+Df_x(D^2\phi_X(H,K))$. Here $x$ is a critical point, that is $Df_x=0$ and $D^2(f\circ\phi)_X(H,K)=D^2f_x(D\phi_X(H),D\phi_X(K))$. With the blue's notation, $D\phi_X=J$, we obtain the equality between symmetric matrices $D^2(f\circ\phi)_X=J^TD^2f_xJ$. This formula permits to study the critical points of $f$; are they really an extremum of the function $f$ ?  
A: Try to compute it directly. Assume $(x,y)\rightarrow(X,Y)$ is linear transformation, we have
$$(H(X))_{ij}=\frac{\partial^2f}{\partial X_i\partial X_j}=\sum_k\sum_l\frac{\partial x_k}{\partial X_i}\frac{\partial^2f}{\partial x_k\partial x_l}\frac{\partial x_l}{\partial X_j}=\sum_k\sum_l J_{ki}(H(x))_{kl}J_{lj}$$
Write in matrix form precisely
$$H(X)=J^TH(x)J$$
Now you may still be unclear why there's a $J^T$ instead of $J^{-1}$. The reason is that Hessian is not a linear transformation as what the other matrices be.
Let's look closer to change of basis of a matrix A by a linear transformation $P$.
If matrix $A$ is a linear transformation, that is it maps a vector to a vector.
$$y=Ax$$
Then change basis means an another linear transformation $A'$ so that
$$Py=A'Px$$
This is precisely $A'=P^{-1}AP$. We say linear transformation is both covariant (so $P$) and contravariant (so $P^{-1}$).
However, if matrix $A$ is a bilinear form, that is it maps two vectors to a scalar.
$$c=A(x,y)$$
Then the change of basis becomes
$$c=A'(Px,Py)$$
So by computation I wrote above, $A'=J^TAJ$. We then say this mapping is covariant(so both $P$)
You may say I can also regard linear transformation as a mapping from two vectors to scalar defined as
$$c=x^TAy$$
Yes, that's the problem. Actually, here $x$ is no longer a vector but covector. You will further check
$$c=(Px)^TA'(Py)=x^T(P^TA'P)y$$
Also transpose instead of inverse right? We can conclude the form of transformation depends on type of vector and matrix. In detail, contravariant, covariant or mix-type.
