Confusion on notation of multivariable differentiation in n-dimensional space for matrices (gradients) I am learning gradients on my own.
One of the rules given in the textbook I am reading states this (without proof):
$$∇_\mathbf{x}\mathbf{A}\mathbf{x} = \mathbf{A}^⊤$$
and then proceeds onwards as if it's trivial to see. I haven't seen this before so I need to go prove that it is correct.
I do not know if the above equation is equal to
$$∇f(\mathbf{x}) = \mathbf{A}^⊤$$
where
$$f(\mathbf{x}) = \mathbf{A}\mathbf{x}$$
I decided to go ahead with this anyways and see where it takes me. I figure doing this with a simple $\mathbb{R}^{2 \times 2}$ like $\mathbf{A} = \begin{bmatrix}a&b\\c&d\end{bmatrix}$ and seeing where it takes me when $\mathbf{x} = \begin{bmatrix}i\\j\end{bmatrix}$.
This means $\mathbf{A}\mathbf{x}$ is:
$$\begin{bmatrix}ai + bj\\ci + dj\end{bmatrix}$$
The gradient would be (for $\mathbf{Ax}$ as I'm passing that in for the parameter):
$$∇f(\mathbf{Ax}) = \begin{bmatrix} \frac{\partial f(\mathbf{Ax})}{\partial i} \frac{\partial f(\mathbf{Ax})}{\partial j}\end{bmatrix}^⊤$$
Which means we want to evaluate $∇_\mathbf{x} \begin{bmatrix}ai + bj\\ci + dj\end{bmatrix}$. By the definition of a gradient, then we should get $∇f(\mathbf{Ax}) = \begin{bmatrix} \begin{bmatrix}a\\c\end{bmatrix} \\ \begin{bmatrix}b\\d\end{bmatrix} \end{bmatrix}$
This however does not look right, or is it? Is that equal to $\begin{bmatrix}a&c\\b&d\end{bmatrix}$?
I wrote it out that way because I suspect that when we ask for $a_{21}$ we are saying "get the 2nd row in the first column, and in a programming fashion, A[2][1] (or A[1][0] for most languages) would get $b$ from the result above. This would make sense because I'd get the transpose I was looking for in the primitive example. My linear algebra class only covered vectors, then jumped to matrices and showed us matrix multiplication without covering matrices as if they were a vector of vectors, so if there is a fundamental gap in my knowledge here and what I've described above is correct... then we've found an issue.
Moving to an n-dimensional proof after figuring out what is wrong (if anything) seems like it'd be straight forward with more rigorous variable and index naming.
 A: Suppose that your matrix A is $m \times n$. First of all, you need to view $Ax$ as a function $f: \mathbb{R}^{n} \mapsto \mathbb{R}^{m}$ such that
\begin{equation}
\forall x \in \mathbb{R}^{n}, f\left(x\right) = Ax.
\end{equation}
Basically, this function is a linear transformation of $\mathbb{R}^{n}$. Then you should understand how derivative is defined for functions from multiple inputs to multiple outputs. The definitions and theorems can typically be found in real analysis. A function is differentiable at $x_{0}$ if and only if there exists a matrix $M$ such that
\begin{equation}
\lim_{x \to x_{0}}\frac{\lVert f\left(x\right)-f\left(x_{0}\right)-M\left(x-x_{0}\right)\rVert}{\lVert x - x_{0}\rVert} = 0.
\end{equation}
Then we write
\begin{equation}
M = f^{\prime}\left(x_{0}\right).
\end{equation}
Here, you may understand $\nabla$ as the concept differential $D$. The derivative of $f$ at a point $x \in \mathbb{R}^{n}$ is described as follows:
\begin{equation}
f^{\prime}\left(x\right) = 
\begin{pmatrix}
\frac{\partial f_{1}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{1}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{1}\left(x\right)}{\partial x_{n}}\\
\frac{\partial f_{2}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{2}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{2}\left(x\right)}{\partial x_{n}}\\
\vdots & \vdots & \ddots & \vdots\\
\frac{\partial f_{m}\left(x\right)}{\partial x_{1}} & \frac{\partial f_{m}\left(x\right)}{\partial x_{2}} & \dots & \frac{\partial f_{m}\left(x\right)}{\partial x_{n}}
\end{pmatrix},
\end{equation}
which is a $m \times n$ matrix. Sometimes, the derivative is defined with a transpose symbol, which is essentially your case. The differential at a point $x_{0}$ is of the following property:
\begin{equation}
D_{x_{0}}f\left(x\right) = f^{\prime}\left(x_{0}\right)x, x \in \mathbb{R}^{n}.
\end{equation}
That is to say, the differential is a function $D_{x_{0}}f: \mathbb{R}^{n} \mapsto \mathbb{R}^{m}$. Again, real analysis is the best reference for these questions. An open-source book I recommend is "Introduction to Real Analysis" by William F. Trench.
A: $\def\p#1#2{\frac{\partial #1}{\partial #2}}\def\d{\delta}$In
index notation, the calculation is clearly
$$\eqalign{
y_i &= A_{ij}x_j \\
dy_i &= A_{ij}dx_j \\
\p{y_i}{x_k} &= A_{ij}\left(\p{x_j}{x_k}\right) = A_{ij}\d_{jk} = A_{ik} \\
}$$
but there is an ambiguity in how the indices on the LHS should be ordered, i.e. is the gradient
$G_{ik} \;{\rm or}\; G_{ki}$?
This really depends on your approach to computing the differential
$$\eqalign{
dy &= \; (G\cdot dx) \;\overset{\;?}{\iff}\; (dx\cdot G)  \\
}$$
I have a strong preference for the first form, but opinions vary.
The ambiguity extends to higher-order tensors as well.
$$\eqalign{
\p{Y_{ij}}{X_{k\ell m}} &=\;\; \Gamma_{ijk\ell m} \;\overset{\;?}{\iff}\; \Gamma_{k\ell mij} \\\\
}$$
These ideas also carry over to one's preference for integral expressions, e.g.
$$\eqalign{
\int G\cdot dx \quad\overset{\;?}{\iff}\quad \int dx\cdot G \\
}$$
Again, I prefer the first form.
