The vector $\mathbf{x}$ is the derivative of $\mathbf{Ax}$ with respect to what? Consider the linear equation
$$\mathbf{A}\mathbf{x}=\mathbf{b},$$
where


*

*$\mathbf{A}=\begin{bmatrix}\mathbf{a}_1 \\ \vdots \\ \mathbf{a}_n\end{bmatrix}$ is an $N \times K$ random matrix

*$\mathbf{x}=\begin{bmatrix}x_1 \\ \vdots \\ x_n\end{bmatrix}$ is a $K \times 1$ vector

*$\mathbf{b}=\begin{bmatrix}b_1 \\ \vdots \\ b_n\end{bmatrix}$ is an $N \times 1$ random vector


For each $K \times 1$ row $\mathbf{a}_n$ of the matrix $\mathbf{A}$, observe that
$$\frac{\partial \mathbf{a}_n \mathbf{x}}{\partial a_k}=\frac{\partial (a_{n1} x_1 + a_{n2} x_2+\cdots+a_{nK} x_K)}{\partial a_{nk}}=x_k$$
Is there some way to generalize this statement for all columns $k$ simultaneously? In other words,

The vector $\mathbf{x}$ is the derivative of the system of equations $\mathbf{Ax}$, taken with respect to what?


Taking a quick look at this list of identities, I see that $\mathbf{A}$ is the derivative of $\mathbf{A}\mathbf{x}$ with respect to $\mathbf{x}$:
$$\mathbf{A}=\frac{\partial \mathbf{Ax}}{\partial{\mathbf{x}}}$$
However, I don't see what derivative would give us $\mathbf{x}$ as an answer:
$$\mathbf{x}=\frac{\partial \mathbf{Ax}}{\partial (?)}$$
I'm not particularly good with the rules of matrix differentiation, so a proof would be especially helpful here. Thank you!
 A: Just rewrite your equation with Einstein notation
$$
b_i(x) =A_{ij}x_j
$$
Now you see that
$$
{\partial b_{\,i}(x) \over \partial A_{ij}} = x_j
$$
So the $j$th-row of the vector $\mathbf{x}$ is the derivative of the $i$th-row of $\mathbf{b}$ with respect to the component $A_{ij}$ of the matrix $\mathbf{A}$. So $\mathbf{x}$ may be seen as the derivative of $\mathbf{b}$ with respect to the columns of $\mathbf{A}$, in fact, observe that
$$
\mathbf{b}=\mathbf{a}^1 x_1 + \mathbf{a}^2 x_2 + \cdots + \mathbf{a}^k x_k
$$
where each $\mathbf{a}^{\,j}$ is the $j$th column of $\mathbf{A}$. Now, abusing of the notation, we may think that
$$
{\partial \mathbf{b} \over \partial \mathbf{a}^i} = x_i
$$ 
Observe that this is not the same of the derivative with respect to the matrix $\mathbf{A}$. This is obtained deriving each component of $\mathbf{b}$ with respect to an arbitrary component of $\mathbf{A}$ and is a third order tensor
$$
{\partial b_{\,i} \over \partial A_{kl}} = \delta_{ik} x_l
$$
where $\delta_{ik}$ is a Kronecker delta.
A: With the usual notations, derivating with respect to a vector increases the dimension by 1, and derivating wrt to a matrix increases the dimension by 2. For example, if $s$ is a scalar, $v$ a vector and $A$ a matrix, $\partial_A s$ is a matrix, $\partial_v s$ is a vector and $\partial_v v$ is a matrix.
In your case, you differentiate a vector with respect to ? to obtain a vector. So, with usual notations, necessarily ? is a scalar. And I don't think any scalar satisfies this.
