I'm currently learning some optimization problems involving matrix calculus and I've encountered a discrepancy in how the gradient of a matrix-vector product is defined. Specifically, I'm looking at the expression $$\nabla_w (Aw)$$ where $$A$$ is a matrix and $$w$$ is a vector.

Some resources state that the gradient of $$Aw$$ with respect to $$w$$ is simply $$A$$. However, others claim that it should be $$A^T$$.

I suspect that this confusion might arise due to differing definitions of a vector (whether it's considered a row vector or a column vector). Could someone please clarify which definition is being used in each case and explain why the gradient would be different based on the vector's orientation?

Thankly kindly for your help and insights.

• See layout conventions. If you use the keywords "numerator layout", "denominator layout" and "mixed layout" you will find tons of questions in this site addressing these differences. Commented Jul 10 at 8:25

You can see the matrix-vector product as a function $$f : \mathbb{R}^d \rightarrow \mathbb{R}^d$$ such that $$f(w)=Aw$$.
What you are looking for is the Jacobian defined by : $$J(f)=(\frac{\partial f_i}{\partial w_j})_{i,j}$$ with $$f_i(w)=\sum_k A_{i,k}w_k$$
Then $$\frac{\partial f_i}{\partial w_j} = A_{i,j}$$. So we can conclude that $$J(f)=A$$