# $\dfrac{\partial y}{\partial\mathbf{x}^\top} = \dfrac{\partial y}{\partial\mathbf{x}}^\top$ in denominator layout?

In the denominator layout (also called Hessian formulation), we have that

$$\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}} = \begin{bmatrix} \dfrac{\partial y_1}{\partial \mathbf{x}} & \dfrac{\partial y_2}{\partial \mathbf{x}} & \cdots & \dfrac{\partial y_m}{\partial \mathbf{x}} \end{bmatrix} = \begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} & \dfrac{\partial y_2}{\partial x_1} & \dots & \dfrac{\partial y_m}{\partial x_1} \\ \dfrac{\partial y_1}{\partial x_2} & \dfrac{\partial y_2}{\partial x_2} & \dots & \dfrac{\partial y_m}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial y_1}{\partial x_n} & \dfrac{\partial y_2}{\partial x_n} & \dots & \dfrac{\partial y_m}{\partial x_n} \\ \end{bmatrix} \in \mathbb{R}^{n\times m}, \tag{1}$$

where $$\mathbf{x} = \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix}^\top \in \mathbb{R}^n$$ and $$\mathbf{y} = \begin{bmatrix} y_1 & y_2 & \cdots & y_m \end{bmatrix}^\top \in \mathbb{R}^m$$, and the gradient matrix given by

$$\dfrac{\partial y}{\partial \mathbf{X}} = \begin{bmatrix} \dfrac{\partial y}{\partial x_{11}} & \dfrac{\partial y}{\partial x_{12}} & \dots & \dfrac{\partial y}{\partial x_{1n}} \\ \dfrac{\partial y}{\partial x_{21}} & \dfrac{\partial y}{\partial x_{22}} & \dots & \dfrac{\partial y}{\partial x_{2n}} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial y}{\partial x_{m1}} & \dfrac{\partial y}{\partial x_{m2}} & \dots & \dfrac{\partial y}{\partial x_{mn}} \\ \end{bmatrix} \in \mathbb{R}^{m \times n}. \tag{2}$$

where $$y \in \mathbb{R}$$ and $$\mathbf{X} = \left[x_{ij}\right] \in \mathbb{R}^{m\times n}$$.

I have noticed that a special case of the gradient matrix is when $$m=1 \therefore \mathbf{X} = \mathbf{x}^\top \in \mathbb{R}^{1\times n}$$, where $$\mathbf{x}= \left[x_{i}\right] \in \mathbb{R}^n$$. Substituting it in the equation (2), we have

$$\dfrac{\partial y}{\partial \mathbf{x}^\top} = \begin{bmatrix} \dfrac{\partial y}{\partial x_{1}} & \dfrac{\partial y}{\partial x_{2}} & \cdots & \dfrac{\partial y}{\partial x_{n}} \end{bmatrix} \in \mathbb{R}^{n} \tag{3}.$$

However, from the Equation (1), when $$\mathbf{y} = y \in \mathbb{R}$$ is a scalar, it is true to state that

$$\dfrac{\partial y}{\partial \mathbf{x}} = \begin{bmatrix} \dfrac{\partial y}{\partial x_1} \\ \dfrac{\partial y}{\partial x_2} \\ \vdots \\ \dfrac{\partial y}{\partial x_n} \end{bmatrix} \in \mathbb{R}^{n}, \tag{4}$$

Hence, I have concluded that $$\dfrac{\partial y}{\partial \mathbf{x}^\top} = \dfrac{\partial y}{\partial \mathbf{x}}^\top$$. Is that right?

• Your question boils down to whether or not $\vec{x}$ is naturally identified with $\vec{x}^T$. This is only true when we make the identification $\vec{x}^T$ and $\vec{x} \cdot$, where $\cdot$ is the usual dot product. In settings where we don't assume the dot product or have any scalar product between vectors, we must carefully distinguish row ($\vec{x}^T$) and column ($\vec{x}$) vectors. Oct 23, 2022 at 17:37
• To maybe more explicitly answer your question, we should identify $\frac{\partial f}{\partial \bf{x}}$ with a row-vector and $\frac{\partial f}{\partial \bf{x}^T}$ with a column-vector. Roughly, this is because $\frac{\partial f}{\partial \bf{x}}$ naturally acts on column vectors (think of the Taylor expansion $f(\bf{x} + \bf{y}) = f(\bf{x}) + \frac{\partial f}{\partial \bf{x}} (\bf{y}) + \dots$). $\frac{\partial f}{\partial \bf{x}^T}$ naturally acts on row vectors (though I know of no canonical examples). Oct 23, 2022 at 17:41
• If you like $\bf{x}$ switches from being a column vector to a row vector in $\frac{\partial}{\partial \bf{x}}$ so that $\frac{\partial}{\partial \bf{x}}$ and $d\bf{x}$ can "cancel" as fractions.There is no natural operation of column vectors on each other (outside of the context of an scalar product), but there is a natural operation of row vectors on column vectors (matrix multiplication). Oct 23, 2022 at 17:45
• @CharlesHudgins "This is because $\frac{\partial f}{\partial\mathbf{x}}$ naturally acts on column vectors". It is not necessarily true, at all. It depends whether you are using Jacobian or Hessian formulation. Oct 23, 2022 at 17:53
• @CharlesHudgins "To maybe more explicitly answer your question, we should identify $\frac{\partial f}{\partial \mathbf{x}}$ with a row-vector and $\frac{\partial f}{\partial \mathbf{x}^\mathsf{T}}$ with a column-vector.". Well, as I have demonstrated, both are column vectors for Hessian for formulation, aren't they? Oct 23, 2022 at 17:56