0
$\begingroup$

In the denominator layout (also called Hessian formulation), we have that

$$\dfrac{\partial \mathbf{y}}{\partial \mathbf{x}} = \begin{bmatrix} \dfrac{\partial y_1}{\partial \mathbf{x}} & \dfrac{\partial y_2}{\partial \mathbf{x}} & \cdots & \dfrac{\partial y_m}{\partial \mathbf{x}} \end{bmatrix} = \begin{bmatrix} \dfrac{\partial y_1}{\partial x_1} & \dfrac{\partial y_2}{\partial x_1} & \dots & \dfrac{\partial y_m}{\partial x_1} \\ \dfrac{\partial y_1}{\partial x_2} & \dfrac{\partial y_2}{\partial x_2} & \dots & \dfrac{\partial y_m}{\partial x_2} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial y_1}{\partial x_n} & \dfrac{\partial y_2}{\partial x_n} & \dots & \dfrac{\partial y_m}{\partial x_n} \\ \end{bmatrix} \in \mathbb{R}^{n\times m}, \tag{1}$$

where $ \mathbf{x} = \begin{bmatrix} x_1 & x_2 & \cdots & x_n \end{bmatrix}^\top \in \mathbb{R}^n $ and $ \mathbf{y} = \begin{bmatrix} y_1 & y_2 & \cdots & y_m \end{bmatrix}^\top \in \mathbb{R}^m $, and the gradient matrix given by

$$\dfrac{\partial y}{\partial \mathbf{X}} = \begin{bmatrix} \dfrac{\partial y}{\partial x_{11}} & \dfrac{\partial y}{\partial x_{12}} & \dots & \dfrac{\partial y}{\partial x_{1n}} \\ \dfrac{\partial y}{\partial x_{21}} & \dfrac{\partial y}{\partial x_{22}} & \dots & \dfrac{\partial y}{\partial x_{2n}} \\ \vdots & \vdots & \ddots & \vdots \\ \dfrac{\partial y}{\partial x_{m1}} & \dfrac{\partial y}{\partial x_{m2}} & \dots & \dfrac{\partial y}{\partial x_{mn}} \\ \end{bmatrix} \in \mathbb{R}^{m \times n}. \tag{2}$$

where $y \in \mathbb{R}$ and $\mathbf{X} = \left[x_{ij}\right] \in \mathbb{R}^{m\times n}$.

I have noticed that a special case of the gradient matrix is when $m=1 \therefore \mathbf{X} = \mathbf{x}^\top \in \mathbb{R}^{1\times n}$, where $\mathbf{x}= \left[x_{i}\right] \in \mathbb{R}^n$. Substituting it in the equation (2), we have

$$\dfrac{\partial y}{\partial \mathbf{x}^\top} = \begin{bmatrix} \dfrac{\partial y}{\partial x_{1}} & \dfrac{\partial y}{\partial x_{2}} & \cdots & \dfrac{\partial y}{\partial x_{n}} \end{bmatrix} \in \mathbb{R}^{n} \tag{3}.$$

However, from the Equation (1), when $\mathbf{y} = y \in \mathbb{R}$ is a scalar, it is true to state that

$$\dfrac{\partial y}{\partial \mathbf{x}} = \begin{bmatrix} \dfrac{\partial y}{\partial x_1} \\ \dfrac{\partial y}{\partial x_2} \\ \vdots \\ \dfrac{\partial y}{\partial x_n} \end{bmatrix} \in \mathbb{R}^{n}, \tag{4}$$

Hence, I have concluded that $\dfrac{\partial y}{\partial \mathbf{x}^\top} = \dfrac{\partial y}{\partial \mathbf{x}}^\top$. Is that right?

$\endgroup$
11
  • $\begingroup$ Your question boils down to whether or not $\vec{x}$ is naturally identified with $\vec{x}^T$. This is only true when we make the identification $\vec{x}^T$ and $\vec{x} \cdot $, where $\cdot$ is the usual dot product. In settings where we don't assume the dot product or have any scalar product between vectors, we must carefully distinguish row ($\vec{x}^T$) and column ($\vec{x}$) vectors. $\endgroup$
    – Charles Hudgins
    Oct 23, 2022 at 17:37
  • $\begingroup$ To maybe more explicitly answer your question, we should identify $\frac{\partial f}{\partial \bf{x}}$ with a row-vector and $\frac{\partial f}{\partial \bf{x}^T}$ with a column-vector. Roughly, this is because $\frac{\partial f}{\partial \bf{x}}$ naturally acts on column vectors (think of the Taylor expansion $f(\bf{x} + \bf{y}) = f(\bf{x}) + \frac{\partial f}{\partial \bf{x}} (\bf{y}) + \dots$). $\frac{\partial f}{\partial \bf{x}^T}$ naturally acts on row vectors (though I know of no canonical examples). $\endgroup$
    – Charles Hudgins
    Oct 23, 2022 at 17:41
  • $\begingroup$ If you like $\bf{x}$ switches from being a column vector to a row vector in $\frac{\partial}{\partial \bf{x}}$ so that $\frac{\partial}{\partial \bf{x}}$ and $d\bf{x}$ can "cancel" as fractions.There is no natural operation of column vectors on each other (outside of the context of an scalar product), but there is a natural operation of row vectors on column vectors (matrix multiplication). $\endgroup$
    – Charles Hudgins
    Oct 23, 2022 at 17:45
  • $\begingroup$ @CharlesHudgins "This is because $\frac{\partial f}{\partial\mathbf{x}}$ naturally acts on column vectors". It is not necessarily true, at all. It depends whether you are using Jacobian or Hessian formulation. $\endgroup$
    – Rubem Pacelli
    Oct 23, 2022 at 17:53
  • $\begingroup$ @CharlesHudgins "To maybe more explicitly answer your question, we should identify $\frac{\partial f}{\partial \mathbf{x}}$ with a row-vector and $\frac{\partial f}{\partial \mathbf{x}^\mathsf{T}}$ with a column-vector.". Well, as I have demonstrated, both are column vectors for Hessian for formulation, aren't they? $\endgroup$
    – Rubem Pacelli
    Oct 23, 2022 at 17:56

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .