0
$\begingroup$

I have $f(x)=\sigma(xA)$ where $x\in 1\times\mathbb R^n, A\in \mathbb R^{n\times m}$ and $\sigma:\mathbb R^m \to \mathbb R^m$.

I tried to calculate the gradient of $f$ with respect to $x$ using the chain rule and came up with: $$\nabla_xf=\frac{\partial \sigma(xA)}{\partial x}=\frac{\partial \sigma(xA)}{\partial (xA)}\frac{\partial (xA)}{\partial x}$$ I started with $\frac{\partial (xA)}{\partial x}=A$ which makes some sense to me in the way that we can look at $xA$ as a function $xA:\mathbb R^n \to \mathbb R^m$ and then the i'th row in $A$ is the partial derivative of $xA$ with respect to $x_i$ (if that makes sense). In other words $A_i=\frac{\partial Ax}{x_i}$ as a row vector.

Then I tried figuring out the dimensions of $\frac{\partial \sigma(xA)}{\partial (xA)}$ and thought it should be some $\mathbb R^{m \times m}$ matrix using the same logic as above.

Obviously I'm doing something wrong (perhaps I mixed the $\nabla$ and $\partial$ notations as well) since the dimensions of the matrices in the multiplication $\frac{\partial \sigma(xA)}{\partial (xA)}\frac{\partial (xA)}{\partial x}$ do not fit.

$\endgroup$
2
  • $\begingroup$ What is $1 \times \mathbb R^N$ ? Also, $\mathbf x \mathbf A,$ doesn't make sense. Do you mean $\mathbf x^T \mathbf A$ ? I think you meant $\mathbf x$ to write $\mathbf x \in \mathbb R^N$ i.e. an $N \times 1$ column vector, so that $\mathbf x^T$ is a row vector. In that situation, it is possible to left multiply $\mathbf A$ by $\mathbf x^T$ ... $\endgroup$ Commented May 23, 2023 at 21:07
  • $\begingroup$ @VivekKaushik It's actually written like this in the assignment I have in my course. I think it means that $x$ s just a row vector. This way $xA$ makes sense $\endgroup$
    – McLovin
    Commented May 23, 2023 at 21:17

1 Answer 1

2
$\begingroup$

Denote $y = xA$, then $f(x) = \sigma(y)$. The function $\sigma: \mathbb{R}^m \rightarrow \mathbb{R}^m$ can be thought of as a function that acts component by component on its input. That is, if $y = (y_1, \ldots, y_m)$, then $\sigma(y) = (\sigma(y_1), \ldots, \sigma(y_m))$. Then, the derivative $\frac{\partial \sigma(y)}{\partial y}$ is a diagonal matrix, where $i$-th diagonal element is $\sigma'(y_i)$, the derivative $\sigma$ on $y_i$

$$\nabla_x f = \frac{\partial \sigma(y)}{\partial y} \frac{\partial y}{\partial x} = \frac{\partial \sigma(xA)}{\partial (xA)} \frac{\partial (xA)}{\partial x}.$$

Since $\frac{\partial \sigma(xA)}{\partial (xA)}$ is $m \times m$ diagonal matrix and $\frac{\partial (xA)}{\partial x}$ is $m \times n$ matrix, their product is $m \times n$ matrix, which is the correct dimension for derivative $f$. The notation $\frac{\partial (xA)}{\partial x} = A$ is an abbreviation. We can write that $y = xA = (y_1, \ldots, y_m)$, where $y_i = xA_i$ and $A_i$ is the $i$th column of $A$, then $\frac{\partial y_i}{\partial x} = A_i^T$, a string vector. Adding these row vectors into a matrix, we get $\frac{\partial y}{\partial x} = A^T$, $m \times n$ matrix.

An important point: in order to calculate the derivative, we need to exploit the fact that the derivative of a composition of functions is given by the product of their derivatives, and that the derivative of a linear function is its matrix transposition. The derivative $\frac{\partial \sigma(xA)}{\partial (xA)}$ is diagonal and the derivative $\frac{\partial (xA)}{\partial x}$ is $A^T$, so the product of these matrices gives $m \times n$ matrix for derivative $f$.

$\endgroup$
1
  • $\begingroup$ Would any of this change if $x$ was a column vector? Meaning we would need the gradient (Jacobian?) of $f=\sigma (Ax)$. More specifically, will the derivative remain $A^T$ or will it become just $A$? $\endgroup$
    – McLovin
    Commented May 25, 2023 at 15:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .