# Calculating the gradient of a multivariable function using chain rule

I have $$f(x)=\sigma(xA)$$ where $$x\in 1\times\mathbb R^n, A\in \mathbb R^{n\times m}$$ and $$\sigma:\mathbb R^m \to \mathbb R^m$$.

I tried to calculate the gradient of $$f$$ with respect to $$x$$ using the chain rule and came up with: $$\nabla_xf=\frac{\partial \sigma(xA)}{\partial x}=\frac{\partial \sigma(xA)}{\partial (xA)}\frac{\partial (xA)}{\partial x}$$ I started with $$\frac{\partial (xA)}{\partial x}=A$$ which makes some sense to me in the way that we can look at $$xA$$ as a function $$xA:\mathbb R^n \to \mathbb R^m$$ and then the i'th row in $$A$$ is the partial derivative of $$xA$$ with respect to $$x_i$$ (if that makes sense). In other words $$A_i=\frac{\partial Ax}{x_i}$$ as a row vector.

Then I tried figuring out the dimensions of $$\frac{\partial \sigma(xA)}{\partial (xA)}$$ and thought it should be some $$\mathbb R^{m \times m}$$ matrix using the same logic as above.

Obviously I'm doing something wrong (perhaps I mixed the $$\nabla$$ and $$\partial$$ notations as well) since the dimensions of the matrices in the multiplication $$\frac{\partial \sigma(xA)}{\partial (xA)}\frac{\partial (xA)}{\partial x}$$ do not fit.

• What is $1 \times \mathbb R^N$ ? Also, $\mathbf x \mathbf A,$ doesn't make sense. Do you mean $\mathbf x^T \mathbf A$ ? I think you meant $\mathbf x$ to write $\mathbf x \in \mathbb R^N$ i.e. an $N \times 1$ column vector, so that $\mathbf x^T$ is a row vector. In that situation, it is possible to left multiply $\mathbf A$ by $\mathbf x^T$ ... Commented May 23, 2023 at 21:07
• @VivekKaushik It's actually written like this in the assignment I have in my course. I think it means that $x$ s just a row vector. This way $xA$ makes sense Commented May 23, 2023 at 21:17

Denote $$y = xA$$, then $$f(x) = \sigma(y)$$. The function $$\sigma: \mathbb{R}^m \rightarrow \mathbb{R}^m$$ can be thought of as a function that acts component by component on its input. That is, if $$y = (y_1, \ldots, y_m)$$, then $$\sigma(y) = (\sigma(y_1), \ldots, \sigma(y_m))$$. Then, the derivative $$\frac{\partial \sigma(y)}{\partial y}$$ is a diagonal matrix, where $$i$$-th diagonal element is $$\sigma'(y_i)$$, the derivative $$\sigma$$ on $$y_i$$
$$\nabla_x f = \frac{\partial \sigma(y)}{\partial y} \frac{\partial y}{\partial x} = \frac{\partial \sigma(xA)}{\partial (xA)} \frac{\partial (xA)}{\partial x}.$$
Since $$\frac{\partial \sigma(xA)}{\partial (xA)}$$ is $$m \times m$$ diagonal matrix and $$\frac{\partial (xA)}{\partial x}$$ is $$m \times n$$ matrix, their product is $$m \times n$$ matrix, which is the correct dimension for derivative $$f$$. The notation $$\frac{\partial (xA)}{\partial x} = A$$ is an abbreviation. We can write that $$y = xA = (y_1, \ldots, y_m)$$, where $$y_i = xA_i$$ and $$A_i$$ is the $$i$$th column of $$A$$, then $$\frac{\partial y_i}{\partial x} = A_i^T$$, a string vector. Adding these row vectors into a matrix, we get $$\frac{\partial y}{\partial x} = A^T$$, $$m \times n$$ matrix.
An important point: in order to calculate the derivative, we need to exploit the fact that the derivative of a composition of functions is given by the product of their derivatives, and that the derivative of a linear function is its matrix transposition. The derivative $$\frac{\partial \sigma(xA)}{\partial (xA)}$$ is diagonal and the derivative $$\frac{\partial (xA)}{\partial x}$$ is $$A^T$$, so the product of these matrices gives $$m \times n$$ matrix for derivative $$f$$.
• Would any of this change if $x$ was a column vector? Meaning we would need the gradient (Jacobian?) of $f=\sigma (Ax)$. More specifically, will the derivative remain $A^T$ or will it become just $A$? Commented May 25, 2023 at 15:27