Gradient of a function that takes in two vectors Say I have a function like
$$f(\mathbf{x}, \mathbf{y})= \mathbf{x}^T\mathbf{y}$$
How can I compute its gradient?
 A: For a scalar function with multivariate inputs, the gradient vector $\nabla f$ is defined as being formed from the partial derivatives of $f$ with respect to each of its inputs. So if we let $\mathbf{x} = [ x_1 x_2 \ldots x_n ]$ and $\mathbf{y} = [y_1 y_2 \ldots y_n ]$, then we get:
$\begin{eqnarray} f(\mathbf{x}, \mathbf{y}) & = & \mathbf{x}^T \mathbf{y} \\
& = & \sum_{i = 1}^n x_i y_i \\
\frac{\partial f}{\partial x_i} & = & y_i \\
\frac{\partial f}{\partial y_i} & = & x_i \\
\nabla f & = & \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \\ x_1 \\ x_2 \\ \vdots \\ x_n
\end{bmatrix} \\
& = & \begin{bmatrix} \mathbf{y} \\ \mathbf{x} \end{bmatrix} \end{eqnarray}$
Which is nice, since it's an extension of the idea that if $f(x, y) = xy$ then $\frac{\partial f}{\partial x} = y$ and $\frac{\partial f}{\partial y} = x$.
A: Suppose $f:\mathbb{R}^{n\times m}\times\mathbb{R}^{n\times k}\to\mathbb{R}^{m\times k}$ is given by
$$f(x,y)=x^Ty.$$
If we fix $y\in\mathbb{R}^{n\times k}$, then the function $x\mapsto x^Ty$ is linear in $x$, and yields a differential of $\xi\mapsto\xi^Ty$.  Similarly, if we fix $x\in\mathbb{R}^{n\times m}$, then the function $y\mapsto x^Ty$ is linear in $y$, and yields a differential of $\eta\mapsto x^T\eta$.  It then follows that for $(x,\xi)\in T\mathbb{R}^{n\times m}$ and $(y,\eta)\in T\mathbb{R}^{n\times k}$ that
\begin{align*}
df_{(x,y)}(\xi,\eta)&=d_xf_{(x,y)}(\xi)+d_yf_{(x,y)}(\eta)\\
&=\xi^Ty+x^T\eta
\end{align*}
If you want a "true gradient", then you probably have $m=k=1$, but the above computation doesn't require this.
To continue, focusing on the case of $f:\mathbb{R}^n\times\mathbb{R}^n\to\mathbb{R}$, we fix $x,y\in\mathbb{R}^n$, and for any $\xi\in T_x\mathbb{R}^n$, $\eta\in T_y\mathbb{R}^n$, the gradient $\nabla f_{(x,y)}$ satisfies
\begin{align*}
\langle \nabla f_{(x,y)},(\xi,\eta)\rangle&=df_{(x,y)}(\xi,\eta)\\
&=d_xf_{(x,y)}(\xi)+d_yf_{(x,y)}(\eta)\\
&=\langle\nabla_xf_{(x,y)},\xi\rangle +\langle\nabla_yf_{(x,y)},\eta\rangle\\
&=\langle((\nabla_xf_{(x,y)},\nabla_yf_{(x,y)}),(\xi,\eta)\rangle,
\end{align*}
so we need only consider each gradient individually and direct-sum them.  To this end
\begin{align*}
\langle\nabla_xf_{(x,y)},\xi\rangle&=\xi^Ty\\
&=\langle y, \xi\rangle
\end{align*}
yielding
$$\nabla_xf_{(x,y)}=y,$$
and similarly,
\begin{align*}
\langle\nabla_yf_{(x,y)},\eta\rangle&=x^T\eta\\
&=\langle x, \eta\rangle,
\end{align*}
yielding
$$\nabla_yf_{(x,y)}=x.$$
Finally, putting all of this together, we conclude that
$$\nabla f_{(x,y)}=(y,x)\in\mathbb{R}^n\times\mathbb{R}^n.$$
=================
As an alternative, i.e., the more direct approach in the case of $f:\mathbb{R}^n\times\mathbb{R}^n\to\mathbb{R}$, we compute directly in coordinates writing
$$f(x,y)=\sum_{j=1}^nx^jy^j,$$
and see that
$$\frac{\partial f}{\partial x^k}=y^k\qquad\frac{\partial f}{\partial y^l}=x^l,$$
and hence
\begin{align*}
\nabla f_{(x,y)}&=\begin{bmatrix}
\frac{\partial f}{\partial x^1}\\
\vdots\\
\frac{\partial f}{\partial x^n}\\
\frac{\partial f}{\partial y^1}\\
\vdots\\
\frac{\partial f}{\partial y^n}
\end{bmatrix}\\
&=\begin{bmatrix}
y^1\\
\vdots\\
y^n\\
x^1\\
\vdots\\
x^n
\end{bmatrix}\\
&=\begin{bmatrix}
y\\
x
\end{bmatrix},
\end{align*}
as previously shown.
