Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [matrix-calculus]

Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.

0
votes
1answer
24 views

How to derive this complex differentiation?

In the finite-deformation theory, the elastic Cauchy-Green strain $\mathbf{E}_e$ is defined as $\mathbf{E}_e=\frac{1}{2}(\mathbf{F}_e^T \mathbf{F}_e-\mathbf{I})$, where the superscript $T$ denotes ...
0
votes
1answer
85 views

How to vector-derivate this matrix-vector multiplication?

Edit: Possible error in the book? See bottom I'm reading "Functional Data Analysis" by Ramsay & Silverman. The text contains the following (p. 87), regarding how smoothing spline coefficients are ...
0
votes
0answers
25 views

Multiplication of matrices in back propagation

I was watching a public available video from Stanford (https://youtu.be/d14TUNcbn1k?t=2720) on the mathematics behind back propagation. They proposed a graph: that was then used as an example of back ...
1
vote
3answers
83 views

Proof of derivative of $x^TBx$ using the product rule

I'm trying to prove that when $f(x) =x^TBx$, then $f'(x) = (B + B^T)x$. I haven't found this formula online but going through the calculations using index notation this is what I came up with. This ...
1
vote
1answer
48 views

Conditions under which a known vector valued function the gradient of some function

Suppose that we have a vector valued function $D(x)$ with derivative $H(x)$ and that both of these are smooth. Under what conditions does there exist a function $f(x)$ such that $\nabla f(x) = D(x)$? ...
0
votes
0answers
31 views

Derivative of depended vector/matrix by matrix multiplication

If $X \in R^{N\times N}$ and $f(X): R^{N\times N} \rightarrow R^N$ How can I compute the derivative (where $\times$ is a vector by matrix multiplication) $\frac{\partial Y(X)}{\partial X} = f(X) \...
0
votes
0answers
41 views

Is it correct for derivatives of matrix completion problem?

I am doing a matrix completion problem. Here is my object function: $$\min_{U\ge0,V\ge0} \frac{\lambda}{2} \Vert \Omega \circ (UV^T - Y) \Vert_F^2 + \frac{1}{2} \Vert U \Vert_F^2 + \frac{1}{2} \Vert ...
0
votes
0answers
19 views

Monotonicity of matrix inverse of positive definite matrices [duplicate]

If $A$ and $B$ are positive definite, $A-B$ is positive definite, can we say $B^{-1}-A^{-1}$ is positive definite? I think this should be true but I don't know how to prove it. It would be great if ...
0
votes
1answer
37 views

How to make sure matrix completion can generate a matrix with values in expected range?

I am doing a matrix completion project. Assume that I have an incomplete matrix like ...
0
votes
1answer
56 views

matrix raised to a matrix [on hold]

i wanted to know if it was possible to raise a matrix to a matrix and i wanted to confirm if i have it. i tested it out with numbers and notation and i want to know if i'm right. $$ x =\left [ \...
2
votes
0answers
58 views

What is the second-order Taylor expansion of a function $f : \mathbb C^n \to \mathbb R$?

Consider for example $f(x)=\|Ax-b\|_2^2$, where $A \in \mathbb C^{m \times n}$, $x \in \mathbb C^n$, $b \in \mathbb C^m$. $y := Ax - b \implies dy = A dx \implies dy^* = A^* dx^*$. Taking the ...
0
votes
0answers
25 views

Differentiation of $\log L(X)$

I want to get the first order derivative of the function $F(X) = \log(L(X))$, where $L$ is a linear mapping from $n\times n$ positive definite matrices to $m\times m$ positive definite matrices. We ...
1
vote
1answer
56 views

Gradient of trace of squared matrix logarithm

I have a simple question that confuses me for a while: $$f(X) = \text{tr} \left( [ \log(X) ]^2 \right)$$ where $X$ is an $m \times m$ symmetric positive definite (SPD) matrix and $\log(X)$ is ...
0
votes
0answers
32 views

What does it mean when $\mathrm{det}(I-Q^2)=0$ where $Q$ is Toeplitz?

Assume $Q$ is a general Toeplitz matrix. Under what conditions can we make sure $$\mathrm{det}(I-Q^2)\neq 0?$$ Let's denote the determinant by $|\cdot|$. We can show that $$|I-Q^2| = |I-Q||I+Q|\...
0
votes
1answer
17 views

Iterating through weak compositions

I'm trying to define $\frac{\mathrm d^m}{\mathrm dt^m}\mathbf P^n$ for a general matrix $\mathbf P$ (where each element is differentiated independently such that $(\dot{\mathbf P})_{ij} = \dot{(\...
0
votes
1answer
60 views

Gradient of trace norm of complex matrix

The problem: Let $S \in \mathbb{C}^{N\times M}$ with $N > M$ and $S^{H}S=\mathbb{I}$, let $\rho$ and $\sigma$ be hermitian matrices of trace $1$ and define the function $D: \mathbb{C}^{N\times M} \...
1
vote
0answers
34 views

Gradient of $f = \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta$ w.r.t. $x$, $\gamma$, and $\beta$

How to compute the gradient of $$\eqalign{ f &= \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \cr \mu(x) &= \alpha \ 1^Tx \cr \sigma(x) &= \...
0
votes
1answer
33 views

Hadamard product tricks of particular entities

Consider the following matrices $\mathbf{Q}_{H}$ order $\left( T\times n\right) $ and $\mathbf{A}$ of order $\left( T\times T\right) $ and $% \mathbf{\hat{u}}$ of order $\left( n\times 1\right) $ and ...
-2
votes
0answers
17 views

Is vectorwise L-1 norm matrix monotone?

I am stuck in a convex optimization problem and need to show this following result for further progress. Suppose, A, B, A-B positive definite. $||vec(A)||_1= \sum_{i,j} |a_{ij}|$, prove or disprove ...
3
votes
0answers
45 views

Matrix geometric sum with a unit eigenvalue

Let $A$ be a complex, square matrix, and define the geometric sum $$S = I+A+\cdots + A^{N-1}. \tag{1} $$ Just like in the scalar case, one can expand and see that $$(A-I)S =A^N-I, \tag{2} $$ and hence,...
0
votes
0answers
14 views

Vector-by-matrix differentation in numerator layout

Let $y\in\mathbb{R}^{T\times1},~ F\in\mathbb{R}^{T\times K}$ and $b\in\mathbb{R}^{K\times1}$. I needed to compute the following: $$\frac{\partial ||y-Fb||_2^2}{\partial F} $$ According to ...
0
votes
1answer
59 views

Gradient of $f(x) = 1^T \left[ \left( x - 1 \left[1^T x\right] \right) \odot \left(x - 1 \left[1^T x\right] \right) \right]$ w.r.t. $x$

How to compute the gradient of $$\eqalign{ f(x) &= 1^T \left[ \left( x - 1 \left[1^T x\right] \right) \odot \left(x - 1 \left[1^T x\right] \right) \right]\cr }$$ where $x \in M_{n,1}(\mathbb{...
-1
votes
1answer
38 views

Does $\frac{\partial^2}{\partial^2 (\vec{r}_2 + \vec{r}_1 )} ?= \frac{\partial^2}{\partial^2(\vec{r}_2)} + \frac{\partial^2}{\partial^2(\vec{r}_1)}$?

Is the following equation true? $$ \frac{\partial^2}{\partial^2 (\vec{r}_2 + \vec{r}_1 )} ?= \frac{\partial^2}{\partial^2(\vec{r}_2)} + \frac{\partial^2}{\partial^2(\vec{r}_1)} ?$$ If not, what is ...
2
votes
2answers
39 views

Let $A,B \in M_{2x2}(\mathbb{K})$, prove that $(AB-BA)^2=-\det(AB-BA)I$

Problem: Let $A,B \in M_{2x2}(\mathbb{K})$, prove that $(AB-BA)^2=-\det(AB-BA)I$. There is no more information Question: Is there some other way to prove this besides brute force?, and if is, ...
8
votes
0answers
236 views

Complexification of compact connected Lie groups: do these curves have the same tangent vector?

I'm trying to understand the complexification of Lie groups from page $207$ here and I'm having trouble with a computation. Assume $A, B$ are hermitian metrices, and $k$ is a unitary matrix. I want ...
1
vote
2answers
60 views

Matrix Minimization

I'd like to minimize the following function, but can't reach a closed form solution with respect to $C$ from the first-order partial derivative. $||A-BCD^T||_F^2 + \frac{1}{2}||C||_F^2$ where A,B,C, ...
0
votes
1answer
27 views

How to get gradient of matrix multiplication with respect to a matrix?

$$X = \left[ \begin{array}{ccc} x_{00} & x_{01} & x_{02} \\ x_{10}& x_{11} & x_{12} \\ \end{array} \right] $$ W = w00 w01 w10 w11 w20 w21 d(XW)/d(W) = ? How to get the gradient ...
0
votes
0answers
22 views

Big-O / smal-o for perturbed matrix-inversion: What is $(A_n + \mathcal O_n(n^{-\alpha}))^{-1}$?

Let $\alpha > 0$ and for each natural number, let $B_n$ be a square matrix $B_n = \mathcal O(n^{-\alpha})$ as $n \rightarrow \infty$. Suppose $A_n$ is ineverible for every $n$ and $A_n \rightarrow ...
0
votes
1answer
35 views

compute gradient of $\dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2}$ via chain rule

Let $A \in \mathbb{R}^{n \times n}$ and $X, Y \in \mathbb{R}^{n \times r}$. Consider the function \begin{equation} H \left( X , Y \right) := \dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2} , ...
0
votes
2answers
28 views

What does it mean by transpose of a vector

I came across this paragraph (confusing transpose signs) in a matrix calculus paper. I'm confused by the double transpose notation of the vectors, in particular t(w)...
2
votes
1answer
86 views

Derivative of a function of matrix

I am trying to derive the gradient of the function $f(X) = AXZ + XZX^TXZ$ where $A,X,Z \in R^{n \times n}$ with respect to $X$ matrix. I read a post Matrix-by-matrix derivative formula about matrix ...
0
votes
1answer
24 views

Why does derivative of sum(u) result in vector of 1s?

I came across these formulas (please see attached) in a paper. Could you please explain why the derivative of sum(u) where u is the dot product of vectors w and x results in vector of 1s? Thanks ...
0
votes
1answer
40 views

Functions of vectors and vectors of functions

In page 22 of the Matrix Calculus For Deep Learning, the authors wrote: It is the nature of neural networks that the associated mathematics deals with functions of vectors not vectors of functions. ...
6
votes
5answers
125 views

Maximal determinant of $3 \times 3$ matrix where sum of squared entries is $\leq 1$

Given a $3 \times 3$ matrix in which the sum of the squares of all the elements is not more than one, what is the maximal determinant of this matrix? I have only one idea that equal elements are ...
2
votes
0answers
35 views

Gradient of $-y^T \log \left(f \left(W x ; \gamma, \beta \right) \right)$ w.r.t. $\left\{W, \gamma, \beta\right\}$?

How to compute the gradient of \begin{align} L\left(W, \gamma, \beta\right) := -y^T \log \left(f \left(W x ; \gamma, \beta \right) \right) \end{align} with respect to $\left\{W, \gamma, \beta\right\...
0
votes
0answers
21 views

$m$-th eigenvalue of continuous matrix function also continuous?

So, I have a continuous matrix function $M : [a,b]\to\mathbb C^{d\times d}$. I assume that $M(t)$ is self-adjoint for every $t$, i.e., $M(t)^* = M(t)$. I would like to show that the $m$-th eigenvalue $...
0
votes
2answers
69 views

Computing the derivative of $\|Ax\|_2$

Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $A$ is an arbitrary matrix and $x$ is a vector. I think somebody said that the result is $2A^TAx$, ...
3
votes
1answer
33 views

Derivative of a multivariate quadratic function

Let's define function $f : \mathbb{R}^n \to \mathbb{R}$ as $$ f(x) = {1\over2}x'Ax + b'x $$ where matrix $A \in \mathbb{R}^{n\times n}$ and vector $b\in \mathbb{R}^n$ are given. Function $f$ is ...
0
votes
1answer
30 views

Gradient of a scalar with respect to a row vector

How is the gradient of a scalar wrt column vector different from the gradient of a scalar wrt to a row vector? Is gradient of a scalar wrt a column vector is a column/row vector?
1
vote
0answers
46 views

Chain rule for matrix-valued functions.

Suppose we have a function $x\mapsto A(x)$ where $A(x)$ is an $n\times n$ matrix with differentiable matrix elements $[A(x)]_{ij}=a_{ij}(x)$. Then it is natural to take \begin{align*} \frac{dA}{dx} := ...
0
votes
0answers
78 views

Double nested nabla operator?

Suppose: $f(\vec{x}, \vec{\theta}):R^m \times R^n \rightarrow R$ Functional $L_(T_i ) (f_θ)=\sum_{x_k^{T_i} \in T_i} loss_{T_i}(f(x_k^{T_i} , \theta)$, where $T_i \sim T$ $\theta_i^{T_i}=\theta-\...
3
votes
0answers
31 views

Gradients of Deep Neural Network With Batch Normalization

How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation) \begin{align} L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=...
0
votes
1answer
50 views

A “unique” solution to an equation over the orthogonal matrices?

Set $D=\text{diag}(-1,1,1,\dots ,1)$ be an $n \times n$ real diagonal matrix (where $D_{11}=-1$ and $D_{ii}=1$ for $i>1$). Let $R,Q$ be special orthogonal matrices, satisfying $RDQ=D$. Is it ...
0
votes
0answers
34 views

Book recommendation on matrix differentiation

As the title says, I am looking for a good reference on matrix differentiation. I have already studied the content of Harville's contribution on the topic, but I would like to know if there are ...
0
votes
1answer
47 views

Gradient of $L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 + \lambda ( \sum_l \| W_l\|_1)$?

Extending this question. How to obtain the gradient of ($\ell1$ penalized) \begin{align} L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 + \lambda \...
0
votes
0answers
28 views

Why is a (Row . Row) matrix multiplication inconsistent?

I came upon the fact that if you defined matrix multiplication such that $\underbrace{\begin{pmatrix} a&b&c \\ a'&b'&c' \\ a''&b''&c'' \end{pmatrix}}_A \begin{pmatrix} x&y&...
0
votes
1answer
43 views

Gradient of $g(x) = f(Ax + b)$

I need the gradient and Hessian of the function $g(x) = f(Ax + b)$. $f:\!R^m \rightarrow \!R$, $x \in \!R^n$, $b \in \!R^m$, $A \in \!R^{mxn}$ I cannot find the expression for the ...
0
votes
1answer
51 views

Dot product of the gradient of a function

In matrix calculus, I keep on seeing things like $\langle \nabla f(x), v\rangle$, which is the dot product of the gradient of a function with a vector. I was wondering if there is any intuitive ...
1
vote
1answer
65 views

How to compute the gradient $\nabla_W \left( x^TW^{-T}W^{-1}x \right)$?

Calculate the following gradient $$\nabla_W \left( x^TW^{-T}W^{-1}x \right)$$ where $W$ is a $\mathbb{R}^{d×d}$ matrix and $x$ is a $\mathbb{R}^d$ vector. The result should be a $\mathbb{R}^{...
0
votes
1answer
34 views

Calculating $\nabla\bigg[ (z_i - c)^{T} \mathbf{A} (z_i - c) - 1 \bigg]$ using matrix calculus

I'm trying to calculate $\nabla\bigg[ (z_i - c)^{T} \mathbf{A} (z_i - c) - 1 \bigg]$ with respect to the following variables; matrix $\mathbf{A}$ and the $c$ vector. I tried calculating it and ended ...