Questions tagged [matrix-calculus]
Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.
0
votes
1answer
24 views
How to derive this complex differentiation?
In the finite-deformation theory, the elastic Cauchy-Green strain $\mathbf{E}_e$ is defined as
$\mathbf{E}_e=\frac{1}{2}(\mathbf{F}_e^T \mathbf{F}_e-\mathbf{I})$,
where the superscript $T$ denotes ...
0
votes
1answer
85 views
How to vector-derivate this matrix-vector multiplication?
Edit: Possible error in the book? See bottom
I'm reading "Functional Data Analysis" by Ramsay & Silverman. The text contains the following (p. 87), regarding how smoothing spline coefficients are ...
0
votes
0answers
25 views
Multiplication of matrices in back propagation
I was watching a public available video from Stanford (https://youtu.be/d14TUNcbn1k?t=2720) on the mathematics behind back propagation. They proposed a graph:
that was then used as an example of back ...
1
vote
3answers
83 views
Proof of derivative of $x^TBx$ using the product rule
I'm trying to prove that when $f(x) =x^TBx$, then $f'(x) = (B + B^T)x$. I haven't found this formula online but going through the calculations using index notation this is what I came up with. This ...
1
vote
1answer
48 views
Conditions under which a known vector valued function the gradient of some function
Suppose that we have a vector valued function $D(x)$ with derivative $H(x)$ and that both of these are smooth. Under what conditions does there exist a function $f(x)$ such that $\nabla f(x) = D(x)$? ...
0
votes
0answers
31 views
Derivative of depended vector/matrix by matrix multiplication
If $X \in R^{N\times N}$ and $f(X): R^{N\times N} \rightarrow R^N$
How can I compute the derivative (where $\times$ is a vector by matrix multiplication)
$\frac{\partial Y(X)}{\partial X} = f(X) \...
0
votes
0answers
41 views
Is it correct for derivatives of matrix completion problem?
I am doing a matrix completion problem. Here is my object function:
$$\min_{U\ge0,V\ge0} \frac{\lambda}{2} \Vert \Omega \circ (UV^T - Y) \Vert_F^2 + \frac{1}{2} \Vert U \Vert_F^2 + \frac{1}{2} \Vert ...
0
votes
0answers
19 views
Monotonicity of matrix inverse of positive definite matrices [duplicate]
If $A$ and $B$ are positive definite, $A-B$ is positive definite, can we say $B^{-1}-A^{-1}$ is positive definite? I think this should be true but I don't know how to prove it.
It would be great if ...
0
votes
1answer
37 views
How to make sure matrix completion can generate a matrix with values in expected range?
I am doing a matrix completion project. Assume that I have an incomplete matrix like
...
0
votes
1answer
56 views
matrix raised to a matrix [on hold]
i wanted to know if it was possible to raise a matrix to a matrix and i wanted to confirm if i have it. i tested it out with numbers and notation and i want to know if i'm right. $$
x =\left [ \...
2
votes
0answers
58 views
What is the second-order Taylor expansion of a function $f : \mathbb C^n \to \mathbb R$?
Consider for example $f(x)=\|Ax-b\|_2^2$, where $A \in \mathbb C^{m \times n}$, $x \in \mathbb C^n$, $b \in \mathbb C^m$.
$y := Ax - b \implies dy = A dx \implies dy^* = A^* dx^*$.
Taking the ...
0
votes
0answers
25 views
Differentiation of $\log L(X)$
I want to get the first order derivative of the function $F(X) = \log(L(X))$, where $L$ is a linear mapping from $n\times n$ positive definite matrices to $m\times m$ positive definite matrices.
We ...
1
vote
1answer
56 views
Gradient of trace of squared matrix logarithm
I have a simple question that confuses me for a while:
$$f(X) = \text{tr} \left( [ \log(X) ]^2 \right)$$
where $X$ is an $m \times m$ symmetric positive definite (SPD) matrix and $\log(X)$ is ...
0
votes
0answers
32 views
What does it mean when $\mathrm{det}(I-Q^2)=0$ where $Q$ is Toeplitz?
Assume $Q$ is a general Toeplitz matrix. Under what conditions can we make sure
$$\mathrm{det}(I-Q^2)\neq 0?$$
Let's denote the determinant by $|\cdot|$. We can show that
$$|I-Q^2| = |I-Q||I+Q|\...
0
votes
1answer
17 views
Iterating through weak compositions
I'm trying to define $\frac{\mathrm d^m}{\mathrm dt^m}\mathbf P^n$ for a general matrix $\mathbf P$ (where each element is differentiated independently such that $(\dot{\mathbf P})_{ij} = \dot{(\...
0
votes
1answer
60 views
Gradient of trace norm of complex matrix
The problem:
Let $S \in \mathbb{C}^{N\times M}$ with $N > M$ and $S^{H}S=\mathbb{I}$, let $\rho$ and $\sigma$ be hermitian matrices of trace $1$ and define the function $D: \mathbb{C}^{N\times M} \...
1
vote
0answers
34 views
Gradient of $f = \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta$ w.r.t. $x$, $\gamma$, and $\beta$
How to compute the gradient of
$$\eqalign{
f &= \gamma \ \left( x - 1\mu(x) \right) \ \left( \sigma(x) + \epsilon \right)^{-1/2} + 1\beta \cr
\mu(x) &= \alpha \ 1^Tx \cr
\sigma(x) &= \...
0
votes
1answer
33 views
Hadamard product tricks of particular entities
Consider the following matrices $\mathbf{Q}_{H}$ order $\left(
T\times n\right) $ and $\mathbf{A}$ of order $\left( T\times T\right) $ and $%
\mathbf{\hat{u}}$ of order $\left( n\times 1\right) $ and ...
-2
votes
0answers
17 views
Is vectorwise L-1 norm matrix monotone?
I am stuck in a convex optimization problem and need to show this following result for further progress.
Suppose, A, B, A-B positive definite.
$||vec(A)||_1= \sum_{i,j} |a_{ij}|$, prove or disprove ...
3
votes
0answers
45 views
Matrix geometric sum with a unit eigenvalue
Let $A$ be a complex, square matrix, and define the geometric sum
$$S = I+A+\cdots + A^{N-1}. \tag{1} $$
Just like in the scalar case, one can expand and see that
$$(A-I)S =A^N-I, \tag{2} $$
and hence,...
0
votes
0answers
14 views
Vector-by-matrix differentation in numerator layout
Let $y\in\mathbb{R}^{T\times1},~ F\in\mathbb{R}^{T\times K}$ and $b\in\mathbb{R}^{K\times1}$. I needed to compute the following:
$$\frac{\partial ||y-Fb||_2^2}{\partial F} $$
According to ...
0
votes
1answer
59 views
Gradient of $f(x) = 1^T \left[ \left( x - 1 \left[1^T x\right] \right) \odot \left(x - 1 \left[1^T x\right] \right) \right]$ w.r.t. $x$
How to compute the gradient of
$$\eqalign{
f(x) &= 1^T \left[ \left( x - 1 \left[1^T x\right] \right) \odot \left(x - 1 \left[1^T x\right] \right) \right]\cr
}$$ where $x \in M_{n,1}(\mathbb{...
-1
votes
1answer
38 views
Does $\frac{\partial^2}{\partial^2 (\vec{r}_2 + \vec{r}_1 )} ?= \frac{\partial^2}{\partial^2(\vec{r}_2)} + \frac{\partial^2}{\partial^2(\vec{r}_1)}$?
Is the following equation true?
$$ \frac{\partial^2}{\partial^2 (\vec{r}_2 + \vec{r}_1 )} ?= \frac{\partial^2}{\partial^2(\vec{r}_2)} + \frac{\partial^2}{\partial^2(\vec{r}_1)} ?$$
If not, what is ...
2
votes
2answers
39 views
Let $A,B \in M_{2x2}(\mathbb{K})$, prove that $(AB-BA)^2=-\det(AB-BA)I$
Problem:
Let $A,B \in M_{2x2}(\mathbb{K})$, prove that $(AB-BA)^2=-\det(AB-BA)I$.
There is no more information
Question: Is there some other way to prove this besides brute force?, and if is, ...
8
votes
0answers
236 views
Complexification of compact connected Lie groups: do these curves have the same tangent vector?
I'm trying to understand the complexification of Lie groups from page $207$ here and I'm having trouble with a computation.
Assume $A, B$ are hermitian metrices, and $k$ is a unitary matrix. I want ...
1
vote
2answers
60 views
Matrix Minimization
I'd like to minimize the following function, but can't reach a closed form solution with respect to $C$ from the first-order partial derivative.
$||A-BCD^T||_F^2 + \frac{1}{2}||C||_F^2$
where A,B,C, ...
0
votes
1answer
27 views
How to get gradient of matrix multiplication with respect to a matrix?
$$X =
\left[
\begin{array}{ccc}
x_{00} & x_{01} & x_{02} \\
x_{10}& x_{11} & x_{12} \\
\end{array}
\right]
$$
W =
w00 w01
w10 w11
w20 w21
d(XW)/d(W) = ?
How to get the gradient ...
0
votes
0answers
22 views
Big-O / smal-o for perturbed matrix-inversion: What is $(A_n + \mathcal O_n(n^{-\alpha}))^{-1}$?
Let $\alpha > 0$ and for each natural number, let $B_n$ be a square matrix $B_n = \mathcal O(n^{-\alpha})$ as $n \rightarrow \infty$. Suppose $A_n$ is ineverible for every $n$ and $A_n \rightarrow ...
0
votes
1answer
35 views
compute gradient of $\dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2}$ via chain rule
Let $A \in \mathbb{R}^{n \times n}$ and $X, Y \in \mathbb{R}^{n \times r}$. Consider the function
\begin{equation}
H \left( X , Y \right) := \dfrac{1}{2} \left\lVert A - XY^{T} \right\rVert _{F}^{2} ,
...
0
votes
2answers
28 views
What does it mean by transpose of a vector
I came across this paragraph (confusing transpose signs) in a matrix calculus paper. I'm confused by the double transpose notation of the vectors, in particular t(w)...
2
votes
1answer
86 views
Derivative of a function of matrix
I am trying to derive the gradient of the function $f(X) = AXZ + XZX^TXZ$ where $A,X,Z \in R^{n \times n}$ with respect to $X$ matrix. I read a post Matrix-by-matrix derivative formula about matrix ...
0
votes
1answer
24 views
Why does derivative of sum(u) result in vector of 1s?
I came across these formulas (please see attached) in a paper. Could you please explain why the derivative of sum(u) where u is the dot product of vectors w and x results in vector of 1s? Thanks
...
0
votes
1answer
40 views
Functions of vectors and vectors of functions
In page 22 of the Matrix Calculus For Deep Learning, the authors wrote: It is the nature of neural networks that the associated mathematics deals with functions of vectors not vectors of functions. ...
6
votes
5answers
125 views
Maximal determinant of $3 \times 3$ matrix where sum of squared entries is $\leq 1$
Given a $3 \times 3$ matrix in which the sum of the squares of all the elements is not more than one, what is the maximal determinant of this matrix?
I have only one idea that equal elements are ...
2
votes
0answers
35 views
Gradient of $-y^T \log \left(f \left(W x ; \gamma, \beta \right) \right)$ w.r.t. $\left\{W, \gamma, \beta\right\}$?
How to compute the gradient of
\begin{align}
L\left(W, \gamma, \beta\right) := -y^T \log \left(f \left(W x ; \gamma, \beta \right) \right)
\end{align}
with respect to $\left\{W, \gamma, \beta\right\...
0
votes
0answers
21 views
$m$-th eigenvalue of continuous matrix function also continuous?
So, I have a continuous matrix function $M : [a,b]\to\mathbb C^{d\times d}$. I assume that $M(t)$ is self-adjoint for every $t$, i.e., $M(t)^* = M(t)$. I would like to show that the $m$-th eigenvalue $...
0
votes
2answers
69 views
Computing the derivative of $\|Ax\|_2$
Compute the following derivative (in matrix form) $$\frac{\partial\, \|Ax\|_2}{\partial x}$$ where $A$ is an arbitrary matrix and $x$ is a vector.
I think somebody said that the result is $2A^TAx$, ...
3
votes
1answer
33 views
Derivative of a multivariate quadratic function
Let's define function $f : \mathbb{R}^n \to \mathbb{R}$ as
$$
f(x) = {1\over2}x'Ax + b'x
$$
where matrix $A \in \mathbb{R}^{n\times n}$ and vector $b\in \mathbb{R}^n$ are given. Function $f$ is ...
0
votes
1answer
30 views
Gradient of a scalar with respect to a row vector
How is the gradient of a scalar wrt column vector different from the gradient of a scalar wrt to a row vector?
Is gradient of a scalar wrt a column vector is a column/row vector?
1
vote
0answers
46 views
Chain rule for matrix-valued functions.
Suppose we have a function $x\mapsto A(x)$ where $A(x)$ is an $n\times n$ matrix with differentiable matrix elements $[A(x)]_{ij}=a_{ij}(x)$. Then it is natural to take
\begin{align*}
\frac{dA}{dx} := ...
0
votes
0answers
78 views
Double nested nabla operator?
Suppose:
$f(\vec{x}, \vec{\theta}):R^m \times R^n \rightarrow R$
Functional $L_(T_i ) (f_θ)=\sum_{x_k^{T_i} \in T_i} loss_{T_i}(f(x_k^{T_i} , \theta)$, where $T_i \sim T$
$\theta_i^{T_i}=\theta-\...
3
votes
0answers
31 views
Gradients of Deep Neural Network With Batch Normalization
How to obtain the gradients of such a complicated Deep network with batch normalization (preferably in matrix/vector notation)
\begin{align}
L\left(\left\{W_\ell, \gamma_\ell, \beta_\ell\right\}_{\ell=...
0
votes
1answer
50 views
A “unique” solution to an equation over the orthogonal matrices?
Set $D=\text{diag}(-1,1,1,\dots ,1)$ be an $n \times n$ real diagonal matrix (where $D_{11}=-1$ and $D_{ii}=1$ for $i>1$).
Let $R,Q$ be special orthogonal matrices, satisfying $RDQ=D$. Is it ...
0
votes
0answers
34 views
Book recommendation on matrix differentiation
As the title says, I am looking for a good reference on matrix differentiation.
I have already studied the content of Harville's contribution on the topic, but I would like to know if there are ...
0
votes
1answer
47 views
Gradient of $L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 + \lambda ( \sum_l \| W_l\|_1)$?
Extending this question.
How to obtain the gradient of ($\ell1$ penalized)
\begin{align}
L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 + \lambda \...
0
votes
0answers
28 views
Why is a (Row . Row) matrix multiplication inconsistent?
I came upon the fact that if you defined matrix multiplication such that
$\underbrace{\begin{pmatrix} a&b&c \\ a'&b'&c' \\ a''&b''&c'' \end{pmatrix}}_A \begin{pmatrix} x&y&...
0
votes
1answer
43 views
Gradient of $g(x) = f(Ax + b)$
I need the gradient and Hessian of the function $g(x) = f(Ax + b)$.
$f:\!R^m \rightarrow \!R$,
$x \in \!R^n$,
$b \in \!R^m$,
$A \in \!R^{mxn}$
I cannot find the expression for the ...
0
votes
1answer
51 views
Dot product of the gradient of a function
In matrix calculus, I keep on seeing things like $\langle \nabla f(x), v\rangle$, which is the dot product of the gradient of a function with a vector.
I was wondering if there is any intuitive ...
1
vote
1answer
65 views
How to compute the gradient $\nabla_W \left( x^TW^{-T}W^{-1}x \right)$?
Calculate the following gradient
$$\nabla_W \left( x^TW^{-T}W^{-1}x \right)$$
where $W$ is a $\mathbb{R}^{d×d}$ matrix and $x$ is a $\mathbb{R}^d$ vector. The result should be a $\mathbb{R}^{...
0
votes
1answer
34 views
Calculating $\nabla\bigg[ (z_i - c)^{T} \mathbf{A} (z_i - c) - 1 \bigg]$ using matrix calculus
I'm trying to calculate $\nabla\bigg[ (z_i - c)^{T} \mathbf{A} (z_i - c) - 1 \bigg]$ with respect to the following variables; matrix $\mathbf{A}$ and the $c$ vector.
I tried calculating it and ended ...
