Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [matrix-calculus]

Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.

0
votes
1answer
31 views

Matrix calculus - incorrect calculation?

I understand that matrix derivatives involve different possible conventions but even with that in mind, I'm not sure how to show the result I have. It is given according to this calculation that $$\...
0
votes
1answer
31 views

Derivative with respect to vector of product of two functions of the vector

I am struggling with the following derivative. Let $\pmb{x} \in \mathbb{R}^{n}$ be a vector, $\pmb{y} \in \mathbb{R}^{m}$ another vector that is a function of $\pmb{x}$, and $\pmb{g}$ and $\pmb{h}$ ...
2
votes
0answers
22 views

Hadamard product derivative

If $\circ$ represents the Hadamard product, and $^*$ the conjugate-transpose operation. Given $$f_{(\mathbf{x})} =(\mathbf{x} \circ \mathbf{x})^*H(\mathbf{x} \circ \mathbf{x}) - (\mathbf{x} \circ \...
2
votes
2answers
50 views

Derivative with respect to diagonal of diagonal matrix

Suppose I have a diagonal matrix $\pmb{D}$ and a symmetric matrix $\pmb{X}$ that is not a function of $\pmb{D}$, and I wish to find the following derivative: $$ \frac{\partial}{\partial \mathrm{diag}(\...
0
votes
1answer
34 views

Help in understanding Matrix Differentiation laws used in Stochastic Gradient Descent

I come from a programming background. I am familiar with scalar calculus but not so much with vector/matrix calculus. I am trying to understand stochastic gradient descent for multiple linear ...
0
votes
1answer
25 views

Gradient of a scalar-valued function with respect to a matrix

Let $\mathbf{x}, \mathbf{y} \in \mathbb{R}^{n}$ and $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{n\times n}$. I am not able to understand why \begin{equation*} \frac{\partial}{\partial \mathbf{B}}(\mathbf{...
1
vote
1answer
47 views

Is there proof of $\frac{dTr(\log(A))}{dA}=A^{-1}$ when A is symmetric using index notation.

So I am a physicist and I encountered the following derivative in my study of the SYK model: $\frac{dTr(\log(A))}{dA}$ where $A$ is a symmetric matrix. I know that Tr$(\log(X))=\log(\det(X))$ and ...
1
vote
2answers
48 views

Derivative of Kronecker product of vector with itself

I'm struggling with the following problem. Suppose $\pmb{x}$ and $\pmb{y}$ are vectors of the same length and $\pmb{y}$ is not a function of $\pmb{x}$. What is the following derivative? $$ \frac{\...
1
vote
2answers
19 views

Hadamard product and being unitary

Let $A\in\mathbb{C}^{n\times n}$ and $A=B\circ B$ where $B$ is a unitary matrix and $\circ$ accounts for the Hadamard product. Can we say any thing about $A$ to be unitary or not?
2
votes
2answers
62 views

Derivation of $\frac{\partial}{\partial A} \left( y^T A x \right) = y x^T$

I would like to see a detailed, step-by-step derivation of the following identity $$\frac{\partial}{\partial A} \left( y^T A x \right) = y x^T$$ where $x, y \in \mathbb R^n$ and $A \in \mathbb R^{n \...
0
votes
1answer
40 views

Differentiate matrix product with respect to vectorized form

I am very new to matrix calculus, but I can't seem to figure this out. Suppose $\pmb{X}$ is a non-square $m \times n$ matrix and $\pmb{Y}$ is a symmetrical positive definite $n \times n$ matrix that ...
0
votes
1answer
42 views

Gradients of $ \sum_{i=1}^N \|W_3 g(W_2 f(W_1 x_i) ) - y_i \|_2^2$ w.r.t. $W_1$, $W_2$, and $W_3$?

How to obtain the gradient and optionally Hessian of \begin{align} L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 \ , \end{align} with respect to $...
1
vote
1answer
20 views

Derivative of scalar field and vector field at $(1,2)$ [closed]

Find the derivative of the function at the point $( 1,2 )$: (a) $f ( x , y ) = e ^ { x y }$ (b) $f ( x , y ) = \left( x^2 + y^2, x y \right)$ Are both derivatives matrices?
1
vote
0answers
23 views

Derivative of matrix w.r.t. its own vectorized version

I am unable to find what would be the derivative of a $m \times m$ real matrix $A$ with respect to $(\mathrm{vec}(A))^T$ (where $T$ is transpose and $\mathrm{vec}$ stacks the columns) without using ...
2
votes
1answer
55 views

How far from a matrix the rank does not decrease?

Let $A$ be a real $n \times n$ matrix, and suppose that $\text{rank}(A)=k$. Consider $$ r_0=\sup \{r>0 \,\, | \,\, |B-A| \le r \Rightarrow \text{rank}(B) \ge k \}.$$ Does $r_0$ depend only on ...
1
vote
1answer
31 views

Derivative of autoencoder

Problem $$\nabla_{\mathbf{W}} \mathcal{L}(\mathbf{W})=\frac{1}{2}\Vert \mathbf{W}^T\mathbf{Wx} - \mathbf{x}\Vert_2 ^2$$ where $\mathbf{W} \in \mathbb{R}^{m\times n}\ (m < n)$. What I Have Done ...
1
vote
1answer
30 views

How to prove that $\langle P,A^2 \rangle \le 0 $ for every positive $P$ and skew-symmetric $A$?

I have stumbled upon the following claim, and I wonder if it has a simple proof: Let $P$ be a real $n \times n$ symmetric positive definite matrix. Then for every real skew-symmetric matrix $A$, $\...
0
votes
0answers
26 views

gradient of a trace of matrix product using chain rule

I would like to know what the gradient of the following real scalar-valued function is: $f(A)=tr(f_{1}(A)f_{2}(A)f_{1}(A)^{T})$, where $A$ is a square matrix and $f_{1}$ and $f_{2}$ is arbitrary ...
3
votes
1answer
57 views

Derivative of $\mathbf{XX}^T$ with respect to $\mathbf{X}$

Problem $$\nabla_{\mathbf{X}}\mathbf{XX}^T$$ What I Have Done I checked matrix cookbook, but there is no luck. So I tried to derive it from scratch. I have $$(\mathbf{XX}^T)_{kl}= \mathbf{X}_k^T\...
2
votes
2answers
69 views

Find the derivative of $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule

So I want to differentiate $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule. Here: $\vec{y}$ is an $n \times 1$ vector $X$ is an $ n \times p$ matrix $\beta$ is a $p \times 1$ ...
1
vote
1answer
31 views

Minimization of a Linear Function with Restrictions

I want to find the $\beta^*\in\mathbb R^k$ which minimizes $f(\beta) = (y - X\beta)'(y-X\beta) = \varepsilon'\varepsilon$ subject to the restriction $R\beta = r$. The standard approach would be to ...
0
votes
1answer
47 views

Gradient and Hessian of $\sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2$?

Is my Gradient and Hessian of the following correct? \begin{align} f &= \sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2 \ , \end{align} where $t_i \in \...
1
vote
1answer
18 views

Going from matrix form to summation for calculating the derivative

I have troubles calculating the derivatives of: $\frac{dy}{dx}$ if $y= \vec{x}^{T}\matrix{W}\vec{x}$ I think it would be easier if I go to summation form and then take it from there. However, i'm not ...
2
votes
3answers
37 views

Vector Space as the set of solutions of matrix equation $AX=O$

One of my professor's lecture notes on Vector Spaces start by the following lines:- We have seen that if $det(A)$ = 0, then system $AX=O$ has infinite number of solutions. We shall now see that in ...
0
votes
1answer
39 views

Fisher Matrix and Hessian matrix

I know that the Fisher matrix is easily obtained from the Hessian matrix $I\left(\hat{\beta}\right)=-H\left(\hat{\beta}\right)$ Why is the covariance variance matrix the inverse of the Fisher ...
0
votes
1answer
40 views

Gradient of matrix valued function

I am wondering if its possible to obtain an analytic expression for the gradient of $$f(B) = (A - B)\left[(A - B)'(A - B)\right]^{-\frac{1}{2}}$$ with respect to $B$ where $A \in \mathbb{R}^{s \times ...
0
votes
0answers
32 views

Velocity of a point expressed by using rotation matrix between two reference frames

I'm trying to express the velocity of a point P by using a rotation matrix between two reference frames (see attached file please). Notation: the vertical bar with $O_1$ or $O_2$ means that the ...
0
votes
0answers
23 views

Clarification: Gradient and Hessian of $g(x) = \sum_i g_i(x_i)$; $x = [x_1,\ldots,x_N]^T \in \mathbb{R}^N$

I would like to clarify the Gradient and Hessian of $$g(x) = \sum_i g_i(x_i) ,$$ where $x = [x_1,\ldots,x_N]^T \in \mathbb{R}^N$. Is the gradient of $g(x)$ correct? $$\nabla g(x) = \left[ \nabla g_1(...
0
votes
0answers
18 views

The logarithm of a symmetric matrix and non-symmetric matrix.

As I was reading a paper, I came across a formulation for the mixing rule of two matrices, where each matrix represents the deformation of a body in a particular phase. It was written that for each ...
0
votes
1answer
43 views

Gradient of a matrix expression

What would be $\nabla f(x)$ if $f(x) = x^tAx$ where A is a n by n matrix and x is a n by 1 matrix. I have gone till $x'Ax + x^tAx'$ (where x' represents derivative) to find the solution but can see ...
0
votes
0answers
21 views

Proof of concavity via matrix differentiation

I have found a maximum likelihood estimator which is given by: $${{argmax }\atop {\vec{\mu} = [\mu_1 \dots \mu_2] \atop \mu_i \geq 0}} -\sum_{j=1}^m \vec{p}_j^T \vec{\mu} + \sum_{j=1}^m y_j ln(\vec{p}...
1
vote
2answers
43 views

Optimizing linear combination of matrices and vectors

Given the function $G(X,y)$ = $y^T$$X$$y+$$b^T$$b$, where $y$ is a vector in $R^n$, $b$ is a constant vector in $R^n$ and $X$ is a constant symmetric, square, and invertible matrix, what value of $y$ ...
0
votes
0answers
24 views

Optimal Assignment With A Dissimilar Earnings Between Actors

Here, "actors" are those preforming the tasks/jobs, which are employees 1-3. Suppose there are three employees who are assigned jobs from a list that has four columns and an infinite ammount of ...
1
vote
1answer
28 views

Differentiation of the determinant of the Jacobian

I am working through A Mathematical Introduction to Fluid Mechanics and I have come to a statement on showing what I am guessing is a corollary to Jacobi's Formula https://en.wikipedia.org/wiki/...
0
votes
1answer
31 views

Gradient and hessian of $\log(x^TAx)$

I am working on a optimization problem which involves the gradient and hessian of $\log(x^TAx)$, where $x$ is an unknown vector and $A$ is a positive definite matrix. How can I derive them? Thanks!
0
votes
2answers
26 views

matrix: solve for square matrix with 0 diagonal

I have an equation of $XI = Y$, where I know $I$ (vector of 1s) and $Y$ (vector of positive integers). I want to find positive square matrix $X$, I also know that the diagonal entries of $X$ are all ...
0
votes
1answer
32 views

Inverse of a Gram matrix

If $G$ is the Gram matrix of $n$ vectors of size $d$ that are linearly independent ($n\leq d$), then $G$ is invertible and its inverse is a Gram matrix. Of what? To be more precise, if $G$ is the ...
0
votes
2answers
29 views

Linear Equation to Matrix form

Hey everyone I'm practicing some linear equation matrix questions, so far they're easy to construct, but I am completely lost on this one since I cannot use the method of putting them in separate ...
2
votes
2answers
67 views

Matrix exponential is differentiable at $0 \in \mathbb{R}^{n,n}$

Given$$\exp : \mathbb{R}^{n,n} \mapsto \mathbb{R}^{n,n} \qquad A \mapsto \sum_{k=0}^{\infty} \frac{A^k}{k!}$$ where $ \mathbb{R}^{n,n}$ is equipped with Operator Norm. I am trying to show that $\exp$ ...
1
vote
1answer
47 views

Book about geometric meaning of matrices and their operations

Is there a book (e.g. linear algebra book, or maybe geometry book) that explains in great details the geometric meaning of matrices and the effect of operations on them? I’m not simply talking about ...
1
vote
1answer
20 views

Compute the gradient of $f(x)=\|\text{diag}(x)\|$ with the chain rule

Consider the function $f:\mathbb{R}^n\to\mathbb{R}$ given by $f(x)=\|\text{diag}(x)\|$, where $\text{diag}(x)\in\mathbb{R}^{n\times{n}}$ is the diagonal matrix with diagonal entries $x_1,x_2,\dots,x_n$...
3
votes
1answer
48 views

Trace of matrix $(I-B)(I-B)^TX^TX$

$I$: identity matrix of $p \times p$ $B$: matrix of size $p \times p$. $X$: a data matrix of size $n \times p$ $\displaystyle \text{Tr}\big((I-B)(I-B)^TX^TX\big) =\sum_{k=1}^{p}\sum_{i=1}^{n} \...
3
votes
1answer
46 views

Why can't I reduce the total differential?

I have encountered the following equation: $g: \mathbb{R}^m \rightarrow \mathbb{R}$ $u: \mathbb{R}^n \rightarrow \mathbb{R}^m$ $z = g(\mathbf{y})$, $\mathbf{y} = u(\mathbf{x})$ then using ...
0
votes
1answer
18 views

Eigenvalues of a block off-diagonal matrix

Let $A_1,A_2 \in \mathbb{R}^{n \times n}$. Construct the block matrix $A$ as follows: $$A: = \left[ {\begin{array}{*{20}{c}} 0&{{A_1}}\\ {{A_2}}&0 \end{array}} \right]$$ My observation is that ...
0
votes
1answer
43 views

On what condition $trace(A) \ge trace(AB)$?

Considering $A$ and $B$ are positive semidefinite real symmetric matrices, on what conditions we can have $trace(A) \ge trace(AB)$?
0
votes
1answer
27 views

inverse matrix search algorithm

For ex, $$A^{-1}\cdot A=\left[\begin{matrix}x_1 & x_2\\x_3 & x_4\end{matrix}\right]\cdot\left[\begin{matrix}1 & 2\\1 & 3\end{matrix}\right]=\left[\begin{matrix}1 & 0\\0 & 1\...
3
votes
1answer
122 views

Derivation of $\frac{\partial }{\partial \Sigma} \left(-\frac{1}{2}\log(\det(M\otimes K +I_T \otimes \Sigma)) \right)$

I've done below a derivation, and I'm wondering if it's correct. If you help me with both derivations, I'll throw in some bonus/bounty points. For ease of notation let's define $U_{M\Sigma}=M\otimes ...
0
votes
1answer
32 views

Matrix integration problem $\int g(Ax)dx=|A| \int g(y)dy$

In p. 96 of Wand & Jones' (1995) book they asserted that the following equation is valid for linear changes of variables $$\int_{\mathbb{R^d}} g(A\mathbf{x})d\mathbf{x}=|A| \int_{\mathbb{R^d}} g(\...
0
votes
1answer
41 views

calculate orthogonal matrix

Given matrix $A(m\times n)$ find matrix $B(n\times m)$ that fulfill the equation $A\,B=0\,(m\times m)$ mean orthogonal m less then n
2
votes
1answer
32 views

Integral representation of log of operators

$\def\1{\mathbb{1}}$ Suppose we're in a "good enough" (finite for example) space, and we have positive (semi)-definite operators $P$ and $Q$. Let $\log{(P)}$ and $\log{(P)}$ be logarithms of $P$ and ...