Questions tagged [matrix-calculus]
Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.
0
votes
1answer
31 views
Matrix calculus - incorrect calculation?
I understand that matrix derivatives involve different possible conventions but even with that in mind, I'm not sure how to show the result I have.
It is given according to this calculation that $$\...
0
votes
1answer
31 views
Derivative with respect to vector of product of two functions of the vector
I am struggling with the following derivative. Let $\pmb{x} \in \mathbb{R}^{n}$ be a vector, $\pmb{y} \in \mathbb{R}^{m}$ another vector that is a function of $\pmb{x}$, and $\pmb{g}$ and $\pmb{h}$ ...
2
votes
0answers
22 views
Hadamard product derivative
If $\circ$ represents the Hadamard product, and $^*$ the conjugate-transpose operation. Given
$$f_{(\mathbf{x})} =(\mathbf{x} \circ \mathbf{x})^*H(\mathbf{x} \circ \mathbf{x}) - (\mathbf{x} \circ \...
2
votes
2answers
50 views
Derivative with respect to diagonal of diagonal matrix
Suppose I have a diagonal matrix $\pmb{D}$ and a symmetric matrix $\pmb{X}$ that is not a function of $\pmb{D}$, and I wish to find the following derivative:
$$
\frac{\partial}{\partial \mathrm{diag}(\...
0
votes
1answer
34 views
Help in understanding Matrix Differentiation laws used in Stochastic Gradient Descent
I come from a programming background. I am familiar with scalar calculus but not so much with vector/matrix calculus.
I am trying to understand stochastic gradient descent for multiple linear ...
0
votes
1answer
25 views
Gradient of a scalar-valued function with respect to a matrix
Let $\mathbf{x}, \mathbf{y} \in \mathbb{R}^{n}$ and $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{n\times n}$.
I am not able to understand why
\begin{equation*}
\frac{\partial}{\partial \mathbf{B}}(\mathbf{...
1
vote
1answer
47 views
Is there proof of $\frac{dTr(\log(A))}{dA}=A^{-1}$ when A is symmetric using index notation.
So I am a physicist and I encountered the following derivative in my study of the SYK model:
$\frac{dTr(\log(A))}{dA}$ where $A$ is a symmetric matrix.
I know that Tr$(\log(X))=\log(\det(X))$ and ...
1
vote
2answers
48 views
Derivative of Kronecker product of vector with itself
I'm struggling with the following problem. Suppose $\pmb{x}$ and $\pmb{y}$ are vectors of the same length and $\pmb{y}$ is not a function of $\pmb{x}$. What is the following derivative?
$$
\frac{\...
1
vote
2answers
19 views
Hadamard product and being unitary
Let $A\in\mathbb{C}^{n\times n}$ and $A=B\circ B$ where $B$ is a unitary matrix and $\circ$ accounts for the Hadamard product. Can we say any thing about $A$ to be unitary or not?
2
votes
2answers
62 views
Derivation of $\frac{\partial}{\partial A} \left( y^T A x \right) = y x^T$
I would like to see a detailed, step-by-step derivation of the following identity
$$\frac{\partial}{\partial A} \left( y^T A x \right) = y x^T$$
where $x, y \in \mathbb R^n$ and $A \in \mathbb R^{n \...
0
votes
1answer
40 views
Differentiate matrix product with respect to vectorized form
I am very new to matrix calculus, but I can't seem to figure this out. Suppose $\pmb{X}$ is a non-square $m
\times n$ matrix and $\pmb{Y}$ is a symmetrical positive definite $n \times n$ matrix that ...
0
votes
1answer
42 views
Gradients of $ \sum_{i=1}^N \|W_3 g(W_2 f(W_1 x_i) ) - y_i \|_2^2$ w.r.t. $W_1$, $W_2$, and $W_3$?
How to obtain the gradient and optionally Hessian of
\begin{align}
L(W_1, W_2, W_3) := \sum_{i=1}^N \| W_3 \ g\left(W_2 \ f\left(W_1 x_i \right) \right) - y_i \|_2^2 \ ,
\end{align}
with respect to $...
1
vote
1answer
20 views
Derivative of scalar field and vector field at $(1,2)$ [closed]
Find the derivative of the function at the point $( 1,2 )$:
(a) $f ( x , y ) = e ^ { x y }$
(b) $f ( x , y ) = \left( x^2 + y^2, x y \right)$
Are both derivatives matrices?
1
vote
0answers
23 views
Derivative of matrix w.r.t. its own vectorized version
I am unable to find what would be the derivative of a $m \times m$ real matrix $A$ with respect to $(\mathrm{vec}(A))^T$ (where $T$ is transpose and $\mathrm{vec}$ stacks the columns) without using ...
2
votes
1answer
55 views
How far from a matrix the rank does not decrease?
Let $A$ be a real $n \times n$ matrix, and suppose that $\text{rank}(A)=k$. Consider
$$ r_0=\sup \{r>0 \,\, | \,\, |B-A| \le r \Rightarrow \text{rank}(B) \ge k \}.$$
Does $r_0$ depend only on ...
1
vote
1answer
31 views
Derivative of autoencoder
Problem
$$\nabla_{\mathbf{W}} \mathcal{L}(\mathbf{W})=\frac{1}{2}\Vert \mathbf{W}^T\mathbf{Wx} - \mathbf{x}\Vert_2 ^2$$
where $\mathbf{W} \in \mathbb{R}^{m\times n}\ (m < n)$.
What I Have Done
...
1
vote
1answer
30 views
How to prove that $\langle P,A^2 \rangle \le 0 $ for every positive $P$ and skew-symmetric $A$?
I have stumbled upon the following claim, and I wonder if it has a simple proof:
Let $P$ be a real $n \times n$ symmetric positive definite matrix. Then for every real skew-symmetric matrix $A$, $\...
0
votes
0answers
26 views
gradient of a trace of matrix product using chain rule
I would like to know what the gradient of the following real scalar-valued function is:
$f(A)=tr(f_{1}(A)f_{2}(A)f_{1}(A)^{T})$,
where $A$ is a square matrix and $f_{1}$ and $f_{2}$ is arbitrary ...
3
votes
1answer
57 views
Derivative of $\mathbf{XX}^T$ with respect to $\mathbf{X}$
Problem
$$\nabla_{\mathbf{X}}\mathbf{XX}^T$$
What I Have Done
I checked matrix cookbook, but there is no luck. So I tried to derive it from scratch. I have
$$(\mathbf{XX}^T)_{kl}= \mathbf{X}_k^T\...
2
votes
2answers
69 views
Find the derivative of $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule
So I want to differentiate $f(\beta) = (\vec{y} -X\beta)^T(\vec{y} -X\beta)$ using the product rule. Here:
$\vec{y}$ is an $n \times 1$ vector
$X$ is an $ n \times p$ matrix
$\beta$ is a $p \times 1$ ...
1
vote
1answer
31 views
Minimization of a Linear Function with Restrictions
I want to find the $\beta^*\in\mathbb R^k$ which minimizes $f(\beta) = (y - X\beta)'(y-X\beta) = \varepsilon'\varepsilon$ subject to the restriction $R\beta = r$. The standard approach would be to ...
0
votes
1answer
47 views
Gradient and Hessian of $\sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2$?
Is my Gradient and Hessian of the following correct?
\begin{align}
f &= \sum_i \log \left(1 + \exp\left\{ -t_i \left(w^T x_i\right)\right\} \right) + \mu \|w \|_2^2 \ ,
\end{align}
where $t_i \in \...
1
vote
1answer
18 views
Going from matrix form to summation for calculating the derivative
I have troubles calculating the derivatives of:
$\frac{dy}{dx}$ if $y= \vec{x}^{T}\matrix{W}\vec{x}$
I think it would be easier if I go to summation form and then take it from there. However, i'm not ...
2
votes
3answers
37 views
Vector Space as the set of solutions of matrix equation $AX=O$
One of my professor's lecture notes on Vector Spaces start by the following lines:-
We have seen that if $det(A)$ = 0, then system $AX=O$ has infinite number of solutions. We shall now see that in ...
0
votes
1answer
39 views
Fisher Matrix and Hessian matrix
I know that the Fisher matrix is easily obtained from the Hessian matrix
$I\left(\hat{\beta}\right)=-H\left(\hat{\beta}\right)$
Why is the covariance variance matrix the inverse of the Fisher ...
0
votes
1answer
40 views
Gradient of matrix valued function
I am wondering if its possible to obtain an analytic expression for the gradient of $$f(B) = (A - B)\left[(A - B)'(A - B)\right]^{-\frac{1}{2}}$$
with respect to $B$ where $A \in \mathbb{R}^{s \times ...
0
votes
0answers
32 views
Velocity of a point expressed by using rotation matrix between two reference frames
I'm trying to express the velocity of a point P by using a rotation matrix between two reference frames (see attached file please).
Notation: the vertical bar with $O_1$ or $O_2$ means that the ...
0
votes
0answers
23 views
Clarification: Gradient and Hessian of $g(x) = \sum_i g_i(x_i)$; $x = [x_1,\ldots,x_N]^T \in \mathbb{R}^N$
I would like to clarify the Gradient and Hessian of $$g(x) = \sum_i g_i(x_i) ,$$ where $x = [x_1,\ldots,x_N]^T \in \mathbb{R}^N$.
Is the gradient of $g(x)$ correct?
$$\nabla g(x) = \left[ \nabla g_1(...
0
votes
0answers
18 views
The logarithm of a symmetric matrix and non-symmetric matrix.
As I was reading a paper, I came across a formulation for the mixing rule of two matrices, where each matrix represents the deformation of a body in a particular phase. It was written that for each ...
0
votes
1answer
43 views
Gradient of a matrix expression
What would be $\nabla f(x)$ if $f(x) = x^tAx$ where A is a n by n matrix and x is a n by 1 matrix. I have gone till $x'Ax + x^tAx'$ (where x' represents derivative) to find the solution but can see ...
0
votes
0answers
21 views
Proof of concavity via matrix differentiation
I have found a maximum likelihood estimator which is given by:
$${{argmax }\atop {\vec{\mu} = [\mu_1 \dots \mu_2] \atop \mu_i \geq 0}} -\sum_{j=1}^m \vec{p}_j^T \vec{\mu} + \sum_{j=1}^m y_j ln(\vec{p}...
1
vote
2answers
43 views
Optimizing linear combination of matrices and vectors
Given the function $G(X,y)$ = $y^T$$X$$y+$$b^T$$b$, where $y$ is a vector in $R^n$, $b$ is a constant vector in $R^n$ and $X$ is a constant symmetric, square, and invertible matrix, what value of $y$ ...
0
votes
0answers
24 views
Optimal Assignment With A Dissimilar Earnings Between Actors
Here, "actors" are those preforming the tasks/jobs, which are employees 1-3.
Suppose there are three employees who are assigned jobs from a list that has four columns and an infinite ammount of ...
1
vote
1answer
28 views
Differentiation of the determinant of the Jacobian
I am working through A Mathematical Introduction to Fluid Mechanics and I have come to a statement on showing what I am guessing is a corollary to Jacobi's Formula
https://en.wikipedia.org/wiki/...
0
votes
1answer
31 views
Gradient and hessian of $\log(x^TAx)$
I am working on a optimization problem which involves the gradient and hessian of $\log(x^TAx)$, where $x$ is an unknown vector and $A$ is a positive definite matrix. How can I derive them? Thanks!
0
votes
2answers
26 views
matrix: solve for square matrix with 0 diagonal
I have an equation of $XI = Y$, where I know $I$ (vector of 1s) and $Y$ (vector of positive integers). I want to find positive square matrix $X$, I also know that the diagonal entries of $X$ are all ...
0
votes
1answer
32 views
Inverse of a Gram matrix
If $G$ is the Gram matrix of $n$ vectors of size $d$ that are linearly independent ($n\leq d$), then $G$ is invertible and its inverse is a Gram matrix. Of what?
To be more precise, if $G$ is the ...
0
votes
2answers
29 views
Linear Equation to Matrix form
Hey everyone I'm practicing some linear equation matrix questions, so far they're easy to construct, but I am completely lost on this one since I cannot use the method of putting them in separate ...
2
votes
2answers
67 views
Matrix exponential is differentiable at $0 \in \mathbb{R}^{n,n}$
Given$$\exp : \mathbb{R}^{n,n} \mapsto \mathbb{R}^{n,n} \qquad A \mapsto \sum_{k=0}^{\infty} \frac{A^k}{k!}$$ where $ \mathbb{R}^{n,n}$ is equipped with Operator Norm.
I am trying to show that $\exp$ ...
1
vote
1answer
47 views
Book about geometric meaning of matrices and their operations
Is there a book (e.g. linear algebra book, or maybe geometry book) that explains in great details the geometric meaning of matrices and the effect of operations on them?
I’m not simply talking about ...
1
vote
1answer
20 views
Compute the gradient of $f(x)=\|\text{diag}(x)\|$ with the chain rule
Consider the function $f:\mathbb{R}^n\to\mathbb{R}$ given by $f(x)=\|\text{diag}(x)\|$, where $\text{diag}(x)\in\mathbb{R}^{n\times{n}}$ is the diagonal matrix with diagonal entries $x_1,x_2,\dots,x_n$...
3
votes
1answer
48 views
Trace of matrix $(I-B)(I-B)^TX^TX$
$I$: identity matrix of $p \times p$
$B$: matrix of size $p \times p$.
$X$: a data matrix of size $n \times p$
$\displaystyle \text{Tr}\big((I-B)(I-B)^TX^TX\big) =\sum_{k=1}^{p}\sum_{i=1}^{n} \...
3
votes
1answer
46 views
Why can't I reduce the total differential?
I have encountered the following equation:
$g: \mathbb{R}^m \rightarrow \mathbb{R}$
$u: \mathbb{R}^n \rightarrow \mathbb{R}^m$
$z = g(\mathbf{y})$, $\mathbf{y} = u(\mathbf{x})$
then using ...
0
votes
1answer
18 views
Eigenvalues of a block off-diagonal matrix
Let $A_1,A_2 \in \mathbb{R}^{n \times n}$. Construct the block matrix $A$ as follows:
$$A: = \left[ {\begin{array}{*{20}{c}}
0&{{A_1}}\\
{{A_2}}&0
\end{array}} \right]$$
My observation is that ...
0
votes
1answer
43 views
On what condition $trace(A) \ge trace(AB)$?
Considering $A$ and $B$ are positive semidefinite real symmetric matrices, on what conditions we can have $trace(A) \ge trace(AB)$?
0
votes
1answer
27 views
inverse matrix search algorithm
For ex,
$$A^{-1}\cdot A=\left[\begin{matrix}x_1 & x_2\\x_3 & x_4\end{matrix}\right]\cdot\left[\begin{matrix}1 & 2\\1 & 3\end{matrix}\right]=\left[\begin{matrix}1 & 0\\0 & 1\...
3
votes
1answer
122 views
Derivation of $\frac{\partial }{\partial \Sigma} \left(-\frac{1}{2}\log(\det(M\otimes K +I_T \otimes \Sigma)) \right)$
I've done below a derivation, and I'm wondering if it's correct. If you help me with both derivations, I'll throw in some bonus/bounty points.
For ease of notation let's define $U_{M\Sigma}=M\otimes ...
0
votes
1answer
32 views
Matrix integration problem $\int g(Ax)dx=|A| \int g(y)dy$
In p. 96 of Wand & Jones' (1995) book they asserted that the following equation is valid for linear changes of variables
$$\int_{\mathbb{R^d}} g(A\mathbf{x})d\mathbf{x}=|A| \int_{\mathbb{R^d}} g(\...
0
votes
1answer
41 views
calculate orthogonal matrix
Given matrix $A(m\times n)$
find matrix $B(n\times m)$ that fulfill the equation
$A\,B=0\,(m\times m)$ mean orthogonal
m less then n
2
votes
1answer
32 views
Integral representation of log of operators
$\def\1{\mathbb{1}}$
Suppose we're in a "good enough" (finite for example) space, and we have positive (semi)-definite operators $P$ and $Q$.
Let $\log{(P)}$ and $\log{(P)}$ be logarithms of $P$ and ...
