Questions tagged [matrix-calculus]
Matrix calculus is about doing calculus, especially derivative and infinite series over spaces of vectors and matrices.
2,951
questions
0
votes
1answer
33 views
Understanding a specific process of finding the derivative of $x^TAx$
I am referring to @copper.hat's response to : Derivative of Quadratic Form. I do not have the reputation to reply directly. My goal is to find a way to better differentiate and understand these ...
0
votes
0answers
12 views
The normal equation for Multivariate Multiple linear regression
I am confused about the normal equation for Multivariate Multiple linear regression. I don't know How could I derive the normal equation for this case.
Consider the multivariate multiple linear ...
1
vote
0answers
34 views
Minimizing trace involving inverse
Given symmetric positive semidefinite rank-$r$ matrix $R \in \mathbb{C}^{m \times m}$, where $r < m$, and scalar $p \geq 0$, I have the following optimization problem in matrix $X \in \mathbb{C}^{n ...
-2
votes
1answer
32 views
What if the sum of the elements of the inverse of a matrix is 0
If the sum of the elements of an inverse of a matrix $A$ is $0$, does it affect somehow the determinant or something else of the matrix $A$, where $A$ is a $3\times 3$ matrix?
I tried for a $2\times 2$...
3
votes
1answer
48 views
Writing Back-Propogation out in terms of Matrix Multiplication
I am trying to write my own backpropogation algorithm for neural networks for a class. For any specific weight of my network I could easily take the derivative, but for computational speed I want to ...
1
vote
2answers
32 views
Matrices and images [closed]
Let T: $\mathbb{R}^4 \to \mathbb{R}^5$ defined by
$T (x_1, x_2, x_3, x_4) = (x_4, x_1, x_3, x_2, x_1 - x_3)$
A) Find the standard matrix for the linear transformation $T$
B) Find the image of $x = (1, ...
2
votes
1answer
34 views
Derivative of trace involving Hadamard product
Let us assume that $A, S\in\mathbb{R}^{n\times n}$, $U\in\mathbb{R}^{n\times k}$, and $V\in\mathbb{R}^{n\times k}$. I am trying to differentiate the following expression: $$\Phi(U,V)=\mathrm{trace}\...
0
votes
0answers
23 views
Full derivative of a scalar function wrt vector - chain rule
This confusion arose from this matrix calculus helper, the chart of total derivatives on page 23. The case with the scalar function of a vector of $n$ variables: it can be written as a product of a $1\...
1
vote
1answer
70 views
How to derive differential gradient descent from cost function using matrix calculus?
I am new to matrix calculus and I hope you can help me with this question!
Iām trying to derive the update law for gradient descent to minimize a cost function
$$j(\mathbf{X}) = \frac{1}{2} \mathbf{e}^...
0
votes
2answers
75 views
Differentiate vector transpose using rules [duplicate]
I am referring to Tom Minka's Old and New Matrix Algebra Useful for Statistics. I don't have the book by Magnus & Neudecker so I can't refer to the details of the theory.
Regarding rules (6): $d(...
1
vote
1answer
39 views
Calculating the partial derivative with respect to a matrix
Say, $y = W^T b$ where, $ W^T \in R^{2 \times 2} $ and $ b \in R^{2 \times 1} $. Now, we want to calculate: $\frac{\partial y}{\partial W}$.
First, if we look at the dimensions: $y \in R^{2 \times 1}$....
1
vote
1answer
35 views
Computing the Jacobian $J_F$ with $F = h \circ f$
Let $f : \mathbb{R}^l \rightarrow{} \mathbb{R}^m$ and $h : \mathbb{R}^m \rightarrow{} \mathbb{R}^o$
and let $F = h \circ f$ with $F : \mathbb{R}^l \rightarrow{} \mathbb{R}^o$
I want to ...
4
votes
2answers
78 views
Find minimum of $f(\mathbf{v})= \mathbf{v}^\top A \mathbf{v} - \sum_{i=1}^N \log((\mathbf{v}^\top \mathbf{u}_i)^2) $
Consider the function:
$$ f(\mathbf{v})= \mathbf{v}^\top A \mathbf{v} - \log((\mathbf{v}^\top \mathbf{u})^2), $$
where $\mathbf{v}, \mathbf{u} \in \mathbb{R}^n$ and $A \in \mathbb{R}^{n \times n}$ is ...
0
votes
0answers
32 views
Graph Theory Question regarding Isomorphism and Matrix Transposes
How would one prove true or false the statement "for an I, incidence matrix of a graph G, and a J, incidence matrix of a graph H, it is impossible for G and H to be isomorphic AND for the product ...
1
vote
0answers
42 views
matrix algebra problem
For every $ n> 2, n\epsilon \mathbb{N}$ let A,B,C,D be matices $ A,B,C,D\epsilon M_{n}(\mathbb{R})$ such that $ AC-BD=I_{n}$ and $AD+BC=0_{n} $. Prove that:
a) $CA-DB=I_{n}$ and $DA+CB=0_{n}$
b)$ ...
0
votes
1answer
32 views
Solution for nonhomogenous linear matrix ODE
For matrices $X, A, B\in \mathbb{R}^{n\times n}$, $X = X^\top, B=B^\top$, I am looking to find an explicit solution to the following linear first-order matrix ODE where $A$ and $B$ are constant;
$
\...
0
votes
2answers
38 views
matrix for this system of non-linear equation?
Let's consider the product of a square matrix and a vector: $
M{\times}v =
\begin{bmatrix} 1 & 2 & 3\\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{bmatrix}
{\times}
\begin{bmatrix} a \\ b \\ c ...
0
votes
1answer
52 views
Does there exist a way to compute matrix “properties” (eigenvalues, determinant, etc.) from its vectorized representation?
I am wondering whether there are some formulas that compute matrix eigenvalues, determinant, rank, etc. directly from its vectorized representation.
More formally: suppose $\bf S \in \mathbb{S}^N$, i....
0
votes
0answers
30 views
Power series expansion for a circulant matrix
Let $A$ the block matrix given by the blocks:
$$\tilde{A}=\begin{pmatrix}
1&-\mu&0&...&0&-\mu\\
-\mu&1&-\mu&...&0&0\\
0&-\mu&1&...&0&0\\
...&...
0
votes
1answer
42 views
Derivative of $\exp[(aX+Y)t]$ w.r.t $a$ where $X,Y$ are matrices of dimension $n\times n$
Let $X,Y\in\mathbb{R}^{n\times n}$ and define
$$
F(a,t) = \exp[(aX+Y)t].
$$
Then, compute
$$
\frac{\partial F}{\partial a} = ?
$$
My attempt: Let $M_a = aX + Y$, then
$$
\frac{\partial F}{\partial a} =...
3
votes
0answers
72 views
KKT conditions for $\max \log \det(X)$ with LMI constraints
I am trying to derive the KKT conditions for the following convex optimization problem where $A$ is a given matrix:
$$\begin{array}{ll} \underset{X,Y,Z}{\text{minimize}} & - \log \det \left(I + Z +...
0
votes
1answer
49 views
Find the gradient (with respect to a matrix) of an expression containing a Frobenius norm and a Hadamard product.
I'm struggling with taking the gradient with respect to (w.r.t) the matrices $H_R$ and $H_I$ in the following expression
$$\left\| Z - I\odot(H_R^TA + H_I^TB) \right\|_F^2 + \left\|W-\begin{pmatrix} ...
3
votes
0answers
40 views
Derivative of $(A+BC^{-T}B^T)^{-1}BC^{-1}$
Suppose $A:\mathbb{R}\rightarrow\mathbb{R}^{m\times m}$, $B:\mathbb{R}\rightarrow\mathbb{R}^{m\times n}$, and $C:\mathbb{R}\rightarrow\mathbb{R}^{n\times n}$. I'm trying to find the derivative with ...
4
votes
0answers
116 views
How to show that a matrix is not full rank using the linear independence definition
I want to show that the matrix $A$ defined below is NOT full column rank. I have a proof but it does not look complete to me. Could you help to fix it?
Consider a matrix $A$ structured as follows
$$
A=...
0
votes
1answer
45 views
Matrix norm differentiation problem.
Consider $J = \|X - AS\|^2$, where $A \in \mathbb{R}^{N \times R}$, $S \in \mathbb{R}^{R \times M}$.
We need to calculate $\dfrac{d J}{d S}$.
I've done the following:
$J = \operatorname{tr}((X - AS)(X ...
1
vote
0answers
30 views
Computational complexity of the Cholesky factorization
According to the Cholesky factorization on Wikipedia, the computational complexity of it is $\frac{n^3}{3}$ FLOPs where $n$ is the size of the considered matrix $\mathbf{A}$.
There are various ...
0
votes
1answer
39 views
Square root of a block matrix
Given the $2n \times 2n $ matrix
$$A=\begin{pmatrix}
\bar{A} & A^* \\
A^* & \bar{A}
\end{pmatrix}$$
let $X$ the matrix such that $XX=A$, that is $X=\sqrt{A}.$
So $$X=\begin{pmatrix}
X_{1,1}&...
0
votes
0answers
26 views
Square root of a circulant block matrix
I'm trying to show the following:
Given the following $n\times n$ symmetric circulant matrices
$$A^*=\begin{pmatrix}
1 & -\mu_a & 0 & ...&0&-\mu_a \\
-\mu_a & 1 & -\mu_a &...
1
vote
1answer
35 views
Is trace( inv(A) B ) a distance?
Given two symmetric p.d. matrices of dimension $(p \times p)$, $\mathbf{A}$ and $\mathbf{B}$, can we interpret
$$
d = \operatorname{trace}(\mathbf{A}^{-1} \mathbf{B})
$$
as a distance between the two?
...
6
votes
2answers
88 views
Matrix equation with diagonal matrix
Let $A \in\mathbb{R}^{n\times n}$ be a given Positive-Semi Definite matrix ,$\theta\in\mathbb{R}^n$ a vector, $x\in\mathbb{R}$ an unknown variable, and $a\in\mathbb{R}^+$ a positive constant. I have ...
0
votes
1answer
47 views
conjugate function of log trace of matrix exponential
Given an n-by-n real symmetric matrix X, define
$$e^X = \sum_{k=0}^{n}\frac{X^k}{k!}$$
Derive the Fenchel conjugate of
$$f(X) = log(Tr(e^X))$$
as a function on n-by-n real symmetric matrices. Here Tr ...
1
vote
0answers
29 views
How to derive the linearized form of the PDE the “singular values are constant”?
$\newcommand{\Cof}{\operatorname{Cof}}$
Let $0<\sigma_1<\sigma_2<\cdots<\sigma_n$ and set $A=\operatorname{diag}(\sigma_1,\dots,\sigma_n)$.
Claim: Let $B $ be a real-valued $n \times n$ ...
3
votes
2answers
28 views
Good definition for $\int_a^\infty X(t)dt$, where $X(t)\in M_n$
Any comments on this? : I'm reading a book, and some exercise says: Provide a good definition for $\int_a^\infty X(t)dt$, where $X(t)\in M_n(\mathbb{R})$ and prove that if $$\int_a^\infty \|X(t)\|dt \ ...
2
votes
1answer
20 views
Equivalence $\|A\|=\inf\left\lbrace c>0:\|Ax\|\leq c\|x\| \right\rbrace$
Prove that if $A$ is a matrix, and we defined $$\|A\|=\sup\left\lbrace\frac{\|Ax\|}{\|x\|}:x\neq 0\right\rbrace $$
then $$\|A\|=\inf\left\lbrace c>0:\|Ax\|\leq c\|x\| \right\rbrace $$
Attempt: $\|A\...
2
votes
2answers
98 views
Implicitly Differentiating a System of Matrix Equations
Setup
Let $\mathbf a$ be an arbitrary $m\times 1$ vector, $\mathbf B$ be an arbitrary $n\times m$ matrix, with $m>n$, and $\mathbf C$ be a symmetric $m\times m$ matrix. The scalar $\lambda$ is also ...
1
vote
1answer
48 views
Loss function gradiant - why the transpose?
I have a product of two matrices, defined as $Y=AB$. I need to find $\nabla_A J$ where $J(Y)$ is a scalar-valued loss function dependent on $Y$. As I understand, this gradient can be written as $\...
1
vote
1answer
39 views
Derivative Matrix with general function and product
Given $F,f:\mathbb R\to\mathbb R$ such that $F'=f$ and $\pmb a,\pmb b\in\mathbb R^n$, compute
$$\frac{d}{d\pmb X}\left(\pmb a^T F\left(\pmb X\right)\pmb b\right)$$
where $\pmb X\in\mathbb R^{n\times n}...
2
votes
0answers
75 views
Why is there a difference between MATLAB and manual calculation for eigenvectors
We have a matrix $A$ as below
$$\begin{bmatrix}1&2&2\\0&1&2\\0&0&2\end{bmatrix}.$$
I know the eigenvalues are 1 and 2 because matrix $A$ is a triangular matrix.
And by working ...
1
vote
0answers
38 views
Spectral norm of $(1/(i-j))$ is smaller than $\pi$
Could someone give me a hint on how to prove that the spectral norm of an $nĆn$ matrix $A = (a_{ij})$, where $a_{ii} = 0$ and $a_{ij} =1/(i-j)$, is smaller than $\pi$.
$$B=\begin{bmatrix}0 & -1 &...
1
vote
1answer
25 views
Applying the chain rule in matrix form to prove Loss function's derivatives…
I want to prove $\nabla_A J = \nabla_Z J \cdot B^T$ where $Z=AB$. $A$ is a $m \times n$ matrix and $B$ is a $n \times k$ matrix. The function $J$ is not given to me.
I began this proof by first ...
0
votes
1answer
40 views
How to take the derivative of $ y = xA^T + b$ w.r.t. $A$ and $b$, where $x,b$ are vectors and $A$ is a matrix
How exactly could I take the derivative of the following expression?
$$ y = xA^T + b$$
Let's say that I have $x \in \mathbb{R}^{n}$, $A \in \mathbb{R}^{m,n}$, $y \in \mathbb{R}^{m}$, and $b \in \...
0
votes
1answer
22 views
Derivative of trace involving hadamard product and product of inverse matrices
I need to find the derivative with respect to $\mathbf{\Omega}$ of
$$
Tr\left(\left(\left(\mathbf{\Omega^{-1}}\mathbf{C}\mathbf{\Omega^{-1}}\right)\circ\mathbf{I}\right)\mathbf{S}\right)
$$
In the ...
2
votes
1answer
41 views
Calculate the gradient and Hessian of $x_0^T(X\operatorname{Diag}(w)X^T)^{-1}x_0$ w.r.t. vector $w$ in matrix calculus?
I'm trying to calculate the Hessian matrix of the formula
$$
f(w) = x_0^T(XWX^T)^{-1}x_0
$$
where $w=(w_1,\cdots ,w_n)^T$, $W=\operatorname{Diag}(w)$ is a diagonal matrix that has $w$ as diagonal ...
2
votes
1answer
30 views
Derivative of submatrix with respect to the whole block matrix
I am reading a research paper and getting stucked with how they derived a formula. Suppose that we have the following block matrix
$$\underset{d \times d}{\boldsymbol{A}} = \begin{bmatrix} \underset{q ...
1
vote
2answers
49 views
Change from differentiation wrt to matrix to wrt to inverse of matrix for symmetric matrices
For the rule below:
$$
\frac{\partial J}{\partial \mathbf{A}}= -\mathbf{A}^{-T} \frac{\partial J}{\partial \mathbf{W}} \mathbf{A}^{-T}
$$
where $\mathbf{A}$ is an invertible square matrix, $\mathbf{W}...
2
votes
1answer
56 views
Is the support of $\rho_{AB}$ always contained in the support of $\rho_A\otimes\rho_B$?
Given a positive unit trace Hermitian matrix, i.e. a density operator, $\rho_{AB}$ on Hilbert spaces $\mathcal{H}_A\otimes\mathcal{H}_B$. Consider its marginals $\rho_A$, $\rho_B$. Do we have the ...
3
votes
4answers
66 views
Matrix Derivative Product Rule $\frac{d}{dx} \langle x,y\rangle Ax $
Let $A\in\mathbb{R^{n\times n}}$,$x,y\in\mathbb{R^{n}}$. What is the derivative of the following expression?
$$
\frac{d}{dx} \langle x,y\rangle Ax
$$
What I tried:
Applying the product rule doesn't ...
0
votes
0answers
36 views
Inequalities for trace of a product
For positive-definite matrices $A$ and $B$ the following inequalities hold:
$$
0\leq\lambda_{\min}(A)\lambda_{\max}(B)\leq\lambda_{\min}(A){\rm Tr}(B)\leq{\rm Tr} (AB)\leq\lambda_{\max}(A){\rm Tr} B\...
-2
votes
2answers
36 views
How may random unique combinations a 4x4 matrix can make
Let's say we have a matrix (grid) of 4x4 that has values from 0 to 15 in it.
0 1 2 3
a = 4 5 6 7
8 9 10 11
12 13 14 15
If this array is ...
0
votes
0answers
17 views
Lower estimation for trace of a product
Let $X\succ0$ and $Q$ are square real matrices.
For $Q\succ0$ we have the estimation
$$
{\rm Tr}(XQ)\succeq{\rm Tr}(X)\lambda_{\min}(Q).
$$
Is it possible to obtain the similar lower estimation for (...
