4

We can use the Schur complement of $A$ to get the formula $$ B^{-1} = \pmatrix{A^{-1} + A^{-1}\mathbf 1 (B/A)^{-1}\mathbf 1' A^{-1} & -A^{-1}\mathbf 1(B/A)^{-1}\\ -(B/A)^{-1}\mathbf 1' A^{-1} & (B/A)^{-1}}, $$ where $$ B/A = 0 - \mathbf 1' A^{-1} \mathbf 1 = -\mathbf 1' A^{-1} \mathbf 1 \in \Bbb R. $$ We can rewrite the above in the following ...


3

You can't. Define $$v_1 = \left[ \begin{array}{c} 1 \\ 0 \\ 0 \end{array} \right]$$ and $$v_2 = \left[ \begin{array}{c} 0 \\ 1 \\ 0 \end{array} \right]$$ By your definition: $$M v_1 = \left[ \begin{array}{c} 1 \\ 4 \\ 7 \end{array} \right]$$ and $$M v_2 = \left[ \begin{array}{c} 2 \\ 5 \\ 8 \end{array} \right]$$ However, $$M (v_1 + v_2) = \left[ \begin{array}...


3

Because $A$ is symetric psd matrix, we can diagonalize $A$ as: $A = UDU^T$ where $U$ is a orthogonal matrix ($U^T = U^{-1}$) and $D$ is a diagonal matrix. Hence, we have \begin{align} (A+xI)^{-1} &= (UDU^T+xUU^T)^{-1} \\ &= (U(D+xI)U^{-1})^{-1} \\ &= U(D+xI)^{-1}U^{-1} \\ \end{align} Then \begin{align} a & =\theta^TA(A+xI)^{-1}(A+xI)^{-1}A\...


2

Using matrix units, you can write $$ X(t)=\sum_{k,j} x_{kj}(t)\,E_{kj}, $$ where $x_{kj}:[a,\infty)\to\mathbb R$ are scalar functions. An integral is a limit of sums of the form "value of the function times size of the region". In particular, you expect an integral to be linear. So you want $$ \int_a^\infty X(t)\,dt=\sum_{kj}\bigg(\int_a^\infty x_{...


2

Let's call this infimum $I$. For all $x$ we have $||Ax||\leq ||A||\cdot ||x||$, and so $I\leq ||A||$. Conversely, if $||Ax||\leq C||x||$ for all $x$, then for all $x\ne 0$ we have $\frac{||Ax||}{||x||}\leq C$. By taking the supremum on $x\ne 0$ we obtain $||A||\leq C$. Now we take the infimum on all such values of $C$ to get $||A||\leq I$.


2

This is a pretty straightforward application of the chain rule. Given $$\begin{aligned} 0 &= f(\lambda, w(\lambda), s(\lambda)) = (1-\lambda s) w - 1 \\[2ex] \implies 0 &= \frac{{\rm d\,} f(\lambda, w(\lambda), s(\lambda))}{{\rm d\,} \lambda} = \frac{\partial f}{\partial \lambda} + \frac{\partial f}{\partial w}\frac{\partial w}{\partial \lambda} + \...


2

The answer is no. Consider the case $A,B$ are diagonal positive definite matrices. It is easy to make $d$ arbitrarily small/arbitrarily close/equal to $p$ without the matrices being close/equal. To make this clear, let $x_1,..., x_p$ be arbitrary positive numbers with $x_1+...+x_p=p$. Set $A=I, B= \mbox{diag}(x_1,...,x_p)$. Then $d(A,B)=p$.


2

One standard approach to computing matrix functions times a vector $f(M)x$ or quadratic forms $x^Tf(M)x$ when $M$ is symmetric is via the Lanczos algorithm. Lanczos computes an orthonormal basis $Q_k = [q_1, \ldots, q_k]$ for Krylov subspace $\operatorname{span}(x,Ax,\ldots, A^{k-1}x)$ by a Gram-Schmidt like procedure. This results in a factorization $AQ_k =...


2

It's just the matrix multiplication of the two Jacobians. In fact, this can be seen as the reason why matrix multiplication is defined like that. If you see a $m\times n$ matrix as a linear function from $\mathbb R^n \to \mathbb R^m$, then the Jacobian of the matrix is the matrix itself. The matrix multiplication is just the composition of the two linear ...


2

$\def\B{\Big}\def\L{\left}\def\R{\right}\def\o{{\tt1}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$Define the all-ones vector $\o$ and a vector $v$ such that $$\eqalign{ v\odot v &= Ax\odot Ax + \delta^2\o \\ 2\,v\odot dv &= 2\,Ax\odot A\,dx \\ dv &= Ax\odot A\,dx\oslash v \\ }$$ where $\odot$ denotes the elementwise/Hadamard product and $\...


2

Rearranging a bit and renaming $\lambda=1/v^T u$ we obtain $$Av=\lambda u\iff v=\lambda A^{-1}u$$ Dotting with $u$ yields the equation that determines $\lambda$: $$\lambda^2=\frac{1}{u^T A^{-1}u}$$ which if $A$ is positive definite yields two solutions: $$v=\pm\frac{A^{-1}u}{\sqrt{u^TA^{-1}u}}$$ both of which are minima of the function. EDIT: To reflect the ...


1

The answer is yes. In particular, see corollary 5 of this document.


1

Ok, lets look at the general definition of a differential $df$ on normed spaces, for $f:\mathbb{S}\to\mathbb{P}$ $$f(x+h)=f(x)+df(x,h)+o(||h||_S)$$ where the differential $df(x,h)$ is linear and continuous in $h$. Lets look at $\mathbb{S}=\mathbb{R}^m,\mathbb{P}=\mathbb{R}^n$. Then any linear operator $\mathbb{R}^m\to\mathbb{R}^n$ can be represented by some ...


1

Your work suggests the problem is equivalent to maximizing $|v^\top u|$ subject to the constraint $v^\top A v = 1$. Making the change of variables $z=A^{1/2} v$ and $y=A^{-1/2} u$, this is equivalent to maximizing $|z^\top y|$ subject to $z^\top z = 1$. By Cauchy-Schwarz, this is attained when $z = \pm y/\|y\|$. Working backwards, this leads to $v = A^{-1/2} ...


1

$\def\m#1{\left[\begin{array}{c}#1\end{array}\right]}\def\p#1#2{\frac{d #1}{d #2}}$Given a scalar function, like $f(x)=\exp(x)$, and the matrices $$\eqalign{ M = M(a)&= aXt + Yt,\qquad &\dot M = \dot M(a) = \p{M}{a} = Xt \\ F = F(a) &= f(M), &\dot F = \dot F(a) = \p{F}{a} \\ }$$ Evaluating the function with a block-triangular argument yields $...


1

$\def\R#1{{\mathbb R}^{#1}}\def\v{{\rm vec}}\def\M{{\rm Reshape}}\def\m#1{\left[\begin{array}{r}#1\end{array}\right]}\def\p#1#2{\frac{\partial #1}{\partial #2}}$For ease of typing, replace the subscripted variables with single-letter names $$\eqalign{ R = H_R \qquad Q = H_I \qquad P=P_d \\ }$$ and define the matrices $$\eqalign{ I_2 &= \m{1&0\\0&...


1

$\def\m#1{\left[\begin{array}{c}#1\end{array}\right]}$Multiplying two block-wise centrosymmetric matrices yields $$\eqalign{ \m{A&B\\B&A}\cdot\m{X&Y\\Y&X} &= \m{(AX+BY)&(AY+BX)\\(BX+AY)&(BY+AX)} \\ }$$ Since matrix addition commutes $\big({\rm i.e.}\;(BX+AY) = (AY+BX)\big)$ the product is also block-wise centrosymmetric. Therefore ...


1

This is not even true for a $2 \times 2$ matrix, take $$A = \begin{pmatrix} 1 & 2 \\ 2 & 3 \end{pmatrix}, \quad A^{-1} = \begin{pmatrix} -3 & 2 \\ 2 & -1 \end{pmatrix}$$


1

First, note that your $W_{n+1}$ should be a vector to get 1d output. So we need to treat the cases where an index $=n+1$ anywhere differently from cases where all indices $<n+1$. Using differentials we have $$dz_i=f'(W_iz_{i-1})\circ dW_iz_{i-1}+f'(W_iz_{i-1})\circ W_idz_{i-1}$$ Using the chain rule for differentials we have $$dMSE[z_{i+1}(z_i),dz_i]=dMSE[...


1

First, it is important to see how matrices act on vectors. In our case, we are transforming vectors from $\mathbb{R}^4$ into vectors in $\mathbb{R}^5$ (i.e. T will transform a vector $x = (x_1,x_2,x_3,x_4)$ into $b =(b_1,b_2,b_3,b_4,b_5)$. If we write a general transformation T: \begin{equation} \begin{bmatrix} t_{11} & t_{12} & t_{13} & t_{14}\\ ...


1

$\def\T{\operatorname{Tr}}\def\p#1#2{\frac{\partial #1}{\partial #2}}$For typing convenience, let's define the matrix variable $$B=S\circ S\circ A$$ and use a colon to denote the trace/Frobenius product $$\eqalign{ A:B &= \T(A^TB) \\ A:A &= \big\|A\big\|_F^2 \\ }$$ The properties of the trace allow the terms in such a product to be rearranged $$\...


1

These rules pertain to differentials not to gradients. Let's use them properly, starting with your second example function. $$\eqalign{ f_2 &= x^TAx \\ df_2 &= dx^TAx+x^TA\,dx \\ &= (Ax)^Tdx+(A^Tx)^Tdx \\ &= (Ax+A^Tx)^Tdx \\ \frac{\partial f_2}{\partial x} &= (Ax+A^Tx) \\ }$$ Setting $A=I$ turns this into your first function. ...


1

Let's use the numerator-layout notation. First note that $\frac{dx}{dx}=I$ but $\frac{dx^T}{dx}=\begin{bmatrix}\begin{pmatrix}1&0&...&0\end{pmatrix},\begin{pmatrix}0&1&0&...&0\end{pmatrix},...,\begin{pmatrix}0&0&...&0&1\end{pmatrix}\end{bmatrix}$, a tensor, technically 1 x n x n. In denominator layout fashion, $\...


1

According to my calculations, $$\begin{matrix}\frac{dy_1}{dW_{11}}=b_1&\frac{dy_1}{dW_{12}}=b_2&\frac{dy_1}{dW_{21}}=0&\frac{dy_1}{dW_{22}}=0\\ \frac{dy_2}{dW_{11}}=0&\frac{dy_2}{dW_{12}}=0&\frac{dy_1}{dW_{21}}=b_1&\frac{dy_2}{dW_{22}}=b_2\end{matrix}$$ So it looks like $b^T\bigotimes I_{2\times 2}$ would give you $$\begin{pmatrix}b_1&...


1

$\def\T#1{\operatorname{tr}(#1)}$I would suggest using Newton's Method, i.e. $$\eqalign{ f(x) &= 0, \quad f_k &= f(x_k), \quad f'_k &= f'(x_k) \quad\implies\quad x_{k+1} &= x_k - \frac{f_k}{f'_k} \\ }$$ For typing convenience, define the matrices $$\eqalign{ B &= A+Ix \quad\implies\quad \frac{dB^n}{dx} = nB^{n-1} \\ M &= A\,\theta\...


1

By definition, the Fenchel conjugate of $f(X)$ is given by: \begin{equation} \label{eq:f}\tag{1} f^{*}(Y) = \sup_{X} X\circ Y - f(X). \end{equation} It can be shown that $g(X) \overset{\text{def}}{=} X\circ Y - f(X)$ is a concave function in terms of $X$. (Actually, you can use the second derivative of $f(X)$ with respect to $X$ to show that $f(X)$ is a ...


1

On way to define the operator-valued integral is as follow: Define $\sigma:\mathbb{R}^{n}\times\mathbb{R}^{n}\rightarrow\mathbb{R}$ by $\sigma(x,y)=\int_{a}^{\infty}\langle X(t)x,y\rangle dt.$ (Assuming that $t\mapsto\langle X(t)x,y\rangle$ is continuous if you are working with Riemann integral. However, this assumption can be weaken and only require that $t\...


1

For ease of typing, define $$\eqalign{ G &= \Gamma = (I+wC) \quad\qquad F=(BGB^T)^{-1} \quad\qquad p=(B^Th-a) }$$ and note that $F$ and $G$ are symmetric since $C^T=C.$ Next, eliminate the variable $s$ in favor of $w$ $$\eqalign{ s &= p^TCp \\ w &= (1-\lambda s)^{-1} \;\doteq\; (1-\lambda p^TCp)^{-1} \\ \tfrac{w-1}{w\lambda} &= p^TCp \\ }$$ ...


1

$\def\p#1#2{\frac{\partial #1}{\partial #2}}$There is a problem with your first derivative $$\eqalign{ y_{ij} &= a_{ik}b_{kj} \\ \p{y_{ij}}{a_{mn}} &= \left(\p{a_{ik}}{a_{mn}}\right)b_{kj} \\ &= \delta_{im}\delta_{kn}\;b_{kj} \\ &= \delta_{im}\,b_{nj} \\ }$$ If you switch the indices to $\Big((m,n)=(i,k)\Big)$, then you obtain $$\eqalign{ \...


1

Let's use a convention where a greek letter denotes a scalar, lowercase latin a vector, and uppercase latin a matrix. So replace your original variable names with $$\eqalign{ \alpha_k = h_k ,\qquad \phi(X) = f(X) ,\qquad x = z }$$ and define two new matrices which will be used later $$\eqalign{ P &= \sum_{k=1}^n \alpha_kX^k \\ M &= 2z(Pz-y)^T\\ }...


Only top voted, non community-wiki answers of a minimum length are eligible