Using gradient descent and Newton's method combined I have this function $f(\mathrm{X})$ where $\mathrm{X=A+B+C}$ where $\mathrm{A}$ is a diagonal element with variable $a$ on its diagonal. $\mathrm{B}$ is another diagonal matrix with variable $b$ on its diagonal. $\mathrm{C}$ is a positive definite matrix with variable $c_{ij}$. Now I want to optimize this function over these variables. I was wondering to keep $\mathrm{A}$ and $\mathrm{B}$ constant first optimize over $\mathrm{C}$ using Newton's method. Then I will keep $\mathrm{C}$ constant and optimize over $a$ and $b$ using gradient descent.
I am not sure if this is applicable or will work. Suggestions guys?
 A: I'm dropping that $\tfrac{1}{2}$ for simplicity; it can be buried in the $y$ vector. Assuming the function is
$$\begin{aligned}
f(X) &= y^T X y + z^T X^{-1} z - \log\det(X) \\
     &= \langle yy^T, X \rangle + \langle zz^T, X^{-1} \rangle - \log\det(X)
\end{aligned}$$
then the gradient is
$$\nabla f(X) = yy^T - X^{-1} zz^T X^{-1} - X^{-1}$$
I'd consider just doing a gradient descent with a backtracking line search to start. Because our variable is a matrix instead of a vector, it takes a bit of care to get the computations right, but it's not too bad. There are plenty of good explanations of gradient descent with backtracking line search available with a simple Google search. In short:


*

*Compute $V=-\nabla f(X)$.

*Find $t>0$ such that $f(X+tV)=f(X)-\tfrac{1}{2}t \|V\|_F^2$

*Update $X\leftarrow X + tV$ and repeat


To find $t$, one typically just starts with $t=t_\max$ (say, 1) and cuts in half until the condition is satisfied. Again, Google is your friend.
A: Here's some sample Matlab code that uses CVX to solve a problem similar to yours.  However, I don't yet know how to handle the $r^T X^{-1} r$ term.
N = 30;
y = randn(N,1);

cvx_begin sdp

    variable X hermitian
    minimize(.5*y'*X*y - log_det(X))
    subject to
        X >= 0

cvx_end 

A: This is a heuristic approach and will get you to a local optima. From comments of OP, I understand that the objective is $$f(X)=y^TXy+r^TX^{-1}r+\log(\det(X))$$ Since $X$ is positive definite, $X$ can be decomposed as $U\Sigma U^T$ where $U$ is a orthonormal matrix of eigenvectors and $\Sigma$ is the diagonal matrix containing the singular values. Thus an heuristic approach would be to alternatively optimize the objective between $U$ and $\Sigma$. This means that we optimize the objective on one variable while fixing the other and then continue this. Note that this is a heuristic approach with no guarantees of global convergence. Now define $Y=yy^T$ and $R=rr^T$. Convince yourself of the following $$y^TXy=\mathrm{Tr}(\Sigma  U^TYU)$$where $\mathrm{Tr}$ is trace of a matrix. Also $$r^TX^{-1}r=\mathrm{Tr}(\Sigma^{-1}  U^TRU)$$ and $$\log(\det(X))=\sum_{i=1}^{N}\log\sigma_i$$ where $\sigma_i=[\Sigma]_{ii}$ are the singular values. Define $Y_u=U^TYU$ and $R_u=U^TRU$. Then fixing $U$, the objective becomes a function of singular values given as $$f(\sigma_i)=\sum_{i=1}^N\sigma_i[Y_u]_{ii}+{1\over \sigma_i}[Y_r]_{ii}+\log(\sigma_i)$$ and the minimization problem is $$\min_{\sigma_i}f(\sigma_i) \\ \sigma_i>0$$ It shouldn't be hard to reduce this further and find solutions for $\sigma_i$. Now fixing the singular values, the objective becomes $$f(U)=\mathrm{Tr}(\Sigma  U^TYU)+\mathrm{Tr}(\Sigma^{-1}  U^TRU)$$ and optimization problem is $$\min_{U}f(U) \\s.t.~~U^TU=I$$ This is a well-studied problem referred to as optimization with orthogonality constraints in literature. Please search for papers in this. You might even find existing packages that do this. Thus alternating between $U$ and $\Sigma$ will get you a local optima for $X$.
