First of all please, let me know if this question is more suited for scicomp.stackexchange.com or or.stackexchange.com, and sorry for my English and math skills.

Pretty often i do some numerical optimization stuff with Gauss-Newton and automatic differentiation. It's pretty handy because Gauss-Newton requires to solve following system: $$ J^TJ\Delta x = -J^Tr $$ where $r(x)$ is residuals function, $J$ is jacobian of $r$, $\Delta x$ is step. This system can be solved via Conjugate gradient method which requires $Av$ like products. Automatic differentiation allow me to compute $Jv$ and $J^Tv$ via forward and reverse mode of graph based automatic differentiation, so i can easily combine it with CG.

Unfortunately CG also needs preconditioner for fast convergence. Obvious choice for preconditioner is diagonal of $J^TJ$.

My question is - how can i quickly compute diagonal of $J^TJ$ if i know expression graph for $r$ and have implementation of $J^Tv$ and $Jv$ for each elementary function? Without symbolic differentiation, if it possible.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.