I have a matrix $A$ which is $n$ by $m$ and a vector $b$ which is $m$ by 1. I have the following expression:

$\frac{\delta}{\delta b} \mathbf{1}^{T}f(Ab)$

Where $\mathbf{1}$ is a vector of ones which is $n$ by $1$, $f$ is a function which is applied elementwise and $^{T}$ represents transpose.

How can I express this just in terms of $f'$, $A$ and $b$? And does anyone have any resources on vector differentiation that could help me.


Let $y=Ab,\,$ then the differential of the elementwise function $f(y)$ is given by $$\eqalign{ df &= f'\odot dy \cr }$$ where $\odot$ is the elementwise/Hadamard product.

Now find the differential and gradient of the cost function. $$\eqalign{ \phi &= 1:f \cr d\phi &= 1:df = 1:(f'\odot dy) = f':dy = f':A\,db = A^Tf':db \cr \frac{\partial\phi}{\partial b} &= A^Tf' \cr }$$ where : is the trace/Frobenius product, i.e. $\,A:B={\rm Tr}(A^TB)$


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.