When taking the $\frac{\partial{\hat{y}}}{\partial{\mathbf{V}}}$ where $\hat{y}$ is a scalar and $\mathbf{V} \in \mathbb{R}^{n \times n}$, how do I calculate the gradient where:

$$ \begin{align} \hat{y} &= \mathbf{W}\mathbf{h_t^T}; \mathbf{W,h_t} \in \mathbb{R}^{n}\\ \mathbf{h_t} &= \mathscr{O} \odot \theta; \mathscr{O},\theta \in \mathbb{R}^{n}\\ \mathscr{O} &= \operatorname{\sigma}(\mathbf{V}\mathbf{h_{t-1}^T}); \end{align} $$

Where $\sigma$ is just the vector version of the expit/sigmoid function.

  • $\begingroup$ Your dimensions don't work for $W$. Or perhaps you meant to write $$y=W^T\cdot h_t$$ Also $\frac{\partial y}{\partial V_3}$ is the gradient of a vector with respect to a matrix, which is a 3rd-order tensor not a matrix. Unless you meant vectorize the matrix, e.g. $$\frac{\partial y}{\partial {\rm vec}(V_3)} = h^T_{t-1}\otimes(WR(Q-Q^2))$$ where $$\eqalign{R &= {\rm Diag}(\theta)\cr Q &= {\rm Diag}(\sigma)\cr}$$Hmm, I can't get mathscr(O) to work in the comment,so I used a sigma. $\endgroup$ – john316 Jan 9 '16 at 17:57
  • $\begingroup$ Just got it worked out. This isn't really correct. The diagonals are adding in terms when you multiply through. $\endgroup$ – donlan Jan 10 '16 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.