24
$\begingroup$

enter image description here

Could someone explain how that derivative was arrived at.

According to me, the derivative of $\log(\text{softmax})$ is
$$ \nabla\log(\text{softmax}) = \begin{cases} 1-\text{softmax}, & \text{if $i=j$} \\ -\text{softmax}, & \text{if $i \neq j$} \end{cases} $$ Where did that expectation come from? $\phi(s,a)$ is a vector, $\theta$ is also a vector. $\pi(s,a)$ denotes the probability of taking action a in state s.

$\endgroup$

1 Answer 1

43
$\begingroup$

The derivation of the softmax score function (aka eligibility vector) is as follows:

First, note that: $$\pi_\theta(s,a) = softmax = \frac{e^{\phi(s,a)^\intercal\theta}}{\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}}$$

The important bit here is that the slide only identifies the proportionality, not the full softmax function which requires the normalization factor.

Continuing the derivation:

Using the $\log$ identity $\log(x/y) = \log(x) - \log(y)$ we can write $$\log(\pi_\theta(s,a)) = \log(e^{\phi(s,a)^\intercal\theta}) - \log(\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}) $$

Now take the gradient:

$$\nabla_\theta\log(\pi_\theta(s,a)) = \nabla_\theta\log(e^{\phi(s,a)^\intercal\theta}) - \nabla_\theta\log(\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta})$$

The left term simplifies as follows:

$$left= \nabla_\theta\log(e^{\phi(s,a)^\intercal\theta}) = \nabla_\theta\phi(s,a)^\intercal\theta = \phi(s,a)$$

The right term simplifies as follows:

Using the chain rule: $$\nabla_x\log(f(x)) = \frac{\nabla_xf(x)}{f(x)}$$

We can write:

$$right = \nabla_\theta\log(\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}) = \frac{\nabla_\theta\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}}{\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}}$$

Taking the gradient of the numerator we get:

$$right = \frac{\sum_{k=1}^N{\phi(s,a_k)}e^{\phi(s,a_k)^\intercal\theta}}{\sum_{k=1}^Ne^{\phi(s,a_k)^\intercal\theta}}$$

Substituting the definition of $\pi_\theta(s,a)$ we can simplify to:

$$right = \sum_{k=1}^N{\phi(s,a_k)}\pi_\theta(s,a_k)$$

Given the definition of Expected Value:

$$\mathrm{E}[X] = X \cdot P = x_1p_1+x_2p_2+ ... +x_np_n$$

Which in English is just the sum of each feature times its probability.

$$X = features = {\phi(s,a)}$$

$$P = probabilities =\pi_\theta(s,a)$$

So now we can write the expected value of the features:

$$right = \mathrm{E}_{\pi_\theta}[\phi(s,\cdot)]$$

where $\cdot$ means all possible actions.

Putting it all together: $$\nabla_\theta\log(\pi_\theta(s,a)) = left - right = \phi(s,a) - \mathrm{E}_{\pi_\theta}[\phi(s,\cdot)]$$

$\endgroup$
6
  • $\begingroup$ Thanks for such an elaborate explanation :) $\endgroup$ Jul 16, 2017 at 15:45
  • 1
    $\begingroup$ Great answer, If I have the right to vote up 100 times, I would do that! $\endgroup$
    – GoingMyWay
    Sep 2, 2017 at 8:45
  • $\begingroup$ What will the equation in case I use State as input and W[state size, actions count] to calculate the SoftMax? $\endgroup$
    – Oleg Dats
    Mar 14, 2018 at 17:21
  • 1
    $\begingroup$ I'm just curious but, how far in a university education does one have to go to see this kind of math? Is this reserved for graduate students? $\endgroup$
    – Lex_i
    Dec 3, 2019 at 19:56
  • $\begingroup$ For me, the main insight was to simplify the gradient of the log sum from the denominator of the softmax using the definition of the softmax: $$\pi_{\theta}(s,a)$$. Thanks! $\endgroup$
    – ady
    Jul 22, 2020 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.