2
$\begingroup$

I'm stuck with understanding pretty simple expression and would appreciate some help on this. The most interesting part for algorithms, it's way how we can come here.

Using the original resources from Andrej Karpathy blog about Policy Gradient. Everything is clear with Monte Carlo credit assignments and supervised algorithms vs reinforcement. We have next expression, how we came up this optimization objective and gradient for it (images from another resources):

enter image description here enter image description here enter image description here

1) I'm familiar with derivation I think, but what was a point for taking log in this case? It's called likehood ratio trick sometimes and also explained here (where I still cannot get it). What is the point of using it here?

2) Can somebody show few Very simple examples of using it with numbers and how it works? Is there anything else about math I need to find or this could exist on Khan Academy?

References:

1) Deep Reinforcement Learning: Pong from Pixels

2) An introduction to Policy Gradients with Cartpole and Doom

3) Deriving Policy Gradients and Implementing REINFORCE

4) Machine Learning Trick of the Day (5): Log Derivative Trick 12

UPDATE

Please consider answering above two points. I don't need to find derivative of softmax and complicated output. I would appreciate some new explanation (different from articles above). And let's say that action space it's continues value and probability of taking action it's liner activation within very simple example.

$\endgroup$
1
$\begingroup$

The point of the logarithm trick is to be able to express $\nabla_\theta J(\theta)$ as an expectation. Computing (or, more correctly, estimating) an expectation is precisely what Monte Carlo methods try to do.

In more detail, $\pi(\:\cdot\:;\theta)$ is a probability distribution on the available actions. That is, $\pi(\tau;\theta) \geq 0$ for each $\tau$ and $\sum_{\tau} \pi(\tau;\theta) = 1$. If we can express $\nabla_\theta J(\theta)$ in the form $\sum_{\tau} \pi(\tau;\theta) \cdot f(\tau)$ for some function $f$, then the expression $\sum_{\tau} \pi(\tau;\theta) \cdot f(\tau)$ takes on the interpretation of the expected value $E_\pi [f]$. Now, to make this trick work, we have to deal with the reality that taking the gradient of $J(\theta)$ doesn't obviously produce such an expression. In particular, it looks like $$ \nabla_\theta J(\theta) = \sum_{\tau} \nabla_\theta \pi(\tau; \theta) R(\tau). $$ So, we employ the classic "multiply by 1" trick: $$ \sum_\tau \nabla_\theta \pi(\tau ; \theta) R(\tau) = \sum_{\tau} \pi(\tau; \theta) \cdot \frac{\nabla_\theta \pi(\tau;\theta)}{\pi(\tau;\theta)} R(\tau). $$ So we want to write $f(\tau) = \frac{\nabla_\theta \pi(\tau ; \theta)}{\pi(\tau ; \theta)} R(\tau)$. And if we are very clever, we recognize that $\frac{\nabla_\tau \pi(\tau;\theta)}{\pi(\tau;\theta)} = \nabla_\theta \log \pi(\tau;\theta)$, which is cleaner. This is why the policy gradient is expressed as an expectation with a logarithm inside.

$\endgroup$
1
  • $\begingroup$ Thank you for the response. However, in your sample there is derivation of softmax functions ∑τπ(τ;θ)=1. Lets abstract from this and say that probability of taking action it's continues space, and choosing action from weights it's just row linear activation. $\endgroup$
    – GensaGames
    Oct 30 '19 at 3:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.