3
$\begingroup$

I don't quite understand the following functional derivative computation when I read a variational inference literature, can someone explain? $$L[q] = E_{q(Y)}[f(Y)]$$ $$\frac{\delta L[q]}{\delta q} = f(Y)$$ Here, $q(Y)$ is a probability density function. The original deduction is here, the relevant steps are equation (14) (17) (under assumption (11)).

I don't know why $q \rightarrow 0 \Rightarrow \int f(Y)q(Y)dY \rightarrow f(Y)$, it seems $q$ acts as a Dirac delta function in the limit case. Am I right? If so, how to argue? Any insights are welcome.

$\endgroup$
1

1 Answer 1

2
$\begingroup$

The paper you linked makes several abuses of notation. Here, since the considered functional doesn't contain any derivative of $q$, the functional differential is given by $$ \delta L[q] = \delta\mathbb{E}_q[f(Y)] = \delta\int q(y)f(y) \,\mathrm{d}y = \int \frac{\partial}{\partial q}\left(q(y)f(y)\right)\delta q(y) \,\mathrm{d}y = \int f(y)\delta q(y) \,\mathrm{d}y, $$ hence $$ \frac{\delta L[q]}{\delta q(x)} = \int f(y)\frac{\delta q(y)}{\delta q(x)} \,\mathrm{d}y = \int f(y)\delta(y-x) \,\mathrm{d}y = f(x). $$ The relation $\frac{\delta q(y)}{\delta q(x)} = \delta(y-x)$ is the functional analog of the more classical $\frac{\mathrm{d}x_j}{\mathrm{d}x_i} = \delta_{ij}$.

$\endgroup$
3
  • $\begingroup$ Clear and convincing. Apparently I am not used to the basic functional calculus operations. Do you have any recommendations for reading, like 'functional calculus cookbook' sort of thing? I only have very limited knowledge to it from some physics lectures. $\endgroup$ Commented Mar 19, 2023 at 7:52
  • 1
    $\begingroup$ Not really, sorry :/ I learnt about functionals mainly on the job. Since you seem to want to use functional derivatives only (instead of constructing all the theory), I guess the following summary should be sufficient, especially Eq.(8) : publish.illinois.edu/yubo-paul-yang/files/2014/09/… $\endgroup$
    – Abezhiko
    Commented Mar 19, 2023 at 9:45
  • $\begingroup$ @Abezhiko Found this great explanation today, and I have one question. Could you comment a bit more on why $\frac{\delta q(y)}{\delta q(x)} = \delta (y - x)$ true? Is $\frac{\delta q(y)}{\delta q(x)} = \delta (y - x)$ true for arbitrary function $q$ or is it because of some special property of $q$ being the probability mass function in this specific example? It seems to me that if, for instance, $q(x) = \cos x$, then for some values of inputs (e.g. $x = 0, y = 2\pi$), $\frac{\delta q(y)}{\delta q(x)} \neq 0$ even if $x \neq y$. $\endgroup$ Commented Jul 19, 2023 at 21:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .