Fourier Transform of quadratic Volterra series Volterra functional series representation of nonlinear random processes
\begin{align}
y(t)&=h[x(t)]=y_1 (t)+y_2(t)+y_3 (t)+\ldots+y_n(t)\\
&=\int_0^th_1(\tau_1)x(t-\tau_1)d\tau_1+\int_0^th_2(\tau_1,\tau_2)x(t-\tau_1)x(t-\tau_2)d\tau_1d\tau_2+\ldots
\end{align}
To obtain the frequency domain response, take the Fourier transform of the above to the 2nd order:
\begin{align}
y(\omega)&=\mathcal{F}[y_1 (t)]+\mathcal{F}[y_2(t)]\\
&=H_1(\omega)X_1(\omega)+\frac{1}{2\pi}\int_{-\infty}^{\infty}H_2(\omega_1,\omega-\omega_1)X(\omega_1)X(\omega-\omega_1)d\omega_1-\overline{y_2}\delta(\omega)
\end{align}
where $\omega_1+\omega_2=\omega$.
It is easy to obtain the $Y_1=H_1(\omega)X_1(\omega)$ from 1st order convolution integral, but the 2nd order seems really hard for me, especially the $\delta(\omega)$ term. I assume $\overline{y_2}$ is the quadratic mean, i.e., constant in time $t$.
Is there any textbook stating the theorem?
Correction: $\frac{1}{2\pi}$
 A: I was working on a similar issue and knowing what the result should have been has helped me find the solution. I’m only going to work on the second order term and I’ll use uppercase letters to denote the Fourier transformed functions for clarity (and, out of laziness, I’m going to omit the bounds of integration).
$$Y_2(\omega) = \iiint h_2(\tau_1,\tau_2)x(t-\tau_1)x(t-\tau_2) e^{-i\omega t} dt d\tau_1 d\tau_2$$
The issue here is that both $x$ depend upont $t$, so I’m plopping in a $\delta$ to deal with that.
$$Y_2(\omega) = \iiiint h_2(\tau_1,\tau_2)x(\theta-\tau_1)x(t-\tau_2)\delta(\theta-t) e^{-i\omega t} d\theta dt d\tau_1 d\tau_2$$
Then I’m going to write the $\delta$ out in its integral form $\delta(\theta - t) = \frac{1}{2\pi} \int e^{-i \omega_1 (\theta-t)} d\omega_1$.
$$Y_2(\omega) = \frac{1}{2\pi} \int\!\!\!\iiiint h_2(\tau_1,\tau_2)x(\theta-\tau_1)x(t-\tau_2) e^{-i \omega_1 (\theta-t)} e^{-i\omega t} d\theta dt d\tau_1 d\tau_2 d\omega_1$$
$$Y_2(\omega) = \frac{1}{2\pi} \int\!\!\!\iiiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} x(\theta-\tau_1) e^{-i \omega_1 (\theta-\tau_1)} x(t-\tau_2) e^{-i (\omega-\omega_1) t} d\theta dt d\tau_1 d\tau_2 d\omega_1$$
With the substitution $\zeta_1 = \theta - \tau_1$ we can finally start getting rid of some of the mess.
$$Y_2(\omega) = \frac{1}{2\pi} \int\!\!\!\iiiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} x(\zeta_1) e^{-i \omega_1 \zeta_1} x(t-\tau_2) e^{-i (\omega-\omega_1) t} d\zeta_1 dt d\tau_1 d\tau_2 d\omega_1$$
$$Y_2(\omega) = \frac{1}{2\pi} \iiiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} X(\omega_1) x(t-\tau_2) e^{-i (\omega-\omega_1) t} dt d\tau_1 d\tau_2 d\omega_1$$
Similarly I can transform the other $x$ by taking $\zeta_2 = t - \tau_2$.
$$Y_2(\omega) = \frac{1}{2\pi} \iiiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} e^{-i (\omega-\omega_1) \tau_2}X(\omega_1) x(t-\tau_2) e^{-i (\omega-\omega_1) (t-\tau_2)} dt d\tau_1 d\tau_2 d\omega_1$$
$$Y_2(\omega) = \frac{1}{2\pi} \iiiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} e^{-i (\omega-\omega_1) \tau_2}X(\omega_1) x(\zeta_2) e^{-i (\omega-\omega_1) \zeta_2} d\zeta_2 d\tau_1 d\tau_2 d\omega_1$$
$$Y_2(\omega) = \frac{1}{2\pi} \iiint h_2(\tau_1,\tau_2) e^{-i\omega_1\tau_1} e^{-i (\omega-\omega_1) \tau_2}X(\omega_1) X(\omega-\omega_1) d\tau_1 d\tau_2 d\omega_1$$
And finally we just have the Fourier transform of $h_2$.
$$Y_2(\omega) = \frac{1}{2\pi} \int H_2(\omega_1,\omega-\omega_1) X(\omega_1) X(\omega-\omega_1) d\omega_1$$
As it has already been commented, the final term with the $\delta(\omega)$ should instead come from the zeroth order term which you omitted: $y_0(t) = h_0$, and the Fourier transform of a constant is indeed just a delta. No idea why it would be denoted as $\overline{y_2}$, but it could be a mistake in the slides you have.
