What are some general properties of $F(x, z) = \int f(x, y) f(y, z) dy$? What are some general properties of $F(x, z) = \int f(x, y) f(y, z) dy$ ?
For example, what can be said about the relation between $F(x,z)$ and $f(x,z)$?
Optical theorem in physics of wave scattering has this form. What other theorems in natural sciences and mathematics also have this form?
 A: I will instead consider the operation taking $f$ and $g$ to $F(x,z)=\int f(x,y) g(y,z)d y$. Everything said below is under mild assumptions, measurability or continuity.


*

*The operation is associative.

*As Noah observed, when the integral is a sum, it is matrix product.

*For symmetric functions: it induces inner product on space of continuous symmetric bivariate real functions.

*The formula reminds me of Möbius transform and incidence algebra, I think it is a continuous form of this.

*It reminds me also of composition of relations and composition of profunctors in category theory

*Also faintly reminds me of convolution
A: It is hard to give a detailed answer without knowing which kind of properties you're seeking.  One idea: it may be helpful to think of the analogous definition for $x$, $y$, and $z$ discrete, taking values in $\{1,\ldots, n\}$.  In this case $f$ is a matrix and $F$ is just $f^2$, the matrix product of $f$ with itself.  Depending on your situation this may yield some insights.  For example, if $f$ is a symmetric matrix then $f^2$ is positive semidefinite, and a similar statement should hold for the case of continuous variables if the terms are reinterpreted appropriately.
