A signal, X, is a random variable with the following density function:

$$f_X(x) =\begin{cases} \frac{3}{25}(x-5)^2, &0 \le x \le 5\\0, &otherwise \end{cases}$$

The signal is transmitted through an additive Gaussian noise channel, where the Gaussian noise has a mean of 0 and a variance of 4. The signal and noise are independent.

Find an expression for the conditional density function of the signal, given the observation of the output.

Perhaps I am just confused by the problem or the wording, but I am totally stuck on what to do.

I believe the output signal should be a convolution where Z = X + Y, and Y is the gaussian(0, 2). But this is a very difficult convolution. I am not able to squeeze a solution from that. And if I did, I am confused where to go from there.

The question is to find the density of $X \vert Z$. You can use Bayes formula $$f_{X \vert Z}(x,z) = f_{Z \vert X}(x,z) \frac{f_X(x)}{f_Z(z)}.$$ Now the density $f_Z$ is indeed the convolution $f_X * f_Y$, which you can calculate as $$f_Z(z) = \int f_{Z \vert X}(x,z) f_X(x) dx.$$