# Minimizing a Least Squared Error cost function

I am looking to minimizing the following function:

$$\min_{f(\cdot)} E_{XY}[(x - f(\cdot))]$$

where $$f(\cdot)$$ describes linear functions of the form $$Ay + d$$. I know the following about distributions:

$$P_X(x) = \frac{1}{a}e^{\frac{-x}{a}}u(x)$$ $$P_{Y|X}(y|x) = xe^{-xy}u(y)$$

How would I go about solving this problem? I think that taking derivatives would be good and I know that I can find the joint distribution to use for the expectation, but the expectation of y is divergent. Any hints?

• It looks like you want to solve $\min_{A, d} E[X-(AY+d)]$, which is the same as choosing $A, d$ to minimize $E[X] - AE[Y]-d$. As long as $E[X]$ and $E[Y]$ are finite then the answer is to let $d\rightarrow\infty$ for an infimum value of $-\infty$. – Michael Oct 10 '18 at 22:26
• Of course, your title says "least squares" but there are no least-squares in your question. Perhaps you really want to choose $A, d$ to minimize $E[||X-(AY+d)||^2]$. This is $$E[||(X-AY) - d||^2] = E[||X-AY||^2] + 2(E[X]-AE[Y])^Td + ||d||^2$$ It would also help to know if $X, Y$ are vectors or not. Anway this is the approach. Standard linear MMSE problem. – Michael Oct 10 '18 at 22:27
• The least squares problem is not well posed when variances are infinite, so you either choose $A=0$ or you assume a different distribution under which variances are finite. When $A=0$ it is well known that the optimal $d$ is $E[X]$. – Michael Oct 10 '18 at 22:38
• Yes that is what I want to do. X and Y are random variables, distributed exponentially. Unfortunately, first and second moments of Y are both infinite, and so I was wondering what I should do. $A = 0$ makes sense, though I also think setting $A = \frac{2(x-d)}{y}$ would work right? – Samyak Shah Oct 10 '18 at 22:41
• If you are allowed to choose $A, d$ as functions of $X, Y$ then just choose $A=0, d=X$, to get $||X-(AY+d)||=0$ always. In standard estimation problems you are not allowed to choose such parameters as functions of the random variables themselves. You might also want to do nonlinear estimation, such as considering $f(Y)=d + A/(1+mY)$ – Michael Oct 10 '18 at 22:44