# Linear Least Squares

I again found myself in trouble understanding the following problem (1b)

I understand linear least squares as that I have a data points, and I am trying to find a line between them which approaches given data the best and thus I want to minimaze the distance between the model and the real system. I know the formula, I know a way but I am quite clueless how to apply that.

My idea is $e[k]=y-\hat{y}$ and thus substituting that to the formula I listed above, and then using the system formula rewrite e as a function of theta and then do the partial derivation with respect to theta and so obtain a minimum.

Can anyone please give me a hint if I am thinking correctly and if so how to get rid of that norm ? Thanks a lot

Note that $$\hat{y}[k] = \theta u[k].$$ Consequently, $$\|y-\hat{y}\|^2 = \sum_{k=0}^{N-1}(y[k]-\theta u[k])^2.$$ In order to compute $\theta$, we take the derivative of the above expression w.r.t. $\theta$ and equate to $0$. That is, $$\frac{\partial }{\partial \theta}\sum_{k=0}^{N-1}(y[k]-\theta u[k])^2 = -2\sum_{k=0}^{N-1}(y[k]-\theta u[k])u[k] = 0 \implies \theta \sum_{k=0}^{N-1}(u[k])^2=\sum_{k=0}^{N-1}y[k]u[k].$$ Therefore, $$\hat{\theta} = \frac{\sum_{k=0}^{N-1}y[k]u[k]}{\sum_{k=0}^{N-1}(u[k])^2}.$$

Obviously, $\theta$ cannot be estimated when $u[k]=0$.

Lastly, use the fact that $E[y[k]]=E[\theta u[k] + e[k]] = \theta u[k] + 1$ to calculate the bias of $\hat{\theta}$.

• Ok thank you, this answers my question what happens to the norm, can you please just explain how is it possible that we do not count with noise at all? It seems a little bit weird to me, why were we given a noise distribution information when I dont get to use that.. – user118331 Sep 25 '17 at 16:38
• You use the noise information to calculate the bias in the part (c). – Math Lover Sep 25 '17 at 16:38
• Thanks a lot :] It really helped me! – user118331 Sep 25 '17 at 16:40

Suppose $u[k] = 0$ for all $k$. By assumption, this means $$y[k] = e[k]$$ Indeed, this means $$\big[ \|\hat y - y\|_2^2\big] = \big[ \|\hat y - e\|_2^2 \big]$$ and in particular, if were minimizing in expectation, it would follow that $$\mathbb{E} \big[ \|\hat y - y\|_2^2\big] = \mathbb{E} \big[ \|\hat y - e\|_2^2 \big]$$ If $e[k]$ were known to be iid according to $\mathcal{N}(1, \sigma^2)$, then the above expected squared error would be smallest for $\hat y = \mathbb{E}[ e ] = 1$. The calculation $\hat y = \mathbb{E} [ e]$ can be done using elementary calculus techniques and knowledge that $e[k] \sim \mathcal{N}(1, \sigma^2)$.

• Thanks for the efford to answer my question, but I think we didnt understand each other. I was asking about the question number2, I dont know how to get rid of that norm in there so I can partialy derivate it with the respect to theta and thus find a minimum. – user118331 Sep 25 '17 at 16:33