Hot answers tagged

5

$X^TX+\lambda I$ is always invertible, if $\lambda>0$. Note that, if $u\in\mathbb R^p\setminus\{0\}$, then $$ \langle(X^TX+\lambda I)u,u\rangle=\lambda\langle u,u\rangle+\langle Xu,Xu\rangle \ge \lambda\langle u,u\rangle+\langle Xu,Xu\rangle \ge \lambda\langle u,u\rangle>0. $$ Hence, $(X^TX+\lambda I)u\ne 0$, for all $u\in\mathbb R^p\setminus\{0\}$, ...


3

Let us consider the model $$y=Ax^B+C$$ for which you want the best estimates of parameters $A,B,C$ in the least square sense based on $n$ data points$(x_i,y_i)$. The model is nonlinear so, at a point, nonlinear regression will be required; but this implies to have good (or at least reasonable) starting values. If there was no $C$, a logarithmic transform ...


3

The Gauss-Markov Theorem is actually telling us that in a regression model, where the expected value of our error terms is zero, $E(\epsilon_{i}) = 0$ and variance of the error terms is constant and finite $\sigma^{2}(\epsilon_{i}) = \sigma^{2} < \infty$ and $\epsilon_{i}$ and $\epsilon_{j}$ are uncorrelated for all i and j the least squares estimator ...


2

You are interested in expectation "$C\epsilon$ given $CX$". If $C$ is nonsingular, given $CX$, I can calculate $X=C^{-1}CX$; hence "given $CX$" and "given $X$" are the same thing. In other words, $X$ is deterministic as soon as $CX$ is known. On the other hand, when $C$ is singular you are losing the following information: $\Pi_{null(C)}(X)$ = Projection of ...


1

This is not true in general. An easy counterexample is when $Z$ orthogonal to $X$, and the residue is just $P_Xy$ which does not vanish. This is true only when the $P_Xy$ happens to be in the projected space or the space spanned by $P_XZ$. The essential point is that OLS is a projection onto a subspace. Thus once the subspace to be projected onto -- no ...


1

The best approach for linear regression when there are some outliers expected and you don't want them to contribute to the error is called RANSAC. It is super easy to implement, and it is designed precisely for such a use case, when you believe there are some erroneous data points which you want to leave out while doing regression. If you want to ...


1

Just simply put $X=[X_1 X_2]$ into $X(X^TX)^{-1}X^T$. $P=[X_1 \ \ X_2]\begin{bmatrix} (X_1^TX_1)^{-1} & 0\\ 0 & (X_2^TX_2)^{-1} \end{bmatrix}\begin{bmatrix} X_1^T\\ X_2^T \end{bmatrix}=X_1(X_1^TX_1)^{-1}X_1^T + X_2(X_2^TX_2)^{-1}X_2^T = P_1 + P_2$. Because $X_1^TX_2=0$, so the inverse is for an block-diagonal matrix, which is easy to compute ...


1

For the model $$z=\ln(y)=\beta_0+\beta_1a+\beta_2a^2+\beta_3a^3+\beta_4a^4+\beta_5b$$ all the $\beta$'s will be obtained using multilinear regression just as you apparently did. The problem is that this is just a first step since what is measured is $y$ and not $\log(y)$. In such a case, you need to start a nonlinear regression of the same data for the ...


1

Based on your initial question and using the values you give, I considered the function $$F(a_0,a_1,b_0,b_1)=\sum_{k=1}^{80} \left( {1 \over y_k -(a_0+a_1 k) } - (b_0+b_1 k) \right)^2 $$ and I just minimized it numerically. The minimum is found for the following parameters $$a_0=36.9437\quad a_1=-0.127637$$ $$b_0=-0.0268729\quad b_1=-9.57943 \times ...


1

I suggest watching Ted Shrifin's lectures from Math 3500 at University of Georgia - Atlanta: Inconsistency and Normal Equations Then Projections and Max/Min


1

consider the minimization problem $$\min_x ||x||_{L^1} = \sum_i |x_i|$$ $$\text{s.t.} \qquad A x = b$$ then yes, if $A x = b$ and $A^T A$ is orthogonal, the smaller is $\sum_i |x_i|$ the sparser will be the representation of $b$ in the basis $A$. note that in computer science what we are really interested in is to find the $x$ minimizing $\sum_i |x|^0$ ...


1

Before answering to the question I would like to make a prelimirary comment. The significance of the regression depends of several factors among them the scatter of the experimental data, the number of adjustable parameters of the model and others are important. In the present example of data, the scatter appears rather large. Considering the data ...


1

I know that this might sound a bit complicated if you are new to the subject, but this is the simplest mathematically sound way that I can think of right now, without introducing gross errors in your estimations. What you are trying to do has extensively been studied, e.g. in the simultaneous localization and mapping (SLAM and visual SLAM) literature if you ...


1

You wan't to show $E\left[ \sum\frac{Y_i}{n-2} \right] \neq \mu_Y$. But $E\left[ \sum\frac{Y_i}{n-2} \right] = \frac{n}{n-2}\mu_Y$ which is different for $\mu_y$ for all $n$. However, note that $\lim_{n\to\infty}E \left[ \sum\frac{Y_i}{n-2} \right] =\mu_Y$. Thus, $\sum\frac{Y_i}{n-2}$ is asymptotically unbiased. That is, for big $n$, it is almost an ...



Only top voted, non community-wiki answers of a minimum length are eligible