New answers tagged

1

Before answering to the question I would like to make a prelimirary comment. The significance of the regression depends of several factors among them the scatter of the experimental data, the number of adjustable parameters of the model and others are important. In the present example of data, the scatter appears rather large. Considering the data ...


3

Let us consider the model $$y=Ax^B+C$$ for which you want the best estimates of parameters $A,B,C$ in the least square sense based on $n$ data points$(x_i,y_i)$. The model is nonlinear so, at a point, nonlinear regression will be required; but this implies to have good (or at least reasonable) starting values. If there was no $C$, a logarithmic transform ...


5

$X^TX+\lambda I$ is always invertible, if $\lambda>0$. Note that, if $u\in\mathbb R^p\setminus\{0\}$, then $$ \langle(X^TX+\lambda I)u,u\rangle=\lambda\langle u,u\rangle+\langle Xu,Xu\rangle \ge \lambda\langle u,u\rangle+\langle Xu,Xu\rangle \ge \lambda\langle u,u\rangle>0. $$ Hence, $(X^TX+\lambda I)u\ne 0$, for all $u\in\mathbb R^p\setminus\{0\}$, ...


2

You are interested in expectation "$C\epsilon$ given $CX$". If $C$ is nonsingular, given $CX$, I can calculate $X=C^{-1}CX$; hence "given $CX$" and "given $X$" are the same thing. In other words, $X$ is deterministic as soon as $CX$ is known. On the other hand, when $C$ is singular you are losing the following information: $\Pi_{null(C)}(X)$ = Projection of ...


0

I think that your notation is a bit off, since you should probably be indicating the first subscript as rows and the second subscript as columns consistently throughout the problem. My main recommendation would be to expand out $\mathbf{X}^T\mathbf{X}$ into a sample mean notation, and then multiply in by $\beta$; it may become clearer how these sample mean ...


1

I suggest watching Ted Shrifin's lectures from Math 3500 at University of Georgia - Atlanta: Inconsistency and Normal Equations Then Projections and Max/Min


1

Based on your initial question and using the values you give, I considered the function $$F(a_0,a_1,b_0,b_1)=\sum_{k=1}^{80} \left( {1 \over y_k -(a_0+a_1 k) } - (b_0+b_1 k) \right)^2 $$ and I just minimized it numerically. The minimum is found for the following parameters $$a_0=36.9437\quad a_1=-0.127637$$ $$b_0=-0.0268729\quad b_1=-9.57943 \times ...


0

If I well understand the question, you have the data $(x_k\:,\:y_k)$ where $k=1$ to $k=n$ ( in your example $n=80$ and $x_k=k$ ). As the problem is settled, this is not exactly the fitting of a function to the data but in fact the fitting of an equation to the data, which is slightly different. The equation is : $$\frac{1}{y-(a_0+a_1x)} - (b_0+b_1x)=0$$ The ...


0

Data which I'm working on. I use $80$ values but put it here in two portions a $40$ (vertically). I'm getting an approximation-formula from the left block of forty values and check the appropriateness by application to the full dataset. The left column of each block is the index which begins at $1$ to allow formulae with reciprocals. The data stem from a ...


0

Maybe the other previous answers are correct but I did not understand them. Looking at the left hand equation 3 in the original post, $x_i$ is a vector and not a scalar which results in $dL/dW$ being a vector, not a scalar which I misunderstood at first. $\Alfa$ remains a scalar but the weights in the W vector are updated differently each iteration.


1

The best approach for linear regression when there are some outliers expected and you don't want them to contribute to the error is called RANSAC. It is super easy to implement, and it is designed precisely for such a use case, when you believe there are some erroneous data points which you want to leave out while doing regression. If you want to ...


0

You can just remove the outliers from your data set manually, and then run the regression on the remaining data. I don't have enough rep to comment Otherwise, you're pretty much stuck with the result you first obtained.


0

Equations are usually used in this form when you're working in a neural network kind setting, where the bias term is also a vector. In the case of linear regression, since the bias term is a single scalar, a more intuitive way to look at these equations is to treat the summation like $\sum_{j=0}^{p}W_jx_{ij}$ by treating $x_{i0}$ as 1, and b as $W_0$. ...


1

This is not true in general. An easy counterexample is when $Z$ orthogonal to $X$, and the residue is just $P_Xy$ which does not vanish. This is true only when the $P_Xy$ happens to be in the projected space or the space spanned by $P_XZ$. The essential point is that OLS is a projection onto a subspace. Thus once the subspace to be projected onto -- no ...


1

Just simply put $X=[X_1 X_2]$ into $X(X^TX)^{-1}X^T$. $P=[X_1 \ \ X_2]\begin{bmatrix} (X_1^TX_1)^{-1} & 0\\ 0 & (X_2^TX_2)^{-1} \end{bmatrix}\begin{bmatrix} X_1^T\\ X_2^T \end{bmatrix}=X_1(X_1^TX_1)^{-1}X_1^T + X_2(X_2^TX_2)^{-1}X_2^T = P_1 + P_2$. Because $X_1^TX_2=0$, so the inverse is for an block-diagonal matrix, which is easy to compute ...


0

In OLS models the covariance matrix of the coefficients is given by $\sigma^2 (X'X)^{-1}$. So, in your case, $$ X = \begin{pmatrix} 1 & x_1 \\ : & :\\1 & x_n \end{pmatrix} $$ hence, $$ (X'X)^{-1} = \frac{1}{n\sum_{i=1}^n(x_i - \bar{x}_n)^2}\begin{pmatrix} \sum_{i=1}^nx^2_i & -\sum_{i=1}^nx_i \\-\sum_{i=1}^nx_i & n \end{pmatrix}, $$ ...


0

I'm answering this on a cell phone so pardon if I don't use lots of notation, but you're essentially just one step away from the answer with your substitution. Note that $$(VD^2V^T+\lambda I)^{-1}=(VD^2V^T+V\lambda IV^T)^{-1}=(V(D^2+\lambda I)V^T)^{-1}$$ From here the final step will look like cheating, but you can actually pull out the $V$ and $V^{T}$ ...


0

Here is the point : In: $\ UDV^T(VD^2V^T+\lambda I)^{-1}VDU^Ty$ you should write $I=VV^T$ giving the factorization: $UDV^T(V(D^2+\lambda I)V^T)^{-1}VDU^Ty$ $=UDV^T(V^T)^{-1}(D^2+\lambda I)^{-1}V^{-1}VDU^Ty$ yielding the looked-for formula.


0

Assuming i.i.d. normal errors, comparing the likelihood of these two models would be equivalent to comparing the $R^2$ values. What would be the null hypothesis of this test? You might consider fitting a model containing all three covariates and separately testing the hypotheses $H_0:\beta_2=0$ and $H_0:\beta_3=0$.


0

In my opinion you can't use a transformation via the logarithmus. I think you need a funciton like $$ f(t) = a e^{bt}. $$ To use least squares on a nonlinear specification some characteristica from the linear regression must be dropped. But in general the asymptotic theory leads to more or less similar results (for the statistical properties). With that ...


0

Better apply linear regression to the logarithms of the quantities. Then you have an approximation of the form $e^{ax+b}$ or equivalently $C.e^{ax}$ In a sense this is the 'best possible' approximation measured by the relative differences between the approximated quantities and the measured quantities.


0

Could I simply make a feature transforming function $\phi(x) = (1, x, x^2, \dots, x^{1337})$ and expect to get the generating polynomial if I have enough (1338?) points? Yes, any polynomial of degree $n$ can be written as a linear combination of $\{x^0, x^1, ... , x^n \}$. Linear regression can learn any linear combination of its features, hence any ...


1

You wan't to show $E\left[ \sum\frac{Y_i}{n-2} \right] \neq \mu_Y$. But $E\left[ \sum\frac{Y_i}{n-2} \right] = \frac{n}{n-2}\mu_Y$ which is different for $\mu_y$ for all $n$. However, note that $\lim_{n\to\infty}E \left[ \sum\frac{Y_i}{n-2} \right] =\mu_Y$. Thus, $\sum\frac{Y_i}{n-2}$ is asymptotically unbiased. That is, for big $n$, it is almost an ...


0

I'm not sure if I understand your question properly but... Take for example series of $\exp$: $$e^x=a+b\,x+c\,x^2+d\,x^3+...$$ Plug $x=0$ to find $a$: $$a=e^0=1$$ Differentiate both sides: $$\frac{d}{dx}e^x=\frac{d}{dx}(a+b\,x+c\,x^2+d\,x^3+...)$$ $$e^x=b+2c\,x+3d\,x^2+...$$ And plug $x=0$ to find $b$: $$b=e^0=1$$ Differentiate again and plug $x=0$: ...


1

I know that this might sound a bit complicated if you are new to the subject, but this is the simplest mathematically sound way that I can think of right now, without introducing gross errors in your estimations. What you are trying to do has extensively been studied, e.g. in the simultaneous localization and mapping (SLAM and visual SLAM) literature if you ...


1

consider the minimization problem $$\min_x ||x||_{L^1} = \sum_i |x_i|$$ $$\text{s.t.} \qquad A x = b$$ then yes, if $A x = b$ and $A^T A$ is orthogonal, the smaller is $\sum_i |x_i|$ the sparser will be the representation of $b$ in the basis $A$. note that in computer science what we are really interested in is to find the $x$ minimizing $\sum_i |x|^0$ ...


3

The Gauss-Markov Theorem is actually telling us that in a regression model, where the expected value of our error terms is zero, $E(\epsilon_{i}) = 0$ and variance of the error terms is constant and finite $\sigma^{2}(\epsilon_{i}) = \sigma^{2} < \infty$ and $\epsilon_{i}$ and $\epsilon_{j}$ are uncorrelated for all i and j the least squares estimator ...


1

For the model $$z=\ln(y)=\beta_0+\beta_1a+\beta_2a^2+\beta_3a^3+\beta_4a^4+\beta_5b$$ all the $\beta$'s will be obtained using multilinear regression just as you apparently did. The problem is that this is just a first step since what is measured is $y$ and not $\log(y)$. In such a case, you need to start a nonlinear regression of the same data for the ...



Top 50 recent answers are included