I have real data $x(t)$ which I model using the linear regression model: $\displaystyle f(t)= \sum_{i=0}^n c_i \phi_i(t)$, where $c_i$ are the regression coefficients I'm trying to find and $\phi_i(t)$ are a basis function (I chose Legendre Polynomials for their orthogonality).

I noticed something strange I couldn't explain i.e. when I fix $n=1$ for my model I get $\hat{c}_0, \hat{c}_1$, normal! Then when I fix $n=2$ for my model I get $\hat{c'}_0=\hat{c}_0, \hat{c'}_1=\hat{c}_1, \hat{c'}_2=0$ (Seems like $n=2$ didn't add anything to the data modeling!)

What is more strange is that when I fix $n=3$, I get $\hat{c''}_0, \hat{c''}_1, \hat{c''}_2, \hat{c''}_3$ that are completely different (no $\hat{c''}_2=0$)

Does anyone understand what is happening here?

This also happens for other even index coefficients $c_4, c_6, \dots$ i.e. when they are the latest regression coefficient in my model they are null but when I add another odd index coefficient they are not. Also I noticed that the coefficient values I obtain are high (as if to indicate an overfitting!)

  • $\begingroup$ Can you list the coefficients so we see, for example, if they just look like a change of basis from Legendre polynomials to odd degree powers of x (or something like that)? $\endgroup$ – zyx Oct 23 '16 at 2:17
  • $\begingroup$ here are the coeff. wrt $n$: ($n=1: \hat{c}_0=-47.85, \hat{c}_1=-29.08$), ($n=2: \hat{c'}_0=-47.85, \hat{c'}_1=-29.08, \hat{c'}_2 = 0.00$), ($n=3: \hat{c''}_0=-2.26\times 10^3, \hat{c''}_1=-2.44\times 10^3, \hat{c''}_2 = -1.14\times 10^3, \hat{c''}_3 = -0.22\times 10^3$) $\endgroup$ – Learn_and_Share Oct 23 '16 at 2:27
  • $\begingroup$ thanks. can you write out what the polynomial approximation to f(t) is at each value of n (the $\sum c_i \phi_i (t)$), to see if it is converging to anything? $\endgroup$ – zyx Oct 23 '16 at 2:30
  • $\begingroup$ my f(t) has more than 10000 points! I don't think I can do that! $\endgroup$ – Learn_and_Share Oct 23 '16 at 2:33
  • $\begingroup$ What I noticed is that I can recover good enough the form of my real data using $n=3$ but not $n=2$ or $n=1$. $\endgroup$ – Learn_and_Share Oct 23 '16 at 2:34

The Legendre polynomials are orthogonal, but your regression results (by changing) show that their values on your data have nonzero covariance.

If $f(t)$ is odd that would explain the odd/even pattern with the $c_i$, where the even degree $c_{2n}$ contribute nothing on their own, but are involved in isolating the contribution of odd powers of $x$.

  • $\begingroup$ what do you mean by nonzero covariance? Since I have different real data measurements for which I use this model, I noticed also that the regression coefficients I'm estimating are a lot correlated (they are linear combinations of each other)! Is this a consequence of this nonzero covariance? $\endgroup$ – Learn_and_Share Oct 23 '16 at 2:31
  • $\begingroup$ Also what is it you mean by "isolating the contribution of odd powers of $x$"? $\endgroup$ – Learn_and_Share Oct 23 '16 at 2:38
  • $\begingroup$ If f(t) is an odd function then adding the next higher even-degree polynomial to the regression will not change the regression output. However there is a unique linear combination of the c_i (of all degrees, even and odd, up to 2k+1, for a fixed value of k) that has a given set of coefficients for odd powers of x, and zero for all the even degree powers. $\endgroup$ – zyx Oct 23 '16 at 3:33
  • $\begingroup$ Nonzero covariance = ($\phi_i(t)$ and $\phi_j(t)$ are correlated, on the set of $t$ used in the regression). $\endgroup$ – zyx Oct 23 '16 at 5:13
  • $\begingroup$ can $phi_i(t)$ and $\phi_j(t)$ be correlated but orthogonal? $\endgroup$ – Learn_and_Share Oct 23 '16 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.