1
$\begingroup$

I'm having trouble understanding that when you minimize a polynomial regression

$$ \min_\beta \left\{ \left \| \frac{1}{2} (y - B\beta )\right \| ^2 \right\} $$

$ B\beta = y $, the B is a (m + 1) x (m + 1) dimension matrix? Where m is the highest exponent in the equation (deg( polynomial ) = m)

Like does the fact that when $ \beta $ is minimized you would need $ \geq$ (m+1) equations to solve for $ \beta $?

If there is a resource that someone could please point me to or a proof showing that this is true, I'd really appreciate it, thank you!

$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

If $B$ is a squared matrix of $(m+1)\times (m+1)$ then you'll have a unique solution (assuming that you don't have any redundant observations). No OLS is needed. You can still minimize and you'll get the same answer, bu the minimization is meaningless in this case. The more common situation is when $B$ is of order $n\times p$, where $n > p$. In this case, the system of linear equations $$ B\beta = y, \quad (1) $$ is "overdetermined", namely you have much more equations $(n)$ than unknown parameters ($p$). In this case, you have to find an approximate solution which can be found by solving the following modified linear system (it is the first order condition of your minimization problem), $$ B'B\beta = B'y, \quad (2) $$ here you'll have $p$ equations with a unique solution $\hat{\beta} = (B'B)^{-1}B'y$. So, by solving $(2)$ you get the "best" solution of $(1)$ which was your inital goal.

$\endgroup$
4
  • $\begingroup$ Thank you, but what do you need an estimate if the solution is over determined? Is it because the column space covers more space than a necessary solution? Is there a proof you could please provide for the fact that $ (m+1)$ x $(m +1) $ gives a unique solution? $\endgroup$
    – Andre Fu
    Feb 21, 2018 at 4:31
  • $\begingroup$ 1. Because in such a case every $p$ equations will give you another solution that will not be consistent with the other equations. 2. What kind of proof do you need? It is basic linear-Algebra statement that $Ax = b$ has a unique solution whenever $\dim(C(A)) = \dim (x)$. $\endgroup$
    – V. Vancak
    Feb 21, 2018 at 9:46
  • $\begingroup$ Sorry, dim(C(A)) = dim(x) gives a unique solution, as in the dimension of the column space is equal to the dimension of x? Which theorem is this? I know of rank(A) = rank (A | b ) = n, in an nxm matrix gives a unique soln $\endgroup$
    – Andre Fu
    Feb 22, 2018 at 0:21
  • $\begingroup$ It's the same as the rank argument, as rank of $A$ is the dim of its column space. $\endgroup$
    – V. Vancak
    Feb 22, 2018 at 0:25
0
$\begingroup$

To understand, you must first consider the polynomial $$y = b_0 + b_1x+ b_2x^2 + \cdots + b_mx^m$$ The equation can be written as $y=B\beta$ where $B$ is the matrix of coefficients $$B = \begin{bmatrix}b_0&0&\cdots&0\\0&b_1&\cdots&0\\\vdots&\vdots&\ddots&\vdots\\0&\cdots&0&b_m\end{bmatrix}$$ and $\beta = [1\;x\;x^2\;\cdots\;x^m]$, a vector of the exponents.

Note there are an $m+1$ number of coefficients, thus the coefficient matrix ends up being $(m+1)\times(m+1)$.

$\endgroup$
1
  • $\begingroup$ That makes sense intuitively but is there a rigorous proof available using the minimization of $ \beta $? How does this apply to the derivative of the $ y - B\beta $ = 0? Is there a relationship between this and the actual derivation of OLS? Thank you $\endgroup$
    – Andre Fu
    Feb 20, 2018 at 4:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.