3
$\begingroup$

Suppose I have $n$ observations of $m$ dependent variables $y_1,\dots,y_m$, and I believe they follow some model wherein they can all be written as linear combinations of some underlying variables $x_1,\dots,x_k$ (with $k<m<n$). In other words, I have the model $$Y=X\beta$$ for a (known) $n\times m$ matrix of observations $Y$, an (unknown) $n\times k$ matrix of underlying variables $X$, and an (unknown) $k\times m$ matrix of coefficients $\beta$.

If $n$ is sufficiently large, then this system is over-determined and I should be able to solve for $X$ and $\beta$ that give the least-squares solution to this equation, right? It seems that this should be solvable with something like linear regression but I'm not sure how.

$\endgroup$
9
  • 1
    $\begingroup$ As you said, least squares. So you want to minimize $\|Y-X\beta\|^2_2$. When does it's gradient vanish? $\endgroup$
    – user251257
    Jul 9, 2015 at 8:40
  • $\begingroup$ Gradient with respect to what? We're trying to minimize with respect to both $X$ and $\beta$. $\endgroup$
    – user2034716
    Jul 9, 2015 at 8:56
  • 1
    $\begingroup$ oh sorry. I overlooked that $X$ is unknown. The problem make no sense without additional info about the model. $\endgroup$
    – user251257
    Jul 9, 2015 at 8:59
  • $\begingroup$ Sorry, can you elaborate? My understanding is that with sufficiently large $n$, the system is be overdetermined and so there must be some $X$ and $\beta$ which minimize squared error. The issue is I have no idea how to find such a solution. $\endgroup$
    – user2034716
    Jul 9, 2015 at 9:06
  • $\begingroup$ You ask howto factorize $Y$. For any minimum $(X^*, \beta^*)$ and $\alpha\ne 0$ the tuple $(\alpha X^*, \frac1\alpha \beta^*)$ will also be a minimum. Also, if rank of $Y$ is $\le k$, then you can compute a factorization using the SVD of $Y$. $\endgroup$
    – user251257
    Jul 9, 2015 at 9:25

1 Answer 1

Reset to default
4
$\begingroup$

In general $\textbf Y$ and $ \textbf X$ are known because you have a sample. This sample have a dataset of m points: $(x_{11},x_{12},\ldots,x_{1m},y_1), (x_{21},x_{22},\ldots,x_{2m},y_2), (x_{31},x_{32},\ldots,x_{3m},y_3), \ldots, (x_{n1},x_{n2},\ldots,x_{nm},y_m)$. The values of $x_{ij}$ are represented by $\textbf X$ and the values of $y_i$ are represented by $\textbf Y$. The observations are always a pair of an m x-value and a y-value. And it is true, that is has to be $n >m >k$.

You have to minimize $V(\beta)=||\textbf Y-\textbf X\beta||_2^2=(\textbf Y-\textbf X \beta)'\times(\textbf Y-\textbf X\beta)=(\textbf Y'- \beta' \textbf X' )\times(\textbf Y-\textbf X\beta)$

Multiplying out

$V(\beta)=\textbf Y'\textbf Y -\textbf Y'\textbf X\beta-\beta' \textbf X' \textbf Y +\beta' \textbf X' \textbf X\beta$

It is $\textbf Y'\textbf X\beta=\beta' \textbf X' \textbf Y$ Therefore

$V(\beta)=\textbf Y'\textbf Y -2\beta' \textbf X' \textbf Y +\beta' \textbf X' \textbf X\beta$

Differentiating w.r.t $\beta$

$\frac{\partial V}{\partial \beta}=-2 \textbf X' \textbf Y +2 \textbf X' \textbf X\beta=0$

$2\textbf X' \textbf X\beta=2\textbf{X}'\textbf{Y}$

Dividing both sides by 2

$\textbf X' \textbf X\beta=\textbf{X}'\textbf{Y}$

Bringing $\textbf X' \textbf X$ to the RHS.

$\beta=(\textbf X' \textbf X)^{-1}\textbf{X}'\textbf{Y}$

$\beta$ are the values of the coefficients, which minimize the (squared) difference between the observed x-values and the observed y-values.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .