I am currently studying the textbook The Elements of Statistical Learning, second edition, by Hastie, Tibshirani, and Friedman. Chapter 2.3.1 Linear Models and Least Squares says the following:

The linear model has been a mainstay of statistics for the past 30 years and remains one of our most important tools. Given a vector of inputs $X^T = (X_1, X_2, \dots, X_p)$, we predict the output $Y$ via the model $$\hat{Y} = \hat{\beta}_0 + \sum_{j = 1}^p X_j \hat{\beta}_j . \tag{2.1}$$ The term $\hat{\beta}_0$ is the intercept, also known as the bias in machine learning. Often it is convenient to include the constant variable $1$ in $X$, include $\hat{\beta}_0$ in the vector of coefficients $\hat{\beta}$, and then write the linear model in vector form as inner product $$\hat{Y} = X^T \hat{\beta}, \tag{2.2}$$ where $X^T$ denotes vector or matrix transpose ($X$ being a column vector). Here we are modelling a single output, so $\hat{Y}$ is a scalar; in general $\hat{Y}$ can be a $K$-vector, in which case $\beta$ would be a $p \times K$ matrix of coefficients. In the $(p + 1)$-dimensional input-output space, $(X, \hat{Y})$ represents a hyperplane.

I'm struggling to make sense of this. If $X^T$ is a vector or matrix transpose and $\hat{Y}$ is a $K$-vector, then what exactly are the dimensions of $X^T$, $\hat{Y}$, and $\hat{\beta}$ for this all to make mathematical sense? Furthermore, how does the $\beta$ relate to this (this might be a typo by the authors)? And lastly, how does this then show that, in the $(p + 1)$-dimensional input-output space, $(X, \hat{Y})$ represents a hyperplane?

  • $\begingroup$ I didn’t check the book, but, as I understood, there are two cases for $X^T$, namely (1) $X^T = (X_1, X_2, \dots, X_p)$ and (2) $X^T = (X_0,X_1, X_2, \dots, X_p)$. Then the equality (2.2) restricts the matrix dimensions as follows. If $\hat Y$ is a $K$-vector, that is $1\times K$-matrix then $\hat \beta$ is $p\times K$-matrix in the first case and $(p+1)\times K$-matrix in the second case. $\endgroup$ Jan 24 at 0:31
  • $\begingroup$ These questions seem to be related: 1, 2, and 3. $\endgroup$ Jan 24 at 0:40

You have to think that $(X, \hat{Y}) $ is a vector with $p+K$ components, given by $ (X_1,\ldots, X_p, \hat{Y}_1, \ldots, \hat{Y}_K ) $.

In general, let $B$ be the $(p+K) \times K$ matrix which consists of two blocks, one over the other:

  • The upper block is the $p\times K$ matrix $\hat{\beta}$;
  • The lower block is the $K \times K$ matrix given by $-Id$.

Let me denote by $\hat{\beta}^i$ the $i$-th column of $\hat{\beta}$. Note that the $i$-th column of $B$ is then given by $(\hat{\beta}^i, -e_i) $; here I am using the same convention of $(X, \hat{Y}) $ by concatening two vectors one after another (the first with $p$ components and the second with $K$ components).

Let's see what does the equation $(X, \hat{Y})^T B = 0$ means. The output is a vector with $K$ components. The $i$-th component is the scalar product of the the input-output with the $i$-th column of $B$, that is

$$ 0= (X, \hat{Y}) \cdot (\hat{\beta}^i, -e_i) = (X, \hat{\beta}^i) + (\hat{Y}, -e_i) $$

Which is equivalent to

$$ \hat{Y}_i = (X, \hat{\beta}^i) $$

In the same spirit of above, note that the RHS is the i-th component of $X^T \hat{\beta}$! This means our equation is equivalent to

$$ \hat{Y} = X^T \hat{\beta}$$

Which is the original equation! All this stuff was a fancy way to demonstrate you that, whenever you have some variables constrained by linear equations, you can always put your equations into the form $(variables) ^T (matrix) = 0$, and this is what happening in your case.

In the pleasant case in which you have just one equation ($K$=1), the matrix $B$ will have just one column; in other words $B$ is a vector, and the $B$ equation reads as

$$ (X, \hat{Y}) \cdot B =0$$

Such an equation describes exactly the hyperplane perpendicular to the vector $B$! Think about this geometrically: if I ask you what is the locus of points in $\mathbb{R}^3$ perpendicular to the vector $(0, 0,1) $, this will be the $x-y$ hyperplane.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.