# In the $(p + 1)$-dimensional input-output space, $(X, \hat{Y})$ represents a hyperplane?

I am currently studying the textbook The Elements of Statistical Learning, second edition, by Hastie, Tibshirani, and Friedman. Chapter 2.3.1 Linear Models and Least Squares says the following:

The linear model has been a mainstay of statistics for the past 30 years and remains one of our most important tools. Given a vector of inputs $$X^T = (X_1, X_2, \dots, X_p)$$, we predict the output $$Y$$ via the model $$\hat{Y} = \hat{\beta}_0 + \sum_{j = 1}^p X_j \hat{\beta}_j . \tag{2.1}$$ The term $$\hat{\beta}_0$$ is the intercept, also known as the bias in machine learning. Often it is convenient to include the constant variable $$1$$ in $$X$$, include $$\hat{\beta}_0$$ in the vector of coefficients $$\hat{\beta}$$, and then write the linear model in vector form as inner product $$\hat{Y} = X^T \hat{\beta}, \tag{2.2}$$ where $$X^T$$ denotes vector or matrix transpose ($$X$$ being a column vector). Here we are modelling a single output, so $$\hat{Y}$$ is a scalar; in general $$\hat{Y}$$ can be a $$K$$-vector, in which case $$\beta$$ would be a $$p \times K$$ matrix of coefficients. In the $$(p + 1)$$-dimensional input-output space, $$(X, \hat{Y})$$ represents a hyperplane.

I'm struggling to make sense of this. If $$X^T$$ is a vector or matrix transpose and $$\hat{Y}$$ is a $$K$$-vector, then what exactly are the dimensions of $$X^T$$, $$\hat{Y}$$, and $$\hat{\beta}$$ for this all to make mathematical sense? Furthermore, how does the $$\beta$$ relate to this (this might be a typo by the authors)? And lastly, how does this then show that, in the $$(p + 1)$$-dimensional input-output space, $$(X, \hat{Y})$$ represents a hyperplane?

• I didn’t check the book, but, as I understood, there are two cases for $X^T$, namely (1) $X^T = (X_1, X_2, \dots, X_p)$ and (2) $X^T = (X_0,X_1, X_2, \dots, X_p)$. Then the equality (2.2) restricts the matrix dimensions as follows. If $\hat Y$ is a $K$-vector, that is $1\times K$-matrix then $\hat \beta$ is $p\times K$-matrix in the first case and $(p+1)\times K$-matrix in the second case. Jan 24 at 0:31
• These questions seem to be related: 1, 2, and 3. Jan 24 at 0:40

You have to think that $$(X, \hat{Y})$$ is a vector with $$p+K$$ components, given by $$(X_1,\ldots, X_p, \hat{Y}_1, \ldots, \hat{Y}_K )$$.

In general, let $$B$$ be the $$(p+K) \times K$$ matrix which consists of two blocks, one over the other:

• The upper block is the $$p\times K$$ matrix $$\hat{\beta}$$;
• The lower block is the $$K \times K$$ matrix given by $$-Id$$.

Let me denote by $$\hat{\beta}^i$$ the $$i$$-th column of $$\hat{\beta}$$. Note that the $$i$$-th column of $$B$$ is then given by $$(\hat{\beta}^i, -e_i)$$; here I am using the same convention of $$(X, \hat{Y})$$ by concatening two vectors one after another (the first with $$p$$ components and the second with $$K$$ components).

Let's see what does the equation $$(X, \hat{Y})^T B = 0$$ means. The output is a vector with $$K$$ components. The $$i$$-th component is the scalar product of the the input-output with the $$i$$-th column of $$B$$, that is

$$0= (X, \hat{Y}) \cdot (\hat{\beta}^i, -e_i) = (X, \hat{\beta}^i) + (\hat{Y}, -e_i)$$

Which is equivalent to

$$\hat{Y}_i = (X, \hat{\beta}^i)$$

In the same spirit of above, note that the RHS is the i-th component of $$X^T \hat{\beta}$$! This means our equation is equivalent to

$$\hat{Y} = X^T \hat{\beta}$$

Which is the original equation! All this stuff was a fancy way to demonstrate you that, whenever you have some variables constrained by linear equations, you can always put your equations into the form $$(variables) ^T (matrix) = 0$$, and this is what happening in your case.

In the pleasant case in which you have just one equation ($$K$$=1), the matrix $$B$$ will have just one column; in other words $$B$$ is a vector, and the $$B$$ equation reads as

$$(X, \hat{Y}) \cdot B =0$$

Such an equation describes exactly the hyperplane perpendicular to the vector $$B$$! Think about this geometrically: if I ask you what is the locus of points in $$\mathbb{R}^3$$ perpendicular to the vector $$(0, 0,1)$$, this will be the $$x-y$$ hyperplane.