2
$\begingroup$

I am studying The Elements of Statistical Learning book and I have a question. On pages 120-121 the logistic regression problems is rewritten in the form of matrix and vectors products as follows:(4.21 transformed to 4.24).

enter image description here enter image description here We know $x_i \in \mathbb{R}^{P+1}$, so based on definition on page 121 $X$ is a $N \times P+1$ matrix, and its columns contain $x_i$ and P is N-dimensional vector contains $p(x_i;\beta)$ in its $i^{th}$ element. When I followed this defenitions I get $ \frac{\partial \mathcal{\ell}(\beta )} {\partial \beta} = \left[ \begin{array} { c } x_1^T \\ \vdots \\ x_n^T \end{array} \right ] \left[ \begin{array} { c } y_1 -p(x_1;\beta) \\ \vdots \\ y_n - p(x_n;\beta) \end{array} \right ] = X^T (y-p) $

I assumed what I wrote above is equivalent to (4.24). But when I multiplied them I don't see how 4.24 is equivalent to 4.21. I would appriciate if anyone could help to understand this. Thanks in advance.

$\endgroup$
6
  • 1
    $\begingroup$ I think your mistake is that it is supposed to be that each $x_i\in\mathbb{R}^{p+1}$ since it is logistic regression with intercept, which is why the the first component if each $x_i$ is $1$. In other words, the sample size is $N$, the number of independent variables is $p$, and an intercept is also being assumed. So $\mathbf{X}$ is the $N\times(p+1)$ data matrix ($N$ rows or observations and $p+1$ columns or variables plus intercept) and thus $\mathbf{X}^T(\mathbf{y}-\mathbf{p})$ makes sense in the book. $\endgroup$
    – g.s
    Sep 19, 2017 at 4:30
  • $\begingroup$ Thanks for your comment. You are right, I have not considered that first element in my writing but I don't see how it will change anything. Am I missing something? My problem is each $x_i$ is multiplied by $y_j$ and $p(x_j)$ that $i \neq j$. $\endgroup$
    – Crimson
    Sep 19, 2017 at 4:34
  • 1
    $\begingroup$ So your matrix equation is wrong. The first matrix should be $\mathbf{X}^T=[x_1...x_N]$, where I'm assuming each $x_i$ is a column vector. If each $x_i$ in the book are row vectors, then it should be $\mathbf{X}^T=[x_1^T...x_N^T]$, but I don't have the book in front of me. $\endgroup$
    – g.s
    Sep 19, 2017 at 4:38
  • $\begingroup$ According to the definition the $x_i$ s are column of $X$. I can update the $x_i$ to be $n+1$ -dimensional just I don't know what should be the first element of $y-p$. and even by adding this I don't see the relation between 4.24 and 4.21. For example in 4.24, $x_2$ will be multiplied by $y_3 - p(x_3)$. which is not correct. $\endgroup$
    – Crimson
    Sep 19, 2017 at 4:48
  • $\begingroup$ See answer below. $\endgroup$
    – g.s
    Sep 19, 2017 at 4:52

2 Answers 2

Reset to default
2
$\begingroup$

$\mathbf{y}-\mathbf{p}$ is a $N$-dimensional column vector whose first entry is $y_1-p(x_1)$, which is a scalar (real number). So 4.21 can be written as $p+1$ equations: \begin{align} \sum^N_{i=1}&[y_i-p(x_i)]\\ &\vdots\\ \sum^N_{i=1}&x_{i1}[y_i-p(x_i)]\\ &\vdots\\ \sum^N_{i=1}&x_{ip}[y_i-p(x_i)], \end{align} where we note that the first entry of each $x_i$ is 1.

$\endgroup$
1
  • $\begingroup$ Thank you for the answer and explanation. $\endgroup$
    – Crimson
    Sep 19, 2017 at 5:19
0
$\begingroup$

The matrix $\bf X$ contains $x_i$ in its $i$th row. So the proper way to view $\bf X$ as a block matrix is $$ {\bf X} = \left[\begin{matrix} x_1^T\\ \vdots\\ x_N^T \end{matrix}\right] $$ and therefore $$ {\bf X}^T=\left[\begin{matrix}x_1&\cdots&x_N\end{matrix}\right]. $$ Now you can properly multiply ${\bf X}^T$ and $y-p$ to obtain (4.21).

$\endgroup$
3
  • $\begingroup$ Thanks for the answer, in the definition by the book, $X$ columns contains $x_i$s. $\endgroup$
    – Crimson
    Sep 19, 2017 at 4:55
  • 1
    $\begingroup$ @Crimson No, as stated $X$ has $N$ rows and $P+1$ columns. If what you are saying is true (i.e. if the columns of $X$ hold $x_i$s), then each $x_i$ would be $N$-dimensional and there would be $P+1$ of them. $\endgroup$
    – grand_chat
    Sep 19, 2017 at 5:00
  • $\begingroup$ Yes, you are right. I was confused. Thanks. $\endgroup$
    – Crimson
    Sep 19, 2017 at 5:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .