3
$\begingroup$

I have n data, $x_i$ are the inputs, and $y_i$ are the labels. $x_i$ is $d$-dimension, and $y_i$ is $p$-dimension. We have a prediction function that is $f(x) = W^Tx$. $W$ is a matrix of dimension $d \times p$.

The squared error loss function of the prediction function is: $$ J(W) = \sum_{i=1}^n||y_i - W^Tx_i||_2^2 $$

I would like to write $J(W)$ in matrix form. How should I convert it to an equation involving $Y$ ($n \times p$ dimension), $W$ ($d \times p$ dimension) and $X$ ($n \times d$ dimension). Hence the loss function will be in matrix form instead of a sum?

$\endgroup$
1
$\begingroup$

Given the set of data $D_{m}:=\left\{\left(x_{i}, y_{i}\right) \mid i=1,2, \cdots, m\right\}$ such that $x_{i} \neq x_{j}$ for $i \neq j .$ we must find the line $y=\alpha x+\beta$ that best fits the data, i.e. find $(\alpha, \beta)$ such that for all $(a, b) \in \mathbb{R}^{2}$ we have \begin{equation} 0 \leq \Phi(\alpha, \beta)=\sum_{i=1}^{m}\left(y_{i}-\alpha x_{i}-\beta\right)^{2} \leq \sum_{i=1}^{m}\left(y_{i}-a x_{i}-b\right)^{2}=\Phi(a, b) \end{equation} in matrix form, $\Phi(\alpha, \beta)=\Phi(z)=\|y-A z\|_{2}^{2}$, thus inequality $(1)$ is equivalent to the least square problem: $$ \Phi(z)=\|y-A z\|_{2}^{2}=\min_{w \in \mathbb{R}^{n}}\|y-A w\|_{2}^{2}=\min_{w \in \mathbb{R}^{n}} \Phi(w) $$ Since $y_{i}-a x_{i}-b=(y-A w)_{i}$ where : $w=[a ; b], z=[\alpha ; \beta], \mathbf{y}=\begin{bmatrix}y_{1} \\ y_{2} \\ y_{3} \\ \vdots \\ y_{m}\end{bmatrix}, \mathbf{A}=\begin{bmatrix}x_{1} & 1 \\ x_{2} & 1 \\ x_{3} & 1 \\ \vdots & \vdots \\ x_{m} & 1\end{bmatrix}, \mathbf{w}=\begin{bmatrix}a \\ b\end{bmatrix},$ and $\mathbf{z}=\begin{bmatrix}\alpha \\ \beta\end{bmatrix}$.

$\endgroup$
1
  • $\begingroup$ So you are saying that $J(W) = \sum_{i=1}^n||y_i - W^Tx_i||_2^2$ = $||Y - WX||_2^2$ ? $\endgroup$ Mar 26 '21 at 2:19
1
$\begingroup$

With $X = \begin{bmatrix}x_1 & \dots & x_n\end{bmatrix}$ and $Y = \begin{bmatrix}y_1 & \dots & y_n\end{bmatrix}$, you simply obtain: $$j(W) = ||Y-W^TX||_{2,1},$$ where the $L_{2,1}$ norm is the sum of the Euclidean norms of the columns of the matrix $Y-W^TX$.

Note however the dimensions: $X \in \mathbb{R}^{d \times n}$ and $Y \in \mathbb{R}^{p \times n}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.