I read that, in multiclass logistic regression, we have a pivot class $K$ and $K-1$ set of $\vec{w}$ weights, then, for the pivot class: \begin{eqnarray} P( C_K | \vec{x} ) &= 1- \sum_\limits{t=1}^{K-1}P(C_{K}|\vec{x})e^{\vec{w}^{(t)}\vec{x}} \implies & P( C_K | \vec{x} )=\frac{1}{ 1+\sum_\limits{t=1}^{K-1}e^{\vec{w}^{(t)}\vec{x}}}\\ \end{eqnarray} and for all the others class: \begin{eqnarray} &P( C_1 | \vec{x} ) = P( C_K | \vec{x} ) e^{ \vec{w}^{(1)} \vec{x} }=\frac{e^{ \vec{w}^{(1)} \vec{x}}}{ 1+\sum_\limits{t=1}^{K-1}e^{\vec{w}^{(t)}\vec{x}}} \\ &P( C_2 | \vec{x} ) = P( C_K | \vec{x} ) e^{ \vec{w}^{(2)} \vec{x} }=\frac{e^{ \vec{w}^{(2)} \vec{x} }}{ 1+\sum_\limits{t=1}^{K-1}e^{\vec{w}^{(t)}\vec{x}}}\\ &\vdots\\ &P( C_{K-1} | \vec{x} ) = P( C_K | \vec{x} ) e^{\vec{w}^{(K-1)} \vec{x}}=\frac{e^{ \vec{w}^{(K-1)} \vec{x} }}{ 1+\sum_\limits{t=1}^{K-1}e^{\vec{w}^{(t)}\vec{x}}} \\ \end{eqnarray}

so, I need to learn $K-1$ set of weights $\vec{w}$, i.e. $\vec{w}^{(1)},\vec{w}^{(2)},\dots,\vec{w}^{(K-1)}$ .

In other references, instead, I found that it is used softmax function for all $K$ classes, i.e. : $ P(C_h|\vec{x})=\frac{e^{ \vec{w}^{(h)} \vec{x} }}{ \sum_\limits{t=1}^{K}e^{\vec{w}^{(t)}\vec{x}}}, $ $\forall\ 1\leq h \leq K.$ So, it could be explained if we take $e^{\vec{w}^{(K)} \vec{x}}=1$, but now the problem need $K$ set of weights to be learned instead of $K-1$, i.e. $\vec{w}^{(1)},\vec{w}^{(2)},\dots,\vec{w}^{(K)}$. My question is: what it the right way to formalize the problem? with $K$ set of weights or $K-1$? Isn't softmax more expansive with an extra weights set to learn? or maybe, in my training process using $K$ logistic regressors, I will found something like $\vec{w}^{(K)}=0$ so that $e^{0 \vec{x}}=1$? I'm confused...


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.