I have a loss function of the form

$L_{multivariate}^j = \sum\limits_{i=1}^{n} \sum\limits_{k=1}^{d} w_k \big[y_{ik} - \sum\limits_{t\ne j} f_t(x_i\beta_t) - f_j(x_i\beta_j)\big]^2$

where $k$ iterates over output dimensions.

I need to minimize it over $f_j$ and $\beta_j$ assuming all other parameters are fixed (as part of an alternating optimization scheme). The good news is I know how to do this for the univariate case:

$L_{univariate}^j = \sum\limits_{i=1}^{n} \big[y_{i} - \sum\limits_{t\ne j} f_t(x_i\beta_t) - f_j(x_i\beta_j)\big]^2$

Aside: It involves finding the optimal $f_j$ with standard regression of the "residuals" against the "projections" and then the optimal $\beta_j$ by Taylor expanding $f_j(x_i\beta_j) \approx f_j(x_i\beta_{j,old}) + \dot{f}(x_i\beta_{j,old})x_i(\beta_j-\beta_{j,old})$, plugging back in to the loss function, doing more algebra than you can possibly believe to put it in the form of a weighted least squares problem, and then solving the weighted least squares problem. Repeat to convergence.

It strikes me that I should be able to use these same strategies to solve for $f_j$ and $\beta_j$ in the multivariate loss function if I can get rid of that sum over $k$ by factoring it in to the terms inside the square.

Clearly the last two terms inside the square do not depend on $k$, so to them the sum over the weights $w_k$ looks like a constant, and I want to be able to simply distribute the square root of that constant to those terms as:

$L_{multivatiate}^j = \sum\limits_{i=1}^{n} \big[(???) - \sqrt{\sum\limits_{k=1}^{d} w_k}\sum\limits_{t\ne j} f_t(x_i\beta_t) - \sqrt{\sum\limits_{k=1}^{d} w_k} f_j(x_i\beta_j)\big]^2$

But I don't think that's right. And what happens to the term that depends on $k$? How can I account for the sum in this term inside the squared expression? What goes in place of ???

  • $\begingroup$ I don't think it matters to the question, but I should say that $x$ and $\beta$ are vectors here, while $w$ and $y$ are scalar. $\endgroup$ Commented Feb 1, 2018 at 0:33

1 Answer 1


I have proven there is no way to put the multivariate loss function in the form I want.

First assume I can, call the terms independent of $k$ by single symbols, $A$ and $B$, assume the sum of the weights is 1 so the root is also 1, and drop the outer sum over $n$ to make the algebra less verbose. The desired expression expands to:

$$((???) - B - A)^2 = (???)^2 - 2(???)B - 2(???)A + B^2 + 2BA + A^2$$

Now expand a version where the sum over $k$ is not factored in, but the sum of only weights still equals 1 for simplicity:

$$\sum\limits_{k=1}^{d}w_k(y_{ik} - B - A)^2 = \sum\limits_{k=1}^{d}w_k y_{ik}^2 + -2B(\sum\limits_{k=1}^{d}w_k y_{ik}) - 2A(\sum\limits_{k=1}^{d}w_k y_{ik}) + B^2 + 2AB + A^2$$

Now set the expansions equal and cancel terms on both sides to get:

$$(???)^2 = \sum\limits_{k=1}^{d}w_k y_{ik}^2$$ $$(???) = \sum\limits_{k=1}^{d}w_k y_{ik}$$

The root of the first expression is not equal to the second, so there is no form for (???). Basically, the sum of the squares is not equal to the square of the sums, which I knew as a general rule and suspected might make my desired form impossible.

The good news is I have found a different way to optimize my multivariate loss function, a way that is more generalized than the special case I have been using to optimize the univariate form.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .