I'm trying to prove that adding a "spectral norm <= $c$" constraint to a least-squares problem will never decrease (relative to the unconstrained solution) a singular value that is already less than $c$.
That is, suppose that $\mathbf{X}_{\text{OLS}}$ is the unconstrained least-squares estimator: $$ \mathbf{X}_{\text{OLS}} = \text{arg}\min_{\mathbf{X}} || \mathbf{Y} - \mathbf{W} \mathbf{X} ||^2_F \tag{1}$$ and $\mathbf{X}_c$ is the spectral-norm-constrained estimator: $$ \mathbf{X}_{\text{c}} = \text{arg}\min_{\mathbf{X}} || \mathbf{Y} - \mathbf{W} \mathbf{X} ||^2_F \quad \text{subject to} \quad ||\mathbf{X}||_2 \le c \tag{2}$$ Let $\sigma^{\text{OLS}}_i$ be the $i$th largest singular value of $\mathbf{X}_{\text{OLS}}$, and let $\sigma^{\text{c}}_i$ be the $i$th largest singular value of $\mathbf{X}_{\text{c}}$
Clearly, if $\sigma_i^{\text{OLS}} \ge c$, then $\sigma_i^{c} = c$. I claim that, moreover, if $\sigma_i^{\text{OLS}} < c$, then $\sigma_i^c \ge \sigma_i^{\text{OLS}}$.
I imagine that the way forward is to assume, by contradiction, that $\sigma_i^{\text{OLS}} < c$ but $\sigma_i^c < \sigma_i^{\text{OLS}}$, and to show somehow that $$ || \mathbf{Y} - \mathbf{W} \mathbf{U} \boldsymbol{\Sigma}' \mathbf{V}^T ||^2_F < || \mathbf{Y} - \mathbf{W} \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T ||^2_F$$ where $\mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T$ is the SVD of $\mathbf{X}_c$, and $\boldsymbol{\Sigma}'$ is $\boldsymbol{\Sigma}$ with the $i$-th entry on the diagonal increased from $\sigma^c_i$ to $\sigma_i^{\text{OLS}}$.
Since $||\mathbf{U} \boldsymbol{\Sigma}' \mathbf{V}^T||_2 \le c$, this would imply that $\mathbf{U} \boldsymbol{\Sigma}' \mathbf{V}^T$ is a better solution to (2) than $\mathbf{X}_c$, which is a contradiction.
