1
$\begingroup$

I need to solve the following problem: $$\min._{0 \le \mu \le 1} \|\mu-z\|_2^2 + \alpha\|\mu\|_\infty$$ I indicate the box constraint on $\mu$ explicitly as a function and get $$\min._{\mu\in\mathbb{R^n}} \|\mu-z\|_2^2 + \alpha\|\mu\|_\infty + I_C(0 \le \mu \le 1)$$

Hence, my optimization problem is about computing the proximal operator of the function $\alpha\|\mu\|_\infty + I_C(0 \le \mu \le 1)$. Inspired by L Infinity ($ {L}_{\infty} $) Norm Regularized Proximity Problem (Or The Proximal Operator of the $ {L}_{\infty} $ (Infinity Norm)), I thought I will use Moreau's decomposition: $I = \text{prox}_f(.) + \text{prox}_{f^*}(.)$

In order to derive the conjugate of the sum of two functions, I used the concept of infimal convolution given in Dimitri P. Bertsekas - Convex Optimization Theory - Chapter 3 - Exercises and Solutions: Extended Version.

Subsequently, I am able to derive closed-form expressions for the proximal operator of the conjugate of the sum of functions and obtain the optimal $\mu$.

In the above notes, the conjugate of the sum of functions, is defined as the closure of the infimal convolution of the conjugates of the individual functions, provided the original sum of functions is proper. What is the practical consequence of dealing with the concept of closure?

$\endgroup$
4
  • 2
    $\begingroup$ I doubt there is any "practical" consequence of requiring closure here. For one thing, most functions one deals with in practice are already closed, and I believe the infimal convolutions of such functions are going to be closed, too. And the conjugate function is always closed anyway. It would be an interesting challenge to come up with a pair of functions whose infimal convolution is not already closed. I'm sure it's possible, but I suspect it would require a manufactured case. So again, in practice, I don't see it likely that the closure requirement changes anything. $\endgroup$ May 29 '17 at 22:29
  • $\begingroup$ Thanks a lot for your inputs. $\endgroup$
    – Hari
    May 30 '17 at 7:00
  • $\begingroup$ Do you have the solution for the Proximal Operator? Please publish it as an answer. $\endgroup$
    – Royi
    Mar 17 '20 at 22:17
  • $\begingroup$ @Royi I have posted a reference in an answer. Hope it is useful for your purpose. $\endgroup$
    – Hari
    Mar 18 '20 at 17:24
1
$\begingroup$

The problem is given by the explicit Proximal Operator:

$$\begin{aligned} \arg \min_{x} \quad & \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \alpha {\left\| x \right\|}_{\infty} + {I}_{0 \preceq \cdot \preceq 1} \left( x \right) \end{aligned}$$

Looking at the Lagrangian:

$$ L \left( x , {\lambda}_{1}, {\lambda}_{2} \right) = \frac{1}{2} {\left\| x - y \right\|}_{2}^{2} + \alpha {\left\| x \right\|}_{\infty} + {\lambda}_{1}^{T} \left( x - \boldsymbol{1} \right) - {\lambda}_{2}^{T} x $$

The KKT Conditions are given by (See the Sub Gradient of the $ {L}_{\infty} $ Norm):

$$ \begin{align*} \nabla L \left( x, {\lambda}_{1}, {\lambda}_{2} \right) = x - y + \alpha \operatorname{sign} \left( {x}_{j} \right) \boldsymbol{e}_{j} + {\lambda}_{1} - {\lambda}_{2} & = 0 && \text{(1) Stationary Point} \\ {\lambda}_{1}^{T} \left( x - \boldsymbol{1} \right) & = 0 && \text{(2) Slackness} \\ {\lambda}_{2}^{T} x & = 0 && \text{(3) Slackness} \\ x & \preceq 1 && \text{(4) Primal Feasibility} \\ -x & \preceq 0 && \text{(5) Primal Feasibility} \\ {\lambda}_{1} & \geq 0 && \text{(6) Dual Feasibility} \\ {\lambda}_{2} & \geq 0 && \text{(7) Dual Feasibility} \end{align*} $$

One need to carefully analyze the different cases of $ y $ and $ x $.

I tried to do a short cut.
Borrowing the idea from Proximal Operator for $ g \left( x \right) = \mu { \left\| x \right\| }_{1} + {I}_{{\left\| x \right\|}_{2} \leq 1} \left( x \right) $ ($ {L}_{1} $ Norm and Unit Ball Constraint) I thought the solution would be a composition of the Proximal Operator of the $ {L}_{\infty} $ Norm and Projection onto the Box Constraint.

I tried once with doing first the $ \operatorname{prox}_{\alpha {\left\| \cdot \right\|}_{\infty}} \left( \cdot \right) $ (See The Proximal Operator of the $ {L}_{\infty} $ (Infinity Norm)) and then $ \operatorname{Proj}_{\boldsymbol{0} \preceq \cdot \preceq \boldsymbol{1}} \left( \cdot \right) $ yet found cases it didn't work and then tried the other way around and still founds cases it didn't work.

The solution I found is doing both and setting the solution to the one which yield lower objective value.

To verify the result I implemented it in MATALB and verified it against CVX.
The code is available at my StackExchange Mathematics Q2301266 GitHub Repository.

$\endgroup$
0
$\begingroup$

When I was facing this problem, I was working on a formulation in which I had to compute the proximal operator of the $L_{\infty}$ norm of a vector, with the elements of the vector being subject to simplex constraints. While working on this problem, I tried a formulation, where I replaced the simplex constraint with box constraints between $0$ and $1$. Later, I realized that simplex constraint is better for me.

Subsequently, I also changed my formulation in order to consider the simplex constraint as a quadratic barrier term in the objective function itself. Then I came to know that such a formulation has already been tried. For example, you could refer this paper: Scalable Exemplar Clustering and Facility Location via Augmented Block Coordinate Descent with Column Generation.

$\endgroup$
1
  • $\begingroup$ I meant your wrote > Subsequently, I am able to derive closed-form expressions for the proximal operator of the conjugate of the sum of functions and obtain the optimal μ. Hence I asked you will publish the derived expression which is the answer. $\endgroup$
    – Royi
    Mar 20 '20 at 13:56
0
$\begingroup$

The closure is not needed here. A bit more details, given functions $f_1,\dots,f_n$, since the original sum of functions is proper, we know that \begin{equation} \text{ri}~(\text{dom}~f_1) \cap \dots \cap \text{ri}~(\text{dom}~f_n) \neq \emptyset. \end{equation} In this sense, the closure is not necessary. You can refer to Rockafaller's book ``Convex Analysis'' (Theorem 16.4) for more details.

For this problem, it is equivalent to \begin{equation} \begin{array}{cl} {\min_{\mu, \beta}} & {\|\mu-z\|^2 + \alpha \beta} \\ {\text{s.t.}} & {-\beta \leq \mu \leq \beta} \\ {} & {0 \leq \mu \leq 1} \end{array} \end{equation} The result is the lower lower objective value of setting $\beta = 1$ and setting $0 < \beta < 1$.

  1. For $\beta=1$, we can easily obtain that optima $\mu^* = \operatorname{proj}_{0 \leq \mu \leq 1}(z)$, i.e., \begin{equation} \mu_i^* = \left\{ \begin{array}{cl} {0} & {z_i \leq 0}\\ {z_i} & {0 < z_i < 1} \\ {1} & {z_i \ge 1} \end{array} \right. \end{equation}

  2. For $0 < \beta < 1$, the problem can be rewritten as \begin{equation} \begin{array}{cl} {\min_{\mu, \beta}} & {\|\mu-z\|^2 + \alpha \beta} \\ {\text{s.t.}} & {0 \leq \mu \leq \beta} \\ \end{array} \end{equation} Thus the closed-form solution of $\mu$ is $\mu^* = \operatorname{proj}_{0 \leq \mu \leq \beta}(z)$, i.e., \begin{equation} \mu_i^* = \left\{ \begin{array}{cl} {0} & {z_i \leq 0}\\ {z_i} & {0 < z_i < \beta} \\ {\beta} & {z_i \ge \beta} \end{array} \right. \end{equation} Then we need to solve the optimization problem in term of $\beta$: \begin{equation} \begin{array}{cl} {\min_{\beta}} & {(z)_{-}^2 + (z - \beta)_{+}^2 + \alpha \beta} \end{array} \end{equation} where $(x)_{-} = \min(x, 0)$ and $x_{+}= \max(x, 0)$. Wlog, suppose that the elements of the vector are in descending order, i.e., $z_1 \ge z_2 \ge \dots \ge z_n$. Let the number of elements in $z$ which are greater than $\beta$ be $m$, and take the gradient be vanished, we have \begin{equation} \sum_{i=1}^m (\beta - z_i) + \alpha = 0. \end{equation} Thus \begin{equation} \beta = \frac{\sum_{i=1}^m z_i - \alpha}{m}. \end{equation} And since there are only $m$ elements in $z$ which are greater than $\beta$, we need \begin{equation} z_m > \beta > z_{m+1}, \end{equation} That is \begin{equation} \begin{aligned} m\cdot z_m &< \sum_{i=1}^m z_i - \alpha, \\ m \cdot z_{m+1} &> \sum_{i=1}^m z_i - \alpha. \end{aligned} \end{equation}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.