0
$\begingroup$

I was watching a talk of Michael Jordan on Youtube. In the lecture, he mentioned that for a convex-Lipschitz function $f$, the gradient descent

$$x^{k+1} = x^k - \beta \nabla f(x^k)$$ for a constant step-size achieves the following rate,

$$f(x^k) - f(x^*) \leq O(1/k)$$

where $x^*$ is a local minimizer of $f$.

And Nesterov came up with an accelerated version, $$y^{k+1} = x^k - \beta \nabla f(x^k)$$ $$x^{k+1} = (1-\lambda^k)x^{k+1} + \lambda^k y^k$$

which acheives the following upper bound

$$f(x^k) - f(x^*) \leq O(1/k^2)$$

At this point, I am terribly confused as to why this upper bound would imply acceleration. Consider the original argument given by Nesterov,

enter image description here

This theorem doesn't say anything about upper bound, it is clearly a theorem on theoretical lower bound $\Omega(1/k^2)$ (unless that $\geq$ should have been a $\leq$). Furthermore, it is not a statement on the existence of a hypothetical algorithm, but that of a particular function. In fact, in the proof, Nesterov explicitly constructs the function that satisfies this condition. It doesn't say there exists an algorithm that achieves this lower bound. Also, there are strange conditions in the theorem such as $t \leq (n-1)/2$ that is never mentioned in the rate of convergence proofs of gradient descent, etc.

I guess I would be more convinced if the result instead was stated as follows: "Prop: there exists a first-order method such that for convex-Lipschitz functions, achieves $f(x^k) - f(x^*) \leq O(1/k^2)$". This seems to be a more logically coherent motivation as to why the rate for gradient descent is suboptimal. However, this is not what the theorem says and I don't see how the theorem implies this proposition.... (as in comment)

Therefore, in order to show that an algorithm achieves acceleration, we must show that it achieves a lower bound of $\Omega(1/k^2)$. So why is that in $>99\%$ of the literature on accelerated algorithms, an upper bound is provided instead of a lower bound?

Plus, an upper bound means the "worst-case" upper bound. Therefore, even if we do show the gradient descent achieves $O(1/k)$ rate, unless that $1/k$ rate is tight, it does not mean that it will experience that worst-case rate in practice. Since all these rates are worst-case rates of convergence, what is preventing the gradient descent from vastly surpassing Nesterovs algorithm in practice?

Can someone please provide a simple argument why upper bound implies acceleration, when the acceleration result is stated neither in terms of an upper bound nor a statement regarding the existence of a particular "accelerated" algorithm.

$\endgroup$
1
$\begingroup$

If method 2 has a better upper bound than method 1, and both bounds are tight, then method 2's worst case is better than the worst case of method 1. The worst cases might be in totally different problems, so method 1 might sometimes be faster than method 2 anyway. In practice, rigorously showing that a method's bound is tight is usually more difficult than proving the bound itself, so usually you just try to not use too blunt of estimates in the course of deriving the upper bound and then don't bother to derive a lower bound.

What you showed in the image seems to be of a somewhat different character, it says "if $x_{s+1}$ is given by $x_1$ plus any linear combination of all the previous gradients, then there is an $f$ which decays at $\Omega(1/t^2)$ up until $t>(n-1)/2$". Thus it is expressing a fundamental limitation on all methods in this class.

$\endgroup$
  • 1
    $\begingroup$ Hi, I don't understand the following arguments used in many references: "Since GD achieves $O(1/k)$ rate, and Nesterov shows that for first-order methods, there exists a function, such that for this function, first-order methods obtain $\Omega(1/k^2)$, therefore there exists a gap between the GD's rate and the optimal rate". $\endgroup$ – Olórin Mar 9 '18 at 22:25
  • $\begingroup$ For precise statement, see p 284 of arxiv.org/pdf/1405.4980.pdf $\endgroup$ – Olórin Mar 9 '18 at 22:27
  • 3
    $\begingroup$ @Olórin There are problems with $\Theta(1/k)$ convergence with gradient descent. There are no methods in this particular class with better than $O(1/k^2)$ convergence by the theorem given. Based on that theorem, you might hope that there is actually a method in this class with $O(1/k^2)$. And there is, but that's not implied by that theorem alone. $\endgroup$ – Ian Mar 9 '18 at 22:28
  • 1
    $\begingroup$ @Olórin Indeed the theorem you quoted in your question does not state that there exists an algorithm which achieves $f(x^k) - f(x^*) \leq O(1/k^2)$. The theorem you quoted is making a different point entirely. $\endgroup$ – littleO Mar 9 '18 at 22:44
  • 1
    $\begingroup$ There are two separate points to be made here: 1) No algorithm of this type can do better than $O(1/k^2)$. 2) There actually exists an algorithm of this type which achieves the $O(1/k^2)$ convergence rate. Both points are true, but the theorem from Nesterov that you quoted only addresses point 1. We have to look elsewhere to find Nesterov's proof of point 2. $\endgroup$ – littleO Mar 9 '18 at 22:49
1
$\begingroup$

Nesterov's theorem that you quoted is showing that no algorithm of this type can do better than $O(1/k^2)$. So a convergence rate of $O(1/k^2)$ is in some sense optimal for this type of algorithm. Nesterov separately provided an algorithm that actually achieves this optimal $O(1/k^2)$ convergence rate.

Of course, if the error decreases like $1/k^2$, that's a dramatic improvement over an algorithm for which the error goes down like $1/k$.

$\endgroup$
  • $\begingroup$ Hi, can you be more explicit as to why showing $f(x^k) - f(x^*) \geq \dfrac{3\beta}{32(k+1)^2\|x_1 - x^*\|^2} = \Omega(1/k^2)$ would imply no algorithm of this type can "do better" than $O(1/k^2)$? I'm still not quite understanding why a lower bound $\Omega$ result implies existence of lower upper bound than $O(1/k)$. $\endgroup$ – Olórin Mar 9 '18 at 22:35
  • $\begingroup$ It seems the more logical implication to the quoted theorem is: "Since no first-order algorithm can break the $\Omega(1/k^2)$ bound, since GD satisfies a 'lower bound' of $\Omega(1/k)$, therefore there exists a gap between the GD and the optimal rate." In other words, everything stated in terms of lower bound, instead the optimal rate stated in terms of lower bound, and the GD result is stated in terms of upper bound. $\endgroup$ – Olórin Mar 9 '18 at 22:55
  • $\begingroup$ The lower bound of $\Omega(1/k^2)$ does not imply a gap with GD. If you only know this lower bound of $\Omega(1/k^2)$ and the fact that GD has a rate of $O(1/k)$, all you can say is that the optimal rate is somewhere between $1/k$ and $1/k^2$ (ignoring constants, and asymptotically). But the fact that Nesterov subsequentially obtained an other algorithm (i.e., an upper bound) with a rate of $O(1/k^2)$ shows the optimal rate is $\Theta(1/k^2)$. $\endgroup$ – Clement C. Mar 9 '18 at 23:06
  • $\begingroup$ @ClementC Great, this makes a lot more sense. I was confused because in this reference arxiv.org/pdf/1405.4980.pdf at the bottom of pg 284, it states that there exists a gap due to my previous reasoning. "So far our results leave a gap in the case of smooth optimization: gradient descent achieves an oracle complexity of $O(1/ε)$ (respectively $O(κ log(1/ε))$ in the strongly convex case) while we proved a lower bound of $Ω(1/\sqrt(\epsilon))$ (respectively $Ω(\sqrt(k) log(1/\epsilon))$). In this section we close these gaps" $\endgroup$ – Olórin Mar 9 '18 at 23:11
  • $\begingroup$ @Olórin If you are talking about the first paragraph of Section 3.6, I don't see why what is written corresponds to what you said. The "gap" mentioned there is not a separation between an algorithm and the other: it literally refers to a gap in our knowledge, as explained above: "the best upper bound we knew was $O(1/\varepsilon)$, the best lower bound we knew was $\Omega(1/\sqrt{\varepsilon})$, so there is a gap between the two where the 'right' answer lies." $\endgroup$ – Clement C. Mar 9 '18 at 23:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.