# Max Min of function less than Min max of function

I can't understand why max min of a function is less than equal to min max of that function i.e
Why $$\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) \leq \underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$$
Here $x,y \in \mathbb{R}$ and $f(x,y)\in \mathbb{R}$
Moreover, I don't understand intuitively what is the effect of just changing the order of max and min.
Suppose $(\hat{x},\hat{y})$ is the solution of $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, then why this is not the same solution for $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$.
One more thing I want to know is do we evaluate inner optimization first or outer ? i.e in $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, do we evaluate $\underset{y}{\text{min}}$ first or $\underset{x}{\text{max}}$ first ?

-
It's a difficult matter. John Forbes Nash has won a Nobel prize for a general theorem about min-max-problems. –  Christian Blatter Aug 25 '12 at 14:18
$\max_x \min_y f(x,y) = \max_x \left( \min_y f(x,y) \right)$, i.e. the maximum (with respect to $x$) of the minimum (with respect to $y$) of $f(x,y)$. It's usually kind of hard to evaluate the outer optimization without first evaluating the inner, since you need to evaluate the inner one just to know what the outer one is supposed to be optimizing. Of course, sometimes one can find clever ways to skip or delay the inner optimization, but those cases are the exception, not the rule. –  Ilmari Karonen Aug 26 '12 at 17:00
add comment

## 5 Answers

Perhaps a simple example will help. Let $f(x,y) = \sin(x+y)$. Then

$\underset{y}{\text{min}} f(x,y) = -1$ for all $x$; and

$\underset{x}{\text{max}} f(x,y) = +1$ for all $y$.

So $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) = \underset{x}{\text{max}} (-1) = -1$; but $\underset{y}{\text{min}}\:\underset{x}{\text{max}} f(x,y) = \underset{y}{\text{min}} (+1) = +1\,$.

-
add comment

One asks that $$\max\limits_x\left(\min\limits_sf(x,s)\right)\leqslant\min\limits_y\left(\max\limits_tf(t,y)\right).$$ The assertion is equivalent to the fact that, for every $x$ and $y$, $$\min\limits_sf(x,s)\leqslant\max\limits_tf(t,y).$$ Since $\min\limits_sf(x,s)\leqslant f(x,y)\leqslant\max\limits_tf(t,y)$ by definition, this holds.

-
what is $\lim_y$? –  Seyhmus Güngören Aug 25 '12 at 12:09
@Seyhmus: A typo. As you guessed, one should read $\min\limits_y$. –  Did Aug 25 '12 at 13:53
add comment

Let $\hat x,\hat y$ be the arguments responsible for the value $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$. Then $f(\hat x,y)\ge f(\hat x,\hat y)$ for all $y$. For every $y$, the maximization $\underset{x}{\text{max}}f(x,y)$ extends over one of these values, and thus $\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)$ for all $y$, and thus also $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)=\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$.

-
Provided that $\hat{x}$ and $\hat{y}$ exist, of course. –  Siminore Aug 25 '12 at 10:23
As Juan and Siminore rightly point out, this only works if the minima and maxima actually exist; else they need to be replaced by infima and suprema, respectively. –  joriki Aug 25 '12 at 10:25
@Joriki I understood first line, but in starting of third line, you wrote $\underset{x}{\text{max}}f(x,y) \geq f(\hat{x},\hat{y})$, isn't $\underset{x}{\text{max}}f(x,y)$ same as $f(\hat{x},y)$ ? –  Happy Mittal Aug 25 '12 at 10:50
@Happy: No, why should it be? $\hat x$ is the value of $x$ that maximizes $\underset{y}{\text{min}} f(x,y)$. I don't see why the same value of $x$ should maximize $f(x,y)$ for every $y$. In fact, since the values of $f$ for different values of $y$ can be chosen entirely independently of each other, it would be weird if it did. –  joriki Aug 25 '12 at 11:26
joriki: Sorry but I do not understand the definition of $\hat y$. Shouldn't this depend on $x$? –  Did Aug 25 '12 at 11:42
show 4 more comments

The inequality should be $\sup_x\inf_yf(x,y)\leq \inf_y\sup_x f(x,y)$. Consider $f(x,y)= x^2y^2$.

-
I downvoted here, because the inequality wasn't reversed. –  mixedmath Aug 26 '12 at 4:53
It's not reversed, I was just (a bit pedantically) pointing out the inequality is in terms of supremums and infimums instead of maxima and minima, the original phrasing implicitly assumed that the minima and maxima actualy exists. –  jkn Aug 26 '12 at 13:25
Oh you're right. I'm sorry. –  mixedmath Aug 26 '12 at 15:22
That's ok, no worries ^^. –  jkn Aug 27 '12 at 15:29
add comment

I personally found this question in ML class of mine and went down to solve it myself at home as the teacher did not give any proof. Here it is:
Let $f(x_{0}, y_{0}) = \max_x\min_y f(x, y)$ and $f(x_{1}, y_{1}) = \min_y\max_x f(x, y)$.
By this definition the problem is to prove that $f(x_{0}, y_{0}) \leq f(x_{1}, y_{1})$ provided that they exist.
By definition of min and max function we have:
$\min_yf(x, y) = f(x, y_{0}) \leq f(x, y) \forall y$. Here $\min_yf(x, y)$ would be a function only of x. $\max_xf(x, y_{0})=f(x_{0}, y_{0}) \geq f(x, y_{0})\forall x$. Here $\max_xf(x, y_{0})$ is a scalar. $\max_xf(x, y)=f(x_{1}, y) \geq f(x,y) \forall x$. Here $\max_xf(x_{1}, y)$ is a function of y. $\min_yf(x_{1}, y)=f(x_{1},y_{1}) \leq f(x_{1}, y) \forall y$. Here $\min_yf(x_{1}, y)$ is a scalar.
From all this equation you get the following inequalities: $f(x, y_{0}) \leq f(x_{0},y_{0}) \leq f(x,y) \leq f(x_{1}, y_{1}) \leq f(x_{1}, y)$ $\min_yf(x,y) \leq \max_x\min_yf(x,y) \leq f(x,y) \leq \min_y\max_xf(x,y) \leq \max_xf(x,y)$ $g(x) \leq Scalar \leq f(x,y) \leq Scalar \leq h(y)$
Hope the last thing as visualization helps you understand the problem.

-
Well just to add a comment that might on the other hand looks really confusing, but the left part of the inequalities are varying on X and the right ones are varying on y... I just hope it helps cause now I realize it might actually confuse you even more. –  Belov Sep 3 '12 at 15:28
add comment