Max Min of function less than Min max of function

Question

I can't understand why max min of a function is less than equal to min max of that function i.e
Why $$\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) \leq \underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$$
Here $x,y \in \mathbb{R}$ and $f(x,y)\in \mathbb{R}$
Moreover, I don't understand intuitively what is the effect of just changing the order of max and min.
Suppose $(\hat{x},\hat{y})$ is the solution of $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, then why this is not the same solution for $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)$.
One more thing I want to know is do we evaluate inner optimization first or outer ? i.e in $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$, do we evaluate $\underset{y}{\text{min}}$ first or $\underset{x}{\text{max}}$ first ?

Answer 1

Perhaps a simple example will help. Let $f(x,y) = \sin(x+y)$. Then

$\underset{y}{\text{min}} f(x,y) = -1$ for all $x$; and

$\underset{x}{\text{max}} f(x,y) = +1$ for all $y$.

So $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y) = \underset{x}{\text{max}} (-1) = -1$; but $\underset{y}{\text{min}}\:\underset{x}{\text{max}} f(x,y) = \underset{y}{\text{min}} (+1) = +1\,$.

Answer 2

One asks that $$ \max\limits_x\left(\min\limits_sf(x,s)\right)\leqslant\min\limits_y\left(\max\limits_tf(t,y)\right). $$ The assertion is equivalent to the fact that, for every $x$ and $y$, $$ \min\limits_sf(x,s)\leqslant\max\limits_tf(t,y). $$ Since $\min\limits_sf(x,s)\leqslant f(x,y)\leqslant\max\limits_tf(t,y)$ by definition, this holds.

Answer 3

Let $\hat x,\hat y$ be the arguments responsible for the value $\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$. Then $f(\hat x,y)\ge f(\hat x,\hat y)$ for all $y$. For every $y$, the maximization $\underset{x}{\text{max}}f(x,y)$ extends over one of these values, and thus $\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)$ for all $y$, and thus also $\underset{y}{\text{min}}\:\underset{x}{\text{max}}f(x,y)\ge f(\hat x,\hat y)=\underset{x}{\text{max}}\:\underset{y}{\text{min}} f(x,y)$.

Answer 4

The inequality should be $\sup_x\inf_yf(x,y)\leq \inf_y\sup_x f(x,y)$. Consider $f(x,y)= x^2y^2$.

Answer 5

I personally found this question in ML class of mine and went down to solve it myself at home as the teacher did not give any proof. Here it is:
Let $ f(x_{0}, y_{0}) = \max_x\min_y f(x, y)$ and $f(x_{1}, y_{1}) = \min_y\max_x f(x, y)$.
By this definition the problem is to prove that $f(x_{0}, y_{0}) \leq f(x_{1}, y_{1})$ provided that they exist.
By definition of min and max function we have:
$\min_yf(x, y) = f(x, y_{0}) \leq f(x, y) \forall y$. Here $\min_yf(x, y)$ would be a function only of x. $\max_xf(x, y_{0})=f(x_{0}, y_{0}) \geq f(x, y_{0})\forall x$. Here $\max_xf(x, y_{0})$ is a scalar. $\max_xf(x, y)=f(x_{1}, y) \geq f(x,y) \forall x$. Here $\max_xf(x_{1}, y)$ is a function of y. $\min_yf(x_{1}, y)=f(x_{1},y_{1}) \leq f(x_{1}, y) \forall y$. Here $\min_yf(x_{1}, y)$ is a scalar.
From all this equation you get the following inequalities: $f(x, y_{0}) \leq f(x_{0},y_{0}) \leq f(x,y) \leq f(x_{1}, y_{1}) \leq f(x_{1}, y)$ $\min_yf(x,y) \leq \max_x\min_yf(x,y) \leq f(x,y) \leq \min_y\max_xf(x,y) \leq \max_xf(x,y)$ $g(x) \leq Scalar \leq f(x,y) \leq Scalar \leq h(y)$
Hope the last thing as visualization helps you understand the problem.