# Equivalent Definitions of the Operator Norm

How do you prove that these four definitions of the operator norm are equivalent? \begin{align*} \lVert A\rVert_{\mathrm{op}} &= \inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}\\ &=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}\\ &=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \}\\ &=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*}

• Just a comment, but there is nothing to show for the last equality: the set of values you are maximizing over is exactly the same. I also claim that $\leq 1$ vs $=1$ also does not require proof, since for $\|v| \leq 1$, $\|Av\| \leq \|A v/(\|v\|)\|$, meaning we can totally disregard the vectors $\|v\| < 1$. The only one that should require any argument is the equivalence between the inf and sup. Feb 3, 2021 at 7:46

Let \begin{align*} I &= \inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}\\ S_1&=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}\\ S_2&=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \}\\ S_3&=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*} Notice that $S_2 \le S_1$ and as $\|Av\| /\|v\| = \| A(v / \|v\|)\|$ we have $S_3 \le S_2$. Now if $\|v\|\le 1$ we have $\|Av\| \le \|Av\| /\|v\|$. Then $S_1 \le S_3$ and $$S_1=S_2=S_3.$$ Now note that $$\|Av\| \le S_3 \|v\| \quad \forall v \in V.$$ Then $I \le S_3$ and by definition of $\sup$ we have $$I \ge \|Av_n\| /\|v_n\| \ge S_3 - 1/n \quad \forall n.$$ Then $S_3 = I$.

• @ user29999 $S_1 \nleq S_3$. You have proved the inequality for $||v||\leq 1$ but $S_3$ is supremm over $X-${0} Aug 25, 2017 at 6:51
• Why $I \ge \|Av_n\| /\|v_n\|$ ? And thank you Jan 23, 2018 at 16:17
• @Schüler if you multiply both sides by the denominator of the RHS, it should be clear from the definition of $I$ Apr 5, 2020 at 4:27

Remember that if $$s=\sup X$$ and $$x\leq t$$ for all $$x\in X$$, then $$s\leq t$$. Also, if $$s=\inf X$$ and $$t\in S$$, then $$s\leq t$$. Now, let us write \begin{align*} a&= \inf\{ k\;\colon\; \lVert Av\rVert\leq k\lVert v\rVert \text{ for all }v\in V\},\\ \\ b&=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\},\\ \\ c&=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \},\\\\ d&=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*}

Note that:

• $$\|Av\|\leq a\|v\|$$ for all $$v\in V$$. Taking $$v\in V$$ with $$\|v\|\leq 1$$ we obtain $$\|Av\|\leq a$$ and thus $$b\leq a$$;

• $$\|Av\|\leq b$$ for all $$v\in V$$ with $$\|v\|\leq 1$$. Take $$v\in V$$ with $$\|v\|=1$$ and define $$v_n=(1-1/n)v$$. Since $$\|v_n\|=1-1/n\leq 1$$, we conclude that $$\|Av_n\|\leq b$$ for all $$n\in\mathbb{N}$$. Taking the limit we obtain $$\|Av\|\leq b$$ and thus $$c\leq b$$;

• $$\|Av\|\leq c$$ for all $$v\in V$$ with $$\|v\|= 1$$. Taking $$v\in V$$ with $$v\neq 0$$, we obtain $$\left\|\frac{v}{\|v\|}\right\|=1$$. Hence $$\frac{\|Av\|}{\|v\|}=\left\|A\left(\frac{v}{\|v\|}\right)\right\|\leq c$$ and thus $$d\leq c$$;

• $$\frac{\|Av\|}{\|v\|}\leq d$$ for all $$v\in V$$ with $$v\neq 0$$. Hence $$\|Av\|\leq d\|v\|$$ for all $$v\in V$$ and thus $$a\leq d$$.

This shows that $$a\leq d\leq c\leq b\leq a$$ and thus $$a=b=c=d$$.

• Why $b\leq a$?Thanks Jan 23, 2018 at 16:22
• @Student As explained in the first item, we have $\|Av\|\leq a$ for all $v\in V$ such that $\|v\|\leq 1$. This shows that $a$ is an upper bound of $\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}$. Since $b$ is the least upper bound (supremum) of this set, we obtain $b\leq a$. Jan 23, 2018 at 17:02
• Thank you very much but my problem: why $\|Av\|\leq a\|v\|$ for all $v\in V$? Jan 23, 2018 at 21:05
• @Student You are right, $a$ does not necessarily belong to the said set. However, there exists a sequence $(k_n)$ in the set such that $k_n\overset{n\to\infty}{\longrightarrow} a$ (see this or this). We have $\|Av\|\leq k_n\|v\|$ for all $n$ and thus, taking the limit as $n\to\infty$, we obtain $\|Av\|\leq a\|v\|$. Jan 23, 2018 at 23:25
• @Student We know that $\inf A=\sup B=\sup C=\inf D$ and you are asking if $\inf A=\inf E$, where $A,B,...,E$ are the appropriate sets. Note that $A\subset E$ and thus $e=\inf E\leq \inf A=a$. As explained above (with sequences), $a\in A$. A similar argumment (with sequences) shows that $e\in E$ and thus $e$ is an upper bound of $D$. This implies that $d=\sup D\leq e$. Therefore $e\leq a=d\leq e$ which implies $a=e$. Jan 25, 2018 at 13:01

I'll give you part of one to give you an idea of the flavor, but you should really do them yourself.

Let $w\neq 0$. Then $\frac{1}{\lVert w\rVert}$ makes sense. Now notice that $$A\left(\frac{1}{\lVert w\rVert}w\right) = \frac{1}{\lVert w\rVert}A(w).$$ Therefore, $$\left\lVert A\left(\frac{w}{\lVert w\rVert}\right)\right\rVert = \frac{\lVert Aw\rVert}{\lVert w\rVert}.$$ But $\frac{w}{\lVert w\rVert}$ is a vector of norm $1$, so...

The four definitions are equivalent only if $V\neq\{0\}$ (which, admittedly, is usually the case).

Reason: When $V=\{0\}$, there is no $v\in V$ such that $\|v\|=1$. So $\sup\varnothing = -\infty$, but we clearly want $\|A\|=0$ if $A$ is the zero operator. This (small but still) mistake can be found in many textbooks.

In summary, only the first two expressions make sense.

• It is not uncommon to take the supremum of an empty set of non-negative real numbers as $0$, generally if $L$ is a lattice with smallest element $s$, $\sup \varnothing = s$ when looking at subsets of $L$. Of course that should be mentioned. Which hardly any text does. And in the first definition, the constraint $c \geqslant 0$ is missing. Without that, $V = \{0\}$ leads to $\lVert A\rVert = -\infty$ for $A \colon V \to W$. Jul 15, 2018 at 8:59

Another equivalent definition, which I can rarely see: $$\lVert A\rVert=\sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$ It's easy to see that $$\sup_{\lVert x \rVert < 1} \lVert Ax \rVert \leqslant \sup_{\lVert x \rVert \leqslant 1} \lVert Ax \rVert$$ For the other direction, let $$(r_n)$$ be a sequence of real numbers, with $$0 \leqslant r_n < 1$$ and $$r_n \to 1$$, and $$x \in V$$ with $$\lVert x \rVert \leqslant 1$$. Then, we have that $$\lVert r_n x\rVert<1$$, which implies that $$\lVert A(r_n x) \rVert \leqslant \sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$, but $$\lVert A(r_n x) \rVert \to \lVert A x\rVert$$, which implies that $$\lVert Ax \rVert \leqslant \sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$.

• Why is it always possible to find a sequence $r_n$ as defined above? Mar 3 at 17:56
• @NatMath What are you trying to point out? Mar 4 at 18:06
• @BotondMy answer is: does it always exist and if yes why a sequence $\{r_n\}$ with the above properties? Mar 7 at 18:49
• @NatMath The point is that you can pick any sequence with that property. Of course, if the underlying set of scalars are the real or the complex numbers. I am only using the properties of the norms, and the limit of real sequences. Is it clear now? Mar 8 at 19:31
• @BotondBut why is it possible to do this? For the completeness of $\mathbb{R}$? Or why are the rationals dense in $\mathbb{R}$? Mar 9 at 9:45

Let $$a=\inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\},b=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\},c=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \},d=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}.$$

$$a=\inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}= \inf\{ c\;\colon\;\frac{\lVert Av\rVert}{\lVert v\rVert}\leq c \text{ for all }v\neq 0\}$$ (easily verified), which is the infimum of the set of upper bounds for the set $$\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}$$, i.e., the least upper bound. So $$a=d$$. Next see that $$\frac{\lVert Av\rVert}{\lVert v\rVert}=\lVert A(\frac{v}{\lVert v\rVert})\rVert$$, and $$\frac{v}{\lVert v\rVert}$$ has norm $$1$$. And if $$v$$ has norm $$1$$, then we can write $$v =\frac{v}{\lVert v\rVert}.$$ This shows that $$c=d$$. Finally I'll show that $$b=c$$ (because I find this more interesting). $$b\geq c$$ is easily seen. For each $$v$$ with $$\lVert v\rVert\leq 1$$, there is $$\tilde v=\frac{v}{\lVert v\rVert}$$ having norm $$1$$, with $$\lVert A\tilde v\rVert=\frac{\lVert Av\rVert}{\lVert v\rVert}\geq \lVert Av\rVert.$$ This proves $$c\geq b.$$

It is of course possible to prove direct equivalence of any two of the definitions as well. Here is one example.
Let \begin{align*} a &= \inf \left\{ k > 0: \left\lVert L v \right\rVert \leq k \left\lVert v \right\rVert \text{ for all } v \in V \right\} \text{ and} \\ b &= \sup \left\{ \left\lVert L v \right\rVert \text{ for all } v \in V \text{ with } \left\lVert v \right\rVert \leq 1 \right\} . \end{align*} Now we prove that $$a = b$$ by proving that $$b \leq a$$ and $$a \leq b$$.

First, we have that for any $$v \in V$$ $$\left\lVert L v \right\rVert \leq a \left\lVert v \right\rVert ,$$ since $$a$$ was the infimum of all numbers that satisfies this equation. In particular, for any $$v$$ with $$\left\lVert v \right\rVert \leq 1$$ we have $$\left\lVert L v \right\rVert \leq a \left\lVert v \right\rVert \leq a .$$ This tells us that $$a$$ is an upper bound for $$\left\lVert L v \right\rVert$$ when $$\left\lVert v \right\rVert \leq 1$$, and hence an upper bound for the set in the definition of $$b$$. Since $$b$$ is the supremum (smallest upper bound), we have that $$b \leq a .$$ The opposite direction requires a simple trick. Let $$v \in V$$ be any vector and choose an $$\epsilon < 1$$. Then the vector defined by $$u = \frac{v}{\left\lVert v \right\rVert + \epsilon} ,$$ fulfils $$\left\lVert u \right\rVert = \frac{\left\lVert v \right\rVert}{\left\lVert v \right\rVert + \epsilon} < 1 .$$ Hence, we see that it is in the set that defines $$b$$, and we must have $$\left\lVert L u \right\rVert \leq b.$$ Inserting the relation between $$u$$ and $$v$$ we get $$\left\lVert L v \right\rVert \leq b \left(\left\lVert v \right\rVert + \epsilon \right) ,$$ which in the limit $$\epsilon \to 0$$ becomes $$\left\lVert L v \right\rVert \leq b \left\lVert v \right\rVert .$$ Since $$a$$, being the infimum of all numbers satisfying this relation, we must have $$a \leq b ,$$ and we are done