# Equivalent Definitions of the Operator Norm

How do you prove that these four definitions of the operator norm are equivalent? \begin{align*} \lVert A\rVert_{\mathrm{op}} &= \inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}\\ &=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}\\ &=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \}\\ &=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*}

• Just a comment, but there is nothing to show for the last equality: the set of values you are maximizing over is exactly the same. I also claim that $\leq 1$ vs $=1$ also does not require proof, since for $\|v| \leq 1$, $\|Av\| \leq \|A v/(\|v\|)\|$, meaning we can totally disregard the vectors $\|v\| < 1$. The only one that should require any argument is the equivalence between the inf and sup. Feb 3, 2021 at 7:46

Let \begin{align*} I &= \inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}\\ S_1&=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}\\ S_2&=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \}\\ S_3&=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*} Notice that $S_2 \le S_1$ and as $\|Av\| /\|v\| = \| A(v / \|v\|)\|$ we have $S_3 \le S_2$. Now if $\|v\|\le 1$ we have $\|Av\| \le \|Av\| /\|v\|$. Then $S_1 \le S_3$ and $$S_1=S_2=S_3.$$ Now note that $$\|Av\| \le S_3 \|v\| \quad \forall v \in V.$$ Then $I \le S_3$ and by definition of $\sup$ we have $$I \ge \|Av_n\| /\|v_n\| \ge S_3 - 1/n \quad \forall n.$$ Then $S_3 = I$.

• @ user29999 $S_1 \nleq S_3$. You have proved the inequality for $||v||\leq 1$ but $S_3$ is supremm over $X-${0} Aug 25, 2017 at 6:51
• Why $I \ge \|Av_n\| /\|v_n\|$ ? And thank you Jan 23, 2018 at 16:17
• @Schüler if you multiply both sides by the denominator of the RHS, it should be clear from the definition of $I$ Apr 5, 2020 at 4:27

Remember that if $$s=\sup X$$ and $$x\leq t$$ for all $$x\in X$$, then $$s\leq t$$. Also, if $$s=\inf X$$ and $$t\in S$$, then $$s\leq t$$. Now, let us write \begin{align*} a&= \inf\{ k\;\colon\; \lVert Av\rVert\leq k\lVert v\rVert \text{ for all }v\in V\},\\ \\ b&=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\},\\ \\ c&=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \},\\\\ d&=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}. \end{align*}

Note that:

• $$\|Av\|\leq a\|v\|$$ for all $$v\in V$$. Taking $$v\in V$$ with $$\|v\|\leq 1$$ we obtain $$\|Av\|\leq a$$ and thus $$b\leq a$$;

• $$\|Av\|\leq b$$ for all $$v\in V$$ with $$\|v\|\leq 1$$. Take $$v\in V$$ with $$\|v\|=1$$ and define $$v_n=(1-1/n)v$$. Since $$\|v_n\|=1-1/n\leq 1$$, we conclude that $$\|Av_n\|\leq b$$ for all $$n\in\mathbb{N}$$. Taking the limit we obtain $$\|Av\|\leq b$$ and thus $$c\leq b$$;

• $$\|Av\|\leq c$$ for all $$v\in V$$ with $$\|v\|= 1$$. Taking $$v\in V$$ with $$v\neq 0$$, we obtain $$\left\|\frac{v}{\|v\|}\right\|=1$$. Hence $$\frac{\|Av\|}{\|v\|}=\left\|A\left(\frac{v}{\|v\|}\right)\right\|\leq c$$ and thus $$d\leq c$$;

• $$\frac{\|Av\|}{\|v\|}\leq d$$ for all $$v\in V$$ with $$v\neq 0$$. Hence $$\|Av\|\leq d\|v\|$$ for all $$v\in V$$ and thus $$a\leq d$$.

This shows that $$a\leq d\leq c\leq b\leq a$$ and thus $$a=b=c=d$$.

• @Student As explained in the first item, we have $\|Av\|\leq a$ for all $v\in V$ such that $\|v\|\leq 1$. This shows that $a$ is an upper bound of $\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\}$. Since $b$ is the least upper bound (supremum) of this set, we obtain $b\leq a$. Jan 23, 2018 at 17:02
• Thank you very much but my problem: why $\|Av\|\leq a\|v\|$ for all $v\in V$? Jan 23, 2018 at 21:05
• @Student You are right, $a$ does not necessarily belong to the said set. However, there exists a sequence $(k_n)$ in the set such that $k_n\overset{n\to\infty}{\longrightarrow} a$ (see this or this). We have $\|Av\|\leq k_n\|v\|$ for all $n$ and thus, taking the limit as $n\to\infty$, we obtain $\|Av\|\leq a\|v\|$. Jan 23, 2018 at 23:25
• @Student We know that $\inf A=\sup B=\sup C=\inf D$ and you are asking if $\inf A=\inf E$, where $A,B,...,E$ are the appropriate sets. Note that $A\subset E$ and thus $e=\inf E\leq \inf A=a$. As explained above (with sequences), $a\in A$. A similar argumment (with sequences) shows that $e\in E$ and thus $e$ is an upper bound of $D$. This implies that $d=\sup D\leq e$. Therefore $e\leq a=d\leq e$ which implies $a=e$. Jan 25, 2018 at 13:01
• I don't understand your second bullet point (nor why you need to define such sequence), like having $\| v\| = 1$ means the third set must be a subset of the previous one so its largest upper bound can only be equal or smaller as the previous one (but not larger). Not sure why we need to invoke the rest of your argument (though Im curious). Mar 18, 2019 at 20:13

I'll give you part of one to give you an idea of the flavor, but you should really do them yourself.

Let $w\neq 0$. Then $\frac{1}{\lVert w\rVert}$ makes sense. Now notice that $$A\left(\frac{1}{\lVert w\rVert}w\right) = \frac{1}{\lVert w\rVert}A(w).$$ Therefore, $$\left\lVert A\left(\frac{w}{\lVert w\rVert}\right)\right\rVert = \frac{\lVert Aw\rVert}{\lVert w\rVert}.$$ But $\frac{w}{\lVert w\rVert}$ is a vector of norm $1$, so...

The four definitions are equivalent only if $V\neq\{0\}$ (which, admittedly, is usually the case).

Reason: When $V=\{0\}$, there is no $v\in V$ such that $\|v\|=1$. So $\sup\varnothing = -\infty$, but we clearly want $\|A\|=0$ if $A$ is the zero operator. This (small but still) mistake can be found in many textbooks.

In summary, only the first two expressions make sense.

• It is not uncommon to take the supremum of an empty set of non-negative real numbers as $0$, generally if $L$ is a lattice with smallest element $s$, $\sup \varnothing = s$ when looking at subsets of $L$. Of course that should be mentioned. Which hardly any text does. And in the first definition, the constraint $c \geqslant 0$ is missing. Without that, $V = \{0\}$ leads to $\lVert A\rVert = -\infty$ for $A \colon V \to W$. Jul 15, 2018 at 8:59

Another equivalent definition, which I can rarely see: $$\lVert A\rVert=\sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$ It's easy to see that $$\sup_{\lVert x \rVert < 1} \lVert Ax \rVert \leqslant \sup_{\lVert x \rVert \leqslant 1} \lVert Ax \rVert$$ For the other direction, let $$(r_n)$$ be a sequence of real numbers, with $$0 \leqslant r_n < 1$$ and $$r_n \to 1$$, and $$x \in V$$ with $$\lVert x \rVert \leqslant 1$$. Then, we have that $$\lVert r_n x\rVert<1$$, which implies that $$\lVert A(r_n x) \rVert \leqslant \sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$, but $$\lVert A(r_n x) \rVert \to \lVert A x\rVert$$, which implies that $$\lVert Ax \rVert \leqslant \sup_{\lVert x \rVert < 1} \lVert Ax \rVert$$.

• Why is it always possible to find a sequence $r_n$ as defined above? Mar 3, 2023 at 17:56
• @NatMath What are you trying to point out? Mar 4, 2023 at 18:06
• @BotondMy answer is: does it always exist and if yes why a sequence $\{r_n\}$ with the above properties? Mar 7, 2023 at 18:49
• @NatMath The point is that you can pick any sequence with that property. Of course, if the underlying set of scalars are the real or the complex numbers. I am only using the properties of the norms, and the limit of real sequences. Is it clear now? Mar 8, 2023 at 19:31
• @BotondBut why is it possible to do this? For the completeness of $\mathbb{R}$? Or why are the rationals dense in $\mathbb{R}$? Mar 9, 2023 at 9:45

Let $$a=\inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\},b=\sup\{ \lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert\leq 1\},c=\sup\{\lVert Av\rVert\;\colon\; v\in V\text{ with }\lVert v\rVert = 1 \},d=\sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}.$$

$$a=\inf\{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}= \inf\{ c\;\colon\;\frac{\lVert Av\rVert}{\lVert v\rVert}\leq c \text{ for all }v\neq 0\}$$ (easily verified), which is the infimum of the set of upper bounds for the set $$\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0\right\}$$, i.e., the least upper bound. So $$a=d$$. Next see that $$\frac{\lVert Av\rVert}{\lVert v\rVert}=\lVert A(\frac{v}{\lVert v\rVert})\rVert$$, and $$\frac{v}{\lVert v\rVert}$$ has norm $$1$$. And if $$v$$ has norm $$1$$, then we can write $$v =\frac{v}{\lVert v\rVert}.$$ This shows that $$c=d$$. Finally I'll show that $$b=c$$ (because I find this more interesting). $$b\geq c$$ is easily seen. For each $$v$$ with $$\lVert v\rVert\leq 1$$, there is $$\tilde v=\frac{v}{\lVert v\rVert}$$ having norm $$1$$, with $$\lVert A\tilde v\rVert=\frac{\lVert Av\rVert}{\lVert v\rVert}\geq \lVert Av\rVert.$$ This proves $$c\geq b.$$

@xxxg: I want to make some further details. In other words, I want to prove the following two things.

(i) The infimum of the first set, i.e., I, also satisfies the condition in the first set, which means that $$\|Av\| \leq I \|V\|, \quad \text{for all v \in V}.$$ Proof of (i): Let us assume that there exists a $$v_0$$ such that $$\frac{\|Av_0\|}{\|v_0\|} > I.$$ Then, by the definition of infimum, there exists $$c_0 \in \{ c\;\colon\; \lVert Av\rVert\leq c\lVert v\rVert \text{ for all }v\in V\}$$ such that $$c_0 < \left(\frac{\|Av_0\|}{\|v_0\|} + I\right) /2 < \frac{\|Av_0\|}{\|v_0\|},$$ which makes a contradiction. End of proof of (i).

(ii) Let $$S_4 = \sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0, \|v\| \leq r\right\},$$ $$S_5 = \sup\left\{ \frac{\lVert Av\rVert}{\lVert v\rVert}\;\colon\; v\in V\text{ with }v\neq 0, \|v\| = r\right\}.$$ Then $$I=S_1=S_2=S_3=S_4=S_5, \quad \text{for any r>0}.$$ Proof of (ii): For any $$v\in V$$, $$\|v\|=1$$, there exists $$g = rv \in V$$, $$\|g\|=r$$,
it holds $$\frac{\|Ag\|}{\|g\|} = \|Av\|.$$ So $$S_2 \leq S_5$$. Then we have $$S_2 \leq S_5 \leq S_4 \leq S_3.$$ End of proof of (ii).

– xxxg
Jan 31 at 13:03
• @Schüler, you can see my answer here.
– xxxg
Jan 31 at 13:05

It is of course possible to prove direct equivalence of any two of the definitions as well. Here is one example.
Let \begin{align*} a &= \inf \left\{ k > 0: \left\lVert L v \right\rVert \leq k \left\lVert v \right\rVert \text{ for all } v \in V \right\} \text{ and} \\ b &= \sup \left\{ \left\lVert L v \right\rVert \text{ for all } v \in V \text{ with } \left\lVert v \right\rVert \leq 1 \right\} . \end{align*} Now we prove that $$a = b$$ by proving that $$b \leq a$$ and $$a \leq b$$.

First, we have that for any $$v \in V$$ $$\left\lVert L v \right\rVert \leq a \left\lVert v \right\rVert ,$$ since $$a$$ was the infimum of all numbers that satisfies this equation. In particular, for any $$v$$ with $$\left\lVert v \right\rVert \leq 1$$ we have $$\left\lVert L v \right\rVert \leq a \left\lVert v \right\rVert \leq a .$$ This tells us that $$a$$ is an upper bound for $$\left\lVert L v \right\rVert$$ when $$\left\lVert v \right\rVert \leq 1$$, and hence an upper bound for the set in the definition of $$b$$. Since $$b$$ is the supremum (smallest upper bound), we have that $$b \leq a .$$ The opposite direction requires a simple trick. Let $$v \in V$$ be any vector and choose an $$\epsilon < 1$$. Then the vector defined by $$u = \frac{v}{\left\lVert v \right\rVert + \epsilon} ,$$ fulfils $$\left\lVert u \right\rVert = \frac{\left\lVert v \right\rVert}{\left\lVert v \right\rVert + \epsilon} < 1 .$$ Hence, we see that it is in the set that defines $$b$$, and we must have $$\left\lVert L u \right\rVert \leq b.$$ Inserting the relation between $$u$$ and $$v$$ we get $$\left\lVert L v \right\rVert \leq b \left(\left\lVert v \right\rVert + \epsilon \right) ,$$ which in the limit $$\epsilon \to 0$$ becomes $$\left\lVert L v \right\rVert \leq b \left\lVert v \right\rVert .$$ Since $$a$$, being the infimum of all numbers satisfying this relation, we must have $$a \leq b ,$$ and we are done