Understanding the operator norm through this example? I'm a better learner through examples so I need help solving this one to better understand the operator norm.
We define first the norm of a matrix as $\|A\|_\infty = \sup_{1\leq i,j\leq n}|c_{ij}|$ .
Questions: Prove that $\Omega(A,B)=tr(AB)$ is continuous and calculate its operator norm.
I have no idea on how to proceed.
I'm not trying to learn the solution. I just want to know the reasoning behind it. Am I supposed to find an inequality of some sort and bound the quantity? What's the method to proceed?
 A: Since you said you only want to know how to approach this type of problem, I'll be a little vague here so you can fill in the details. First, to define the operator norm, as Teresa Lisbon mentioned in the comments, we can't really address this problem until we understand the spaces and norms we are talking about. You equipped the space $\mathbb{R}^{n\times n}$ with the $\ell^\infty$ norm. However, $\Omega$ is a mapping from $(\mathbb{R}^{n\times n})^2 \to \mathbb{R}$. To define the operator norm, we need to have a norm on the space $(\mathbb{R}^{n\times n})^2$. For the purposes of this problem, let $(A,B) = (a_{ij},b_{ij})_{i,j=1,\dots, n}$ and say that,
$$\|(A,B)\|_{\infty} = \sup_{i,j}\max\{|a_{ij}|,|b_{ij}|\}.$$
Then the definition of the operator norm is,
\begin{equation}
\tag{1}
\|\Omega\| = \sup_{\|(A,B)\|_{\infty} = 1} |\Omega(A,B)|.
\end{equation}
From here on, I'm going to be a little vague. We can prove continuity directly. For any $\epsilon > 0$, introduce two perturbation matrices $\|(A_\epsilon,B_\epsilon)\|_\infty < \epsilon$. Then,
$$(A + A_\epsilon)(B+B_\epsilon) = AB + AB_\epsilon + A_\epsilon B + A_\epsilon B_\epsilon.$$
You can use this and the definition of $\Omega$ to directly bound the quantity $\|\Omega(A,B) - \Omega(A+A_\epsilon,B+B_\epsilon)|$ by some function of $\epsilon$. This would show that for any sequence $\{(A_n,B_n) \in (\mathbb{R}^{n\times n})^2\}$ such that $(A_n,B_n) \to (A,B)$, $\Omega(A_n,B_n) \to \Omega(A,B)$. Thus, $\Omega$ is continuous.
To compute the operator norm, we again consider equation (1). To compute $\|\Omega\|$, we want to show two things. First, we need a bound $M$ such that for all $(A,B)$ with $\|(A,B)\|_\infty=1$, $|\Omega(A,B)| \leq M$. Second, we need an example of matrices $\|(A,B)\|_\infty = 1$ such that $|\Omega(A,B)| = M$.
The easiest way to do this is to explicitly write out $\Omega$ in terms of the entries of $A$ and $B$. Note that,
$$\text{tr}(AB) = \sum_{k=1}^n (AB)_{k,k} = \sum_{k=1}^n \sum_{i=1}^n a_{k,i}b_{i,k}.\tag{2}$$
Then I'll leave this for you. Given that $|a_{i,j}|,|b_{i,j}| \leq 1$ for every $i,j \in \{1,\dots,n\}$, is there a simple choice of $M$ such that
$$\text{tr}(AB) \leq M?$$
Second, are there matrices $(A,B)$ with norm 1 such that $\text{tr}(AB) = M$?
Hint:

 Given that $-1\leq a_{ij},b_{ij}\leq 1$ for every $i,j$, is there a choice of entries $(a_{ij},b_{ij})_{i,j = 1,\dots,n}$ that clearly maximize equation (2)?

With this as an example, let's talk more generally. Suppose you have a normed vector space $X$ such that the set $\{x \in X: \|x\|_X= 1\}$ is compact and a continuous operator $\Omega: X \to \mathbb{R}$ and you want to compute the operator norm. Then try investigating what $\Omega(x)$ looks like and see if there is an $x \in X$ such that $\|x\|_X = 1$ and $|\Omega(x)| \geq |\Omega(x')|$ for any other $x' \in X$ such that $\|x'\|_X = 1$. If you can find that value of $x$, then $\|\Omega\| = |\Omega(x)|$. Note that the existence of this value $x$ follows from the continuity of $\Omega$ and the compactness of $\{\|x\|_X = 1\}$ in $X$.
