Equivalence of different definitions of sup in expectations So the context of my question comes from the proof of Theorem 7.3.1(Norms of Gaussian random matrices) from High Dimensional Probability by Vershynin.
In particular assume $A$ is an $m$ by $n$ matrices with independent $\mathcal{N}(0,1)$ entries. Define $T=\mathbb{S}^{n-1}\times \mathbb{S}^{m-1}$. Let $X_{uv}=\langle Au,  v\rangle$ where $u\in\mathbb{S}^{n-1}$ and $v\in \mathbb{S}^{m-1}$.
We know that the operator norm of $A$ is equivalently defined as $||A|| = \sup_{(u,v)\in T} X_{uv}$. Now the book claims the following equivalence.
$$E||A|| = \sup_{S\subset T} E \left(\sup_{(u,v)\in S} X_{uv}\right)$$
where the outer $sup$ is over all finite subsets $S$ of $T$. I do not see why this is true at all. Could someone please explain?
More generally, my concern is when it is true for a random process $X_t$ that
$$E \sup_{t\in T} X_t= \sup_{S\subset T} E\left( \sup_{t\in S} X_t\right)$$
where again the outer $sup$ is over finite subsets $S$ of $T$. I ask this because this is how Vershynin defines these expectations in the book by taking the sup over finite subsets to avoid measurability issues. I was wondering what merits these definitions because it seems to be commonplace in a lot of introductory high dimensional statistics/probability books.
 A: In general, this is not true. Let $U$ be a uniformly distributed random variable on $T:=[0,1]$. Define,
$$X_t = \begin{cases} 0 &\text{ if } t \neq U\\ 1 &\text{ if } t = U\end{cases}.$$
Then for any finite $S\subset T$, $\mathbb{E}\left[\sup_{t\in S}X_t\right] = 0$, so,
$$\sup_{S\subset T} \mathbb{E}\left[\sup_{t \in S} X_t\right] = 0 \neq 1 = \mathbb{E}\left[\sup_{t \in T} X_t\right].$$
Splitting the supremum works in the proof you mentioned because $(u,v)\mapsto X_{(u,v)}$ is continuous in $T:=\mathbb{S}^{n-1}\times\mathbb{S}^{m-1}$.
Let $\tilde{S} \subset T$ be a countable dense subset of $T$. Then by continuity of $(u,v) \mapsto X_{(u,v)}$,
$$\sup_{(u,v) \in \tilde{S}} X_{(u,v)} = \sup_{(u,v) \in T} X_{(u,v)} \text{ almost surely}.$$
Let $S_1 \subset S_2 \subset \cdots$ be a sequence of finite sets growing to $\tilde{S}$, and suppose $(0,0) \in S_1$. Then for all $n$,
$$\sup_{(u,v) \in S_n} X_{(u,v)} \geq X_{(0,0)} = \langle A0,0\rangle = 0\text{ almost surely,}$$
so the quantity $\sup_{(u,v) \in S_n} X_{(u,v)}$ is non-negative and almost surely non-decreasing. Applying monotone convergence,
$$\sup_{S \subset T} \mathbb{E}\left[\sup_{(u,v) \in S} X_{(u,v)}\right] \geq \sup_{n \in \mathbb{N}}\mathbb{E}\left[\sup_{(u,v)\in S_n}X_{(u,v)}\right] = \lim_{n \rightarrow \infty} \mathbb{E}\left[\sup_{(u,v)\in S_n}X_{(u,v)}\right] = \mathbb{E}\left[\sup_{(u,v) \in \tilde{S}} X_{(u,v)}\right] = \mathbb{E}\left[\sup_{(u,v) \in T} X_{(u,v)}\right].$$
The other direction of the inequality is trivial. For any finite $S$,
$$\mathbb{E}\left[\sup_{(u,v)\in S} X_{(u,v)}\right] \leq \mathbb{E}\left[\sup_{(u,v) \in T} X_{(u,v)}\right].$$
Taking a supremum,
$$\sup_{S \subset T}\mathbb{E}\left[\sup_{(u,v)\in S} X_{(u,v)}\right] \leq \mathbb{E}\left[\sup_{(u,v) \in T} X_{(u,v)}\right].$$
We can then conclude,
$$\sup_{S \subset T}\mathbb{E}\left[\sup_{(u,v)\in S} X_{(u,v)}\right] = \mathbb{E}\left[\sup_{(u,v) \in T} X_{(u,v)}\right].$$
Edit: I just realized I made a pretty big mistake. I'll leave the original up anyway. The problem is that $(0,0) \notin \mathbb{S}^{n-1}\times \mathbb{S}^{m-1}$. However, notice that for any (not necessarily finite) $S \subseteq T$,
$$\left|\sup_{(u,v) \in S} X_{(u,v)}\right| \leq \sum_{i=1}^m\sum_{j=1}^n|A_{ij}|,$$
which is integrable. Therefore we can switch the limit and the expectation using the Dominated Convergence Theorem instead of the Monotone Convergence Theorem.
