I was wondering about the truth of the following:

$$ \text{if} \lim_{n\to \infty} \| f_n - f \|_{\infty,S}=0 \implies \forall p\in \{ 2,3,4,...\}, \lim_{n\to\infty} \| f_n - f \|_{p,S} = 0$$

or at least in $L^2$ norm.

I think this statement is false. The reason is by re-writing the first part:

$$ \lim_{n\to \infty} \left[ \lim_{p \to \infty} \| f_n - f \|_{p,S} \right] =0 $$

and then reading what it means. It is saying that the as n goes to infinity the limit inside goes to zero only as the limit inside itself has $p$ go to infinity. Thus, my suspicion is that since the limit inside implicitly requires $p$ to grow and grow (for sufficiently large $p$) or "be at infinity already" (since we are talking about the actual limit value). Then means that the outer limit with $n$ is not true with $n$ going to infinity. This is my intuition of why its false, informally because the outer limit is only true when "$p=\infty$".

I bet that a counter example would be the easiest way to prove it. To construct it I tried sort of unraveling what the limit definitions where saying rigorously. I open up the out limit first (i.e. definition using for sufficiently large N...):

$$ \forall \epsilon_n, \exists N': \text{if } n \geq N' \implies | \lim_{p\to \infty} \| f_n - f \|_{p,S} - 0 | < \epsilon_n$$

where $\epsilon_n$ is one symbol denoting the outer limit (i.e. its not a function of $n$, the $n$ denotes its the epsilon for the outer limit).

But then when I tried to open up the second I got confused where the new quantifiers for the inner limit where suppose to go and thus, gave up because I didn't know how to let the statement still be true without writing non-sense. Does someone know how to do this? Maybe this is the wrong way to do it but its what I have...

Anyway, I thought it would be nice to put my question in context too since I believe I am not dealing with arbitrary functions. I was reading the following approximation theory paper in the context of Neural Networks. I am no expert in approximation theory, thus my question. In the paper section 3.1 they have the following sentences:

Notice, however, that from the point of view of machine learning, the relevant norm is the L2 norm. In this sense, several of our results are stronger than needed.

I guess since the quote says:

several of our results are stronger than needed.

I assumed that since the authors have convergence results in the sup norm, then it must mean that it somehow automatically implies its true for the truly relevant norm (i.e. the $L^2$ norm) in their machine learning context. Since a stronger statement $A$ implies weaker statements $B$ (check Why is the definition of stronger statement the way it is in logic?) then it was natural to assume that indeed, the authors where implying that it automatically extended to the $L^2$ norm.

Is that true? Can the results in that paper extend to the relevant loss functions $L^2$ norms in machine learning?

As a summary the context of target functions $f$ and the functions used to approximate (i.e. the one that forms the sequence of functions $f_n$) can be found in section 3.1 and 3.2 I believe. The target class $W^n_{m}$ is a smoothness class with the following property:

$$ \| f \|_{\infty} + \sum_{ 1 \leq | k |_1 \leq m}\| D^{k} f \|_{\infty} \leq 1 $$

i.e. all functions of $n$ variables and $m < \infty $ partial derivatives indicated by the multi-integer $k\geq 1$ and $|k|_1$ is the sum of the components of $k$.

These function are approximated with Neural Networks, I believe with smooth activation functions.

  • $\begingroup$ I've never seen this notation before. How is the norm $|\!|f|\!|_{p,S}$ defined? $\endgroup$ – uniquesolution Dec 10 '17 at 21:08
  • $\begingroup$ @uniquesolution oh really? I thought it was standard since I looked up wikipedia before asking this question (note this is not my field of expertise). The definition is $\| f \|_{\infty,S} = \sup_{x \in S} |f(x)| = \sup \{ |f(x)| : x \in S \}$ in the wikipedia article its the first equation: en.wikipedia.org/wiki/Uniform_norm $\endgroup$ – Pinocchio Dec 10 '17 at 21:12
  • $\begingroup$ On a set of finite measure, convergence in $L^p$ implies convergence in $L^q$ for $q < p$ because of Holder's inequality. In particular convergence in $L^\infty$ implies convergence in every other $L^p$. This is not true on sets of infinite measure; one can find functions converging in $L^1$ that don't converge in $L^\infty$ and those that converge in $L^\infty$ that don't converge in $L^1$. Think of examples where mass escapes to the left/right or escapes to infinity. $\endgroup$ – user217285 Dec 10 '17 at 21:20
  • $\begingroup$ @Nitin what about in the context that I provided? I guess since its Sobolev space and the norms seem to be bounded (the input space is a bounded hypercube $[-1,1]^n$). Then it seems its also bounded in the context of the paper I was reading? $\endgroup$ – Pinocchio Dec 10 '17 at 21:23
  • 1
    $\begingroup$ @Pinocchio. Wikipedia is not a standard mathematical reference. The definition you mention is absent from the leading 20 texts in functional analysis, for example. $\endgroup$ – uniquesolution Dec 10 '17 at 21:26

If the underlying measure space $X$ has infinite measure then consider $f_n(x) := 1/n \mathbb 1_{A_n}$ where $A_n$ is a set with measure tending to infinity faster than $n$. $f_n \to 0$ in $L^\infty(X)$ but there is no hope for any $L^p$ convergence, $p\in[1,\infty)$.

If the underlying measure space $X$ has finite measure then note that $$ \| f \|_{L^p(X)}^p = \int_X |f|^p ≤ \|f\|^p_{L^\infty(X)} |X| $$ so convergence in $L^\infty$ implies convergence in any $L^p$, $p\in [1,\infty]$.

Its true that for functions $f\in \bigcap_{p\in[1,\infty]}L^p$, $\|f\|_{L^\infty} = \lim_{p\to\infty} \|f\|_{L^p}$ but this is a restrictive assumption on $f$. For example, without extra information, its possible that none of the $L^p$ spaces are contained in another.

  • $\begingroup$ what Im still trying to digest is if the proof for sup norm in the paper I linked does indeed imply it holds for every other $L^p$ norm specially $L^2$. $\endgroup$ – Pinocchio Dec 11 '17 at 2:25
  • 1
    $\begingroup$ @Pinocchio It does, the space considered there is $I^n=[-1,1]^n$ which is finite measure. By the way, in more standard PDE references, the space this paper calls $W^n_m$ would be called $W^{m,\infty}(I^n)$. $\endgroup$ – Calvin Khor Dec 11 '17 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.