This question (and much more) is settled by the Gelfand–Mazur theorem for normed algebras over the ground field $\mathbb{F} \in \{\mathbb{R},\mathbb{C}\}$. For our purposes, let us say that an algebra over a field $\mathbb{F}$ is a vector space over $\mathbb{F}$ which is at the same time a ring (with unit).¹ Furthermore, a normed algebra is an algebra over $\mathbb{F} \in \{\mathbb{R},\mathbb{C}\}$ equipped with a vector space norm which is submultiplicative ($\lVert ab\rVert \leq \lVert a\rVert\lVert b\rVert$). Finally, a division algebra is a non-zero algebra with unit in which every non-zero element is invertible.
(¹: Normed algebras are not usually assumed to have a unit in the literature, but in the context of the Gelfand–Mazur theorem we will need a unit anyway.)
We use the following version of the Gelfand–Mazur theorem:
Theorem (Gelfand–Mazur). Let $A$ be a real normed division algebra. Then $A$ is isomorphic with $\mathbb{R}$, $\mathbb{C}$, or the algebra of quaternions $\mathbb{H}$.
For a proof, see theorem 14.7 in [F.F. Bonsall, J. Duncan, Complete Normed Algebras, Springer–Verlag, Berlin Heidelberg New York 1973].
In order to apply the above theorem to the question at hand, note that we can extend any multiplicative norm $N$ on $\mathbb{R}[X]$ to a multiplicative vector space norm $N' : \mathbb{R}(X) \to \mathbb{R}_{\geq 0}$, where $\mathbb{R}(X)$ denotes the field of rational functions over $\mathbb{R}$. This is done by defining
$$ N'\left(\frac{f(X)}{g(X)}\right) \: := \: \frac{N(f(X))}{N(g(X))}. $$
It is not very hard to prove that $N'$ is well defined, multiplicative, and a vector space norm (use that $N$ is multiplicative). But now $\mathbb{R}(X)$ would be an infinite-dimensional normed division algebra, which is impossible by the Gelfand–Mazur theorem. We conclude that there is no multiplicative norm on $\mathbb{R}[X]$.
More generally, a slight modification of this argument shows that there does not exist a multiplicative norm on any non-zero $\mathbb{R}$-algebra $A$ satisfying both of the following properties:
- $A$ is a commutative ring with unit;
- $A$ is not isomorphic with $\mathbb{R}$ or $\mathbb{C}$.
Proof. If $A$ has non-trivial zero divisors ($a,b\neq 0$, $ab = 0$), then there is no hope of defining a multiplicative norm on $A$, for any norm on $A$ satisfies $\lVert ab\rVert = 0$ but $\lVert a\rVert \lVert b\rVert \neq 0$. So assume that $A$ is an integral domain. We distinguish two cases:
- If $A$ is finite-dimensional, then we know from abstract algebra that $A$ is a field. But then $A$ is a finite field extension of $\mathbb{R}$, so it must be isomorphic with $\mathbb{R}$ or $\mathbb{C}$, contrary to our assumption.
- If $A$ is infinte-dimensional, then we use the argument from before. Any multiplicative norm on $A$ extends to a multiplicative norm on $\text{Frac}(A)$, the field of fractions of $A$. Furthermore, the natural map $A \to \text{Frac}(A)$, $a \mapsto \frac{a}{1}$ is injective and $\mathbb{R}$-linear, so we find that $\text{Frac}(A)$ is infinite-dimensional as well. Again it follows from the Gelfand–Mazur theorem that there does not exist a multiplicative norm on $\text{Frac}(A)$, so we conclude that does not exist a multiplicative norm on $A$ either. $\quad\Box$
The result might even extend to the non-commutative case (with the added assumption that $A$ is not isomorphic with $\mathbb{H}$), since the Gelfand–Mazur theorem does not assume commutativity. However, I'm not too well acquainted with “division rings of fractions” in the non-commutative setting, and I hear that their theory features more than a few subtleties.
Despite these more general results, I can't help but feel that there must be a more elementary proof for the special case $A = \mathbb{R}[X]$ from the original question...