Not a rigorous proof at all, but you could take the $xe$th root of both sides, and then we have
$$
e^{1/e} \geq x^{1/x}
$$
That $x^{1/x}$ attains a maximum at $x = e$ can be shown fairly straightforwardly.
ETA2: A picture is worth—well, a lot of words, if not quite a thousand:

ETA: OK, a discussion of why $x^{1/x}$ attains a maximum at $x = e$. To obtain the actual value requires calculus, but we can get some intuition for the fact that it attains a maximum somewhere near $e$, as follows.
Let $f(x) = x^{1/x} = \sqrt[x]{x}$. For $x = 1$, we clearly have $f(x) = 1$. For $x = 2$, we have $f(x) = \sqrt{2} > 1$. So far, it's going up.
Now, for $x = 4$, we have $f(x) = \sqrt[4]{4} = \sqrt{2} = f(2)$. And for $x$ really large—let's say, $x = 256$, we have $f(x) = \sqrt[256]{256} = \sqrt[128]{16} = \sqrt[64]{4} = \sqrt[32]{2}$. We may not know exactly what the $32$nd root of $2$ is, but it's clearly quite a bit less than the square root of $2$. (It's about $1.02190$.) However, no matter how large $x$ is, $f(x)$ is always the $x$th root of a number greater than $1$, so $f(x)$ must be greater than $1$.
The general shape we have, then, is a function that (over the range from $1$ to $\infty$) starts at $1$, rises to a peak at some point between $2$ and $4$, and then falls off toward $1$ again (though never arriving at it). We can even find a value of $x$ in the interval $(2, 4)$ for which $f(x) > \sqrt{2}$. That is,
$$
f(\sqrt{8}) = \sqrt{8}^{1/\sqrt{8}}
$$
How does this compare to $f(2) = f(4) = \sqrt{2}$? At first glance, it is difficult to say. However, if we cube $f(\sqrt{8})$, we get
$$
[f(\sqrt{8})]^3 = \sqrt{8}^{3/\sqrt{8}}
$$
Note that that exponent is greater than $1$ (since $3 = \sqrt{9} > \sqrt{8}$), so
$$
[f(\sqrt{8})]^3 = \sqrt{8}^{3/\sqrt{8}} > \sqrt{8} = [f(2)]^3 = [f(4)]^3
$$
so (again, not rigorously)
$$
f(\sqrt{8}) > \sqrt{2} = f(2) = f(4)
$$
The actual value is about $1.44426$.
As I said previously, proving that $x^{1/x}$ attains a maximum specifically at $x = e$ requires calculus, but it seems that may not be exactly what you want? Hoping the above serves as some kind of intuitive basis.
\begin{align}
\frac{d}{dx} x^{1/x} & = \frac{d}{dx} e^{\frac{\ln x}{x}} \\
& = \left( \frac{d}{dx} \frac{\ln x}{x} \right)
e^{\frac{\ln x}{x}} \\
& = \frac{1-\ln x}{x^2} e^{\frac{\ln x}{x}} \\
& = \frac{1-\ln x}{x^2} x^{1/x}
\end{align}
and $f(x)$ attains a maximum when that derivative is equal to $0$, which happens when $1 = \ln x$, which happens when $x = e$. (There might be a bit of circularity there—I have to give that some thought—but that's how you obtain the maximum.)