Since $X$ is a metric space, the compact-open topology on $C(X,X)$ coincides with the topology of compact convergence. With the latter viewpoint, it is in some respects easier to treat.
We choose a normal exhaustion of $X$ by compact sets, say $K_n = \left[3^{-n},1-3^{-n}\right]$ for $n \geqslant 1$, and since $X$ is homeomorphic to $\mathbb{R}$, we see that $C(X,X)$ is homeomorphic to a (closed) subspace of
$$\prod_{n=1}^\infty C(K_n,\mathbb{R}).$$
By Weierstraß' approximation theorem, the space of polynomials is a dense subspace of $C(K_n,\mathbb{R})$, and it is easy to see that the space of polynomials with rational coefficients is then also dense. Hence $C(K_n,\mathbb{R})$ is separable, therefore also the countable product of such spaces, and since it is metrisable, also its subspace (corresponding to) $C(X,X)$. Since $\mathbb{R}$ is complete in the usual metric, we now have the
Proposition: $C(X,X)$ is a Polish space.
So it remains to see whether $\operatorname{Homeo}(X)$ is a $G_\delta$-set in $C(X,X)$, or equivalently some Polish subspace of that, since by Mazurkiewicz' theorem, a subspace of a Polish space is Polish if and only if it is a $G_\delta$.
Lemma: $\overline{\operatorname{Homeo}(X)} = \operatorname{Mon}(X)$, where $\operatorname{Mon}(X)$ is the set of monotonic functions in $C(X,X)$.
Proof: The pointwise limit of a sequence of monotonically nondecreasing functions is monotonically nondecreasing, and similarly for monotonically nonincreasing functions. Hence $\operatorname{Mon}(X)$ is closed. To see that the set of homeomorphism is dense in the monotonic functions, let $f \in \operatorname{Mon}(X)$, without loss of generality let $f$ be nondecreasing. Choose a compact $K \subset X$ - without loss of generality $K$ is an interval - and $\varepsilon > 0$. Since $K$ is compact, there is an $\eta > 0$ such that $f(K) \subset [2\eta,1-2\eta]$. Choose $\eta < \varepsilon$. On the interval $K$, define $g(x) = f(x) + \eta\cdot x$. Then $g(K) \subset [2\eta,1-\eta]$, and $g$ is strictly increasing on $K$. Extend $g$ to a homeomorphism of $X$ by linear interpolation on $(0,\min K)$ and $(\max K,1)$. On $K$, we have $\lvert g(x) - f(x)\rvert = \eta\cdot x < \eta < \varepsilon$, hence $g\in U(f;K,\varepsilon)$, and the set of homeomorphisms is seen to be dense in $\operatorname{Mon}(X)$.
There are two ways in which a function in $\operatorname{Mon}(X)$ can fail to be a homeomorphism, it can be not injective, hence constant on some subinterval of $X$, or it can be not surjective. The first failure mode is easy to deal with:
Lemma: $\operatorname{SMon}(X)$, the set of strictly monotonic functions in $C(X,X)$ is a $G_\delta$ in $\operatorname{Mon}(X)$ (and hence also a $G_\delta$ in $C(X,X)$).
Proof: For any $0 < a < b < 1$, the set $Co(a,b) := \{ f\in C(X,X) : f\lvert_{[a,b]} \equiv \operatorname{const}\}$ is closed. Since there are only countably many pairs of rational numbers, and every non-injective monotonic function is constant on some non-degenerate interval with rational endpoints,
$$\operatorname{SMon}(X) = \bigcap_{\substack{0 < a < b < 1 \\ a,b \in\mathbb{Q}}} \left(\operatorname{Mon}(X)\setminus Co(a,b)\right)$$
is a $G_\delta$ in $\operatorname{Mon}(X)$.
Now it remains to see that $\operatorname{Homeo}(X)$ is a $G_\delta$ in $\operatorname{SMon}(X)$. A strictly monotonic $f \in C(X,X)$ is a homeomorphism if and only if it is surjective, that is if and only if $\inf f(X) = 0$ and $\sup f(X) = 1$.
For any $a \in X$, the sets $A(a) = \{ f\in C(X,X) : \sup f(X) \leqslant a\}$ and $B(a) = \{ f \in C(X,X) : \inf f(X) \geqslant a\}$ are closed. Hence
$$\operatorname{Homeo}(X) = \bigcap_{a \in X\cap\mathbb{Q}} \left(\operatorname{SMon}(X) \setminus \left(A(a) \cup B(a)\right)\right)$$
is a $G_\delta$ in $\operatorname{SMon}(X)$.
Wrapping it up, we have proved the
Theorem: $\operatorname{Homeo}(X)$ is a Polish space in the compact-open topology.
(There's probably a more elegant proof, but this is what I came up with.)