For a probability space $(\Omega, \mathcal{F}, \mathbb{P})$, we say $X: \Omega \rightarrow \mathbb{R}$ is a random variable if $X^{-1}(B) \in \mathcal{F} ~\forall B \in \mathcal{B}$, where $\mathcal{B}$ is the Borel $\sigma$-field. In other words, $X$ is a random variable if $X$ is measurable.
The intuition I've gathered so far is that since we can only talk about measurable subsets of $\Omega$, we want the preimage of any set in the image of X to be measurable. However, my question is, why are we only concerned with preimages of Borel sets, rather than all subsets of $\mathbb{R}$?
(For context, my experience with measure theory is only in the view of probability theory)