In Kaplansky's Set Theory And Metric Spaces, he mentions a useful example of a neighborhood of $x$ is a closed ball with center $x$. However, one of the theorems in baby Rudin is "Every neighborhood is an open set". I'm confused?
You’re seeing two different definitions of neighborhood. Kaplansky is using the more inclusive definition: $N$ is a nbhd of $x$ if $x$ is in the interior of $N$, or, equivalently, if there is an open set $U$ such that $x\in U\subseteq N$. Rudin is using the narrower definition: $N$ is a nbhd of $x$ if $N$ is an open set containing $x$. Kaplansky would call Rudin’s nbhds open neighborhoods.
Neighborhoods are mathematical objects more general than Rudin's definition. A ball is a type of neighborhood (I think more specifically, a ball is a type of neighborhood in a metric space). So you can think of a ball as a special type of neighborhood. Similar as to how a metric space is a special type of topological space.