Understand closure and limit points I study Rudin's book on analysis. I want to check with you that I understand the definition of closure. Therefore I must check with you that I understand the definition of limit points. 


*

*A neighborhood is for example an interval, a circle or a sphere around a point or a set. 

*A limit point is any point that "can be approached as a limit" in contrast to an isolated point for example the set $\{\{1\}, [2,3]\}$ i.e. the union of the point $1$ and the interval between $2$ and $3$ has an isolated point $1$ which is not a limit point but any points between $2$ and $3$ are limit points. 
Do you agree?
What I don't understand about closure is: How can there be limit points of a set $E$ which are outside of $E$? I think that the example is an open set but I don't get the picture. I think that the closure by definition always is a closed set and therefore the union of an open set and its limit points is not closed, but I still don't understand how a set can have a limit point not in the set. 
I must have misunderstood.
 A: Here's an important point that some of the other answers aren't putting clearly, and I think it's worth stating explicitly.
The phrase "the limit points of the set $A$" doesn't make sense on its own. People will say it, but it's shorthand for something more precise: as it stands, this phrase is missing a crucial piece of information. A correct way of putting this might be "the limit points of the subset $A$ of the topological space $X$", or "the limit points of the set $A$ as a subspace of $X$", or similar.
Limit points are always relative to an ambient space. For example, you can ask the question "what are the limit points of the set $(0,1)$ inside the space $X$?" for each of the following four spaces $X$, and you'll get four different answers:


*

*$X = (0,1)$

*$X = \mathbb{R}_{>0}$

*$X = \mathbb{R}$ (with the usual topology)

*$X = \mathbb{R}$, but now with the Zariski topology

A: 
A neighborhood is for example an interval, a circle or a sphere around a point or a set. 

Intuitively you have the right idea but you really must become more comfortable with a much more abstract concept.  
An neighborhood is a collection of points within a certain distance of a point.  The reason circles and spheres are so commonly used is that using the the Euclidean metric for distance, is because  all points within a give distance make the shape of a circle (or sphere, or interval).  But if we used a different method of distance we'd get different sets. For example if we used the "taxi cab" metric where distance is not measured by a straight line but by adding the vertical offset to the horizontal offset and neighborhood (all the points where the vertical offset plus the horizontal offset or within a certain value) would not be a circle; it would be a diamond.  We can even have a "distance" where every two different points are always $1$ distance apart.  Thus a neighborhood could be either just the point itself (that's the only point within $\frac 12$) or every single point in the universe (that's all the points withing $2$).

A limit point is any point that "can be approached as a limit" in contrast to an isolated point for example the set {{1}, [2,3]} i.e. the union of the point 1 and the interval between 2 and 3 has an isolated point 1 which is not a limit point but any points between 2 and 3 are limit points. 

Yes, you are missing the point here.  Points in and of themselves are not limit points.  The point $1$ isn't a limit point by itself. It must be (or not be) a limit point of a set.  $1$ is a limit point of $(0,1)$; it is a limit point of $[-3,7]$.  It is a limit point of $\mathbb Q$.  It is a limit point of the irrational numbers.   It is not a limit point of $\mathbb Z$. It is not a limit point of $(2,3)$.  It is a limit point of $\{ 1-\frac 1n| n\in \mathbb N\}$.
If you want to think intuitively of what a limit point is, a limit point of a set $E$ is a point, $x$, so that there always points of $E$ that are "close" to $x$.
$1$ is a limit point of $(0,1)$.  Even though $1 \not \in (0,1)$ we can always find points in $(0,1)$ that are close to $1$.  If we want to find a number that is within hundred-billionth ($0.00000000001$) of $1$ we'd pick $0.9999999999$.  If we need something within $\epsilon$ where $\epsilon$ were as tiny as we wanted it, we'd pick... some $x$ so that $1-\epsilon < x < 1$.
$1$ is a limit point of $[-3,7]$ for pretty much the same reason.  $1$ is smack dab in the middle of $[-3,7]$ and we can find $x\in [-3,7]$ as close as we want.
It is a limit point of $\mathbb Q$.  For any $\epsilon > 0$ we can find rational numbers $q$ so that $1-\epsilon < q < 1 + \epsilon$.
It is a limit point of the irrationals, as well.  For any $\epsilon > 0$ we can find irrational numbers $z$ so that $1-\epsilon < z < 1+\epsilon$.
$1$ is not a limit point of $\mathbb Z$. We can not find ANY integers (other than $1$ itself) within $\frac 14$ of $1$.  Take the neighborhood of all real numbers within $\frac 14$ of $1$ and there are not any integers at all (other than $1$.)
$1$ is not a limit point of $(2,3)$.  We can not find any elements of $(2,3)$ withing $\frac 14$ of $1$.
Notice: $1$ does not have to be in the set $E$ to be a limit point nor does being in the set guarantee that it is a limit point (it won't be if there aren't any other points nearby.)

I think that the example is an open set but I don't get the picture.

$1$ is a limit point of $(0,1)$ because for every neighborhood $(1-\epsilon, 1 + \epsilon)$ there will always be points, $y$ so that $y\in (0,1)$ and $y\in (1-\epsilon, 1+\epsilon)$.

I think that the closure by defintion always is a closed set .

That is true but not by definition.  Rudin proves it in Th. 2.27 a)

therefore the union of an open set and its limit points is not closed,

Oh, but it is!.
Example $(0,1)$ is an open set.  $0$ and $1$ are limit points of $(0,1)$.  Also $(0,1)$ are also limit point.  But other than that there are not any other limit points.
So $(0,1)'=$ the set of limit points of $(0,1)$ is $[0,1]$
So $(0,1)\cup (0,1)' = (0,1) \cup [0,1] = [0,1]$.
And $[0,1]$ is closed.
A: Ultimately, it really depends on the topology in question.
For example, given any set $X,$ the power set of $X$ is a topology on $X,$ and given any subset $A$ of $X,$ we have that $A$ is a neighborhood of all of its elements.
As for your second question, I will say that your understanding of "limit point" makes sense in many cases (metric spaces, for example), but doesn't hold universally. Let $X=\{1\}\cup[2,3],$ as in your example, and define a topology $\mathcal T$ on $X$ by $$\mathcal T=\{\emptyset\}\cup\{A\subseteq X:1\in A\}.$$ Then it turns out that $1$ is the only limit point in $X$! For any other point $x$ of $X,$ the set $X\setminus\{x\}$ is closed. However, $X\setminus\{1\}$ is not closed. Thus, $1$ is a limit point in $X.$ In fact, it is a limit point of every non-empty subset of $X,$ except for $\{1\}.$

Let $A\subseteq X$ such that $A\neq\{1\}$ and $A\neq\emptyset.$ Then there is some $r\in A$ such that $r\neq 1.$ Now, take any neighborhood $U$ of $r.$ Since $U\neq\emptyset,$ then by definition of $\mathcal T,$ we have that $1\in U.$ Since $r\neq 1$ and since $U$ was an arbitrary neighborhood of $r\in A,$ then $1$ is a limit point of $A.$ $\Box$

A: Consider the set $ (0, 1) \subseteq \mathbb R $. Then $ 1 $ is a limit point of this set, but is not an element.
EDIT: A set with its limit points is always closed. One can show that $ \overline{A} = A \cup A' $ where $ A' $ is the set of limit points of $ A $, and the closure of a set is always closed.
A: The closure of a set is indeed closed because it contains all its limit points. That directly follows from the definition of being closed in metric spaces. At least the way Rudin defines it. 
However, for showing you an example where the limit point of a set might not be an element of it, consider $(0,1]$. $0$ is a limit point because $a_n = \frac{1}{n}$ is a sequence that approaches $0$ but it's not in the set. 
A: The term "neighborhood" to many authors (Wikipedia, for instance) is often broader than just an open/closed ball, and this actually confused me for a while in my topology course$^\dagger$.  If we have a metric space $X$ and a point $x \in X$, then a neighborhood $A$ of $x$ is any subset $A \subset X$ such that there exists an open ball of some radius $\delta >0$ that is contained entirely inside of $A$.  
If we take $X$ to be instead a general topological space, then $A$ is a neighborhood of $x$ whenever there is an open set $Y \subset A$ that contains $x$. 
In particular, a neighborhood doesn't need to be open nor does it need to be closed nor does it need to be connected (even inside a connected space).  This loose definition allows for things like $(0,1) \cup (2,3]$ to be a neighborhood of $2.5 \in \mathbb{R}$ despite containing lots of "irrelevant" points.  To make things more precise, you'll often see authors opt for the term open neighborhood of $x$, in which case what's being referred to is an open set containing $x$, but this still allows for things like $(0,1) \cup (2,3)$ to be a neighborhood of $\displaystyle \frac{1}{2} \in \mathbb{R}$.

Forewarning Footnote: 
Point-set topology (which, in a sense, generalizes much of the first couple chapters of Rudin) is notorious for sometimes having terms whose definitions vary slightly from author to author, so it's important to check how your particular author has defined their terms.  An example that immediately springs to mind is "locally compact": when the space isn't Hausdorff, there are a handful of inconsistent definitions that you can read about here.
More examples arise due to the prefix "quasi-" (you can thank Bourbaki for this, IIRC).  Whereas many people use "compact" to describe any space in which an open cover admits a finite subcover, others take it to mean this definition plus Hausdorff.  To such people, a space that fulfills the covering condition but is not necessarily Hausdorff is instead called quasi-compact.  Likewise, you'll sometimes encounter terms like quasi-regular, quasi-normal, and so forth—all referring to regular / normal / what-have-you spaces that are not necessarily Hausdorff.

$^\dagger$For example, one sometimes encounters the term "compact neighborhood" (here, e.g.).  While "neighborhood of a point" is interchangeable with "an open set containing that point" in most contexts, a compact set in a Hausdorff space is necessarily closed and not necessarily open!  For instance, there are no compact open sets in $\mathbb{R}^n$.
