Why is the derivative a limit? 
Start by assuming that function curves are made of an infinite amount of lines (i.e. look at the image above but instead of approximating it using a finite number of lines, use infinite lines). This intuition is supported by the arc length formula (integral of infinite "line approximations").
Let's call the point at the top ("the highest point") point $x$. Refer to the $n^{th}$ point before that as $x-n$ and the $n^{th}$ point after that as $x+n$. I want to calculate the derivative at point $x$. In other words I want to calculate the slope of the line connecting point $x$ and $x+1$.
I can start be drawing a secant line from point $x$ to $x+n$ and using the slope to approximate the derivative. As $n$ approaches $1$, you get better approximations of the derivative. As $n=1$ you exactly have the derivative. 
Now draw a secant line from point $x$ to $x-n$. As n approaches 1 you get infinitely better approximations of the slope of the line between $x$ and $x-1$, but not $x$ and $x+1$. No matter how close to n=1 you get you will never get the slope of the line that connects $x$ and $x+1$ (which is what the derivative is: the rate of change of the function between a point and the next infinitesimal point).
So why is the derivative the limit:
$$\lim _{h \rightarrow 0} \frac{f(x+h) - f(x)}{h}$$
instead of:
$$\lim _{h \rightarrow 0^+} \frac{f(x+h) - f(x)}{h}$$
Since using the above intuition shows that the left hand limit doesn't give you the slope of the line between $x$ and the next infinitesimal point.
 A: It is an interesting question. Your understanding of the situation is quite good, you are just missing a small element.
Suppose your function is $$f(x) = \left| x \right  |$$

Using your technique, the limit from the right, you find that the slope at $x=0$ is simply $1$. But what if you use the same technique starting from the left? You get that the slope at $x=0$ is $-1$. This is a contradiction. You find that the slope at a certain point can be two different things, but this is far off the general definition of the tangeant line and its slope!
In fact, in the case of $f(x) = \left| x \right  |$, the derivative at $x=0$ is said to be undefined, and you can easily understand why by taking a look at the graph. But, in some cases, you might only be interested in what is going on with the limit from the right, or maybe with the limit from the left.

Added:
A limit $\lim\limits_{x\rightarrow a} f(a)$ is only defined if $\lim\limits_{x\rightarrow a^+} f(a) = L$ and $\lim\limits_{x\rightarrow a^-} f(a) = L$.
A: As others have noted, your definition means that $f(x) = |x|$ is differentiable at zero. But why is the positive direction any more important than the left? You could define a "right derivative" like this if you wanted to, it's just that the two-sided derivative has nicer properties.
You say "Start by assuming that function curves are made of an infinite amount of lines". This is actually really close to the (in my opinion) best way to think about the derivative: as the linear function that best approximates $f$ at that point. You take a small neighborhood around $x_0$, and as that neighborhood gets smaller and smaller, $y - f(x_0) = f'(x_0) (x - x_0)$ looks more and more like $y = f(x)$.

However, for a function like $f(x) = |x|$, there's no line you can draw at $(0,0)$ that approximates it well. No matter how close you zoom in, it will look like this:

Lastly, your intuition about "infinite line segments" is wrong. You're breaking $f$ into finitely many pieces, looking at the slope between $x_0$ and $x_n$, and taking the limit as $n \to 1$. But this depends a lot on how many pieces you break it into, and where you break them! Plus, since it's finite, you could just cut to the chase and find the slope between $x_0$ and $x_1$.
The limit you should be looking at is "the slope between $x_0$ and $x_1$ as the number of slices increases" (well, more accurately, as $|x_0 - x_1| \to 0$). In many cases, this will be the same whether $x_1$ is on the left or the right. But as in the first paragraph, why is right any more interesting than left?
A: According to your intuition the function 
$$
f(x)=\begin{cases}1,&\ x\geq0\\0,&\ x<0\end{cases}
$$
is differentiable everywhere, while $$
g(x)=\begin{cases}1,&\ x>0\\0,&\ x\leq0\end{cases}
$$
is not. 
In particular, in your intuition a function can be differentiable at a point while failing to be continuous. 
A: Your intuition is indeed flawed. Say we can somehow deal with infinitesimal values. Let $\epsilon$ be such an infinitesimal value, greater than $0$, but smaller than any positive real number. Then the derivative of $f$ at $x$ would be $\frac{f(x+\epsilon)-f(x)}{\epsilon}$ (this is probably what you want to convey with your $f(x+1)$ above), and there would be no need for limits at all.  Now indeed for a differentiable $f$ the value of $\frac{f(x-\epsilon)-f(x)}{-\epsilon}$ would differ from $\frac{f(x+\epsilon)-f(x)}{\epsilon}$ only by an infinitesimal value, so it would not matter which one you took. The size of that infinitesimal difference would of course determine the second derivative.
