First of all, there is no such thing a correct definition. When we pick a definition we just want to be sure that it satisfies some properties that we like. In this way, definitions can be good, but not correct. For this discussion, it suffices to discuss functions $f:\mathbb{R}\to\mathbb{R}$. We want differentiable to have the following properties as a bare minimum.
- $f$ is differentiable implies $f$ is continuous.
- The converse is false, as we don't want "kinks" in our graph.
So, based on what you've said, it does seem like a bad definition of the derivative.
However, I think the mistake you have made is to confuse the "left hand derivative" and the "limit of the derivative from the left." So let's just clear that up right now. The "left hand derivative" means
$$
\lim_{h\to 0^-} \frac{f(x+h)-f(x)}{h}
$$
I believe you looked at
$$
\lim_{h\to 0^-} f'(x+h)
$$
which would be called the "limit of the derivative from the left". These are quite different, as you've discovered. Clearly the "limit of the derivative from the left" is not a good definition of derivative.
However, it turns out that the difference quotient makes for a decent definition. It's easy to show from this that $f$ will be continuous at $x$. So this makes us happy. Furthermore, $f$ won't have a kink at $x$. Otherwise, the left and right limit would disagree. So this makes us happy as well.
Then there's the tangent definition. This one is slightly more troublesome, as you'd have to define tangent without mentioning derivative. This is possible, but what does it buy you? What most people do is define tangent in terms of derivatives!
To address your last question, if a function is not defined at a point, how could it be anything at that point? Continuity is not defined outside the domain of a function.