What is the definition of differentiability? Some places define it as:

If the Left hand derivative and the Right hand derivative at a point are equal then the function is said to be differentiable at that point.

Others define it based on the condition of the existence of a unique tangent at that point.
Which one of these is correct? Or are both of them wrong? 
According to the first definition, the curve need not be continuous at that point and can have a point discontinuity or a hole, like this:

Moreover it doesn't stop a curve, with a jump discontinuity but with same slope on both sides of it, from being differentiable.
And lastly, if a function is not defined at a point, then is the function discontinuous there too?
 A: First of all, there is no such thing a correct definition. When we pick a definition we just want to be sure that it satisfies some properties that we like. In this way, definitions can be good, but not correct. For this discussion, it suffices to discuss functions $f:\mathbb{R}\to\mathbb{R}$. We want differentiable to have the following properties as a bare minimum.


*

*$f$ is differentiable implies $f$ is continuous.

*The converse is false, as we don't want "kinks" in our graph.


So, based on what you've said, it does seem like a bad definition of the derivative.
However, I think the mistake you have made is to confuse the "left hand derivative" and the "limit of the derivative from the left." So let's just clear that up right now. The "left hand derivative" means
$$
\lim_{h\to 0^-} \frac{f(x+h)-f(x)}{h}
$$
I believe you looked at 
$$
\lim_{h\to 0^-} f'(x+h)
$$
which would be called the "limit of the derivative from the left". These are quite different, as you've discovered. Clearly the "limit of the derivative from the left" is not a good definition of derivative.
However, it turns out that the difference quotient makes for a decent definition. It's easy to show from this that $f$ will be continuous at $x$. So this makes us happy. Furthermore, $f$ won't have a kink at $x$. Otherwise, the left and right limit would disagree. So this makes us happy as well.
Then there's the tangent definition. This one is slightly more troublesome, as you'd have to define tangent without mentioning derivative. This is possible, but what does it buy you? What most people do is define tangent in terms of derivatives! 
To address your last question, if a function is not defined at a point, how could it be anything at that point? Continuity is not defined outside the domain of a function.
A: A more general definition of differentiability is: 
$$\text{Function $f:\mathbb R\to \mathbb R$ is said to be differentiable if $\exists\,a\in \mathbb R$ such that $\lim_{h\to 0}\frac{f(x+h)-f(x)-ah}{\lvert h\rvert}=0\;$}$$
It can be shown that this definition is equivalent to the conventional definition that relies on the existence of $\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$ (in fact, $a$ is just $\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$ here!).

This definition can be extended naturally to multivariate function. We can define a multivariate function $g:\mathbb R^n\to\mathbb R$ as differential if:
$$\text{$\exists\,\mathbf A(\mathbf X)\in\mathbb R^n$ such that $\lim_{\|\mathbf H\|\to 0}\frac{g(\mathbf X+\mathbf H)-g(\mathbf X)-\mathbf A(\mathbf X)\cdot\mathbf H}{\|\mathbf H\|}=0$}$$
Likely, $\mathbf A(\mathbf X)$ can be proved to be exactly $(\nabla g)(\mathbf X)$, i.e., if $g$ is differentiable at $\mathbf X$, then the first order partial derivatives exist at $\mathbf X$ under above definition.
A: A function is differentiable (has a derivative) at point x if the following limit exists: 
$$
\lim_{h\to 0} \frac{f(x+h)-f(x)}{h}
$$
The first definition is equivalent to this one (because for this limit to exist, the two limits from left and right should exist and should be equal). But I would say stick to this definition for now as it's simpler for beginners. The second definition is not rigorous, it is quite sloppy to say the least.  
Also, there's a theorem stating that: if a function is differentiable at a point x, then it's also continuous at the point x.  
