Find the directional derivative of $x^2-xy-2y^2$ at $(1,2)$ along the vector that goes from this point to $(3,4)$ I used this formula(in blue, you have to scroll down a bit), and did this:
$$\lim_{h \rightarrow 0} \frac{f(1+3h,2+4h)-f(1,2)}{h} = \\
\lim_{h\rightarrow 0}\frac{(1+3h)^2-(1+3h)(2+4h)-2(2+4h)^2+11}{h} = \\
\lim_{h\rightarrow 0} \frac{2-36h-35h^2}{h} = \lim_{h \rightarrow 0 } 2/h - 36 - 35h = \infty$$
What went wrong?
 A: One mistake you made is that $f(1,2)$ should be $-9$, not $-11$, so $2$ in the last line should be $0$.
A: The directional derivative of $f(x,y)$ in the direction of $\mathbf v$ at the point $P$ is $$\nabla f(P) \bullet \frac{\mathbf v}{||\mathbf v||}$$ $$=([2x-y,-x-4y] \text { at $P$=(1,2)}) \bullet \frac{[3-1,4-2]}{||[3-1,4-2]||}$$
$$=[0,-9]\bullet \frac{[2,2]}{||[2,2]||}$$
$$=[0,-9]\bullet \frac{[1,1]}{\sqrt 2}$$
$$=\frac{-9}{\sqrt 2}=-\frac{9\sqrt2}{2}.$$
A: There are some algebraic and arithmetic errors in your work, but you made a more fundamental error from the get-go: $(3,4)$ isn’t the direction in which you’re supposed to compute the derivative. It’s another point that you’re supposed to use to compute the direction. So, even had you done the calculation correctly, you wouldn’t have gotten the right answer since you’re taking the derivative in the wrong direction.  
Your next error in setting up the limit was that you didn’t use a unit vector: as noted in one of the comments to your question and as it says right at the top of the blue box that contains the formula that you’re using, $(a,b)$ must be a unit vector. So again, had you computed the limit correctly, you would’ve gotten the wrong value anyway.  
So, first you need to compute the direction vector to use. That’s $(3,4)-(1,2) = (2,2)$. You then need to normalize this to get a unit vector: ${(2,2)\over\lVert(2,2)\rVert} = {(2,2)\over2\sqrt2} = \frac1{\sqrt2}(1,1)$. The limit that you should be computing is therefore $$\lim_{h\to0}{f\left(1+h/\sqrt2,2+h/\sqrt2\right)-f(1,2)\over h}.$$ Try that and be careful with your algebra and arithmetic.  
Then again, since $f$ is differentiable at $(1,2)$, you could compute the dot product of the gradient of $f$ at this point and the unit direction vector. You should do that anyway to check your answer using the limit definition.
