Regarding direction of gradient of a function

I was reading this book related to optimizing a function f(x,y) with constraints g(x,y) = 0. The book says that let us suppose g(x,y) define a surface, then gradient of the g(x,y) will be orthogonal to the surface. But I am a bit confused how is it possible. I mean gradient always points in the direction of maximum increase. It is not necessary for it to be orthogonal.

For eg lets take the function

f(x) = x^2


If I take the gradient of the function at x=1, then its value is 2i with i giving the direction. This direction is not perpendicular to the function or its tangent at the point (1,f(1)).

I am a bit confused. So any clarifications?

-
The function that you give is a function of $1$ variable only. So there is only $1$ way that the function can increase at $x=1$.
Suppose we had, instead, $f(x,y)=x^2-y$. This is the actual graph you are describing: $y=x^2$. In particular, we are interested with the level curve $f(x,y)=0$.
This is now a function of two variables. Now take the gradient $\nabla f=\langle2x,-1\rangle$, and at $(1,1)$, this becomes $\nabla f=\langle2,-1\rangle$. Look where this graph is pointing! It is perpendicular to the tangent line of the level curve $f(x,y)=0$.
Additional remark: if $f(x,y)=x^2$, the gradient $\langle 1,0\rangle$ is orthogonal to the line $x=1$, as claimed. (BTW notice that \ langle \ rangle make better angle brackets than inequality signs.) –  user31373 Jun 24 '12 at 2:09
I have fixed this. Thank you for this $\langle \rangle$ tip. –  Andrew Salmon Jun 24 '12 at 2:12