Is there actually a difference when we say 'perpendicular' vs 'tangent' I find myself often getting these two words mixed up a lot. 
So let's say I have a simple graph of $y = t^2$ and $x = t$. If a line is tangent to the curve at the origin, it would only be the line y = 0. If a line is perpendicular to the curve (let's think this in $\mathbb{R}^3$ now), it would be coming out to you assuming if you are looking at it on top of the xy-plane. Is this true or can we actually say that the tangent line is also y = 0?
I know this is actually a really simple question but I couldn't find a good place to distinguish the two words. It's just one of those times you pick up and kinda 'know' by heart when you've finished with calculus.
 A: There is a very big difference, in fact I would be tempted to call the two concepts opposites. Two lines in Euclidean space are perpendicular if they intersect at right angles. This is extended to parametric curves by calling them perpendicular at a point if their tangent lines are perpendicular at the point of intersection. Note that one or both of these parametric curves may be a line. Two lines in Euclidean space are tangent if they are the same line (and hence tangent lines are never perpendicular). This is extended to parametric curves by saying two parametric curves are tangent at a point if their tangent lines are the same at that point. Again, one or both of these curves may be a line.
A: In your example, the $x$-axis is tangent to the curve, the $y$-axis is what is called normal to the curve, and the line you describe coming straight up out of the page is what is called binormal to the curve.  The normal line lies in the same plane as the curve, while the binormal direction is the cross product of the tangent and normal directions.  Both the normal and binormal lines to a curve are perpendicular to the tangent line as lines, and that motivates why those lines are both called perpendicular to the curve.
If $\vec{r}:\mathbb{R}\to\mathbb{R}^3$, the unit tangent vector $\vec{T}(t)$ is defined as $\frac{\vec{r}'(t)}{\|\vec{r}'(t)\|}$.  It is one unit long and points in the same direction as the curve "is moving".
$\vec{T}(t)$ itself changes as $t$ changes.  In your example, as $t$ moves through $0$, $\vec{T}(t)$ points downward-right, then to the right, then upward-right.  Anyway, that means we can consider $\frac{d}{dt}\vec{T}(t)$, which I'll just call $\vec{T}'(t)$ for short.  The unit normal vector $\vec{N}(t)$ is defined as $\frac{\vec{T}'(t)}{\|\vec{T}'(t)\|}$.  It is one unit long, and points in the direction that $\vec{T}(t)$ is changing, which is the direction that the curve "is bending".  In your example, that's straight up.
Locally, $\vec{T}(t)$ and $\vec{N}(t)$ define a plane that contains the second degree Taylor approximation of the curve.  This plane is called the osculating plane.
In $\mathbb{R}^3$, there are $3$ dimensions.  Locally, $\vec{T}(t)$ and $\vec{N}(t)$ account for two orthogonal directions.  A third orthogonal direction can be written down as $\vec{T}(t)\times\vec{N}(t)$, which we call the unit binormal vector $\vec{B}(t)$.  It is automatically one unit long and orthogonal to $\vec{T}(t)$ and $\vec{N}(t)$.
Lastly, in your example, $\vec{B}(t)$ is constant, pointing always straight up.  Imagine a curve that took full use of all $3$ dimensions and writhes around more.  $\vec{B}(t)$ would change as $t$ changes.  In fact the more writhing there is, the more $\vec{B}(t)$ would change.  This motivates why $\|\frac{d}{dt}\vec{B}(t)\|$ is called the torsion of the curve, denoted $\tau(t)$.
That's space curve vector calculus in a nutshell.
