I would like to compliment the answers already posted, especially fgp's excellent answer.
The reason we don't want to add quantities like meters and feet is indeed scale invariance. But this isn't the reason why we don't add meters and seconds--we aren't worried about issues of scaling, it just doesn't make any sense to add the underlying physical quantities. Yet it makes perfect sense to multiply any two quantities. Why? It turns out there is some fairly sophisticated mathematics going on here.
Generally, physical quantities are elements of an abelian group, or if they are continuous a vector space. This means we can add and subtract values of the same quantity, and if they are continuous we can scale them. To go beyond this, we need to construct new vector spaces. It turns out that there are natural constructions which allow us to multiply elements (even in different spaces!), and in some cases to invert elements, but not to add elements in different spaces.
Multiplying elements: Given two vector spaces $V$ and $W$, we can form the tensor product $V\otimes W$ which consists of linear combinations of elements of the form $v\otimes w$ with $v\in V,w\in W$, and obeys the following rules:
$$v\otimes w+v'\otimes w=(v+v')\otimes w$$
$$v\otimes w + v\otimes w'=v\otimes (w+w')$$
$$\lambda(v\otimes w)=(\lambda v)\otimes w=v\otimes (\lambda w)$$
The product of $v$ and $w$ is then just $v\otimes w$. One important thing to note is that if $\dim V=1$ or $\dim W=1$, then all elements of $V\otimes W$ are of the form $v\otimes w$ (which is called an elementary tensor), but if either space has dimension greater than $1$ there are elements of $V\otimes W$ which are not elementary.
Inverting elements: Given a vector space $V$, we can form the dual space $V^*$ consisting of all linear maps $f:V\to \mathbb R$. If $\dim V=1$, for any $v\ne 0$ in $V$ there is a unique linear map $v^{-1}:V\to\mathbb R$ such that $v^{-1}(v)=1$, which we call the inverse of $v$. This behaves much the way we'd expect of an inverse: we have a natural map $V\otimes V^*\to \mathbb R$ defined by $v\otimes f\mapsto f(v)$, and so $v\otimes v^{-1}\mapsto 1$. If $\dim V>1$, we need more structure in order to define an inverse. If $V$ is an inner product space, we can take an orthonormal basis $e_1,\ldots,e_n$ of $V$ and define the inverse of $v=a_1e_1+\cdots+a_ne_n$ to be the function $v^{-1}$ such that
$$v^{-1}(e_i)=\frac{a_i}{\|v\|^2}=\frac{a_i}{a_1^2+\cdots+a_n^2}$$
which satisfies $v^{-1}(v)=1$ and turns out to be independent of our choice of orthonormal basis.
Let's bring these together in an example. Let $V$ be the space of displacements, which has units of $(m_x,m_y,m_z)$ where $m_x$ is meters in the $x$-direction, etc. Let $W$ be the space of times, which has units $s$. Then $W^*$ has units $s^{-1}$, and $V\otimes W^*$ has units $(m_x\cdot s^{-1},m_y\cdot s^{-1}, m_z\cdot s^{-1})$, so $V\otimes W^*$ is the space of velocities. If a displacement $d$ occurs over a time $t$, we then get a velocity $d\otimes t^{-1}$. Conversely, given a velocity $v$ and a time $t$, $v\otimes t$ is an element of $V\otimes W^*\otimes W$. The map $W^*\otimes W\to \mathbb R$ gives us a map $V\otimes W^*\otimes W\to V$; this is just cancelling the units $s$ and $s^{-1}$ to get a displacement.
If this formulation seems cumbersome and unnecessary, that's because when one of the quantities we are working with is $1$-dimensional we can ignore most of the details of tensor products and everything becomes much easier. Let's examine what would happen if time were $2$-dimensional. Let $V$ and $W$ be as before. The space of velocities is still $V\otimes W^*$, but this is now $6$-dimensional. Now not every element of $V\otimes W^*$ is elementary, so we can no longer interpret a velocity as a displacement over a period of time! Indeed, two different velocities can have the same displacement over the same (nonzero) time. Things get even worse when we consider acceleration (which is $12$-dimensional) or even higher derivatives. In these cases, the algebra of tensor products is really necessary.
