"Vectors aren't really numbers" - how sound is that statement? Since I first learned about vectors, I noticed something interesting: almost any numeric formula can be replaced by a vectorial formula by just replacing addition, multiplication, etc., with their vectorial elementwise versions. For example:
average a b = (a + b) / 2

not only gets the average of 2 numbers, but the mid-point between 2 vectors if you use the elementwise addition. On Calculus, many formulas for integrals can be extended to triple, quadruple integrals unchanged the same way. Even more interestingly, some formulas gain a dimensional generality by doing so.
distance a b = modulus (a - b)

That formula holds for numbers, returning their difference, but also works for any n-dimensional vectors, returning their n-dimensional distance. Even complicated formulas such as:
intersectAABB (Ray r_pos r_dir) (AABB aabb_min aabb_max) 
    = [tmin, tmax] where
        t1   = (aabb_min - r_pos) / r_dir
        t2   = (aabb_max - r_pos) / r_dir
        tmin = foldr1 max (liftI2 min t1 t2)
        tmax = foldr1 min (liftI2 max t1 t2)

get the same benefit. On this case, intersectAABB, used for numbers, returns the intersection distance between a line and a segment. Used with 2d vectors, it returns the intersection distances between a line and a rectangle. With 3d vectors, the intersection distances between a line and a cuboid. And so on.
All that leads me to believe it makes a lot of sense to use vectors just like number. My question is: why is nobody doing so? Why are dot and cross considered the vectorial version of multiplication when they're mostly completely different operations? Is there any case where using vectors in place of numbers stops making sense? 
 A: Generally the word "number" should refer to not a single object, but a set. Then, you might say the set is a set of "numbers" if you can add and multiply any two elements of the set to get a third element of the set, satisfying certain relations (distributivity, associativity,..etc). Note that this definition relies on being able to add two numbers in the set, which is why any single element of the set can't really be called a number without reference to the ambient "number set". As such, a set of numbers thus "defined" is really just a ring (or if it's especially nice, it's a field).
Similarly, the word "vector" shouldn't refer to a single object, but again to a set. Ie., a set is a "vector space" over some field $K$ if it satisfies the axioms of a vector space (see the wikipedia definition of vector space).
Note that a field $K$ itself satisfies the axioms of a vector space, and thus elements of $K$ can be both considered numbers, and vectors. This becomes especially useful in the study of field extensions $L/K$, where now $L$ is a multi-dimensional vector space over $K$, so elements of $L$ are both numbers and vectors.
Anyway, the answer to your question, as I see it, is that the typical definitions/notions of numbers and vectors are simply different. A ring is sometimes a vector space (if contains a field), and otherwise isn't. Similarly, a vector space generally isn't a ring/field, because you can't always multiply vectors, but some vector spaces are. In other words, number vs vector isn't like the difference between apple and orange. Numbers vs vectors are more like "tall people" vs "skinny people". Someone can be tall, skinny, both, or neither.
A: Your question mentions addition and subtraction, and exhibits analogies between addition and subtraction of numbers on the one hand and addition and subtraction of vectors on the other hand.
But numbers, as usually understood, have more operations and relations than just addition and subtraction. For instance, numbers have a binary order relation, and the order obeys laws such as 


*

*$a<b$ and $b<c$ implies $a<c$

*$a<b$ implies $a+c<b+c$ 


Vectors do not, in general, have an order relation like that. So vectors aren't really numbers.
A: One way to see that vectors 'are numbers' is to trace the evolution of the concept - or one might say its genealogy which doesn't pretend to an accurate history but articulates mathematical significance. 
Now starting from the positive integers one can close them under addition to get the integers; and then under multiplication to get the rationals - but this has 'gaps'; thus we complete them by including the irrationals which give us the real numbers.
Now we enter the realm of geometry by observing that this is just the (real) straight line, but there are geometric objects such as planes and volumes from which we discover the notion of dimension; and thus we 'complete' by dimension to get the notion of an n-dimensional vector space.
It's worth noting that the inner, or dot product generalises to arbitrary dimensions in the obvious manner; but the cross-product doesn't, at least not directly; the correct concept that replaces this is the wedge product. 
A: Vectors are not numbers, the differerences are quite more important than similarities.


*

*Vectors do not represent quantities like numbers, but directions. While mathematicians are defining vectors more abstract as any list of numbers (tuples), physicists are more strict: Each component of a vector has the same dimension and you can move (translate and rotate) a vector by applying
rotational and translational matrices (these are two-dimensional objects, you can also see them as vectors of vectors). A number is dimensionless, you cannot use it to model directions or rotations.

*You cannot divide by a vector. What is called multiplication for numbers is scaling for vectors and because we have this direction/rotation part, there are operations called "multiplication" which precisely model this.
(I do not add the four-vector product here).


*

*Scalar product: You have two vectors and you want to know if they show in the same direction. Applying the scalar product gives you a number (!) as result of two vectors and returns a value in the range of -(length of vector) to (length of vector). If it is 0, the two vectors are perpendicular; this is also one case different from numbers. If neither a nor b is zero, then a*b cannot be zero if a and b are numbers; this is not the case with vectors.  It also allows that a matrix can be multiplied with a vector to get a matrix.

*Cross product: You have two vectors and you want to have a vector which is perpendicular to the given two vectors. The length of the resulting vector depends how perpendicular the two vectors are themselves. If they are perpendicular together, then the length will be the product of their lengths, if they are equal, the resulting vector vanishes (logical: There is no possible perpendicularity anymore). 

*Dyadic product: Creates a matrix from two vectors. An explanation what it does would be lengthy. No numbers at all involved, so not interesting.
What is really interesting: If you combine numbers, the result is always not defined or a number itself. Not so with vectors: The result can be a number, a vector or a matrix.
A: As far as addition and subtraction are concerned, vectors do behave just like any notion of number you care to name. But the elementwise multiplication is trickier: the problem of division-by-zero gets complicated to division-by-vectors-with-any-entry-zero, for instance. More to the point, there are less uses for the elementwise multiplication than for the dot and cross product. The latter have physical and geometric meaning that the former lacks. That's the single most important reason we use them more.
A: The reason you notice these similarities is because a lot of these structures are algebraic, Vectors are under addition a group like normal numbers and even a ring with vector-product. One important thing though is that multiplication by a scalar in vector spaces is only belonging to modules while numbers are not modules.
Another example of something that is "like a number" is symmetry. Take an equilateral triangle, and look at the ways those symmetries can be composed. This is associative and has a zero, like numbers, but also has differences between numbers. In general, there are many things like numbers (considered in abstract algebra) but not called numbers.
