Checking convexity

I know that the function $(\mathbf{a}-\mathbf{b})'(\mathbf{a}-\mathbf{b})$ is convex in $\mathbf{a}$ ($\mathbf{a}$ and $\mathbf{b}$ are vectors, not scalars). Would $(\mathbf{a}-\mathbf{b})'(\mathbf{a}-\mathbf{b})\mathbf{a}'$ which is a cubic function be convex too? Is there a simple way to check this?

Edit (after martini's comment): the function is $(\mathbf{a}-\mathbf{b})'\mathbf{a}'S \mathbf{b}(\mathbf{a}-\mathbf{b})$ where the term in the middle $\mathbf{a}'S\mathbf{b}$ is a scalar (S is a square matrix and constant like $\mathbf{b}$).

-
This function is vector-valued, what do you mean by convexity in this case? – martini May 18 '12 at 12:13
... and it doesn't hold in 1d: $a\mapsto (a-b)^2$ is convex on $\mathbb R$, but $a \mapsto (a-b)^2a$ isn't. – martini May 18 '12 at 12:42
I guess that $(a-b)'(a-b)$ means $(a-b)^T(a-b)$, which is the same thing as $\|a-b\|^2=\langle a-b,a-b \rangle$, where $\langle\cdot,\cdot\rangle$ is the standard scalar product. – Martin Sleziak May 18 '12 at 12:49
@martini: Sorry, the function was actually (a-b)'a'Sb(a-b) where the term in the middle a'Sb is a scalar (S is a square matrix and constant like b). Would this function be convex? – Nick May 18 '12 at 13:31
Not in general. If I didn't a computation error, if e. g. $b^TSb < 0$, $b \ne 0$, then $f''(b/2)[b,b] < 0$. – martini May 18 '12 at 13:47

No. It is still false in one dimension (if it is true for vectors, it has to be true for scalars too).

Let $S = [1]$ the $1\times 1$ positive definite square matrix. Let $b = [1]$ the $1\times 1$ vector. Then your function reduces to

$$a\mapsto (a-1)^2 a = a^3 - 2a^2 + a$$

Its second derivative is

$$6a - 4$$

which is negative for all $a < 2/3$ and so the function is certainly not convex.

-