There is a general method to determine whether two matrices are similar, which involves computing and comparing the Rational Canonical Form of the two matrices; however this is rather a lot of work (and in addition does not play well with the presence of undetermined entries like $x$) while one can do a lot better in small size examples like this one.
You calculated the characteristic polynomials and found both to be $(X-1)^2(X-2)$, independently of$~x$. The eigenvalue $\lambda=2$ will necessarily given an eigenspace of dimension$~1$ and causes no concern. For the eigenvalue $\lambda=1$ with algebraic multiplicity$~2$ you will have a generalised eigenspace $\ker((M-I)^2)$ that has dimension$~2$ for both matrices$~M$, but two possibilities exist for the (ordinary) eigenspace $\ker(M-I)$: either it is a subspace of dimension$~1$ of the generalised eigenspace, or (much less likely) it has dimension$~2$ and fills the generalised eigenspace. You can check that the latter happens for $M=A$ (the vectors $(0,1,0)$ and $(1,0,2)$ generate the eigenspace, but not for$~B$ (the vector $(1,0,1)$ spans the kernel) except when $x=0$, in which case (for instance) $(0,1,0)$ which always belongs to the generalised eigenspace actually belong to$~\ker(B-I)$.
In conclusion, for $x\neq0$ one has $A$ diagonalisable but $B$ not, so they are not similar. But for $x=0$ both matrices are diagonalisable, and also having the same characteristic polynomial, they are clearly similar.