Double bars (or sometimes even single bars) tend to denote a norm in Mathematics. Most likely, the double bars here are denoting the Euclidean norm. This is just the length of the vector. So for example, the vector (I shall write it horizontally for compactness) $(1,2,3)$ has length
$$
\|(1,2,3) \|=\sqrt{1^2+2^2+3^2}=\sqrt{14}
$$
and the vector
$$
\|(3,-1,2) \|=\sqrt{3^2+(-1)^2+2^2}=\sqrt{14}
$$
Notice that $A\mathbf{x}$ is just a vector, so $\|A\mathbf{x}\|$ is just the length of the vector. $\|\mathbf{x}\|$ is just the length of $\mathbf{x}$. So here you are looking for scaling of $\mathbf{x}$ under transformation by $A$ to be between $m$ and $M$. (Look at $\frac{\|A\mathbf{x}\|}{\|\mathbf{x}\|}$ and think about what it means 'pictorially' to see what I am talking about).
How should I solve this
What are solving for? what is the known and what is the unknown? $\endgroup$