I need to find a nonlinear function $f:\mathbb{R}^2\rightarrow\mathbb{R}^2$ such that $f(\alpha (a,b))=\alpha f(a,b)$ for all $(a,b)\in\mathbb{R}^2$ and $\alpha\in\mathbb{R}$.
I can't find anything.
The requirement $f(\alpha (a,b))=\alpha f(a,b)$ says that $f$ respects the scalar multiplication, just as linear maps do. In particular, $f$ is homogeneous of degree $1$. To make it nonlinear, one has to somehow destroy the additive property $f(a+c,b+d)=f(a,b)+f(c,d)$.
The hypothesis that $f(\alpha\vec x)=\alpha f(\vec x)$ is equivalent to requiring that $f$ is linear on each line through the origin. You can map these lines to other lines in nonlinear ways. E.g., you could rotate a vector with polar angle $\theta$ by an angle of $\sin^2\theta$ (or any other nonconstant $\pi$-periodic function of $\theta$):
$$(r\cos(\theta),r\sin(\theta))\mapsto (r\cos(\theta+\sin^2(\theta)),r\sin(\theta+\sin^2(\theta)).$$
You can define f as follows : $f(0,y)=(0,y)$ for any $y$, and $f(x,y)=(y,0)$ for any $y$,whenever $x\neq 0$.
As observed by others, in order for this to happen the function must fail to be additive. Now, one way to think about such problems is to ask the following: given my hypotheses on the function, how much data determines the function?
For example, in a linear function it is a standard observation that specifying $f$'s values at a basis determines the function. Now, given this homogeneity constrain we observe that specifying $f(a,b)$ determines $f$ along the entire ray by $(a,b)$ but nowhere else. So let's choose a value of $f(a,b)$ for every $(a,b)$ on the unit circle and such that $f(-a,-b) = -f(a,b)$. Then by scaling we have a function on the whole plane satisfying your constraint.
Now, final problem: figure out which values on the unit circle give linear functions (or equivalently, what values do we get on the unit circle if we start with a linear function). Choose something that isn't that.
Hint: find a function $g:\mathbb R^2 \to \mathbb R$ that is non linear and satisfies your condition. Then take $$f(a,b) = (g(a,b),0)$$
Try $$g(a,b) = \begin{cases}\frac{a^2}b & b \ne 0\\0&\text{otherwise}\end{cases}$$
