I was reading a book in which it is mentioned that:
Rotate coordinate axes by $45$ degrees so that a point $(x,y)$ becomes $(x+y,y-x)$ .
I can't understand how the new coordinates became $(x+y,y-x)$ . If I apply the formula for rotation of axes I'll get $\sqrt{2}$ $(x-y, x+y) $.
As noted, that transformation isn't an isometry (in the Euclidean metric); it's a $45^\circ$ rotation composed with a dilation by $\sqrt{2}$.
But then, looking at the pictures, we're not looking at Euclidean geometry - we're instead looking at the distances derived from the 1-norm $\|(x,y)\|_1=|x|+|y|$ and the $\infty$-norm $\|(x,y)\|=\max(|x|,|y|)$. From that perspective, we're just fine leaving that scale factor in so the transformation has rational coefficients. The formula for $T^{-1}$ will have factors of $\frac12$, and that's just not a big deal.
Let $T(x,y)=(x+y,y-x)$. The linked pictures say that $\|T(v)\|_{\infty}=\|v\|_1$. We also get $\|T(v)\|_1=2\|v\|_{\infty}$. Taking two steps, $T^2(x,y)=(2y,-2x)$, which doubles both norms. What this is saying is that the geometry we get from the $1$-norm and the geometry we get from the $\infty$-norm are isomorphic - because this transformation $T$ transforms one distance into the other. Using the irrational version that's an isometry in the Euclidean metric would only make things worse by introducing a factor of $\sqrt{2}$ to the distances we're looking at. Pass.
You are rotating the coordinate axes, not the points. Let $(x,y)$ be a point in the plane. Then $(x,y)=x(1,0)+y(0,1)$. A rotation of the coordinates by $45^{\circ}$ takes $(1,0)$ to $(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})$ and $(0,1)$ to $(\frac{-1}{\sqrt{2}},\frac{1}{\sqrt{2}})$.
To find the new coordinates $(x',y')$ we must solve $$x'(\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})+y'(\frac{-1}{\sqrt{2}},\frac{1}{\sqrt{2}})=(x,y)$$
Grouping terms on the left of the equation and equating coordinates gives $$x'-y'=\sqrt{2}x$$ $$x'+y'=\sqrt{2}y$$
The $\sqrt{2}$ corresponds to the length of the diagonal of a unit square which is $2$ in taxicab geometry. Replacing $\sqrt{2}$ with $2$ in out system of equations gives us $$x'-y'=2x$$ $$x'+y'=2y$$
We can solve this system for $x'$ and $y'$ to obtain $$x'=x+y$$ $$y'=y-x$$.
Edit:
I see many saying that this transformation is a rotation followed by a dilation but this is not accurate. A rotation must preserve length. In this case the length is the taxicab length. Take $(1,0)$. It has a taxicab length of $1$ from the origin. If we rotate this by $45^{\circ}$ under the standard metric we get $(\frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}})$. But the same rotation under the taxicab metric must give $(\frac{1}{2},\frac{1}{2})$ to preserve the length of the rotated vector.
My proof above can be confusing because I don't make that adjustment until after the rotation with respect to the standard metric.
I'd think that the author was more concerned to the underlayed square lattice grid, as being displayed in your linked pictures. That is, integer coordinates ought be transformed into integer coordinates only, and not into something irrational, involving a sqrt(2) factor.
--- rk
In this text, "rotate" is used in a loose sense, as it is actually combined with a scaling by $\sqrt2$, to keep integer coordinates.
To rotate is the verb that corresponds to rotation. There is no verb for "similarity transformation".
