Rotate and scale a point around different origins I am trying to rotate an arbitrary, 2D point (x,y) around another point (a,b), and at the same time, scale it from a different point (c,d). To transform the point, I must use a 3x3 transformation matrix.
For example, say I have a rectangle. I want to rotate its topleft point around its center, and scale it around its topleft:
Given point: \begin{pmatrix} x\\ y \end{pmatrix} rotational origin: \begin{pmatrix} a\\ b \end{pmatrix} and scaling origin: \begin{pmatrix} c\\ d \end{pmatrix}
(1) rotate (x,y) around (a,b)
(2) scale (x,y) from (c,d)
I know I can transform it around one origin using homogeneous coordinates, but how I can incorporate different origins for scaling and rotating in just one transformation matrix?
 A: As you state in your question, you require the transformation matrix. You will need the concept of homogeneous coordinates to perform the translation components in matrix form. You also need to know how to construct a rotation matrix and scaling matrix in two dimensions.
Rotation and scaling matrices are usually defined around the origin. To perform these transformations about an arbitrary point, you would translate the point about which the transformation is to occur to the origin, perform the transformation as normal, and then undo the translation. This process is then repeated for each transformation required.
The counter-clockwise rotation of a point $p=(x,y)$ to its image $p^\prime$ by an angle of $\theta$ around the point $(a,b)$ is
$$
p^\prime=Rp=\overbrace{\underbrace{\begin{bmatrix}1&0&a\\0&1&b\\0&0&1\end{bmatrix}}_{\text{Translate back}}
\underbrace{\begin{bmatrix}\cos\theta&-\sin\theta&0\\\sin\theta&\cos\theta&0\\0&0&1\end{bmatrix}}_{\text{Normal rotation matrix}}
\underbrace{\begin{bmatrix}1&0&-a\\0&1&-b\\0&0&1\end{bmatrix}}_{\text{Translate}}}^{\text{Complete rotation matrix $R$}}
\underbrace{\begin{bmatrix}x\\y\\1\end{bmatrix}}_{p}.
$$
Similarly, to scale a point $p=(x,y)$ to its image $p^\prime$ in the $x$ direction by $s_x$ and $y$ direction by $s_y$ from the point $(a,b)$ is
$$
p^\prime=Sp=\overbrace{\underbrace{\begin{bmatrix}1&0&a\\0&1&b\\0&0&1\end{bmatrix}}_{\text{Translate back}}
\underbrace{\begin{bmatrix}s_x&0&0\\0&s_y&0\\0&0&1\end{bmatrix}}_{\text{Normal scalingmatrix}}
\underbrace{\begin{bmatrix}1&0&-a\\0&1&-b\\0&0&1\end{bmatrix}}_{\text{Translate}}}^{\text{Complete scaling matrix $S$}}
\underbrace{\begin{bmatrix}x\\y\\1\end{bmatrix}}_{p}.
$$
The combined transformation matrix I will denote $T$ to perform the complete transformation will be either $T=SR$, if the rotation is performed first, followed by the scaling, or $T=RS$, if the scaling is performed first, followed by the rotation. The complete transformation of a point $p$ to its image $p^\prime$ is then $p^\prime=Tp$.
Edit: In response to comments below, the point about which the transformation is performed can give different resulting outcomes. Compare scaling the rectangle with bounding box $(2,2)$, $(4,6)$ by a factor of $2$ about the points $(4,6)$ and $(3,5)$. The distances from the scaled point are doubled, which produces the same size rectangle, but it is positioned differently as the centre of the transformation was different.
Result from scaling the rectangle about the point $(4,6)$

Result from scaling the rectangle about the point $(3,5)$

A: Given $(x,y)$
Proceeding in the order you mention plug each relation into the next tagged equation if so required in an algorithm.
Rotate by angle $t$ counter clockwise i.e., multiply by rotation matrix $A$
$$ A=[ (\cos(t), - \sin (t)) ,\, (\sin(t),  \cos (t)) ]$$
$$ c=x \cos(t)-y \sin (t); \quad d=x \sin(t)+ y \cos (t); \tag2$$
Scale factor $m$
$$ x3 =m. c;\,y3= m.d; \tag3 $$
Shift  back to starting origin
$$ x_{new}= x3-h;\,y_{new}= y3-k. \tag4 $$
EDIT1  (after OP's edit)
Given $(x,y)$
Rotate by angle $t$ counter clockwise i.e., multiply by rotation matrix $A$
$$ A=[ (\cos(t), - \sin (t)) ,\, (\sin(t),  \cos (t)) ]$$
$$ p=x \cos(t)-y \sin (t); \quad q=x \sin(t)+ y \cos (t); \tag5$$
$$ r= c + (p-c).{Xscale} ; \, s= d + (q-d).{Yscale}\tag6 $$
