Find the equations of the circles tangent to the lines A and B, and their centers are on the line C. I can't find the solution to this problem, whatever the angle I look at it.
We need to find the equations of the circles tangent to two lines (A and B), knowing that their centers are on the line C.
$$
A = x - 2y + 2 = 0
$$
$$
B = 2x-y+1=0
$$
$$
C = 5x-y-11=0
$$
 A: The key fact here is that the radius of a circle is perpendicular to the corresponding tangent line.
If we want the circles tangent to both lines $A$ and $B$, with centres on line $C$ then we want points $P$ on the line $C$ whose perpendicular distance from line $A$ and line $B$ is equal. 
The perpendicular distance of a point $(p,q)$ from the line $ax+by+c=0$ is given by
$$\frac{|ap+bq+c|}{\sqrt{a^2+b^2}}$$
The line $C$ has equation $5x-y-11=0$, i.e. $y=5x-11$ and so a general point is parametrised by $(p,q)=(t,5t-11)$, for some parameter $t$.
Line $A$ has $a=1$, $b=-2$ and $c=2$. Line $B$ has $a=2$, $b=-1$ and $c=1$.
It follows that we need to solve, for $t$, the following equation:
\begin{eqnarray*}
\frac{|1t+(-2)(5t-11)+2|}{\sqrt{1^2+(-2)^2}} &=& \frac{|2t+(-1)(5t-11)+1|}{\sqrt{2^2+(-1)^2}} \\ \\
\frac{|24-9t|}{\sqrt 5} &=& \frac{|12-3t|}{\sqrt 5}
\end{eqnarray*}
Solving $|24-9t|=|12-3t|$ gives $t=2$ or $t=3$, and the centre of the circles are $(p,q)=(2,-1)$ and $(p,q)=(3,4)$.
The radius is given by substituting $t=2$ and $t=3$ into either of the perpendicular distance formulae:
\begin{eqnarray*}
\frac{|24-9(2)|}{\sqrt 5} &=& \frac{6}{\sqrt 5} \\ \\
\frac{|24-9(3)|}{\sqrt 5} &=& \frac{3}{\sqrt 5}
\end{eqnarray*}
The equations of the circles are given by $(x-2)^2+(y+1)^2=\frac{36}{5}$ and $(x-3)^2+(y-4)^2=\frac{9}{5}$.
Here's a picture. The lines $A$ and $B$ are in black. The line $C$ is in red. The circles, and their centres are in blue.

A: Assume the general equation of a circle: $x^2 + y^2 + 2gx + 2fy + c = 0$.
The center of this circle is $(-g,-f)$ which lies on line $C$. Hence we have our first equation 
$$-5g + f -11 = 0 \ \ \ \ \ \ \ \ --------(1)$$
And the length of the perpendicular from the center $(-g,-f)$ to lines $A$ and $B$ is equal to the radius of the circle due to the the tangency condition. Hence we have the following 2 relations, 
$$\lvert \frac{-g+2f+2}{\sqrt{5}} \rvert = \sqrt{g^2 + f^2 - c} \ \ \ \ \ \ \ \ --------(2)$$ and
$$\lvert \frac{-2g+f+1}{\sqrt{5}} \rvert = \sqrt{g^2 + f^2 - c} \ \ \ \ \ \ \ \ --------(3)$$
From $(2)$ and $(3)$ we get 
$$\lvert -2g+f+1 \rvert = \lvert -g+2f+2 \rvert \ \ \ \ \ \ \ \ --------(4)$$
Now once solve $$-2g+f+1=-g+2f+2$$ and equation $(1)$ to get
$$g=-2; \ f=1; \ c=\frac{-11}{5}$$
And then solve $$-2g+f+1=-(-g+2f+2)$$ and equation $(1)$ to get 
$$g=-3; \ f=-4; \ c=\frac{116}{5}$$
And hence you have obtained the equations of the $2$ possible circles.
On a side note, it is interesting to see the geometry of the problem to see why $2$ circles are possible with the given conditions.
A: Here’s a sketch of a solution using the approach suggested by N74’s comment.  
The centers of the circles must lie on the bisectors of the angles formed by the tangent lines. Normals to $A$ and $B$ can be read from their equations: they are $[1,-2]$ and $[2,-1]$, respectively, from which the normals of the two bisectors are easily found to be $(1,1)^T$ and $(1,-1)^T$. The lines $A$ and $B$ intersect at $(0,1)$, so the equations of the two bisectors are $x+y-1=0$ and $x-y+1=0$. Solving for the intersections of these lines with $C$, we find that the centers of the two circles are $(2,-1)$ and $(3,4)$ respectively. The corresponding radii can be found by using the formula for the distance of a point from a line as in other answers: $6/\sqrt5$ and $3/\sqrt5$, respectively. Putting this all together, the two circles are given by the equations $$(x-2)^2+(y+1)^2=\frac{36}5 \\ (x-3)^2+(y-4)^2=\frac95.$$
