How to get the center of this circle I now realize that I have over-simplified the problem in my last post because it is hard to explain in words.
This is the original problem that I am fighting with for more than a week:
drawing
There are two lines originating from the origin. The angle of each line in relation to the diagonal is the same and is known. A circle is centered on the diagonal and only a 90 degrees arc of the circle is drawn between the lines. On the arc of the circle is a point P of which the coordinates are known. The dimension and location of the circle is not known, the only thing that is known is that point P lies on the arc and the arc is a 90 degrees segment of a circle with its center on the diagonal.
How can I find the coordinates of the circle?
 A: First, let us identify all the elements:

*

*The diagonal $d$ is the line $y=x$.

*The point $P$ has coordinates $(p_x,p_y)$.

*The angles are defined by lines $r_A : y=ax$ and $r_B:y=bx$.

*The circle intersects the lines $r_A$ and $r_B$ in points $T : (t_x, t_y)$ and $U : (u_x,u_y)$ respectively (there are two more points of intersection, but you should be able to discard them by inspection).

*You need to find the center $C: (c_x,c_y)$ and the radius $r$ of the circle, which has equation:
$$(x-c_x)^2 + (y-c_y)^2 = r^2$$

Now, here there are several key points that you should try to justify:

*

*Since the angles to the diagonal are equal, $b=1/a$. Use the angle to deduce the value of a.

*The center lies on the diagonal, so $c_x=c_y$.

*$d$ is the bisector of the angle between lines $r_A$ and $r_B$. Use its properties to conclude that $(t_x,t_y)=(c_x,c_x+r)$ and $(u_x,u_y)=(c_x+r,c_x)$.

Finally, replacing coordinates in the equations of the lines $r_A$, $r_B$ and the circle, you should be able to deduce the values of $c_x$, $c_y$ and $r$ in terms of $a$, $p_x$ and $p_y$.

Detailed steps:

*

*The slope of $r_A$ is given by the tangent of the angel with respect to the horizontal axis:
$$a=\tan(\frac{\pi}{4} + A),$$
where $A$ is the angle between $d$ and $r_A$. Analogously, the slope of $r_B$ is
$$b=\frac{1}{\tan(\frac{\pi}{4} + A)} = \tan(\frac{\pi}{4} - A).$$

*Since the center $(c_x,c_y)$ lies on $d$ (of equation $y=x$) then it clearly satifies that $c_x=c_y$.

*The points of $d$ are equidistant from $r_A$ and $r_B$ because $d$ is the angle bisector. Moreover, $\triangle TCU$ needs to be a rectangular and isosceles triangle (with legs of length $r$). So the only way for this to be possible is that $\overline{TC}$ is parallel to the vertical axis (so that $t_x=c_x$) and $\overline{UC}$ is parallel to the horzontal axis (that is, $u_y=c_y=c_x$).

*Note also that $T$ and $U$ lie on the circle, so the distance to $C$ is $r$. Therefore, $t_y=c_x+r$ and $u_x=c_y+r$.

*Taking all previous steps into account, knowing that $T$ lies on both $r_A$ an the circle, its coordinates have to satisfy both equations:
$$\begin{cases}
c_x + r &= c_x \tan(\frac{\pi}{4} + A),\\
(p_x-c_x)^2 + (p_y-c_x)^2 &= r^2.
\end{cases}$$

*Finally, you need to solve for $r$ in the first equation and replace it in the second one:
$$(p_x-c_x)^2 + (p_y-c_x)^2 = \left(c_x \tan(\frac{\pi}{4} + A) - c_x\right)^2.$$
Expand that expression to obtain a quadratic equation on $c_x$ which should be easy to solve (see the quadratic formula) in terms of $p_x$ and $p_y$. The rest of the values can be obtained by replacing $c_x$ with that value.

A: Reusing the nice picture and notations of AugSB, you have
$$\begin{cases}
c_x &= (r+c_x)\tan\left(\frac{\pi}{4} -A\right)\\
(p_x-c_x)^2 + (p_y-c_x)^2 &= r^2
\end{cases}$$
as $c_x=c_y=u_y$ and where the angle $A$ is given in radians.
Which is equivalent to
$$\begin{cases}
r &=c_x \frac{1 - \tan\left(\frac{\pi}{4} -A\right)}{\tan\left(\frac{\pi}{4} -A\right)}\\
(p_x-c_x)^2 + (p_y-c_x)^2 &= r^2
\end{cases}$$
$A$ is given. Replacing $r$ from the first equation into the second one you get an equation of the second degree where $c_x$ is the only unknown as $p_x, p_y$ are known. Solving it you get the desired center $C=(c_x,c_x)$.
A: A Straightedge and compass construction
After algebra, let's come back to geometry. Here is a Straightedge and compass construction.
The construction comes from following basic idea:

*

*It is easy to construct an arc $A_0$ having all the required properties... except that it isn't passing through $P$.

*From such an arc, the desired arc is the one obtained as the image of previous arc under the homothetic transformation $T$.

*The homothety $T$ is the one having for center the origin $O$ and that transforms $P_0$ into $P$.

*Where $P_0$ is the intersection between the arc $A_0$ and the line $(O, P)$.

Base on that, the center $C$ of the desired arc $A$ is the intersection of the line parallel to $(C_0, P_0)$ passing through $P$ with the main diagonal.
See picture below.

Geogebra file
