Your question reads like you're looking for an implementation of the transformation whose existence is stipulated by the Riemann mapping theorem; particularly as you tagged it conformal-geometry.
Unfortunately, these beasts are rather hard to compute even for simple shapes, with huge numerical issues. The best continuous solutions seem to be based on Schwartz-Christoffel mappings. There is (at least) a whole book on the subject, which might be useful to you:
Driscoll and Trefethen Schwarz-Christoffel Mapping, Cambridge press, 2002
Recent developments in discrete differential geometry provide discretized concepts of conformal maps. One that I have intimate knowledge of is described in this article:
Springborn, Schröder and Pinkall, Conformal Equivalence of Triangle Meshes, ACM Transactions on Graphics 27:3
So what I'd do is the following:
- Triangulate both your input shape and the circle in an arbitrary way
- Choose an arbitrary triangle as intermediate step
- Map both triangulations onto the triangle using my own implementation of the algorithm from the paper referenced above
- Use the projective interpolation scheme described in that paper to map points first onto the triangle and from there onto the circle
There are a lot of arbitrary choices in the above approach. There is an arbitrary choice of a triangle, and for both partial mappings there is a choice which vertices of the triangulation should correspond to the corners of the triangle. In the continuous theory, the choice of the triangle is arbitrary, as is the choice of one set of corner vertices. The second choice of corner vertices fixes a Möbius transformation, as the conformal map is unique only up to Möbius transformations.
In practice, however, these choices will have significant impact on numeric stability. In a first attempt, I'd try the following heuristics. From the polygonal outline of the triangulated input shape, choose those vertices where the interior angle is minimal, and map those onto the corners of the triangle. Sum up the edge lengths along the rim of the polygon between these points, and use these three lengths as the edge lengths of your triangle. Also use these lengths as arc lengths along the rim of the circle to choose the corresponding vertices there as well. Make the triangulation of your shapes reasonably regular. With a bit of luck, this will work well enough. If not, new ideas might be required.