A circle with infinite radius is a line I am curious about the following diagram: 

The image implies a circle of infinite radius is a line. Intuitively, I understand this, but I was wondering whether this problem could be stated and proven formally? Under what definition of 'circle' and 'line' does this hold?
Thanks!
 A: Well, this is true in the ordinary Euclidean plane. It is false in the ordinary hyperbolic plane. The "limit" of circles going through a common point, as radius grows without bound, is a horocycle. Sometimes this is spelled horocircle. 
In your image above, there would be two horocycles tangent to your line at the point $C$, one on either side of the line. 
Anyway, there are pages of stuff to be described about this. Maybe I should just say that there is an "intrinsic" axiomatic definition of a hyperbolic plane, modernized by David Hilbert. You are likely to first come across some of the models, the most popular being the Poincaré disk, the Poincaré upper half plane, the Beltrami-Klein model, the one-sheet hyperboloid model in Minkowski 3-space. The first two and the fourth are linked to in the wikipedia link.
A: There is no such thing as a circle of infinite radius. One might find it useful to use the phrase "circle of infinite radius" as shorthand for some limiting case of a family of circles of increasing radius, and (as the other answers show) that limit might give you a straight line. 
A: Algebraic description
A generic algebraic curve of degree $2$ is the set of points satisfying
$$ax^2 + by^2 + cxy + dx + ey + f = 0$$
You can define circles to be those curves with $a=b$ and $c=0$. You can compute the radius as
$$r=\sqrt{\frac{d^2 + e^2}{4a^2} - f}$$
For the case of $a=b=0$ your equation describes a line, and your radius becomes infinite due to the division by zero.
Note that the definition also includes circles with imaginary radius, which have no points in the real plane. You may exclude these by restricting the range of parameters.
Complex plane and cross ratios
Interpreting the points of your plane as complex numbers, you can define that four points lie on a common circle or line if their cross ratio is a real number. Using this definition, a line is just a circle, and the only reasonable value to assign to its radius is $\infty$.
Möbius Geometry
A Möbius transformation will map circles and lines in the complex plane (or complex line, depending on your use of the word) $\mathbb C$ to other circles and lines. So the above definition of a circle will suit these transformations as well. There is a scientific topic called Möbius Geometry which uses 4-dimenstional vectors to describe both lines and circles. In this setup again a line is a special case of circle, and performing any radius computation will lead to infinity.
You may extend Möbius geometry a bit further to obtain Lie geometry, where even points are special cases of circles, namely those with zero radius. Lie transformations may transform generic circles to lines or points and vice versa.
Constant curvature
You may define a circle as a line of constant curvature, and without endpoints. You may imagine curvature as the inverse of the radius, but you can define it in other ways as well. Obviously, for the case of zero curvature and therefore infinite radius, you will obtain a line.
Projective geometry
You can define a circle to be a conic incident with the two complex circle points
$$\begin{pmatrix}1\\i\\0\end{pmatrix}
\text{ and }
\begin{pmatrix}1\\-i\\0\end{pmatrix}$$
where the points are given in homogenous coordinates in $\mathbb{CP}^2$. There are conics with real coefficients which degenerate into a pair of lines. One of them is the line at infinity incident with the two points above, and the other may be a finite line. So unless your conic degenerates to a double line at infinity, you get a single line as the finite portion of a conic which by definition is a circle. You can compute the center of that circle using the points above, and the result will be an infinite point, indicating the infinite radius.
A: A circle of radius $r$ whose center is at $(r,0)$ has the parametric form
$$
\begin{array}{}x=r(1-\cos(\theta/r))&y=r\sin(\theta/r)\end{array}\tag{1}
$$
the limit of the curve in $(1)$ as $r\to\infty$ is
$$
\begin{array}{}x=0&y=\theta\end{array}\tag{1}
$$
which is the vertical line in your image.
Addendum:
In Inversive Geometry, circles and lines are considered the same. The inverse of a circle which passes through the center of the inversion is a line which doesn't pass through the center and vice-versa. The inverse of a line which passes through the origin is the line itself.
In the following image, the red and green circles are inverses with respect to the grey circle. Notice that when the red circle passes through the center of the inversion, the green circle becomes a line.

A: Most people are going about this from a projective geometry perspective. I will approach this from a differential geometry viewpoint instead, since I feel I can give a more intuitive understanding from this vantage point. Unfortunately I can't speak about $-\infty = \infty$ from this viewpoint without dealing with stereographic projection or projectivization, but I don't think this is the meat of the understanding anyway.
For a smooth curve (that is, no sharp turns) in the plane there is the notion of its curvature; it is a numerical quantity defined at each point along the curve that, for lack of better phrasing, describes how curved the curve is at that point. If you imagine yourself as on a bike traveling along the curve at $10$ mph, then where the curvature is very high you are turning very fast. Where the curvature is small you are not turning very fast, and where the curvature is zero you aren't turning at all.
From this intuitive description, you can guess quite easily that a straight line has curvature $0$ everywhere, and in fact this uniquely characterizes straight lines. You can also show, if you make the proper precise definitions, that a circle of radius $R$ has constant curvature $1/R$, and this uniquely characterizes circles as well. If you know any physics, this is closely related to the fact that uniform circular motion is the result of constant acceleration toward a center. Essentially, there are only two kinds of curves of constant curvature: lines (zero curvature) and circles (nonzero curvature).
The statement that a line is an infinitely large circle can then be framed in terms of curvature. For each $R>0$ let $C_R$ be a circle of radius $R$; then it has curvature $1/R$. Then in the limit as $R\to\infty$, the curvature goes to $0$. In some sense, at "$R=\infty$" we end up with curvature "$1/\infty = 0$," a straight line. The way this is interpreted more formally is that as $R\to\infty$, small segments of the circle $C_R$ become better and better approximations to straight lines. Armed with this interpretation, we can amend the above bolded statement: there is only one kind of curve of constant curvature: circles, including the infinite ones (lines).
Addendum: This is only one way to view the statement that lines are "infinite circles." Another common way to think about this is in the context of projective geometry, where lines can "become circle" by "closing them up" by adding "points at infinity." Also, you can think about them in the context of inversion geometry (geometry of the map $z\mapsto 1/\overline{z}$ in the complex plane) and more generally Mobius transformations, which overall tend to map lines and circles to other lines and circles; if one makes the convention that a line is an infinite circle, then Mobius transformations can be said to map circles to circles, which makes their description somewhat neater.
A: If you take the equation of a circle tangent to the vertical axis to be
$$(x-r)^2+y^2=r^2$$
or
$$\frac{x^2+y^2}{r}-2x=0$$
and let $r\to \infty$, you find that you obtain the equation $x=0$, which is precisely the vertical axis...
A: This obtains a clear interpretation in terms of the hyperreals.  An equation like $(x-r)^2+y^2=r^2$ for a positive infinite hyperreal number $r$ gives a circle of an infinite radius in the hyperreal plane.  One can take the finite part $F$ of the circle, i.e., points with both coordinates finite.  Then the standard part of $F$ is a line, namely the $y$-axis.
A: The standard definition of a circle is the locus at a distance r from the centre O. It’s obvious that a line on Euclidean plane is not a circle. But have we really see the centre to check whether a piece of curve is a piece of some circle? We haven’t, because, on Euclidean plane, line of constant non-zero curvature is a criterion of a circle (let us consider only curves with only one sign of curvature, i.e. all chords must lie on one side of the curve). For example, we can check whether Menger curvature for all triples of distinct points is constant. If it is, and is a non-zero quantity, then our curve is a piece of circle. There are different definitions of curvature, such as differentially-geometric one, but it isn’t important which definition we use if it can work for a small piece of curve. Local definition of curvature implies a local criterion for a circle. On Euclidean plane, circles with very small curvature (i.e. great radius) are locally almost indistinguishable from straight lines, where straight lines are lines of zero curvature.
How two definitions, of r,O-circles and of lines of constant curvature, are related? We see, on Euclidean plane, each (unterminated) line of constant curvature is either an r,O-circle or a straight line.
We also have another two homogeneous geometries of a plane: elliptic geometry and Lobachevski’s geometry. They also have distances, and differential geometry works there as well. In any of these geometries both straight lines and r,O-circles have constant curvature, but how they are related? In elliptic geometry there is no r,O-circles with r > πR/2 (where R is constant) and, curiously, circles of radius πR/2 are the same as straight lines (great circles). So, in elliptic geometry, (unterminated) lines of constant curvature and r,O-circles define the same class of curves.
In Lobachevski’s geometry opposite conditions exist: there are r,O-circles for any r > 0, but any r,O-circle has curvature greater than 1/R, where R is constant (sorry, do not remember exact dependence of curvature on r). But there are lines of constant curvature for any curvature. Lines of constant curvature between 0 and 1/R inclusively extend indefinitely (are not closed). The case of curvature 1/R was mentioned in the answer of @Will Jagy. So, in Lobachevski’s geometry, both straight lines and r,O-circles are proper subsets of the set of all (unterminated) lines of constant curvature, and there are also some lines in between.
That’s how relation between straight lines and r,O-circles demonstrates Euclidean geometry’s intermediate position between elliptic geometry and Lobachevski’s geometry. Euclidean straight lines are not r,O-circles, unlike situation in elliptic spaces. But Euclidean straight lines are adjacent to r,O-circles (can be obtained as a limit of r,O-circles, in the sense of pictures presented), unlike situation in hyperbolic spaces.
P.S. please, in case of problems with English grammar or terminology, report them.
