An infinite group with proper subgroups being conjugate I am wondering whether there exists an infinite group $G$ such that any two proper non-trivial subgroups are conjugate.
Using HNN extensions one can show that there are infinite groups such that any two non-trivial elements are conjugate. Also there are 2-generated groups such that any proper subgroup is of prime order and such that any two proper subgroups of the same order are conjugate, due to Olshanskii.
For the moment, it is irrelevant whether the group $G$ is finitely generated or not.
 A: Assume that $G$ is such a group and non-abelian. Take $x$ a non-trivial element, $H$ the group generated by $x$. It is a proper subgroup of $G$, non-trivial. By assumption any proper non-trivial subgroup is conjugate with $H$, in particular they are all abelian.
Take now $N_G(H)$ its normalizer, we know that $H\subseteq N_G(H)$. If $N_G(H)=G$ then $H$ is normal in $G$ and then $H$ is the unique proper non-trivial subgroup of $G$, clearly, this imposes that $G$ is abelian.
So $N_G(H)$ is proper, since it is non-trivial, it follows that $N_G(H)=gHg^{-1}=<gxg^{-1}>$. Since $H\subseteq N_G(H)$ we can find $k\neq 0$ such that $x=gx^kg^{-1}$. It follows from this that $g\in N_G(H)$ (since $g^{-1}xg=x^k$) (Warning, this last sentence is false).
It follows that $N_G(H)=gHg^{-1}=H$. Hence $N_G(H)=H\subseteq Z_G(H)$ since $H$ is abelian. Hence :
$$H=Z_G(H)=N_G(H) $$
Finally take $l\geq 2$, $H':=<x^l>$ and $H:=<x>$. You have :
$$H'=Z_G(H')\supseteq Z_G(H)=H\supseteq H' $$
hence $H=H'$. So you can find $k_l>0$ such that $x=x^{lk_l}$. Hence $x$ is of finite order.
This shows that any non-trivial element must be of finite order $n$.
Now because of what we wrote $x^n=1$ and there also exists $k_n$ such that $x^{k_nn}=x$ combining both, we get that $x$ is trivial. Since we have done it for any $x\in G$ we get that $G$ is trivial, which is a contradiction (we assumed $G$ to be non-abelian).

Edit : 

We saw that the proof above is not correct. I think this second proof will be better, the aim is to show that for any proper non-trivial subgroup $H$ of $G$, $H=Z_G(H)$ :
Assume $G$ is not cyclic. Take $H:=<x>$ where $x\in G$  is non-trivial. It is proper  ($G$ not cyclic) and not trivial. Any proper  non-trivial subgroup being conjugate to this one, we see that they are all cyclic. Furthermore it is clear that the center of $G$ must be trivial.
In particular, define $H':=Z_G(H)$, since $x$ cannot be central and $H\subseteq H'$ it is both proper and non-trivial hence $H'=<y>$. Take now $H'':=Z_G(H')$ since $y$ cannot be central and $H'\subseteq H''$ it is both proper and non-trivial hence $H''=<z>$.
Now because $H\subseteq H'$ there exists $k$ such that $x=y^k$ and because $H'\subseteq H''$  there exists $k'$ such that $y=z^{k'}$. From this, it follows that $x=y^k=z^{kk'}$. Hence we see that $z\in Z_G(H)=H'$ so that $H''\subseteq Z_G(H)=H'$. 
Remember now that $H'\subseteq H''$ so $H'=H''=Z_G(H')$.
We found a proper, non-trivial subgroup $H'$ of $G$ which is its own centralizer. Since they are all conjugate it is true for any proper non-trivial subgroup. Then we can do the same thing as in the proof before.
A: As indicated in the comments, the proof of Clément Guérin only shows that such a group necessarily has to be a torsion group. And in fact I found out that there are examples of infinite groups whose proper non-trivial subgroups are all conjugate. The proper reference is:
Some Applications of Graded Diagrams in Combinatorial Group Theory by Ivanov and Ol'shanskii (Groups St Andrews 1989, Vol. 2)
There, Corollary 10 on p. 289 says:
Given a prime $p \gg 1$, there exists an infinite group whose proper non-trivial subgroups are all conjugate and have order $p$.
