Side of the largest possible cube inside a cone? Question:

What is the side of the largest possible cube inside a cone of height $12$ units and radius $3\sqrt 2$ units?

Now, when this question came up (in my class), I instantly thought to myself that $1^{st}$ I must check how the cube must be placed. With the limited time I had, I drew the following cases:

I reasoned to myself that any intermediate variations must ultimately settle at these $2$ supposedly extremities. Let the figure on the right be called $R$ and that on left, $L$. Then by looking at the diagram itself, I concluded that since $R$ makes better use of the extended width of the cone down below, so it must be the largest possible cube that one may fit inside the cone.
But my teacher directly proceeded with $L$ and no one else minded it so I questioned, to which, I was told by teacher that $L$ IS better. Thus, here. Please help.

Finally, using similar triangles and the common vertex angle in $L$, we arrived at $4$cm for the side of the cube (in $L$).
 A: I think your teacher is right, though it's very disappointing that he/she didn't seem to take your question seriously.  First thing to note is that your right-hand diagram is not accurate.  If you stand a cube upright on one of its corners (I am assuming that's what you meant), the profile will be a hexagon, not a square.  One corner will be on the base of the cone.  Three corners will be at height $s/\sqrt3$, where $s$ is the side of the cube, and will lie on a circle with radius $s\sqrt{2/3}$.  Three more will be at height $2s/\sqrt3$ and will lie on a circle with the same radius.  The top corner will be at height $s\sqrt3$ above the base.
By similar triangles, the radius of the cone at height $h$ is given by
$$r=3\sqrt2\Bigl(1-\frac{h}{12}\Bigr)\ ,$$
and the second condition above means that for the cube to fit inside the cone we need
$$3\sqrt2\Bigl(1-\frac{s}{6\sqrt3}\Bigr)\ge s\sqrt{\frac23}\ .$$
Solving gives $s\le2\sqrt3$ which is less than $4$, so the cube in the left-hand diagram is bigger.
If you meant the cube to be standing on an edge then the profile is a square as in your diagram (or a pair of rectangles if you look from a different angle).  In this case four vertices form a rectangle with circumradius $s\sqrt3/2$ at height $s/\sqrt2$.  Using the same idea as above gives
$$s\le\frac{12\sqrt2}{1+2\sqrt3}\ ,$$
which is actually bigger than before, but still less than $4$.
A: The General Forms:



Consider the above diagram. Let the radius of the cone be $r$ and its height be $h$.
Let the leftmost triangle be called $L$, rightmost triangle be called $R$.
Let the center $2$ triangles be $C_1$ and $C_2$ respectively from left to right.
Let $C_1$ and $C_2$ collectively be called $C$. (Check Note $5$, at the end)
Note:

*

*Cube inside  $L$ is sitting on its face; in $C$, on edge; and in $R$, on a vertex.

*In $C_1, \ S$ touches the cone's lateral surface but since it is closer to us (remember it lies on a circle) so we don't see it that way from the front.

*$L,C,R$ are their respective best positions with respect to ease of observing them and maximizing the cube's side.


Cube's Side Length:




Diagram
Similar Triangles
$a$ in terms of $r$ and $h$




$L$
$\frac{r}{h}= \frac{\frac{a}{\sqrt{2}}}{h-a}$
$\frac{\sqrt{2}rh}{h+2\sqrt{r}}=a=a_L$


$C_1$
$\frac rh=\frac{\frac{\sqrt3a}2}{h-\frac a{\sqrt2}}$
$\frac{2rh}{\sqrt3h+\sqrt2r}=a=a_{C_1}$


$C_2$
$\frac rh=\frac{\frac a2}{h-a\sqrt2}$
$\frac{2rh}{h+2\sqrt2r}=a=a_{C_2}$


$R$
$\frac rh=\frac{a\sqrt{\frac23}}{h-\frac {2a}{\sqrt3}}$
$\frac{\sqrt3rh}{\sqrt2(h+\sqrt2r)}=a=a_R$




Note:
To arrive at the results as discussed above we use similar triangles as OP has mentioned.
From the diagram, the similar triangles are: For $L: \Delta \text{ABQ}\sim\Delta \text{VBU}$, 
For $C_1: \Delta \text{ABP}\sim\Delta \text{QBK}$, 
For $C_2: \Delta \text{ABK}\sim\Delta \text{VBL}$, 
For $R: \Delta \text{ABP}\sim\Delta \text{UBR}$

Results:


(Click the images to enlarge)
In descending order of the length of the side of the cube:

*

*From $O$ to $X$: $\rightarrow L>C>R$

*From $X$ to $Y\rightarrow L>R>C$

*From $Y$ to $Z\rightarrow L>C>R$

*From $Z$ to $\infty\rightarrow L>R>C$
where $O$ is origin and the value of $\frac hr$ at:

*

*$X= 2(\sqrt3-\sqrt2)\simeq0.636$

*$Y= \frac{2\sqrt6-4}{2\sqrt2-\sqrt3}\simeq0.820$

*$Z= \frac{4-\sqrt6}{3-2\sqrt2}\simeq9.037$
Note:
From $O(0)$ to $I(\simeq0.707)$, the side of the cube in $R$ is not given by the formula mentioned in the table. Because before $I$, $h<\frac r{\sqrt2}$, here, cube's length is restricted by the height of the cone. So, it's instead given by $a_R=\frac h{\sqrt3}$.
Similarly, in case of $C$, from $O$ to $J(\simeq1.932)$, $h<\frac{\sqrt2}{\sqrt3-1}r$, here, the cube's length is given by $C_2$ (as $C_1$ is not optimal here) and after $J$ its given by $C_1$ (as $C_2$ is not optimal here).

Conclusion:
Side of the largest possible cube inside a cone is always given by $\bf L$.
The only competition is between $R$ and $C$ for the $2^{nd}$ and $3^{rd}$ place.

Example:
Given, $h=12$ cms and $r=3\sqrt2$ cms. We know that the setting of L will give us the cube of largest side.
$\Rightarrow a_L=a=\frac{\sqrt2rh}{h+\sqrt2r}=\frac{\sqrt2⋅3\sqrt2⋅12}{12+\sqrt2⋅3\sqrt2}=12$
Thus, the side of the largest possible cube inside the given cone is $12$ cms.

Other Notes:
Note $\bf1$: (Why $\bf UV=a/\sqrt2$)
To know why $UV=a/\sqrt2\neq a/2$, check this post.
Similar reasoning is involved to calculate $KQ,LV$ and $RU$.
Note $\bf2$: (Diagram colors' meaning)
Cone's cross-section vertices: orange
Visible, hidden vertices of the cube: dark blue, light blue respectively
Visible, hidden edges of the cube: solid line, dotted line respectively
Imaginary lines/point to show the similar triangles we considered: red
Note $\bf3$: (Note of Thanks)
Special thanks to @David for this answer. It was especially helpful to figure out everything related to $R$.
Note $\bf4$: (Fun Fact)
In the given diagram, $h=3r$ and surprisingly, though it wasn't intended, the cubes inside look as if drawn to scale.
Just by observing, one can see that $a_L>(a_{C_1}\text{ & }a_{C_2})>a_R$ which is in accordance with our results.
Note $\bf5$: (Collectively called $\bf C$?)
Observe $C_1$ and then $C_2$. Notice anything? If not, then it would be best to take a dice and keep it on the table on $1$ edge and observe from front while rotating it.
You'll notice that $C_1$ and $C_2$ are same orientations (i.e. standing on an edge), we just rotated the cone along the $y$-axis passing through the vertex and the center of the base of the cone (that is, rotation along the red line, from the diagram).
The difference between $C_1$ and $C_2$ is that, in $C_1: SQUW$ touch the cone while in $C_2: VW$ touch the cone.
Note $\bf6$: (Note of Gratitude)
My heartiest thanks to @peterwhy for his immeasurable patience and for providing his great insights and proofreading each and every word and even diagrams! He helped me get clarity over various things too (and it wasn't easy). 
Do check this post where he lucidly explains everything about $\bf R$.

Tl;dr: Side of the largest possible cube inside a cone is always given by $L$ (when the cube is sitting on its face).
