Hint: If we additionally make the (reasonable, though not necessarily true) assumption that the minimum happens when $ a = b$, then reduce it to a 1 variable inequality and hence show that the minimum is 4 obtained at $ ( 1/3, 1/3, 4/3)$.
Note that we've not proven this is the minimum yet.
Use this to figure out what AM-GM's to create.
Modifying your approach, we have the following inequalities:
$a^2 + b^2 \geq 2ab \longrightarrow$ This reinforces $a=b$ at the equality case.
$Xa^2 + 0.5c^2 \geq 2\sqrt{X/2}ac \longrightarrow$ What does the (assumed) equality case tell us about $X$?
$Yb^2 + 0.5c^2 \geq 2\sqrt{Y/2}bc \longrightarrow$ What does the (assumed) equality case tell us about $Y$?
From the equality case, we require
$ X = 8$, $ Y = 8$.
Then, weight the inequalities to get
$ 2a^2 + 2b^2 \geq 4ab $
$ 8 a^2 + 0.5 c^2 \geq 4 ac $
$ 8b^2 + 0.5c^2 \geq 4bc $
Summing them up gives:
$$ 10a^2 + 10b^2 + c^2 \geq 4(ab+bc+ca) = 4,$$
with equality when $ a = b = c/4, ab+bc+ca = 1$, IE $ ( 1/3, 1/3, 4/3)$.
(Thankfully, our initial assumption is true.)
I wanted to show you how to derive these inequalities (assuming that your stated approach could work). It isn't just "magic" or "by observation" or "by luck".
For the actual solution, you just need to write out the 3 AM-GM inequalities, sum them up, and verify the equality case.
Using this approach, try your hand at minimizing (say) $3a^2 + 2b^2 + c^2 $ given $ab+bc+ ca = 1$.
(Note: I've not actually done this, so I can't guarantee that the values will look nice.)