No: the issue is that the spectrum can have "multiplicity". For instance, in the finite dimensional case, $\sigma(A)=\sigma(B)$ means that $A$ and $B$ have the same set of eigenvalues, but they could have those eigenvalues with different multiplicities, and thus fail to be unitarily equivalent. (For a really simple example, you could have $A=0$ and $B=0$ but $H_A$ and $H_B$ have different dimensions!)
In general, "multiplicity" is the only obstruction to self-adjoint operators with the same spectrum being unitarily equivalent: two self-adjoint operators with the same spectrum and the "same multiplicity" everywhere on the spectrum are unitarily equivalent. However, precisely stating and proving this idea in the infinite-dimensional case (where the spectrum may be continuous and you cannot just simply use the multiplicity of an eigenvalue as usually defined) is quite nontrivial. In the separable case, you really can reasonably define a "multiplicity function" on the spectrum but in the inseparable case the formulation is more delicate and measure-theoretic. For more details, you may want to look up the Hahn-Hellinger theorem (in the separable case) or Halmos's nice little book Introduction to Hilbert space and the theory of spectral multiplicity (in the inseparable case).