If by a prime in this set you mean something that cannot be factored non-trivially in this set, the result is false. For we have $(9)(49)=(21)(21)$.
The numbers $9$, $49$, and $21$ are prime in this set.
You can produce an infinite family of examples by taking $4$ distinct ordinary primes of the form $4k+3$, say $s,t,u, v$ and considering the number $stuv$, which has three distinct "prime" (in the $4k+1$ sense) factorizations.
Remark: This answers the question if we define prime as you did, as a number $p$ which has no non-trivial factorization in our set. It does not answer the question if by prime we mean a number $p$ which is not a unit, such that $p$ divides $ab$ implies that $p$ divides $a$ or $p$ divides $b$.