To convince yourself that
when multiplying two matrices, if scale k-th column of the matrix on the left by $\alpha$ and the k-th row of the matrix on the right by $1/\alpha$ then the product does not change.
So we can always multiply the column of $Q$ by $-1$ and the corresponding row of $R$ also by $-1$, the we have another QR factorization. So requiring the diagonal to be positive prevents this ambiguity.
By the way, the ambiguity actually comes from taking square roots when calculating the columns of Q. Depending on the sign of the square root, we can end up with either positive value or a negative value on the diagonal of $R$.
Also, if $A$ is singular, then one or more of the diagonal entries of $R$ would be zero. In this case, scaling by $-1$ the column of $Q$ that goes with the diagonal value of zero does not change the diagonal value of $R$. This is why we need $A$ to be nonsingular.