Your basic idea is correct, but you are missing an important bit: There is an implicit universal quantification in the definitions:
Soundness: For all formulas $A, B$, if $A \vdash B$, then $A \Rightarrow B$
Completeness: For all formuas $A, B$, if $A \Rightarrow B$, then $A \vdash B$
So when a proof system is not sound, what is negated is the universality of the implication:
Unsoundness: Not for all formulas $A, B$, if $A \vdash B$, then $A \Rightarrow B$
Incompleteness: Not for all formuas $A, B$, if $A \Rightarrow B$, then $A \vdash B$
This is in turn equivalent to an existential negated statement:
Unsoundness: There exist formulas $A, B$ such that not if $A \vdash B$, then $A \Rightarrow B$
Incompleteness: There exist formuas $A, B$ such that not if $A \Rightarrow B$, then $A \vdash B$
And this is in turn equivalent ot the following:
Unsoundness: There exist formulas $A, B$ such that $A \vdash B$ but not $A \Rightarrow B$
Inompleteness: There exist formuas $A, B$ such that $A \Rightarrow B$ but not $A \vdash B$
So if a proof system is unsound, then some of the proofs it produces are not semantically valid. It doesn't have to be per se that everything it proves is nonsense.
This may be even more convincing on the incompleteness side: The word "incomplete" just means that some sequents are missing from the proof system; it doesn't have to be that that it fails to prove all sequents whatsoever.
If a proof sytem is unsound and complete, then all the semantically valid inferences can be proven, but in addition it proves some sequents that are not actually valid.
Edit (changing my last pargraph thanks to Malice Vendrine's comment):
Note also that those provable but invalid sequents do not have to be contradictory: It might just be that they are not true in all structures. For example, a system that proves $\vdash A \lor B \to A$ would be unsound, because this inference is not universally valid. But neither is its negation (there may well be structures in which the formula is satisfied, for example, in any structure in which $A$ is true).
So proving non-valid formulas does not immediately lead to an inconsistency. Only if we can prove the negation of a formula that is valid (and hence, by completeness, also provable) or, vice versa, a formula whose negation is valid, unsoundness in combination with completeness makes the system inconsistent.