I am trying to get a better grasp on invertibility of matrices. My current mental model is basically that, a Matrix is invertible if every value of b can be mapped back to a unique x. Do I have the right idea here?
As remarked in the comments, your idea is correct.
Specifically, a linear transformation $T: V \to W$ is invertible iff (that is, if and only if) there is a linear transformation $S: W \to V$ such that $T\circ S: V \to V$ and $S \circ T: W \to W$ are both the identity linear transformation.
In matrix notation, if $[T]$ is an $n \times m$-matrix, we need a $m \times n$-matrix $[S]$ such that $[T][S]$ and $[S][T]$ are identity matrices (this relates to the above by $[T][S] = [T\circ S]$). It follows by e.g. Rank-Nullity that necessarily $n = m$.