By compactness, first order $\mathsf{PA}$ is not enough to describe the least upper bound property. This follows from the existence of non-standard models of $\mathsf{PA}$, which is an obvious consequence of compactness (let $c$ be a new constant symbol, and consider the result of adding to $\mathsf{PA}$ the axioms "$c>\underline n$" for all $n$). Now, any non-standard model has an initial segment isomorphic to $\mathbb N$. This segment has no supremum. Because if $a$ is larger than all the standard integers, then so is $a-1$. What you have is that any definable set with an upper bound has a supremum, since for each definable set, the corresponding statement is an easy consequence of an appropriate instance of (first-order) induction.
By the way, this gives us the nice and very useful property of overspill in non-standard models: Given a non-standard model $\mathcal N$, if $\phi(x)$ is a first order property (perhaps with parameters from $\mathcal N$), and $\phi(n)$ holds in $\mathcal N$ for infinitely many standard integers, then it holds of some non-standard integers as well. Otherwise, there would be a first $m$ such that $\phi(k)$ fails for all $k\ge m$, and this $m$ would be a supremum of the standard part.