This goes against our intuition because we think of suprema as being 'big' and infima as being 'small'. The crux is that suprema are the 'least big big thing', and when anything is big $-$ as is the case compared to the empty set $-$ then the 'least big big thing' is the smallest thing you can have!
Given a set $X$ equipped with some ordering $\le$, and some subset $Y \subseteq X$, we can define $\sup(Y)$, if it exists, to be an element $s \in X$ such that:
- $y \le s$ for each $y \in Y$; and
- If $y \le t$ for each $y \in Y$, then $s \le t$.
Why define it like this? Well (1) says it's an upper bound, and (2) says it's a least upper bound. (That's what a supremum is: a least upper bound.)
So what about when $Y = \varnothing$? Then (1) is satisfied trivially, since $\varnothing$ has no elements; and similarly (2) is only satisfied if $s$ is a minimal element of the set $X$. [Stare hard at (1) and (2) to see why these are the case.] So when $X$ is taken to be the (extended) real line, this gives $\sup(\varnothing)=-\infty$.
The same goes for $\inf(Y)$ and $\infty$.