Both categories are denoted by $\mathrm{Rel}$. In most cases, however, the author either explicitly tells which $\mathrm{Rel}$ he/she is using, or reproduces the definition of $\mathrm{Rel}$. (Note: this applies to any definition/term in math which can be used for two different things.) As egreg has mentioned in the comments, Categories, Allegories, Freyd and Ščedrov covers this.
Perhaps what you're looking for is a relation between $\mathrm{Rel}$ as defined in $\mathit{ACC}$, p. $22$ and $\mathrm{Rel}$ in the standard sense. If so, then note that $\mathrm{Rel}$ as defined in $\mathit{ACC}$, p. $22$ is $\mathrm{Rel}$ in the standard sense with an extra condition that the $2$-morphisms in $\mathrm{Rel}$ (in the standard sense) are relation-preserving maps, which are also morphisms in $\mathrm{Set}$, since the $2$-morphisms in $\mathrm{Rel}$ in the standard sense not only take relations $\rho\to\sigma$, but also takes the sets $X\to Y$.