This result is true for any compact Lie group $G$, and is a consequence of the Peter-Weyl theorem. This theorem says that the space $L^2(G)$ of square-integrable functions, with inner product $\langle f,\,k\rangle = \int_G f(g)\overline{k(g)}\,dg$ ($dg$ denoting the normalised Haar metric), has an orthonormal (Schauder) basis
$$
\{\sqrt{\dim\rho}\,\rho_{ij} \mid \rho \text{ an irrep of }G,\ 1\le i,j\le \dim\rho\},
$$
where $\rho_{ij}$ denotes the $ij$ matrix element of $\rho$ (with respect to some bases of the representation space). Orthonormality of these elements says that
$$
\langle \sqrt{\dim\rho}\,\rho_{ij}, \sqrt{\dim\rho'}\,\rho'_{mn}\rangle = \delta_{\rho\rho'}\delta_{im}\delta_{jn}.
$$
Completeness says that any $f\in L^2(G)$ can be decomposed into this orthonormal basis
$$
f = \sum_{\rho \text{ irrep}}\sum_{i,j=1}^{\dim\rho}\langle f,\sqrt{\dim\rho}\rho_{ij}\rangle \sqrt{\dim\rho}\rho_{ij}
$$
and this can be written informally as
$$
\sum_{\rho\text{ irrep}}\dim\rho\sum_{i,j=1}^{\dim\rho}\overline{\rho_{ij}(h)}\rho_{ij}(g) = \delta(h^{-1}g).
$$
Taking $h=e$ and using the fact that $\rho_{ij}(e) = [I]_{ij}=\delta_{ij}$ gives the result you are using.
The statement is precise in the sense that the Peter-Weyl theorem is precise (and, for example, infinite sums are defined as limits relative to the topology in $L^2(G)$ derived from the inner product). Writing the sum in terms of a delta function is a bit informal; formally, the sum has to be understood as only making sense when integrated against a function $f\in L^2(G)$ (although I'm sure one can make it precise using distributions).
