While the Dirac $\delta$ is not a classical pointwise-valued function, it can be manipulated legitimately as though it were, in many situations, especially linear differential questions. As a distribution, it can be differentiated. Sometimes derivatives of classical, pointwise-valued functions really are distributions. For example, the second derivative of $|x|$ on the real line really is $2\delta$... not just in some fantasy sense, but in the sense that this conclusion behaves correctly with respect to integration by parts and such.
In two or more dimensions, the "Green's function" is harder to understand, for more than one reason, in contrast to the derivation of it for second-order ODEs on an interval (Sturm-Liouville problems). It is not obvious, for example, that in two dimensions the Laplacian applied to $\log |x|$ is a constant multiple of $\delta$ (although this has been known for a long time).