I would like to expand on the other answer somewhat, as this is a popular area of investigation in ML currently.
Firstly, non-parametric Bayesian methods are likely among the most principled approaches to obtaining uncertainty measures for ML models. In general, they can not only give you the variance at a data point, but in fact the entire predictive posterior distribution at that point (meaning you can get the entropy as well, for example). Such methods, which include Gaussian processes for example, are often not used in practice due to computational constraints, however.
Furthermore, the question asks specifically about models through which one can perform backprop. Many people view softmax and sigmoid outputs as implicitly giving a form of certainty measure. However, it is a bad one, generally speaking, and unprincipled. Why is that? It is because they are uncalibrated. For instance, consider a binary classifier that gives $C(x_i) = (0.1,0.9)$. We would expect that, if we took 100 $x_i$ examples, which all have $(0.1,0.9)$ as output, then 10 of them would be class $A$ and 90 of them would be class $B$. Neural networks do not satisfy this by just adding a softmax. However, there are methods that try to correct this lack of calibration with simple modifications. For instance, see
Note that the reason for this is that the network training procedure doesn't really reward proper uncertainty estimates well. Since (by definition) we are training on in-sample data, getting a good uncertainty estimate on out-of-sample data isn't really something the network is concerned with (even if one uses e.g. cross entropy loss). Even on in-sample data, it may well be (and in fact is, according to some of the papers above) that being confidently wrong can lead to a lower loss than hedging - and networks only care about loss. In other words, nothing really forces the softmax outputs to be calibrated confidences or principled uncertainty measures. They just happen to give results that seem like they are, sometimes.
Other related questions include
Ok, so if softmax and sigmoid values are not good confidence estimates, then what should we use? Ideally, we want the principled uncertainties given by Bayesian methods, i.e. with predictive posteriors. But it also has to be more scalable that Bayesian non-parametrics. The most common answer I hear is to use Bayesian neural networks (BNNs). The idea is to maintain probability distributions over the parameters of the neural network, which we can integrate over to get predictive posteriors.
Classical methods such as Laplace Approximations, Hamiltonian Monte Carlo, and Stochastic gradient langevin dynamics are still rather slow, however, though they are used in popular frameworks like the Tensorflow-based Edward.
How to make this more scalable?
A good paper is to look at is Hernandez-Lobato & Adams, Probabilistic Backpropagation for Scalable
Learning of Bayesian Neural Networks, which discusses getting calibrated predictions via a modified form of backprop.
Another popular approach is variational inference, where the posterior over weights is approximated by e.g. a mean-field approach. Yet even this is not scalable in many cases! Hence we come to the approach mentioned in a link, on approximate variational inference (e.g. see Gal, Uncertainty in deep learning) using methods such as dropout-based estimators.
One other thing to mention is that BNNs are closely tied to Bayesian non-parametrics. Indeed, deep Bayesian ANNs of infinite size are Gaussian processes. See here or here for more. In other words, using a BNN is how we bring such Bayesian methods to neural networks, in a rigorous manner.
All of this is to say that (1) creating certainty measures of ANNs is not as easy as one might think and is an active research area, and (2) Bayesian methods can be brought to bear on ANNs to construct good uncertainty measures in a scalable manner. Posterior predictive distributions can then be used to estimate some form of "ignorance".
Note that the idea is not to
avoid getting the network to extrapolate into areas we actually don't know much
because it is very hard to delineate such areas. And we usually want the network to extrapolate. Any element of the test set is unlikely to have a training set element right beside it in data space, and this problem is often exacerbated by the high dimensionality of our data. (Incidentally, perhaps there are ways to do this, but I do not know of any.) Instead, we find a function approximator as before, but supplement it with meta-information concerning the uncertainty of our function's values in a given area.