# How to preserve ignorance in backpropagation learning?

In machine learning it is important not only to correctly classify things based on observations that have been made, but also to know how unsure one is in an area where not many observations have been made. How can this be achieved? Which methods exist to avoid getting the network to extrapolate into areas we actually don't know much?

(I am particularly interested in approaches for neural networks trained by error back propagation).

Here is an example of what I want to achieve which I just accidentally accomplished in this case The training data are the cluster points and the colored image is the prediction map. Pure blue, red or green color means close to 100% confidence and in between colors like purple and yellow mean a large uncertainty between the classes of which colors are mixed :

• Better fit for stats.stackexchange.com – Gabriel Romon Oct 28 '17 at 21:01
• I don't agree. Ideas for methods I am interested in which could be useful could include algebra or analysis concepts which stats folks would not be as knowledgeable about. – mathreadler Oct 28 '17 at 21:05
• Can you be more specific? – Qudit Oct 28 '17 at 21:12
• No it is definitely not unanswerable. You are just being obnoxious. You are free to give any mathematical or algorithmic description of how to push a neural network into something that can represent "the response to this input is unknown". I have come up with at least 6 ways to do it so far so I am sure you can too, if you know what you are talking about, that is. – mathreadler Nov 1 '17 at 18:03
• The "Better fit for stat.stackexchange" is such a lazy and unhelpful response - this is a perfectly reasonable question for this forum. Anyway you may already be aware of this but there might be something of interest in this blogpost and links within? – Nadiels Nov 1 '17 at 21:40

You might be better off using (non-parametric) Bayesian methods such as Gaussian Processes or kernel methods. They provide a posterior distribution that not only gives you a prediction for a new data point, but also the certainty in the form of the variance.

With neural networks it is also possible, but less rigorous. You can use the softmax activation function in the output layer to produce something that resembles a probability distribution for each class. All outputs will be between zero and one and they sum up to one if you sum over all classes.

• Say the OP trained the RNN to learn the couples $(x_i,y_i=f(x_i))$ with $x_i \in \mathbb{R}^2$ and $y_i \in \{0,1\}^4$ (a vector with only one $1$ indicating the class of $x_i$), using the objective function $E = \sum_i \|f(x_i)-y_i\|^2$. We can instead train the RNN to learn $x_i,f(x_i),e^{-p(x_i)^2}$ with $e^{-p(x_i)^2}\in (0,1]$ using $E = \sum_i \|f(x_i)-y_i\|^2 e^{-p(x_i)^2} + \frac{1}{10} (1-e^{-p(x_i)^2})$, thus the RNN will output $p(x_i)$ very large if it isn't sure of $f(x_i)$. – reuns Oct 30 '17 at 13:58
• It is not so difficult to create certainty measures for ANNs. Many of them use sigmoids as activation functions which provide a smooth variation between two values "yes" and "no" in a vector for any one of the classes.The issue is mostly how to decide the points to encourage to be undefined and when/how to push the network in that direction as compared to backpropagating the actual training samples for the known classes. – mathreadler Oct 30 '17 at 15:46

I would like to expand on the other answer somewhat, as this is a popular area of investigation in ML currently.

Firstly, non-parametric Bayesian methods are likely among the most principled approaches to obtaining uncertainty measures for ML models. In general, they can not only give you the variance at a data point, but in fact the entire predictive posterior distribution at that point (meaning you can get the entropy as well, for example). Such methods, which include Gaussian processes for example, are often not used in practice due to computational constraints, however.

Furthermore, the question asks specifically about models through which one can perform backprop. Many people view softmax and sigmoid outputs as implicitly giving a form of certainty measure. However, it is a bad one, generally speaking, and unprincipled. Why is that? It is because they are uncalibrated. For instance, consider a binary classifier that gives $$C(x_i) = (0.1,0.9)$$. We would expect that, if we took 100 $$x_i$$ examples, which all have $$(0.1,0.9)$$ as output, then 10 of them would be class $$A$$ and 90 of them would be class $$B$$. Neural networks do not satisfy this by just adding a softmax. However, there are methods that try to correct this lack of calibration with simple modifications. For instance, see

Note that the reason for this is that the network training procedure doesn't really reward proper uncertainty estimates well. Since (by definition) we are training on in-sample data, getting a good uncertainty estimate on out-of-sample data isn't really something the network is concerned with (even if one uses e.g. cross entropy loss). Even on in-sample data, it may well be (and in fact is, according to some of the papers above) that being confidently wrong can lead to a lower loss than hedging - and networks only care about loss. In other words, nothing really forces the softmax outputs to be calibrated confidences or principled uncertainty measures. They just happen to give results that seem like they are, sometimes.

Other related questions include

Ok, so if softmax and sigmoid values are not good confidence estimates, then what should we use? Ideally, we want the principled uncertainties given by Bayesian methods, i.e. with predictive posteriors. But it also has to be more scalable that Bayesian non-parametrics. The most common answer I hear is to use Bayesian neural networks (BNNs). The idea is to maintain probability distributions over the parameters of the neural network, which we can integrate over to get predictive posteriors. Classical methods such as Laplace Approximations, Hamiltonian Monte Carlo, and Stochastic gradient langevin dynamics are still rather slow, however, though they are used in popular frameworks like the Tensorflow-based Edward.

How to make this more scalable? A good paper is to look at is Hernandez-Lobato & Adams, Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks, which discusses getting calibrated predictions via a modified form of backprop. Another popular approach is variational inference, where the posterior over weights is approximated by e.g. a mean-field approach. Yet even this is not scalable in many cases! Hence we come to the approach mentioned in a link, on approximate variational inference (e.g. see Gal, Uncertainty in deep learning) using methods such as dropout-based estimators.

One other thing to mention is that BNNs are closely tied to Bayesian non-parametrics. Indeed, deep Bayesian ANNs of infinite size are Gaussian processes. See here or here for more. In other words, using a BNN is how we bring such Bayesian methods to neural networks, in a rigorous manner.

All of this is to say that (1) creating certainty measures of ANNs is not as easy as one might think and is an active research area, and (2) Bayesian methods can be brought to bear on ANNs to construct good uncertainty measures in a scalable manner. Posterior predictive distributions can then be used to estimate some form of "ignorance".

Note that the idea is not to

avoid getting the network to extrapolate into areas we actually don't know much

because it is very hard to delineate such areas. And we usually want the network to extrapolate. Any element of the test set is unlikely to have a training set element right beside it in data space, and this problem is often exacerbated by the high dimensionality of our data. (Incidentally, perhaps there are ways to do this, but I do not know of any.) Instead, we find a function approximator as before, but supplement it with meta-information concerning the uncertainty of our function's values in a given area.