Erhan et al. in their 2010 paper discusses how pre-training improves deep networks: http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf#page=15

In there, they compare different neural network models by visualizing the function representation for each network. For a given model, Function representation is defined as a vector of outputs for a given set of inputs (the link above points directly to the description of function representation).

Inline below is a figure comparing model trajectories:

enter image description here

Question: Since, function representation is defined as a vector of output values for a given set of inputs, I assume there exists a vector $V_{c}$ with all the correct outputs. Now ideally, both the model trajectories (with pre-training and without pre-training) should try to converge towards this correct vector $V_{c}$ with progression in training iterations.

However, if we look at the figure above, the model trajectories seem to diverge, instead of converging towards a correct vector $V_{c}$. Why are the trajectories diverging instead of converging?

PS: Here is a link to a video that describes this figure: https://youtu.be/MJs9JHr8C-s?t=177


1 Answer 1


However, at some point (after about 7 epochs) the different trajectories (corresponding to different random seeds) diverge (slowing down into elongated jets) and never get back close to each other (this is more true for trajectories of networks without pre-training). This suggests that each trajectory moves into a different apparent local minimum.

So yes, there exists some correct vector $V_c$, however, both paths encounter completely different local minima and they get stuck there.

  • $\begingroup$ So you mean to say that both these models are stabilizing in different local minimas. Hmmm. Good point !! $\endgroup$ Jun 12, 2017 at 18:17
  • $\begingroup$ @TheWanderer exactly $\endgroup$ Jun 12, 2017 at 18:25
  • $\begingroup$ Hey Thomas, on second thought, I see a problem with this reasoning. The space that we are looking at is the function representation space, ie it just shows the number of true and false outputs. And as we train, we are expected to increase the number of true outputs. Therefore both the models: models with pre-training and models without pre-training should merge towards the case where the output is all ones. $\endgroup$ Jun 13, 2017 at 19:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .