0
$\begingroup$

I am training my neural network (2 layers GRU x 512 neurons each, softmax output layer, cross-entropy error - total of ~2.5M parameters) language model on J.R. Martin's book (network tries to predict next letter based on previous steps). My training set consists of 1.3MB of text, divided in 50 chunks (batches). Sequence length is 50. Network memory is reset at start of each epoch - when I start to feed my training data to it from the beginning once more. With these settings each epoch is 267 iterations (each iteration - one sequence of 50 steps) I use RMSProp with momentum and weight decay for optimization.

So, I got this graph of error (not smoothed by any filters for debug reasons): Cross entropy vs iterations You can clearly see peaks at rate of exactly epoch restart. Is it normal? Why there is a peak in error when I reset memory and data set starts over? Does it affect training performance?

Thanks.

P.S. Also, another thing bothers me: very first iterations always looks like this (in any task, with any training data): Start err peak Is it OK to have this very high peak in the very begining, after ~10-15 iterations? It does not appear later in training, only on very start.

$\endgroup$
  • $\begingroup$ Are you computing the error on the mini-batch or the full dataset ? Does the y-axis represent error in the second graph ? $\endgroup$ – Curious Aug 19 '16 at 23:17
  • $\begingroup$ I am computing mini-batch error. Both graphics are the same - mini-batch error vs iteration. Luckily, I found what caused that wierd spikes - it was just bug in training program, dataset position was reset few iteration before network memory reset, so network started to "read" the same text again, but with some old context, wich lead to prediction erros. So, all is fine now. (But second question still stands, this one wierd peak at very begining of training is still there) $\endgroup$ – R. A. Aug 20 '16 at 20:13
0
$\begingroup$

Ok, I found a bug in training algorithm, wich lead to dataset position reset few iterations before network memory reset. So network started to read the same text once more but with some old unrelated context, and this caused prediction errors. So it's all fine now.

But question about second graph still stands, this peak is still there (does not seem to cause any trouble in later training, though)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.