0
$\begingroup$

Initial Situation

I have a Classification Model which I train on a Dataset consisting of 1400 samples where train on a training set (80%) and validate on another validation set (20%). As Model I use a Neural Network. During training I plot the train- and validation-accuracy curves. Both accuracies grow until the training accuracy reaches 100% - Now also the validation accuracy stagnates at 98.7%.

My Assumptions

I think the behavior makes intuitively sense since once the model reaches a training accuracy of 100%, it gets "everything correct" so the failure needed to update the weights is kind of zero and hence the modes "does not know what to further learn". Nonetheless the validation Accuracy has not flattened out and hence there is some potential to further increase the Validation Accuracy.

My Question

What can I possibly do to further increase the validation accuracy? As a side note: I still implement slight Data Augmentation (slight noise, rotation) on the training set (not on the validation set).

Thanks for your suggestions :)

$\endgroup$
4
  • 1
    $\begingroup$ How many samples do you have in total, what is the split proportion, what model are you using? $\endgroup$ Nov 16, 2020 at 6:08
  • $\begingroup$ I have updated the question $\endgroup$
    – 2Obe
    Nov 16, 2020 at 6:16
  • 1
    $\begingroup$ Get more training data if you can. Try using regularization to avoid overfitting. Try dropout and batch normalization. Try further data augmentation. Try using a pretrained model. Try using a simpler architecture that might be less prone to overfitting. 98.7 % validation accuracy sounds already quite good. $\endgroup$
    – littleO
    Nov 16, 2020 at 6:25
  • $\begingroup$ Thank you for your suggestions. That means in turn that my suggestion that the training stops once the training accuracy reaches 100% is correct? $\endgroup$
    – 2Obe
    Nov 16, 2020 at 6:31

1 Answer 1

1
$\begingroup$

It appears that your network very quickly learns how to classify the data. To check your train/validation errors are not just anomalies, shuffle the data set repeatedly and again split it into train/test sets in the 80/20 ratio as you have done before. If you continue to observe the same behaviour, then it is indeed possible that your model learns very quickly and would continue to improve if only it had more data.

To test that, do a Leave-One-Out-Crossvalidation (LOOC). That is, for each $i=1,\ldots, 1400$ take your test set to be the $i$-th sample, and your training set to be the other $1399$ samples. If the average training accuracy over these $1400$ models is $100$% and the average test accuracy is again very high (and higher than $98.7$%) then we have reason to suspect that even more data would help the model.

Another way to improve the model, related to what you said about your model not knowing "what further to learn" once the training accuracy reaches $100$%, is to add a regularisation term into your error function, so that even when a set of weights gives a training accuracy of $100$%, you can continue to find even simpler weights which also do the same, instead of stagnating.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.