I have a data set with which I am trying to find correlations.
I split the data into a training set and a validation set. I also have a solver I built which finds the "best coefficients" to give me the best results on the training set.
After solving for the training set, the validation set shows completely different results which do not support the results on the training set.
I then made my solver output all of the results, as opposed to only the best results. There are cases which show positive, very similar results in both the training and validation sets, and there are results that show different results in both sets.
Is it okay to choose, by hand, the results that show the most similar and most positive results in both validation and training sets, or does this defeat the purpose of the validation set and invalidate the results?