I am currently writing a thesis in machine learning and I am trying to use t-test to show that my model is better than the current state of the art. The dataset I used has 16 different tasks and each of the mean performance and the number of dataset for each task are as below:
dataset = [2690,76,55,898,3758,69,787,392,1547,451,202,184,283,66,152,5271]
comparison1= [84.6,86.3,87.2,81.1,91.1,76.5,92.6,88.4,82.7,96.2,78.1,95.8,85.4,69.0,82.0 ,83.6]
proposed = [85.8,87.2,90.8,81.4,91.4,81.5,92.8,88.2,84.9,96.4,77.9,96.5,84.8,69.1, 78.2,84.1]
I would like to find out whether my model has improved or not using t-test and so far I calculated it by creating a new list that repeats or replicate the mean performance by the corresponding number of dataset.
(e.g. full_comparison1 = [comparison1[0]]*dataset[0]+[comparison1[1]]*dataset[1]+[comparison1[2]]*dataset[2]...
Then, I used scipy from python to calculate p-value:
scipy.stats.ttest_ind(full_proposed , full_comparison1 , equal_var=True)
Ttest_indResult(statistic=13.868700994408345, pvalue=1.2916487642703654e-43)
This result tells that the difference is significant, but it seems that p-value is way too small. Could somebody please let me know whether I have done it correctly or not? Thank you so much!!
full_comparison1
represent? $\endgroup$