0
$\begingroup$

I'm currently working on an optimisation problem where a simulation can be run with several parameters and I need to find which parameters are the best. Since each simulation takes a long time to run, I need to discard bad sets of parameters as soon as possible.

Each simulation is scored (lower is better), and the results are added to a list.

For example, for four different setups, the simulation values are:

[0.31771960854530334, 0.3411135673522949, 0.3310917615890503, 0.3159765601158142, 0.33600446581840515, 0.3352251946926117, 0.33761146664619446, 0.32225948572158813, 0.3350348472595215, 0.3457653820514679, 0.32325679063796997, 0.326958030462265, 0.3291585445404053, 0.3235035836696625, 0.3258521854877472, 0.3204689919948578, 0.3229052722454071, 0.315879225730896, 0.31862542033195496, 0.33796921372413635, 0.32604268193244934, 0.34038183093070984, 0.3183555006980896, 0.3175279498100281, 0.32143139839172363, 0.3224892318248749, 0.34159010648727417, 0.33131226897239685, 0.3263885974884033, 0.31594401597976685, 0.3327217102050781, 0.32813698053359985, 0.3278810679912567, 0.31822341680526733, 0.34068915247917175, 0.32860109210014343, 0.33968520164489746, 0.3319733440876007, 0.32811060547828674, 0.31651514768600464, 0.32799118757247925, 0.31894755363464355, 0.3412730395793915, 0.32626068592071533, 0.3176010847091675, 0.33038097620010376, 0.3207707405090332, 0.31058192253112793, 0.3066112995147705, 0.3303760886192322, 0.3223210871219635, 0.33629271388053894, 0.33516937494277954, 0.3147951066493988, 0.3211946189403534, 0.32553717494010925, 0.333586722612381, 0.31509238481521606, 0.3251512944698334, 0.33351272344589233, 0.3334641754627228, 0.3285764455795288, 0.31418612599372864, 0.31757283210754395, 0.3138282001018524, 0.33097517490386963, 0.3072074055671692, 0.330384761095047, 0.32259663939476013, 0.33039024472236633, 0.33015933632850647, 0.30926862359046936, 0.32790517807006836, 0.3298207223415375, 0.32572630047798157, 0.3271850645542145, 0.3353654742240906, 0.33518925309181213, 0.3298708498477936, 0.31685906648635864, 0.32341158390045166, 0.3397740125656128, 0.3357153832912445, 0.33256030082702637, 0.3344813585281372, 0.3193385601043701, 0.3442055583000183, 0.3175262212753296, 0.3320333957672119, 0.3278960585594177, 0.31660929322242737, 0.33225393295288086, 0.31808194518089294, 0.3276512920856476, 0.3289024829864502, 0.3260359764099121, 0.31691187620162964, 0.31895992159843445, 0.32702019810676575, 0.3133307695388794, 0.33777090907096863, 0.31016212701797485, 0.3192517161369324, 0.3378731608390808, 0.33338403701782227, 0.33280327916145325, 0.3471103310585022, 0.3226596415042877, 0.32838064432144165, 0.32759958505630493, 0.32864367961883545, 0.32859107851982117, 0.3304615914821625, 0.3392229378223419, 0.31688782572746277, 0.3223646581172943, 0.3196653127670288, 0.3321816623210907, 0.3239646852016449, 0.33385634422302246, 0.3212287425994873, 0.341138631105423, 0.33112257719039917, 0.33745497465133667, 0.328122079372406, 0.3257265090942383, 0.3174751400947571, 0.3313937783241272, 0.3291671872138977, 0.3429141938686371, 0.3296595811843872, 0.3168943226337433, 0.31397369503974915, 0.3285788595676422, 0.3362148702144623, 0.3139144480228424, 0.33292356133461, 0.3266337215900421, 0.3262031078338623, 0.3298202455043793, 0.3371160328388214, 0.3162180483341217, 0.3364160656929016, 0.3440856337547302, 0.33859172463417053, 0.3409537672996521, 0.3220680058002472, 0.3370307981967926, 0.32390737533569336, 0.3329233229160309, 0.33171454071998596, 0.32725751399993896, 0.3364013135433197, 0.31892862915992737, 0.32104170322418213, 0.33342257142066956, 0.324175089597702, 0.3192666172981262, 0.32224178314208984, 0.33359402418136597, 0.32298949360847473, 0.32580050826072693, 0.3253556787967682, 0.32010194659233093, 0.32726871967315674, 0.31733739376068115, 0.34169745445251465, 0.3263677954673767, 0.31995689868927, 0.316768616437912, 0.32777139544487, 0.357673317193985, 0.3344147503376007, 0.32023680210113525, 0.34205177426338196, 0.33622756600379944, 0.3366716504096985, 0.31310343742370605, 0.33088022470474243, 0.3324989378452301, 0.3340208828449249, 0.33667221665382385, 0.33015182614326477, 0.3351299464702606, 0.3493722081184387, 0.35016506910324097, 0.31731393933296204, 0.3396014869213104, 0.33443284034729004, 0.3282649517059326, 0.32744741439819336, 0.3251930773258209, 0.3237573206424713, 0.33440178632736206, 0.31835541129112244, 0.3139073848724365, 0.33747008442878723, 0.3255719542503357, 0.3291328549385071, 0.3283381462097168, 0.326183021068573, 0.3443581759929657, 0.32772335410118103, 0.3262929916381836, 0.3197605013847351, 0.3139354884624481, 0.3296361267566681, 0.32264745235443115, 0.3289695084095001, 0.311590313911438, 0.3294873535633087, 0.3286621570587158, 0.33963173627853394, 0.32591935992240906, 0.3243439197540283, 0.346100777387619, 0.3155832886695862, 0.3194901943206787, 0.3262616991996765, 0.3232318162918091, 0.32922855019569397, 0.33100077509880066, 0.329399436712265, 0.32703420519828796, 0.31251150369644165, 0.3232514262199402, 0.325417697429657, 0.32872843742370605, 0.32970234751701355, 0.325783371925354, 0.32775282859802246, 0.3299684226512909, 0.3423644006252289, 0.31883227825164795, 0.3266981244087219, 0.32093918323516846, 0.32331332564353943, 0.3233243227005005, 0.33228716254234314, 0.31604820489883423, 0.3312731981277466, 0.32008159160614014, 0.3313177227973938, 0.33330607414245605, 0.32773950695991516, 0.323442667722702, 0.3311300277709961, 0.32210227847099304, 0.3235970437526703, 0.33470338582992554, 0.3385612666606903]
[0.3267229199409485, 0.32743632793426514, 0.3379046320915222, 0.3192898631095886, 0.3243163824081421, 0.31881511211395264, 0.33305877447128296, 0.3382623791694641, 0.31462612748146057, 0.3215361535549164, 0.323795348405838, 0.3384469747543335, 0.32207536697387695, 0.3416607677936554, 0.31890445947647095, 0.3318520486354828, 0.34684109687805176, 0.3353455662727356, 0.3477597236633301, 0.32024338841438293, 0.34402525424957275, 0.32256367802619934, 0.3309565782546997, 0.33743816614151, 0.3308350145816803, 0.3184194564819336, 0.33140555024147034, 0.31232574582099915, 0.30976834893226624, 0.3154352605342865, 0.3383341133594513, 0.3322511315345764, 0.3395189344882965, 0.31730926036834717, 0.33276185393333435, 0.3410451114177704, 0.3145223557949066, 0.3371468782424927, 0.34231001138687134, 0.3266512155532837, 0.3317577540874481, 0.3369251787662506, 0.33937856554985046, 0.3243086338043213, 0.33703064918518066, 0.3240451514720917, 0.32600873708724976, 0.3160380721092224, 0.32518136501312256, 0.34050101041793823, 0.31199783086776733, 0.32796674966812134, 0.3090508282184601, 0.3316683769226074, 0.3190543055534363, 0.3295072317123413, 0.32485708594322205, 0.32997167110443115, 0.3235652446746826, 0.33609041571617126, 0.3368695080280304, 0.3402504026889801, 0.33218008279800415, 0.3152736723423004, 0.32928070425987244, 0.32465416193008423, 0.3271172046661377, 0.3246077597141266, 0.3263513743877411, 0.3309141993522644, 0.3227037489414215, 0.32698488235473633, 0.34095796942710876, 0.3373194932937622, 0.32564684748649597, 0.3281288146972656, 0.34161245822906494, 0.3324119746685028, 0.3160533607006073, 0.3425150513648987, 0.3319264352321625, 0.33455589413642883, 0.3432563543319702, 0.32094091176986694, 0.3395601212978363, 0.3127990961074829, 0.32510772347450256, 0.3366890251636505, 0.32935991883277893, 0.31454864144325256, 0.3233325779438019, 0.32630276679992676, 0.31492000818252563, 0.3266668915748596, 0.3274754285812378, 0.3304188549518585, 0.33483192324638367, 0.33228811621665955, 0.349879652261734, 0.33940431475639343, 0.3177168369293213, 0.3212905526161194, 0.3304230570793152, 0.3346741795539856, 0.33448565006256104, 0.3424493372440338, 0.3224787712097168, 0.3284454345703125, 0.3369140028953552, 0.30988961458206177, 0.32718050479888916, 0.3204021155834198, 0.3266516327857971, 0.3354157507419586, 0.33314698934555054, 0.32606416940689087, 0.3144015073776245, 0.32933250069618225, 0.34145551919937134, 0.3194434940814972, 0.31916865706443787, 0.32313302159309387, 0.3259444236755371, 0.3320809304714203, 0.3194102644920349, 0.3517061769962311, 0.32393112778663635, 0.3093551695346832, 0.3187052607536316, 0.33517155051231384, 0.3415912091732025, 0.34153974056243896, 0.34034159779548645, 0.3147021234035492, 0.3277668356895447, 0.3327922224998474, 0.3261421322822571, 0.3345797657966614, 0.3185528516769409, 0.3284032940864563, 0.3316884934902191, 0.332044392824173, 0.32601290941238403, 0.3254467248916626, 0.324466735124588, 0.3143446743488312, 0.3280661106109619, 0.3198263645172119, 0.33755022287368774, 0.32410794496536255, 0.33001866936683655, 0.33350929617881775, 0.3437245190143585, 0.3209250569343567, 0.32605138421058655, 0.327309250831604, 0.3251086175441742, 0.33221352100372314, 0.3134590983390808, 0.309801310300827, 0.32777610421180725, 0.3143211901187897, 0.34568271040916443, 0.3289255201816559, 0.32307568192481995, 0.3394985795021057, 0.33417317271232605, 0.35315218567848206, 0.33393919467926025, 0.3232193887233734, 0.3336452543735504, 0.31885212659835815, 0.31463339924812317, 0.33677753806114197, 0.3339676260948181, 0.320152223110199, 0.3280581533908844, 0.3307224214076996, 0.332164466381073, 0.3410584032535553, 0.3176063895225525, 0.32713887095451355, 0.3278213441371918, 0.330533891916275, 0.33382999897003174, 0.32583391666412354, 0.31684643030166626, 0.32354992628097534, 0.3226381242275238, 0.3322112560272217, 0.31852445006370544, 0.3268941342830658, 0.32859161496162415, 0.31969207525253296, 0.34099286794662476, 0.32638439536094666, 0.315696120262146, 0.34351271390914917, 0.3354894518852234, 0.32553502917289734, 0.332570880651474, 0.3274179697036743, 0.3266271948814392, 0.3194597661495209, 0.31992536783218384, 0.33337461948394775, 0.32929253578186035, 0.3233630657196045, 0.33480459451675415, 0.31341734528541565, 0.3299284279346466, 0.3140990138053894, 0.32049453258514404, 0.3212701976299286, 0.3378102779388428, 0.32193851470947266, 0.327597975730896, 0.3306169807910919, 0.3273265063762665, 0.3364575207233429, 0.3323219120502472, 0.33423301577568054, 0.32122907042503357, 0.3457379937171936, 0.3297920227050781, 0.3203602731227875, 0.32442525029182434, 0.3364064693450928, 0.32620587944984436, 0.3200381398200989, 0.326015442609787, 0.3124906122684479, 0.334214448928833, 0.3377543091773987, 0.3333757519721985, 0.3231061100959778, 0.32188844680786133, 0.324896901845932, 0.32873085141181946, 0.31589481234550476, 0.3319867253303528, 0.3147903382778168, 0.3351294696331024, 0.320241242647171, 0.3409782946109772, 0.3226363956928253, 0.3367580771446228, 0.3292776048183441, 0.33287107944488525, 0.3366411626338959, 0.3345683813095093]
[0.33624929189682007, 0.3287142515182495, 0.3282497823238373, 0.3231293261051178, 0.3276967704296112, 0.3159177303314209, 0.3232215642929077, 0.31050199270248413, 0.33882132172584534, 0.31627219915390015, 0.328301340341568, 0.323494017124176, 0.33648091554641724, 0.3302154541015625, 0.32252809405326843, 0.325831800699234, 0.3291361629962921, 0.33038660883903503, 0.3243918716907501, 0.3314204514026642, 0.31982478499412537, 0.31441012024879456, 0.31564193964004517, 0.32808980345726013, 0.3281863331794739, 0.324187695980072, 0.3282264471054077, 0.33002328872680664, 0.32661059498786926, 0.3380882441997528, 0.33419668674468994, 0.3230492174625397, 0.3348229229450226, 0.3574739694595337, 0.33011552691459656, 0.3326999843120575, 0.3287033438682556, 0.3321249186992645, 0.31782105565071106, 0.32974880933761597, 0.31068962812423706, 0.3354730010032654, 0.33830124139785767, 0.3275121748447418, 0.3350403904914856, 0.3289044201374054, 0.32169339060783386, 0.3415501117706299, 0.3231006860733032, 0.3376041054725647, 0.331894189119339, 0.3367205560207367, 0.3307500183582306, 0.32555049657821655, 0.32826390862464905, 0.32624563574790955, 0.336374968290329, 0.3409147262573242, 0.33605843782424927, 0.33033502101898193, 0.33736634254455566, 0.31489112973213196, 0.3447684347629547, 0.3127114474773407, 0.3219720423221588, 0.3324788510799408, 0.33304640650749207, 0.3233388662338257, 0.344908207654953, 0.32772520184516907, 0.3202908933162689, 0.3341575860977173, 0.3242793679237366, 0.31357690691947937, 0.342279851436615, 0.3376206159591675, 0.33145490288734436, 0.3231625556945801, 0.34458813071250916, 0.35161229968070984, 0.3341100811958313, 0.31792309880256653, 0.3350130617618561, 0.3269343078136444, 0.3274449110031128, 0.3332025110721588, 0.33890464901924133, 0.3310836851596832, 0.3260648846626282, 0.3320920169353485, 0.31796830892562866, 0.3316657841205597, 0.32678696513175964, 0.3243063986301422, 0.3389805853366852, 0.3378528952598572, 0.32963958382606506, 0.3291800022125244, 0.3229588270187378, 0.3158859610557556, 0.33378612995147705, 0.32980960607528687, 0.3314034640789032, 0.3495618999004364, 0.34350207448005676, 0.33728474378585815, 0.3223113715648651, 0.3273019790649414, 0.3375053107738495, 0.3219207525253296, 0.31646591424942017, 0.3290528655052185, 0.3236597180366516, 0.33656299114227295, 0.31681203842163086, 0.33121103048324585, 0.3391268849372864, 0.3329790234565735, 0.3316926658153534, 0.3406834602355957, 0.3411814272403717, 0.3304559290409088, 0.3344014883041382, 0.31848758459091187, 0.3220844864845276, 0.3284826874732971, 0.3178800642490387, 0.3257116675376892, 0.3284456729888916, 0.3307386338710785, 0.3249015510082245, 0.3346707224845886, 0.3350428342819214, 0.3272017538547516, 0.3285459578037262, 0.3333998918533325, 0.3338611423969269, 0.3345821499824524, 0.32539471983909607, 0.32254770398139954, 0.3264961838722229, 0.33045777678489685, 0.3462536633014679, 0.33078157901763916, 0.32030102610588074, 0.3491971492767334, 0.3487035036087036, 0.3317318260669708, 0.3164506256580353, 0.3425511419773102, 0.3261355459690094, 0.3322986662387848, 0.3051031827926636, 0.31799325346946716, 0.3363778293132782, 0.3247794210910797, 0.32316914200782776, 0.31646087765693665, 0.32941049337387085, 0.3170713484287262, 0.32127413153648376, 0.3235956132411957, 0.33041346073150635, 0.3275195360183716, 0.3408852219581604, 0.33364754915237427, 0.3346582055091858, 0.3150944411754608, 0.3334123492240906, 0.32507532835006714, 0.3130708932876587, 0.3386007249355316, 0.3217112720012665, 0.33478912711143494, 0.3175964057445526, 0.33688923716545105, 0.330964058637619, 0.3255104124546051, 0.33298707008361816, 0.3214297592639923, 0.3196769952774048, 0.3227819800376892, 0.3238556385040283, 0.32693472504615784, 0.3445647656917572, 0.3323732614517212, 0.32003507018089294, 0.33475562930107117, 0.3320826590061188, 0.3399401903152466, 0.3327116072177887, 0.3162004351615906, 0.3291887044906616, 0.3206031620502472, 0.32628875970840454, 0.3226355314254761, 0.32289108633995056, 0.33406588435173035, 0.3300870954990387, 0.3349706530570984, 0.3140355944633484, 0.32819029688835144, 0.31147170066833496, 0.3432897627353668, 0.3340785503387451, 0.3283796012401581, 0.3312298357486725, 0.33994895219802856, 0.322643518447876, 0.3271510601043701, 0.3148044943809509, 0.3369317650794983, 0.3361482620239258, 0.31631866097450256, 0.3364940285682678, 0.331266850233078, 0.3319823145866394, 0.32205238938331604, 0.3201162815093994, 0.325533926486969, 0.3220116198062897, 0.33426058292388916, 0.32188719511032104, 0.32607758045196533, 0.3368567228317261, 0.3341996967792511, 0.33817270398139954, 0.3485211133956909, 0.31580621004104614, 0.3350232243537903, 0.33642539381980896, 0.3443624675273895, 0.33930182456970215, 0.3162977993488312, 0.3302444815635681, 0.33965036273002625, 0.33180543780326843, 0.3386842906475067, 0.3192340135574341, 0.3184114396572113, 0.33170008659362793, 0.34038981795310974, 0.32190370559692383, 0.32587191462516785, 0.3249591290950775, 0.3265175223350525, 0.32746967673301697, 0.33073827624320984, 0.32750192284584045, 0.3206477761268616, 0.32076528668403625]
[0.33303067088127136, 0.3494284152984619, 0.33405354619026184, 0.3282312750816345, 0.3211935758590698, 0.33927488327026367, 0.327896386384964, 0.34072694182395935, 0.3367060124874115, 0.32884159684181213, 0.33350279927253723, 0.32315438985824585, 0.34809088706970215, 0.3294481337070465, 0.33831289410591125, 0.31815847754478455, 0.33279183506965637, 0.3479308784008026, 0.3168991208076477, 0.33735811710357666, 0.3396143913269043, 0.3400998115539551, 0.3414795994758606, 0.3205634355545044, 0.3740873336791992, 0.3353399336338043, 0.3305051922798157, 0.3272343575954437, 0.3491530120372772, 0.3284094035625458, 0.3373633325099945, 0.329039990901947, 0.3336949944496155, 0.32058975100517273, 0.333803653717041, 0.34136584401130676, 0.3283222019672394, 0.333801805973053, 0.3265749514102936, 0.3335794508457184, 0.3308466374874115, 0.32078322768211365, 0.31576845049858093, 0.3423018753528595, 0.32843002676963806, 0.3371984660625458, 0.3320159614086151, 0.33756551146507263, 0.33608195185661316, 0.3506498634815216, 0.3427160680294037]

I'm trying to find which setup is the best, which I'll take as the setup with the lowest median. To discard bad sets of parameters, I need to somehow tell if a list is not likely to be the best.

(I chose the median because it isn't affected by outliers as much as the mean, although I would be alright to use the mean instead.)

Which brings me to my question:

How do I find the list with the highest probability of having the lowest median? And how can I use this to discard bad setups early on?


Where I'm at so far

I've found the median and the IQR for each list, but I'm not sure of where to go from there. The probability that one list has a lower median will surely depend on how many values there are in the list, and neither the median or the IQR contains that information.

Update: I've also tried modelling each list with a normal distribution, then using Welch's t-test to find the probability that one mean is lower than the other, although the distribution of values in the lists doesn't seem to be a normal distribution so it sometimes underestimates the probability massively and causes the best candidate to be removed. For example, after ~50 simulations it gave one set of parameters a 0.1% chance of being the best, which later changed to 40% after ~200 simulations.

Here are some of the distributions of simulation results, each graph is for different setup parameters for the simulation.

Distribution 1 Distribution 2 Distribution 3 Distribution 4

Thanks! -Rlz

$\endgroup$
10
  • $\begingroup$ I've followed everything in the 'ask new questions' guide, and also checked it against math.meta.stackexchange.com/questions/34059/…. If there's anything I can do to improve my question please let me know! $\endgroup$
    – Rlz
    May 27, 2022 at 13:34
  • 4
    $\begingroup$ (I am not the downvoter). How does the question arise ? We know with certainty what the median of each list is. Why probability/likely ? $\endgroup$
    – Kurt G.
    May 27, 2022 at 13:46
  • $\begingroup$ @KurtG. Thanks for the comment. Each time I run a simulation I get a different value, and I need to find what setup gives the best simulation results on average (hence the median). The more simulations I run, the more accurate the median calculated is, but the longer it takes. So I'd like to find the probability that a simulation setup is the best, so I can remove bad setups early on, and speed up the process significantly! $\endgroup$
    – Rlz
    May 27, 2022 at 13:49
  • 2
    $\begingroup$ Can you divulge how the simulations are produced and what role the parameters play in such? It might be possible to get approximations to the density as a function of the parameters and then get approximations for the median as a function of the parameters. Because, obviously, if you had an exact density $f(x, \theta)$ we would just be after $\min_\theta m(\theta)$ assuming the median depends on the parameters. Depending on how the simulation is made, one might be able to find a heuristic function that approximates the density. $\endgroup$ May 27, 2022 at 15:17
  • 1
    $\begingroup$ @Rlz Thanks for the context. You have multiple groups to be dealing with here, so it gets a bit harder to quantify. One place I'd start with is a non-parametric ANOVA, such as the Kruskal-Wallis test. $\endgroup$ May 30, 2022 at 1:56

0

You must log in to answer this question.

Browse other questions tagged .