Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests
<p>Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , |
| منشور في: |
2023
|
| الموضوعات: | |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1864513528421941248 |
|---|---|
| author | Kenneth V. Price (17877002) |
| author2 | Abhishek Kumar (156479) P.N. Suganthan (16518528) |
| author2_role | author author |
| author_facet | Kenneth V. Price (17877002) Abhishek Kumar (156479) P.N. Suganthan (16518528) |
| author_role | author |
| dc.creator.none.fl_str_mv | Kenneth V. Price (17877002) Abhishek Kumar (156479) P.N. Suganthan (16518528) |
| dc.date.none.fl_str_mv | 2023-04-01T00:00:00Z |
| dc.identifier.none.fl_str_mv | 10.1016/j.swevo.2023.101287 |
| dc.relation.none.fl_str_mv | https://figshare.com/articles/journal_contribution/Trial-based_dominance_for_comparing_both_the_speed_and_accuracy_of_stochastic_optimizers_with_standard_non-parametric_tests/25117088 |
| dc.rights.none.fl_str_mv | CC BY 4.0 info:eu-repo/semantics/openAccess |
| dc.subject.none.fl_str_mv | Mathematical sciences Applied mathematics Benchmarking Two-variable non-parametric tests Evolutionary algorithms Dominance Stochastic optimization Numerical optimization Mann-Whitney test |
| dc.title.none.fl_str_mv | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| dc.type.none.fl_str_mv | Text Journal contribution info:eu-repo/semantics/publishedVersion text contribution to journal |
| description | <p>Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both the time that a trial takes to reach the target value (or not) and its final fitness value characterize its outcome. This paper describes how trial-based dominance can totally order this two-variable dataset of outcomes so that traditional non-parametric methods can determine the better of two algorithms when one is faster, but less accurate than the other, i.e. when neither algorithm dominates. After describing trial-based dominance, we outline its benefits. We subsequently review other attempts to compare stochastic optimizers, before illustrating our method with the Mann-Whitney U test. Simulations demonstrate that “U-scores” are much more effective than dominance when tasked with identifying the better of two algorithms. We validate U-scores by having them determine the winners of the CEC 2022 competition on single objective, bound-constrained numerical optimization.</p><h2>Other Information</h2> <p> Published in: Swarm and Evolutionary Computation<br> License: <a href="http://creativecommons.org/licenses/by/4.0/" target="_blank">http://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1016/j.swevo.2023.101287" target="_blank">https://dx.doi.org/10.1016/j.swevo.2023.101287</a></p> |
| eu_rights_str_mv | openAccess |
| id | Manara2_5d301428a933a31830cd6f542c71f032 |
| identifier_str_mv | 10.1016/j.swevo.2023.101287 |
| network_acronym_str | Manara2 |
| network_name_str | Manara2 |
| oai_identifier_str | oai:figshare.com:article/25117088 |
| publishDate | 2023 |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | CC BY 4.0 |
| spelling | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric testsKenneth V. Price (17877002)Abhishek Kumar (156479)P.N. Suganthan (16518528)Mathematical sciencesApplied mathematicsBenchmarkingTwo-variable non-parametric testsEvolutionary algorithmsDominanceStochastic optimizationNumerical optimizationMann-Whitney test<p>Non-parametric tests can determine the better of two stochastic optimization algorithms when benchmarking results are ordinal—like the final fitness values of multiple trials—but for many benchmarks, a trial can also terminate once it reaches a prespecified target value. In such cases, both the time that a trial takes to reach the target value (or not) and its final fitness value characterize its outcome. This paper describes how trial-based dominance can totally order this two-variable dataset of outcomes so that traditional non-parametric methods can determine the better of two algorithms when one is faster, but less accurate than the other, i.e. when neither algorithm dominates. After describing trial-based dominance, we outline its benefits. We subsequently review other attempts to compare stochastic optimizers, before illustrating our method with the Mann-Whitney U test. Simulations demonstrate that “U-scores” are much more effective than dominance when tasked with identifying the better of two algorithms. We validate U-scores by having them determine the winners of the CEC 2022 competition on single objective, bound-constrained numerical optimization.</p><h2>Other Information</h2> <p> Published in: Swarm and Evolutionary Computation<br> License: <a href="http://creativecommons.org/licenses/by/4.0/" target="_blank">http://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1016/j.swevo.2023.101287" target="_blank">https://dx.doi.org/10.1016/j.swevo.2023.101287</a></p>2023-04-01T00:00:00ZTextJournal contributioninfo:eu-repo/semantics/publishedVersiontextcontribution to journal10.1016/j.swevo.2023.101287https://figshare.com/articles/journal_contribution/Trial-based_dominance_for_comparing_both_the_speed_and_accuracy_of_stochastic_optimizers_with_standard_non-parametric_tests/25117088CC BY 4.0info:eu-repo/semantics/openAccessoai:figshare.com:article/251170882023-04-01T00:00:00Z |
| spellingShingle | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests Kenneth V. Price (17877002) Mathematical sciences Applied mathematics Benchmarking Two-variable non-parametric tests Evolutionary algorithms Dominance Stochastic optimization Numerical optimization Mann-Whitney test |
| status_str | publishedVersion |
| title | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_full | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_fullStr | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_full_unstemmed | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_short | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| title_sort | Trial-based dominance for comparing both the speed and accuracy of stochastic optimizers with standard non-parametric tests |
| topic | Mathematical sciences Applied mathematics Benchmarking Two-variable non-parametric tests Evolutionary algorithms Dominance Stochastic optimization Numerical optimization Mann-Whitney test |