Self-Distillation for Randomized Neural Networks
<p dir="ltr">Knowledge distillation (KD) is a conventional method in the field of deep learning that enables the transfer of dark knowledge from a teacher model to a student model, consequently improving the performance of the student model. In randomized neural networks, due to the...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , |
| منشور في: |
2023
|
| الموضوعات: | |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1864513527006363648 |
|---|---|
| author | Minghui Hu (2457952) |
| author2 | Ruobin Gao (16003195) Ponnuthurai Nagaratnam Suganthan (11274636) |
| author2_role | author author |
| author_facet | Minghui Hu (2457952) Ruobin Gao (16003195) Ponnuthurai Nagaratnam Suganthan (11274636) |
| author_role | author |
| dc.creator.none.fl_str_mv | Minghui Hu (2457952) Ruobin Gao (16003195) Ponnuthurai Nagaratnam Suganthan (11274636) |
| dc.date.none.fl_str_mv | 2023-08-16T09:00:00Z |
| dc.identifier.none.fl_str_mv | 10.1109/tnnls.2023.3292063 |
| dc.relation.none.fl_str_mv | https://figshare.com/articles/journal_contribution/Self-Distillation_for_Randomized_Neural_Networks/25243387 |
| dc.rights.none.fl_str_mv | CC BY 4.0 info:eu-repo/semantics/openAccess |
| dc.subject.none.fl_str_mv | Information and computing sciences Artificial intelligence Biological neural networks Knowledge engineering Neurons Training Pipelines Closed-form solutions Predictive models Knowledge distillation (KD) random vector functional link (RVFL) randomized neural network self-distillation |
| dc.title.none.fl_str_mv | Self-Distillation for Randomized Neural Networks |
| dc.type.none.fl_str_mv | Text Journal contribution info:eu-repo/semantics/publishedVersion text contribution to journal |
| description | <p dir="ltr">Knowledge distillation (KD) is a conventional method in the field of deep learning that enables the transfer of dark knowledge from a teacher model to a student model, consequently improving the performance of the student model. In randomized neural networks, due to the simple topology of network architecture and the insignificant relationship between model performance and model size, KD is not able to improve model performance. In this work, we propose a self-distillation pipeline for randomized neural networks: the predictions of the network itself are regarded as the additional target, which are mixed with the weighted original target as a distillation target containing dark knowledge to supervise the training of the model. All the predictions during multi-generation self-distillation process can be integrated by a multi-teacher method. By induction, we have additionally arrived at the methods for infinite self-distillation (ISD) of randomized neural networks. We then provide relevant theoretical analysis about the self-distillation method for randomized neural networks. Furthermore, we demonstrated the effectiveness of the proposed method in practical applications on several benchmark datasets.</p><h2>Other Information</h2><p dir="ltr">Published in: IEEE Transactions on Neural Networks and Learning Systems<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1109/tnnls.2023.3292063" target="_blank">https://dx.doi.org/10.1109/tnnls.2023.3292063</a></p> |
| eu_rights_str_mv | openAccess |
| id | Manara2_e3bbfbb0ac09bb3bd7fe43247b43c1de |
| identifier_str_mv | 10.1109/tnnls.2023.3292063 |
| network_acronym_str | Manara2 |
| network_name_str | Manara2 |
| oai_identifier_str | oai:figshare.com:article/25243387 |
| publishDate | 2023 |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | CC BY 4.0 |
| spelling | Self-Distillation for Randomized Neural NetworksMinghui Hu (2457952)Ruobin Gao (16003195)Ponnuthurai Nagaratnam Suganthan (11274636)Information and computing sciencesArtificial intelligenceBiological neural networksKnowledge engineeringNeuronsTrainingPipelinesClosed-form solutionsPredictive modelsKnowledge distillation (KD)random vector functional link (RVFL)randomized neural networkself-distillation<p dir="ltr">Knowledge distillation (KD) is a conventional method in the field of deep learning that enables the transfer of dark knowledge from a teacher model to a student model, consequently improving the performance of the student model. In randomized neural networks, due to the simple topology of network architecture and the insignificant relationship between model performance and model size, KD is not able to improve model performance. In this work, we propose a self-distillation pipeline for randomized neural networks: the predictions of the network itself are regarded as the additional target, which are mixed with the weighted original target as a distillation target containing dark knowledge to supervise the training of the model. All the predictions during multi-generation self-distillation process can be integrated by a multi-teacher method. By induction, we have additionally arrived at the methods for infinite self-distillation (ISD) of randomized neural networks. We then provide relevant theoretical analysis about the self-distillation method for randomized neural networks. Furthermore, we demonstrated the effectiveness of the proposed method in practical applications on several benchmark datasets.</p><h2>Other Information</h2><p dir="ltr">Published in: IEEE Transactions on Neural Networks and Learning Systems<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1109/tnnls.2023.3292063" target="_blank">https://dx.doi.org/10.1109/tnnls.2023.3292063</a></p>2023-08-16T09:00:00ZTextJournal contributioninfo:eu-repo/semantics/publishedVersiontextcontribution to journal10.1109/tnnls.2023.3292063https://figshare.com/articles/journal_contribution/Self-Distillation_for_Randomized_Neural_Networks/25243387CC BY 4.0info:eu-repo/semantics/openAccessoai:figshare.com:article/252433872023-08-16T09:00:00Z |
| spellingShingle | Self-Distillation for Randomized Neural Networks Minghui Hu (2457952) Information and computing sciences Artificial intelligence Biological neural networks Knowledge engineering Neurons Training Pipelines Closed-form solutions Predictive models Knowledge distillation (KD) random vector functional link (RVFL) randomized neural network self-distillation |
| status_str | publishedVersion |
| title | Self-Distillation for Randomized Neural Networks |
| title_full | Self-Distillation for Randomized Neural Networks |
| title_fullStr | Self-Distillation for Randomized Neural Networks |
| title_full_unstemmed | Self-Distillation for Randomized Neural Networks |
| title_short | Self-Distillation for Randomized Neural Networks |
| title_sort | Self-Distillation for Randomized Neural Networks |
| topic | Information and computing sciences Artificial intelligence Biological neural networks Knowledge engineering Neurons Training Pipelines Closed-form solutions Predictive models Knowledge distillation (KD) random vector functional link (RVFL) randomized neural network self-distillation |