Entropy, Free Energy, and Work of Restricted Boltzmann Machines
<p dir="ltr">A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy func...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , |
| منشور في: |
2020
|
| الموضوعات: | |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1864513511346929664 |
|---|---|
| author | Sangchul Oh (10980923) |
| author2 | Abdelkader Baggag (16864140) Hyunchul Nha (17871053) |
| author2_role | author author |
| author_facet | Sangchul Oh (10980923) Abdelkader Baggag (16864140) Hyunchul Nha (17871053) |
| author_role | author |
| dc.creator.none.fl_str_mv | Sangchul Oh (10980923) Abdelkader Baggag (16864140) Hyunchul Nha (17871053) |
| dc.date.none.fl_str_mv | 2020-05-11T09:00:00Z |
| dc.identifier.none.fl_str_mv | 10.3390/e22050538 |
| dc.relation.none.fl_str_mv | https://figshare.com/articles/journal_contribution/Entropy_Free_Energy_and_Work_of_Restricted_Boltzmann_Machines/26176837 |
| dc.rights.none.fl_str_mv | CC BY 4.0 info:eu-repo/semantics/openAccess |
| dc.subject.none.fl_str_mv | Information and computing sciences Machine learning Software engineering Mathematical sciences Statistics restricted Boltzmann machines entropy subadditivity of entropy Jarzynski equality machine learning |
| dc.title.none.fl_str_mv | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| dc.type.none.fl_str_mv | Text Journal contribution info:eu-repo/semantics/publishedVersion text contribution to journal |
| description | <p dir="ltr">A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate thermodynamic quantities such as entropy, free energy, and internal energy as a function of the training epoch. We demonstrate the growth of the correlation between the visible and hidden layers via the subadditivity of entropies as the training proceeds. Using the Monte-Carlo simulation of trajectories of the visible and hidden vectors in the configuration space, we also calculate the distribution of the work done on the restricted Boltzmann machine by switching the parameters of the energy function. We discuss the Jarzynski equality which connects the path average of the exponential function of the work and the difference in free energies before and after training.</p><h2>Other Information</h2><p dir="ltr">Published in: Entropy<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.3390/e22050538" target="_blank">https://dx.doi.org/10.3390/e22050538</a></p><p dir="ltr">Additional institutions affiliated with: Arts and Sciences Program - TAMUQ</p> |
| eu_rights_str_mv | openAccess |
| id | Manara2_b8f22daa17cef3476163cb43947cd983 |
| identifier_str_mv | 10.3390/e22050538 |
| network_acronym_str | Manara2 |
| network_name_str | Manara2 |
| oai_identifier_str | oai:figshare.com:article/26176837 |
| publishDate | 2020 |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | CC BY 4.0 |
| spelling | Entropy, Free Energy, and Work of Restricted Boltzmann MachinesSangchul Oh (10980923)Abdelkader Baggag (16864140)Hyunchul Nha (17871053)Information and computing sciencesMachine learningSoftware engineeringMathematical sciencesStatisticsrestricted Boltzmann machinesentropysubadditivity of entropyJarzynski equalitymachine learning<p dir="ltr">A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. Given training data, its learning is done by optimizing the parameters of the energy function of the network. In this paper, we analyze the training process of the restricted Boltzmann machine in the context of statistical physics. As an illustration, for small size bar-and-stripe patterns, we calculate thermodynamic quantities such as entropy, free energy, and internal energy as a function of the training epoch. We demonstrate the growth of the correlation between the visible and hidden layers via the subadditivity of entropies as the training proceeds. Using the Monte-Carlo simulation of trajectories of the visible and hidden vectors in the configuration space, we also calculate the distribution of the work done on the restricted Boltzmann machine by switching the parameters of the energy function. We discuss the Jarzynski equality which connects the path average of the exponential function of the work and the difference in free energies before and after training.</p><h2>Other Information</h2><p dir="ltr">Published in: Entropy<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.3390/e22050538" target="_blank">https://dx.doi.org/10.3390/e22050538</a></p><p dir="ltr">Additional institutions affiliated with: Arts and Sciences Program - TAMUQ</p>2020-05-11T09:00:00ZTextJournal contributioninfo:eu-repo/semantics/publishedVersiontextcontribution to journal10.3390/e22050538https://figshare.com/articles/journal_contribution/Entropy_Free_Energy_and_Work_of_Restricted_Boltzmann_Machines/26176837CC BY 4.0info:eu-repo/semantics/openAccessoai:figshare.com:article/261768372020-05-11T09:00:00Z |
| spellingShingle | Entropy, Free Energy, and Work of Restricted Boltzmann Machines Sangchul Oh (10980923) Information and computing sciences Machine learning Software engineering Mathematical sciences Statistics restricted Boltzmann machines entropy subadditivity of entropy Jarzynski equality machine learning |
| status_str | publishedVersion |
| title | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| title_full | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| title_fullStr | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| title_full_unstemmed | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| title_short | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| title_sort | Entropy, Free Energy, and Work of Restricted Boltzmann Machines |
| topic | Information and computing sciences Machine learning Software engineering Mathematical sciences Statistics restricted Boltzmann machines entropy subadditivity of entropy Jarzynski equality machine learning |