Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications
<p dir="ltr">Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designe...
محفوظ في:
| المؤلف الرئيسي: | |
|---|---|
| مؤلفون آخرون: | , , , |
| منشور في: |
2019
|
| الموضوعات: | |
| الوسوم: |
إضافة وسم
لا توجد وسوم, كن أول من يضع وسما على هذه التسجيلة!
|
| _version_ | 1864513513587736576 |
|---|---|
| author | Rumen Dangovski (6658229) |
| author2 | Li Jing (445177) Preslav Nakov (17760905) Mićo Tatalović (18619243) Marin Soljačić (3961958) |
| author2_role | author author author author |
| author_facet | Rumen Dangovski (6658229) Li Jing (445177) Preslav Nakov (17760905) Mićo Tatalović (18619243) Marin Soljačić (3961958) |
| author_role | author |
| dc.creator.none.fl_str_mv | Rumen Dangovski (6658229) Li Jing (445177) Preslav Nakov (17760905) Mićo Tatalović (18619243) Marin Soljačić (3961958) |
| dc.date.none.fl_str_mv | 2019-04-01T03:00:00Z |
| dc.identifier.none.fl_str_mv | 10.1162/tacl_a_00258 |
| dc.relation.none.fl_str_mv | https://figshare.com/articles/journal_contribution/Rotational_Unit_of_Memory_A_Novel_Representation_Unit_for_RNNs_with_Scalable_Applications/25908373 |
| dc.rights.none.fl_str_mv | CC BY 4.0 info:eu-repo/semantics/openAccess |
| dc.subject.none.fl_str_mv | Information and computing sciences Artificial intelligence Machine learning Language modeling Text summarization Long-range dependencies Memory representation Rotational Unit of Memory (RUM) Unitary learning Associative memory |
| dc.title.none.fl_str_mv | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| dc.type.none.fl_str_mv | Text Journal contribution info:eu-repo/semantics/publishedVersion text contribution to journal |
| description | <p dir="ltr">Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art.</p><p><br></p><h2>Other Information</h2><p dir="ltr">Published in: Transactions of the Association for Computational Linguistics<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1162/tacl_a_00258" target="_blank">https://dx.doi.org/10.1162/tacl_a_00258</a></p> |
| eu_rights_str_mv | openAccess |
| id | Manara2_e99ca4ab715686fe31a81cb13e486301 |
| identifier_str_mv | 10.1162/tacl_a_00258 |
| network_acronym_str | Manara2 |
| network_name_str | Manara2 |
| oai_identifier_str | oai:figshare.com:article/25908373 |
| publishDate | 2019 |
| repository.mail.fl_str_mv | |
| repository.name.fl_str_mv | |
| repository_id_str | |
| rights_invalid_str_mv | CC BY 4.0 |
| spelling | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable ApplicationsRumen Dangovski (6658229)Li Jing (445177)Preslav Nakov (17760905)Mićo Tatalović (18619243)Marin Soljačić (3961958)Information and computing sciencesArtificial intelligenceMachine learningLanguage modelingText summarizationLong-range dependenciesMemory representationRotational Unit of Memory (RUM)Unitary learningAssociative memory<p dir="ltr">Stacking long short-term memory (LSTM) cells or gated recurrent units (GRUs) as part of a recurrent neural network (RNN) has become a standard approach to solving a number of tasks ranging from language modeling to text summarization. Although LSTMs and GRUs were designed to model long-range dependencies more accurately than conventional RNNs, they nevertheless have problems copying or recalling information from the long distant past. Here, we derive a phase-coded representation of the memory state, Rotational Unit of Memory (RUM), that unifies the concepts of unitary learning and associative memory. We show experimentally that RNNs based on RUMs can solve basic sequential tasks such as memory copying and memory recall much better than LSTMs/GRUs. We further demonstrate that by replacing LSTM/GRU with RUM units we can apply neural networks to real-world problems such as language modeling and text summarization, yielding results comparable to the state of the art.</p><p><br></p><h2>Other Information</h2><p dir="ltr">Published in: Transactions of the Association for Computational Linguistics<br>License: <a href="https://creativecommons.org/licenses/by/4.0/" target="_blank">https://creativecommons.org/licenses/by/4.0/</a><br>See article on publisher's website: <a href="https://dx.doi.org/10.1162/tacl_a_00258" target="_blank">https://dx.doi.org/10.1162/tacl_a_00258</a></p>2019-04-01T03:00:00ZTextJournal contributioninfo:eu-repo/semantics/publishedVersiontextcontribution to journal10.1162/tacl_a_00258https://figshare.com/articles/journal_contribution/Rotational_Unit_of_Memory_A_Novel_Representation_Unit_for_RNNs_with_Scalable_Applications/25908373CC BY 4.0info:eu-repo/semantics/openAccessoai:figshare.com:article/259083732019-04-01T03:00:00Z |
| spellingShingle | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications Rumen Dangovski (6658229) Information and computing sciences Artificial intelligence Machine learning Language modeling Text summarization Long-range dependencies Memory representation Rotational Unit of Memory (RUM) Unitary learning Associative memory |
| status_str | publishedVersion |
| title | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_full | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_fullStr | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_full_unstemmed | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_short | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| title_sort | Rotational Unit of Memory: A Novel Representation Unit for RNNs with Scalable Applications |
| topic | Information and computing sciences Artificial intelligence Machine learning Language modeling Text summarization Long-range dependencies Memory representation Rotational Unit of Memory (RUM) Unitary learning Associative memory |