Explorations on chaotic behaviors of Recurrent Neural Networks
Loading...
Date
2019-04-29
Authors
Myrzakhmetov, Bagdat
Journal Title
Journal ISSN
Volume Title
Publisher
Nazarbayev University School of Science and Technology
Abstract
In this thesis work we analyzed the dynamics of the Recurrent Neural Network architectures.
We explored the chaotic nature of state-of-the-art Recurrent Neural Networks:
Vanilla Recurrent Network, Recurrent Highway Networks and Structurally
Constrained Recurrent Network. Our experiments showed that they exhibit chaotic
behavior in the absence of input data. We also proposed a way of removing chaos
chaos from Recurrent Neural Networks. Our findings show that initialization of the
weight matrices during the training plays an important role, as initialization with
the matrices whose norm is smaller than one will lead to the non-chaotic behavior
of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable
dynamics. At the end, we tested our chaos-free version of the Recurrent Highway
Networks (RHN) in a real-world application.
In a sequence-to-sequence modeling experiments, particularly in the language
modeling task, chaos-free version of RHN perform on par with the original version by
using the same hyperparameters.
Description
Submitted to the Department of Mathematics on Apr 29, 2019, in partial fulfillment of the
requirements for the degree of Master of Science in Applied Mathematics
Keywords
Research Subject Categories::MATHEMATICS::Applied mathematics