Reusing Weights in Subword-Aware Neural Language Models

Abstract

The authors introduce methods for reusing subword embeddings and other parameters in subword-aware neural language models. Techniques improve syllable- and morpheme-aware models' performance while greatly reducing model size. A practical principle is identified: when reusing embedding layers at the output, they should be tied consecutively from bottom up. The best morpheme-aware model significantly outperforms word-level baselines across languages with 20–87 % fewer parameters.