In:
The Prague Bulletin of Mathematical Linguistics, Walter de Gruyter GmbH, Vol. 108, No. 1 ( 2017-06-1), p. 13-25
Abstract:
Training neural networks is a non-convex and a high-dimensional optimization problem. In this paper, we provide a comparative study of the most popular stochastic optimization techniques used to train neural networks. We evaluate the methods in terms of convergence speed, translation quality, and training stability. In addition, we investigate combinations that seek to improve optimization in terms of these aspects. We train state-of-the-art attention-based models and apply them to perform neural machine translation. We demonstrate our results on two tasks: WMT 2016 En→Ro and WMT 2015 De→En.
Type of Medium:
Online Resource
ISSN:
1804-0462
DOI:
10.1515/pralin-2017-0005
Language:
Unknown
Publisher:
Walter de Gruyter GmbH
Publication Date:
2017
detail.hit.zdb_id:
2589108-X
Permalink