An improved Adam algorithm using look-ahead
Document Type
Conference Proceeding
Publication Date
6-2-2017
Abstract
Adam is a state-of-art algorithm to optimize stochastic objective function. In this paper we proposed the Adam with Look-ahead (AWL), an updated version by applying look-ahead method with a hyperparameter. We firstly performed convergence analysis, showing that AWL has similar convergence properties as Adam. Then we conducted experiments to compare AWL with Adam on two models of logistic regression and two layers fully connected neural network. Results demonstrated that AWL outperforms the Adam with higher accuracy and less convergence time. Therefore, our newly proposed algorithm AWL may have great potential to be widely utilized in many fields of science and engineering.
Publication Title
ACM International Conference Proceeding Series
First Page Number
19
Last Page Number
22
DOI
10.1145/3094243.3094249
Recommended Citation
Zhu, An; Meng, Yu; and Zhang, Changjiang, "An improved Adam algorithm using look-ahead" (2017). Kean Publications. 1612.
https://digitalcommons.kean.edu/keanpublications/1612