An improved Adam algorithm using look-ahead

Document Type

Conference Proceeding

Publication Date

6-2-2017

Abstract

Adam is a state-of-art algorithm to optimize stochastic objective function. In this paper we proposed the Adam with Look-ahead (AWL), an updated version by applying look-ahead method with a hyperparameter. We firstly performed convergence analysis, showing that AWL has similar convergence properties as Adam. Then we conducted experiments to compare AWL with Adam on two models of logistic regression and two layers fully connected neural network. Results demonstrated that AWL outperforms the Adam with higher accuracy and less convergence time. Therefore, our newly proposed algorithm AWL may have great potential to be widely utilized in many fields of science and engineering.

Publication Title

ACM International Conference Proceeding Series

First Page Number

19

Last Page Number

22

DOI

10.1145/3094243.3094249

This document is currently not available here.

Share

COinS