Near optimal step size and momentum in gradient descent for quadratic functions

Authors

ENGİN TAŞ
MEMMEDAĞA MEMMEDLİ

DOI

10.3906/mat-1411-51

Abstract

Many problems in statistical estimation, classification, and regression can be cast as optimization problems. Gradient descent, which is one of the simplest and easy to implement multivariate optimization techniques, lies at the heart of many powerful classes of optimization methods. However, its major disadvantage is the slower rate of convergence with respect to the other more sophisticated algorithms. In order to improve the convergence speed of gradient descent, we simultaneously determine near-optimal scalar step size and momentum factor for gradient descent in a deterministic quadratic bowl from the largest and smallest eigenvalues of the Hessian. The resulting algorithm is demonstrated on specific and randomly generated test problems and it converges faster than any previous batch gradient descent method.

Keywords

Gradient descent, step size, momentum, convergence speed, stability

First Page

110

Last Page

121

Recommended Citation

TAŞ, ENGİN and MEMMEDLİ, MEMMEDAĞA (2017) "Near optimal step size and momentum in gradient descent for quadratic functions," Turkish Journal of Mathematics: Vol. 41: No. 1, Article 11. https://doi.org/10.3906/mat-1411-51
Available at: https://journals.tubitak.gov.tr/math/vol41/iss1/11