Alec Radford’s animations for optimization algorithms[FW]

Alec Radford has created some great animations comparing optimization algorithms SGDMomentumNAGAdagradAdadeltaRMSprop (unfortunately no Adam) on low dimensional problems. Also check out his presentation on RNNs.

Noisy moons: This is logistic regression on noisy moons dataset from sklearn which shows the smoothing effects of momentum based techniques (which also results in over shooting and correction). The error surface is visualized as an average over the whole dataset empirically, but the trajectories show the dynamics of minibatches on noisy data. The bottom chart is an accuracy plot.”

Beale’s function: Due to the large initial gradient, velocity based techniques shoot off and bounce around – adagrad almost goes unstable for the same reason. Algos that scale gradients/step sizes like adadelta and RMSProp proceed more like accelerated SGD and handle large gradients with more stability.”

Long valley: Algos without scaling based on gradient information really struggle to break symmetry here – SGD gets no where and Nesterov Accelerated Gradient / Momentum exhibits oscillations until they build up velocity in the optimization direction. Algos that scale step size based on the gradient quickly break symmetry and begin descent.”

Saddle point: Behavior around a saddle point. NAG/Momentum again like to explore around, almost taking a different path. Adadelta/Adagrad/RMSProp proceed like accelerated SGD.”

Source Link: http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

☆版权☆

* 网站名称:obaby@mars
* 网址:https://lang.ma/
* 个性:https://oba.by/
* 本文标题: 《Alec Radford’s animations for optimization algorithms[FW]》
* 本文链接:https://danteng.me/2020/02/6966
* 短链接:https://oba.by/?p=6966
* 转载文章请标明文章来源,原文标题以及原文链接。请遵从 《署名-非商业性使用-相同方式共享 2.5 中国大陆 (CC BY-NC-SA 2.5 CN) 》许可协议。


You may also like

发表回复

您的电子邮箱地址不会被公开。 必填项已用 * 标注