Asynchronous SGD for DNN training on Shared-memory Parallel Architectures

Florent Lopez; Edmond Chow; Stanimire Tomov; Jack Dongarra

doi:10.1109/IPDPSW50202.2020.00168

Asynchronous SGD for DNN training on Shared-memory Parallel Architectures

Florent Lopez, Edmond Chow, Stanimire Tomov, Jack Dongarra

Department of Mathematics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.

Original language	English
Title of host publication	2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Pages	1-4
DOIs	https://doi.org/10.1109/IPDPSW50202.2020.00168
Publication status	E-pub ahead of print - 28 Jul 2020
Event	2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) - New Orleans, LA, USA Duration: 18 May 2020 → 22 May 2020

Conference

Conference	2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
Period	18/05/20 → 22/05/20

Access to Document

10.1109/IPDPSW50202.2020.00168

https://ieeexplore.ieee.org/document/9150336/

Cite this

@inproceedings{92930b32ec984ad09bc6a8383f0948f0,

title = "Asynchronous SGD for DNN training on Shared-memory Parallel Architectures",

abstract = "We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.",

author = "Florent Lopez and Edmond Chow and Stanimire Tomov and Jack Dongarra",

year = "2020",

month = jul,

day = "28",

doi = "10.1109/IPDPSW50202.2020.00168",

language = "English",

pages = "1--4",

booktitle = "2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)",

note = "2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) ; Conference date: 18-05-2020 Through 22-05-2020",

}

Lopez, F, Chow, E, Tomov, S & Dongarra, J 2020, Asynchronous SGD for DNN training on Shared-memory Parallel Architectures. in 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). pp. 1-4, 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 18/05/20. https://doi.org/10.1109/IPDPSW50202.2020.00168

TY - GEN

T1 - Asynchronous SGD for DNN training on Shared-memory Parallel Architectures

AU - Lopez, Florent

AU - Chow, Edmond

AU - Tomov, Stanimire

AU - Dongarra, Jack

PY - 2020/7/28

Y1 - 2020/7/28

N2 - We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.

AB - We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.

U2 - 10.1109/IPDPSW50202.2020.00168

DO - 10.1109/IPDPSW50202.2020.00168

M3 - Conference contribution

SP - 1

EP - 4

BT - 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

T2 - 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

Y2 - 18 May 2020 through 22 May 2020

ER -

Asynchronous SGD for DNN training on Shared-memory Parallel Architectures

Abstract

Conference

Access to Document

Fingerprint

Cite this