Global Convergence of SGD On Two Layer Neural Nets

Pulkit Gopalani; Anirbit Mukherjee

doi:10.1093/imaiai/iaae035

Global Convergence of SGD On Two Layer Neural Nets

Pulkit Gopalani, Anirbit Mukherjee^*

^*Corresponding author for this work

Machine Learning and Robotics

University of Michigan

Research output: Contribution to journal › Article › peer-review

Abstract

In this note, we consider appropriately regularized ℓ2−empirical risk of depth 2 nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it – for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are “Villani functions” and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.

Original language	English
Number of pages	22
Journal	Information and Inference: a Journal of the IMA
DOIs	https://doi.org/10.1093/imaiai/iaae035
Publication status	Published - 20 Jan 2025

Keywords

deep-learning
stochastic optimization
stochastic differential equations
functional analysis

Research Beacons, Institutes and Platforms

Institute for Data Science and AI
Christabel Pankhurst Institute

Access to Document

10.1093/imaiai/iaae035Licence: CC BY

https://arxiv.org/abs/2210.11452Licence: CC BY

1 Active

MCAIF: Centre for AI Fundamentals
Kaski, S. (PI), Alvarez, M. (Researcher), Pan, W. (Researcher), Mu, T. (Researcher), Rivasplata, O. (PI), Sun, M. (PI), Mukherjee, A. (PI), Caprio, M. (PI), Sonee, A. (Researcher), Leroy, A. (Researcher), Wang, J. (Researcher), Lee, J. (Researcher), Parakkal Unni, M. (Researcher), Sloman, S. (Researcher), Menary, S. (Researcher), Quilter, T. (Researcher), Hosseinzadeh, A. (PGR student), Mousa, A. (PGR student), Glover, L. (PGR student), Das, A. (PGR student), DURSUN, F. (PGR student), Zhu, H. (PGR student), Abdi, H. (PGR student), Dandago, K. (PGR student), Piriyajitakonkij, M. (PGR student), Rachman, R. (PGR student), Shi, X. (PGR student), Keany, T. (PGR student), Liu, X. (PGR student), Jiang, Y. (PGR student), Wan, Z. (PGR student), Harrison, M. (Support team), Machado, M. (Support team), Hartford, J. (PI), Kangin, D. (Researcher), Harikumar, H. (PI), Dubey, M. (PI), Parakkal Unni, M. (PI), Dash, S. P. (PGR student), Mi, X. (PGR student) & Barlas, Y. (PGR student)
1/10/21 → 30/09/26
Project: Research

1 Article
1 Preprint

Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data
Kumar, D., Jha, S. & Mukherjee, A., 14 Mar 2025, (In preparation) arXiv.org.
Research output: Preprint/Working paper › Preprint
Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets
Gopalani, P., Jha, S. & Mukherjee, A., 25 Feb 2024, In: Transactions on Machine Learning Research . 21 p.
Research output: Contribution to journal › Article › peer-review

Open Access
File

2 Invited talk

Provable Training of Neural Nets With One Layer of Activation
Mukherjee, A. (Speaker)
8 Mar 2023
Activity: Talk or presentation › Invited talk › Research
Provable Training of Neural Nets With One Layer of Activation
Mukherjee, A. (Speaker)
19 Jan 2023
Activity: Talk or presentation › Invited talk › Research

Cite this

@article{d3ba0ea4659742ef8fa5ab1f2f817d1a,

title = "Global Convergence of SGD On Two Layer Neural Nets",

abstract = "In this note, we consider appropriately regularized ℓ2−empirical risk of depth 2 nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it – for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are “Villani functions” and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.",

keywords = "deep-learning, stochastic optimization, stochastic differential equations, functional analysis",

author = "Pulkit Gopalani and Anirbit Mukherjee",

year = "2025",

month = jan,

day = "20",

doi = "10.1093/imaiai/iaae035",

language = "English",

journal = "Information and Inference: a Journal of the IMA",

issn = "2049-8764",

publisher = "Oxford University Press",

}

TY - JOUR

T1 - Global Convergence of SGD On Two Layer Neural Nets

AU - Gopalani, Pulkit

AU - Mukherjee, Anirbit

PY - 2025/1/20

Y1 - 2025/1/20

N2 - In this note, we consider appropriately regularized ℓ2−empirical risk of depth 2 nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it – for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are “Villani functions” and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.

AB - In this note, we consider appropriately regularized ℓ2−empirical risk of depth 2 nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it – for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are “Villani functions” and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.

KW - deep-learning

KW - stochastic optimization

KW - stochastic differential equations

KW - functional analysis

U2 - 10.1093/imaiai/iaae035

DO - 10.1093/imaiai/iaae035

M3 - Article

SN - 2049-8764

JO - Information and Inference: a Journal of the IMA

JF - Information and Inference: a Journal of the IMA

ER -

Global Convergence of SGD On Two Layer Neural Nets

Abstract

Keywords

Research Beacons, Institutes and Platforms

Access to Document

Fingerprint

Projects

MCAIF: Centre for AI Fundamentals

Research output

Langevin Monte-Carlo Provably Learns Depth Two Neural Nets at Any Size and Data

Global Convergence of SGD For Logistic Loss on Two Layer Neural Nets

Activities

Provable Training of Neural Nets With One Layer of Activation

Provable Training of Neural Nets With One Layer of Activation

Cite this