Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry

Jonas gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, David Rügamer

Research output: Chapter in Book/Conference proceedingChapterpeer-review

Abstract

Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. Local methods, which have emerged as a popular alternative, focus on specific parameter regions that can be approximated by functions with tractable integrals. While these often yield satisfactory empirical results, they fail, by definition, to account for the multi-modality of the parameter posterior. Such coarse approximations can be detrimental in practical applications, notably safety-critical ones. In this work, we argue that the dilemma between exact-but-unaffordable and cheap-but-inexact approaches can be mitigated by exploiting symmetries in the posterior landscape. These symmetries, induced by neuron interchangeability and certain activation functions, manifest in different parameter values leading to the same functional output value. We show theoretically that the posterior predictive density in Bayesian neural networks can be restricted to a symmetry-free parameter reference set. By further deriving an upper bound on the number of Monte Carlo chains required to capture the functional diversity, we propose a straightforward approach for feasible Bayesian inference. Our experiments suggest that efficient sampling is indeed possible, opening up a promising path to accurate uncertainty quantification in deep learning.
Original languageEnglish
Title of host publicationJoint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD 2023)
PublisherSpringer Nature
Chapter27
Pages459-474
DOIs
Publication statusPublished - 17 Sept 2023

Publication series

NameMachine Learning and Knowledge Discovery in Databases: Research Track
Volume14169
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Keywords

  • Uncertainty quantification
  • Predictive uncertainty
  • Bayesian inference
  • Monte Carlo sampling
  • Posterior symmetry

Fingerprint

Dive into the research topics of 'Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry'. Together they form a unique fingerprint.

Cite this