A statistical mechanics analysis of Gram matrix eigenvalue spectra

David C. Hoyle, Magnus Rattray

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    The Gram matrix plays a central role in many kernel methods. Knowledge about the distribution of eigenvalues of the Gram matrix is useful for developing appropriate model selection methods for kernel PCA. We use methods adapted from the statistical physics of classical fluids in order to study the averaged spectrum of the Gram matrix. We focus in particular on a variational mean-field theory and related diagrammatic approach. We show that the mean-field theory correctly reproduces previously obtained asymptotic results for standard PCA. Comparison with simulations for data distributed uniformly on the sphere shows that the method provides a good qualitative approximation to the averaged spectrum for kernel PCA with a Gaussian Radial Basis Function kernel. We also develop an analytical approximation to the spectral density that agrees closely with the numerical solution and provides insight into the number of samples required to resolve the corresponding process eigenvalues of a given order.
    Original languageEnglish
    Title of host publicationLecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science)|Lect Notes Artif Intell
    EditorsJ. Shawe-Taylor, Y. Singer
    PublisherSpringer Nature
    Number of pages14
    Publication statusPublished - 2004
    Event17th Annual Conference on Learning Theory, COLT 2004 - Banff
    Duration: 1 Jul 2004 → …

    Publication series

    NameLecture Notes in Computer Science


    Conference17th Annual Conference on Learning Theory, COLT 2004
    Period1/07/04 → …
    Internet address


    Dive into the research topics of 'A statistical mechanics analysis of Gram matrix eigenvalue spectra'. Together they form a unique fingerprint.

    Cite this