Infinite-Dimensional Probability

Цели урока

• Understand Gaussian measures on Banach spaces through one-dimensional projections
• Master Minlos's theorem and the role of nuclearity of the covariance operator
• Analyze the construction of Wiener space via Kolmogorov's theorem
• Connect Fernique concentration and Cameron-Martin to modern ML

Предварительные знания

• Hilbert and Banach spaces
• Weak convergence of probability measures
• Gaussian distributions and covariance matrices
• Compact and nuclear operators

• Exchangeability and de Finetti's Theorem

How does one build a probability measure on the space of all continuous functions - and why does DALL-E and Stable Diffusion need it?

• **Diffusion models:** DALL-E, Stable Diffusion generate images by inverting a Gaussian diffusion process in image space
• **Quantum field theory:** Feynman path integrals are measures on quantum-particle trajectory spaces
• **Neural Network Gaussian Process:** the limit of an infinitely wide network is a Gaussian process with NTK kernel
• **Gaussian Process Regression:** a measure on function space for Bayesian ML - the standard in Bayesian optimization

Gaussian Measures on Banach Spaces

In finite dimensions a Gaussian is defined by the density exp(-x^2/2). In infinite dimensions neither Lebesgue measure nor a density exists; a Gaussian measure is characterized through all one-dimensional projections. This is the mathematical foundation of the Neural Network Gaussian Process - the limiting model of infinitely wide neural networks.

A standard Gaussian vector X = (X_1, X_2, ...) on R^infty has ||X||^2 = sum X_i^2 = infinity almost surely. So a 'standard' Gaussian measure on R^infty does not exist; one needs weighted norms or restriction to subspaces with finite covariance trace.

Wiener Space

Norbert Wiener in 1923 constructed a probability measure on C([0,1]) of continuous functions, realizing Brownian motion as a standard object. Today this measure underlies diffusion generative models: DALL-E 3 and Stable Diffusion are interpreted as measures on C([0,1] x R^d) - the path space in the image space.

Karhunen-Loeve construction of Wiener measure: W_t = sum (xi_n / (n*pi)) sin(n*pi*t), where xi_n are iid N(0,1). This is the expansion in eigenfunctions of the covariance operator - a standard tool in infinite-dimensional probability.

Gaussian Concentration and Cameron-Martin

A remarkable property of Gaussian measures in Banach spaces is strong concentration of the norm around its mean. Fernique's inequality gives exponentially small tails for ||X||, the analog of classical Gaussian concentration in R^n. This underlies generalization bounds for infinitely wide neural networks in NTK theory.