Functional Analysis

Introduction to Spectral Theory

Why does the conjugate gradient method converge in a finite number of steps? Why do kernel methods (kernel SVM) work? The answer is spectral theory: the structure of an operator's spectrum determines both the convergence speed of iterative solvers and the expressive power of kernel methods.

  • **Iterative solvers**: convergence rate of Jacobi iterations = spectral radius of the iteration matrix; preconditioned CG minimizes iteration count by optimizing the spectrum of the preconditioned matrix
  • **Kernel methods (SVM, GP)**: the Gram matrix generates a compact integral operator; Mercer's theorem guarantees k(x,y) = sum lambda_i phi_i(x) phi_i(y) via the operator's eigenfunctions
  • **PageRank**: the stationary distribution is the eigenvector for lambda = 1; computed by power iterations in O(n) due to the spectral gap |lambda_1| - |lambda_2|

Предварительные знания

  • Open Mapping and Closed Graph Theorems

Spectrum and Resolvent

For a bounded linear operator T: X -> X on a Banach space X, define: **resolvent set** rho(T) = {lambda in C : (T - lambda I)^{-1} exists and is bounded}, **spectrum** sigma(T) = C minus rho(T). Unlike matrices, the spectrum in infinite dimensions splits into three disjoint parts.

**Three parts of the spectrum**: 1. **Point spectrum** sigma_p(T): lambda such that T - lambda I is not injective (eigenvalues, Tx = lambda x). 2. **Continuous spectrum** sigma_c(T): T - lambda I is injective, its range is dense, but (T - lambda I)^{-1} is unbounded. 3. **Residual spectrum** sigma_r(T): T - lambda I is injective but its range is not dense.

0

1

Sign In

**Theorem**: for a bounded operator T the spectrum sigma(T) is a compact non-empty subset of C, contained in the disk |lambda| <= ||T||. The resolvent set rho(T) is open. The resolvent R(lambda) = (T - lambda I)^{-1} is an analytic (holomorphic) function of lambda on rho(T).

How does the spectrum of an operator in infinite dimensions differ from that of a matrix?

Spectral Radius

**Spectral radius**: r(T) = sup{|lambda| : lambda in sigma(T)}. **Gelfand formula**: r(T) = lim_{n->inf} ||T^n||^{1/n} = inf_{n>=1} ||T^n||^{1/n}. Always r(T) <= ||T||, and strict inequality is possible. For normal operators on a Hilbert space: r(T) = ||T||.

**Applications**: the Neumann series (T - lambda I)^{-1} = -sum_{k=0}^{inf} T^k / lambda^{k+1} converges when |lambda| > r(T). In numerical methods: Jacobi/Gauss-Seidel iterations converge if and only if the spectral radius of the iteration matrix is less than 1.

When does the Neumann series (T - lambda I)^{-1} = -sum T^k / lambda^{k+1} converge?

Compact Operators

**Compact operator**: T: X -> Y such that the image of any bounded set is precompact (has compact closure). Equivalently: from any bounded sequence {x_n} one can extract a subsequence on which {Tx_n} converges.

**Riesz-Schauder spectral theorem**: if T is a compact operator, then: 1. sigma(T) minus {0} consists only of eigenvalues 2. eigenvalues form a countable set converging to 0 3. each nonzero eigenvalue has finite algebraic multiplicity. Compact operators behave like matrices-except for 'infinity near zero'.

Compact operators in ML: the kernel matrix K_ij = k(x_i, x_j) generates a compact integral operator T_k f(x) = integral k(x,y) f(y) dy. The spectral decomposition of this operator is the foundation of Mercer's theorem and RKHS (reproducing kernel Hilbert space) analysis.

Spectral theory in infinite dimensions is just eigenvalue theory, the same as for matrices

The spectrum can contain points with no eigenvectors (continuous spectrum). The multiplication operator (Tf)(x) = x*f(x) on L^2[0,1] has sigma(T) = [0,1] with empty point spectrum

Continuous spectrum corresponds to 'approximate eigenfunctions'-sequences approaching an eigenvector without reaching it. This is fundamental in quantum mechanics (scattering states)

What is the spectrum of a compact operator T on an infinite-dimensional Banach space?

Key Ideas

  • **Spectrum**: sigma(T) = C minus rho(T)-compact non-empty set; splits into point (eigenvalues), continuous, and residual subspectra
  • **Spectral radius**: r(T) = lim ||T^n||^{1/n} <= ||T||; Neumann series converges for |lambda| > r(T); iterative methods converge if and only if r < 1
  • **Compact operators**: sigma(T) minus {0} = discrete eigenvalues -> 0 (Riesz-Schauder); kernel matrices in ML are compact integral operators

Related Topics

Spectral theory runs through all of functional analysis:

  • Spectral Theorem — Self-adjoint operators on Hilbert spaces: complete spectral decomposition via a spectral measure E(lambda)
  • Banach Theorems — Open mapping theorem: lambda in rho(T) if and only if (T - lambda I)^{-1} is bounded-the definition uses the bounded inverse theorem

Вопросы для размышления

  • The multiplication operator (Tf)(x) = x*f(x) on L^2[0,1] has sigma(T) = [0,1] with no eigenvalues. How is this consistent with the operator being 'diagonal' in function space?
  • The spectral radius r(T) <= ||T|| and strict inequality is possible. Give an example of an operator where r(T) = 0 but ||T|| > 0.
  • PageRank uses the power method to find the leading eigenvector. How is the convergence rate related to the spectral gap |lambda_1| - |lambda_2|?

Связанные уроки

  • la-13-eigenvectors
Introduction to Spectral Theory