Dynamical Systems
Neurodynamics
The 2024 Nobel Prize in Physics went to Hopfield and Hinton for neural networks, but the committee justified it through spin glass physics and statistical mechanics. Neural networks have always been dynamical systems. Now it's officially recognized.
- **Neural navigation:** place cells in the hippocampus are oscillators with phase coding; their synchronization builds a cognitive map of space
- **Neuroprosthetics:** deep brain stimulation for Parkinson's disease disrupts pathological synchronization in the basal ganglia by delivering an antiphase signal
- **ML architectures:** Neural ODE, Continuous Normalizing Flow, Hamiltonian NN are modern architectures that explicitly use dynamical systems theory
Предварительные знания
Hopfield
**How does the brain recall a face from partial information?** The Hopfield network is a mathematical model of associative memory: patterns are stored as attractors of a dynamical system. Give it a noisy or incomplete input and the system will "roll down" to the nearest stored memory.
**Hopfield network** is a recurrent neural network with symmetric weights W. Dynamics: **dσᵢ/dt = −σᵢ + tanh(β Σⱼ Wᵢⱼσⱼ)** (continuous version). Energy: **E = −(1/2)Σᵢⱼ Wᵢⱼσᵢσⱼ** decreases monotonically, so the system always converges to a local minimum of E. Memory capacity: ≈ 0.14N patterns for N neurons.
| Parameter | Hopfield Network | Biological analog |
|---|---|---|
| Neurons σᵢ = ±1 | Binary spins | Activity patterns |
| Weights Wᵢⱼ = WⱼI | Symmetric connections | Synapses (simplified) |
| Energy E | Decreases monotonically | Memory landscape |
| Attractor | Local minimum of E | A memory |
John Hopfield, 1982
John Hopfield published his model in 1982 in PNAS, drawing an analogy with spin glass physics. The paper became one of the most cited in the history of neuroscience. In 2024, Hopfield shared the Nobel Prize in Physics with Geoffrey Hinton "for foundational discoveries and inventions that enable machine learning with artificial neural networks."
Why does Hopfield network dynamics always converge to an attractor rather than oscillating?
Oscillators
**A neuron is not just a yes/no element.** Real neurons generate rhythmic spikes: pacemaker neurons fire ~60 times per minute, gamma oscillators in the hippocampus run at 40 Hz. Mathematically, such a neuron is a nonlinear oscillator, and the whole neural circuit is a system of coupled oscillators.
**Hodgkin-Huxley neuron model (1952):** four equations describe the action potential: C·dV/dt = I − gₙₐm³h(V−Eₙₐ) − g_K n⁴(V−E_K) − g_L(V−E_L). Simplified models: **FitzHugh-Nagumo** (2D): dv/dt = v − v³/3 − w + I, dw/dt = ε(v + a − bw). When I > I_threshold, the neuron produces a limit cycle (sustained spiking).
| Neuron model | Dimension | Features |
|---|---|---|
| Hodgkin-Huxley (1952) | 4D | Accurate biophysics; 4 variables |
| FitzHugh-Nagumo | 2D | Simplified, analytically tractable |
| Izhikevich (2003) | 2D | 20+ types of neural behavior |
| Van der Pol | 2D | Classic nonlinear oscillator |
What mechanism causes a neuron to start spiking when the current I increases?
Synchronization
**Fireflies in Southeast Asian jungles flash in unison: hundreds of thousands of insects, perfectly synchronized with no conductor.** Neurons in the thalamus synchronize during an epileptic seizure. A pacemaker synchronizes heart cells. Synchronization is a universal phenomenon in coupled oscillators.
**Kuramoto model (1975):** N phase oscillators with different natural frequencies ωᵢ, coupled in a mean-field fashion: **dθᵢ/dt = ωᵢ + (K/N)Σⱼ sin(θⱼ − θᵢ)**. Order parameter: r = |(1/N)Σⱼ exp(iθⱼ)|. When K > K_c = 2/πg(0) (g is the frequency distribution), a **phase transition** occurs: r > 0 indicates partial synchronization.
| K/K_c | Order parameter r | State |
|---|---|---|
| K < K_c | r ≈ 0 (→ 1/√N) | Full incoherence |
| K = K_c | r ≈ 0 (phase transition) | Critical point |
| K > K_c | 0 < r < 1 | Partial synchronization |
| K >> K_c | r → 1 | Near-complete synchronization |
In the Kuramoto model with K > K_c, an order parameter r > 0 means:
Neural Networks
**Deep neural networks are also dynamical systems.** The forward pass is an iterated map. Backpropagation is a gradient flow. ResNet is a discretized ODE. This connection opens a new perspective on neural architectures through the lens of dynamical systems theory.
**Neural ODEs (Chen et al., 2018):** interpreting ResNet as a discretization of an ODE: **dh/dt = f(h(t), t, θ)**, where h(t) is the hidden state and θ are the network parameters. The forward pass solves the ODE with a numerical integrator. The backward pass uses the adjoint method instead of standard backprop, requiring O(1) memory.
| Neural network concept | Dynamical systems analog |
|---|---|
| ResNet forward pass | Discrete approximation of ODE dh/dt = f(h,t,θ) |
| Depth → ∞ | Neural ODE: continuous time |
| Activation space | Phase space |
| Training (gradient descent) | Gradient flow in parameter space |
| Layer normalization | Stabilization of dynamics (Lyapunov) |
Neural ODE: Chen et al., NeurIPS 2018
Ricky Chen's paper "Neural Ordinary Differential Equations" received the Outstanding Paper Award at NeurIPS 2018. The key idea: ResNet x_{k+1} = x_k + f(x_k) is Euler's method for an ODE. Taking the limit (infinitely many layers with infinitesimally small step size) gives a continuous Neural ODE. This enables adaptive numerical integrators, normalizing flows, and new architectures. This direction remains highly active as of 2024.
Neural networks are just matrix operations, unrelated to dynamical systems
Deep neural networks are discrete dynamical systems. ResNet = Euler's method, LSTM = a gated dynamical system, Transformer = a discrete state-space transformation. Dynamical systems theory provides new analytical tools.
Training instability (vanishing/exploding gradients) is a question about the Lyapunov exponents of the system. BatchNorm and LayerNorm stabilize the dynamics. Adversarial examples exploit sensitivity to initial conditions. Understanding DNNs as dynamical systems opens the path to theoretically grounded architectural choices.
ResNet with skip connections x_{k+1} = x_k + f(x_k, θ_k) corresponds to:
Key ideas
- **Hopfield network:** associative memory as a dynamical system; patterns are attractors, recall is descent on the energy landscape
- **Neural oscillators** (FHN, HH): a neuron begins spiking through a Hopf bifurcation when the input current exceeds a threshold
- **Kuramoto synchronization:** at K > K_c a system of coupled oscillators self-organizes; r is the order parameter of synchronization
- **Neural ODE:** ResNet = Euler's method; a neural network is a dynamical system with parameters θ; ODE theory provides tools for analyzing deep networks
Related topics
Neurodynamics brings together all previous topics in the course:
- Bifurcations — Hopf bifurcation is the mechanism behind the onset of neural spiking
- Hamiltonian Systems — Hamiltonian Neural Network (HNN) is an architecture that preserves symplectic structure
- Dynamical Systems in ML — Neural ODE and reservoir computing are direct applications of this lesson
Вопросы для размышления
- Epilepsy is pathological hypersynchronization, yet normal synchronization (gamma oscillations during attention) is beneficial. How does the brain regulate the level of synchronization?
- The Hopfield network has a memory capacity of ~0.14N. How can capacity be increased? Modern Hopfield Networks (2020) achieve exponential capacity. How?
- If neural networks are dynamical systems, then training changes the "landscape" of the system. What does a "well-trained" network look like from a dynamics perspective?