Differential Equations

Second-Order ODEs

November 7, 1940. Washington State. Tacoma Narrows Bridge sways in 64 km/h winds and collapses four hours later. NeurIPS 2018, Montreal. Neural ODE reshapes neural architecture by showing that ResNet is an Euler solver for $dh/dt = f(h, t, \theta)$. Both events are described by the same equation $y'' + ay' + by = f(x)$. The only difference is whether the right-hand side destroys the system - or trains it.

  • **Neural ODE (Chen et al. NeurIPS 2018):** hidden state as a continuous ODE. ResNet is a special case with Euler step delta_t=1. Understanding second-order dynamics explains why this works
  • **Diffusion models (DDPM, Stable Diffusion, FLUX):** the forward process adds noise via a stochastic ODE. Reverse generation is integration of the reverse ODE. ODE theory is unavoidable in understanding the architecture
  • **Kalman filter:** optimal filter for linear ODEs with Gaussian noise. Used in GPS, Tesla Autopilot, Boston Dynamics robots for sensor fusion
  • **Structural engineering:** seismic analysis of buildings - second-order ODE with periodic forcing. Every skyscraper is a solved resonance problem

Предварительные знания

  • First-Order ODEs

The Characteristic Equation

NeurIPS 2018, Montreal. Tian Qi Chen presents Neural ODE - a continuous neural network where the hidden state evolves according to the differential equation $dh/dt = f(h, t, \theta)$. ResNet is a discretized version of this very equation: $h_{t+1} = h_t + F(h_t)$ is an Euler step with $\Delta t = 1$. One insight - and a discrete architecture becomes a numerical ODE solver. To understand why it works, one must first understand the second order.

0

1

Sign In

The second-order linear ODE with constant coefficients: $y'' + ay' + by = 0$. Substitute the trial solution $y = e^{rx}$: the derivatives give $y' = re^{rx}$, $y'' = r^2 e^{rx}$. After substitution and division by $e^{rx} \neq 0$, what remains is the **characteristic equation** $r^2 + ar + b = 0$. Differentiation has been replaced by multiplication by $r$ - this is the seed of the Laplace transform.

DiscriminantRootsSolutionPhysics
D > 0r1 != r2 (real)C1*e^(r1*x) + C2*e^(r2*x)Transient processes
D = 0r1 = r2 = r(C1 + C2*x)*e^(r*x)Critical damping
D < 0alpha +/- beta*ie^(alpha*x)*(C1*cos(beta*x) + C2*sin(beta*x))Damped oscillations

Substituting $y = e^{rx}$ converts a differential equation into an algebraic one. This is not a trick - it is the principle underlying all of operational calculus. Engineers use the Laplace transform for exactly this reason: complex ODEs become polynomial equations in $r$.

Characteristic equation: r^2 + 6r + 9 = 0. What are the roots?

Three Regimes: Transient, Critical, Oscillatory

Tacoma Narrows Bridge, November 7, 1940. Wind speed: 64 km/h - unremarkable by load standards. The bridge begins oscillating and collapses four hours later. The cause: wind locked into resonance with the structure's natural frequency. The three regimes of a second-order ODE are three scenarios for any oscillatory system - from a suspension bridge to an electrical circuit.

**Case D > 0** - two distinct real roots $r_1, r_2$. General solution: $y = C_1 e^{r_1 x} + C_2 e^{r_2 x}$. A sum of two exponentials - a transient process. If both roots are negative, the system decays without oscillations. This regime describes transient dynamics in RLC circuits: the capacitor charge after voltage is applied.

**Case D = 0** - repeated root $r$. A single $e^{rx}$ is insufficient - two linearly independent solutions are required. The second is $xe^{rx}$. General solution: $y = (C_1 + C_2 x)e^{rx}$. This is **critical damping** - return to equilibrium without a single oscillation, as fast as physically possible. Engineers seek this regime for door closers and car shock absorbers.

**Case D < 0** - complex roots $\alpha \pm \beta i$. Euler's formula $e^{i\theta} = \cos\theta + i\sin\theta$ converts the complex exponential into a real solution: $y = e^{\alpha x}(C_1 \cos(\beta x) + C_2 \sin(\beta x))$. With $\alpha < 0$ - damped oscillations. With $\alpha = 0$ - pure undamped oscillations. With $\alpha > 0$ - growing resonance, as with Tacoma.

d'Alembert and Euler (1747-1749)

Jean le Rond d'Alembert derived the wave equation for string vibrations - a second-order ODE at its core. His substitution method $y = e^{rx}$ became the standard. Two years later Euler added complex roots and the formula $e^{i\theta} = \cos\theta + i\sin\theta$ - from which a real solution emerges from complex algebra. Neural ODE (2018) is the same d'Alembert, but with a learnable right-hand side $f(h, t, \theta)$.

The damping ratio $\zeta$ determines the regime: $\zeta < 1$ - oscillations, $\zeta = 1$ - critical, $\zeta > 1$ - overdamped. In gradient descent there is a direct analogy: momentum $\beta_1 = 0.9$ in Adam corresponds to a system with $\zeta \approx 0.16$ - slightly underdamped, which helps escape saddle points.

Equation y'' + 4y = 0. What type of solution results?

Undetermined Coefficients and Resonance

Diffusion models - Stable Diffusion, DALL-E 3, FLUX - add noise according to the equation $dx = -x\,dt + \sigma\,dW$. This is a stochastic ODE. The reverse generative process is the integration of the reverse ODE. The deterministic part of this equation is a non-homogeneous ODE: $y' + y = f(t)$, where $f(t)$ is the driving signal. The method of undetermined coefficients answers the question: how to find a particular solution for a specific right-hand side.

For the non-homogeneous equation $y'' + ay' + by = f(x)$, the general solution equals the homogeneous part plus a **particular solution**. The method of undetermined coefficients: guess the form of the particular solution from the shape of $f(x)$, substitute, and solve for the coefficients. The guess is not arbitrary - the form is rigidly determined by the type of right-hand side.

Right-hand side f(x)Trial solution y_pCondition
Pn(x) - polynomial of degree nQn(x) - polynomial of degree n0 is not a root of char. eq.
e^(alpha*x)A*e^(alpha*x)alpha is not a root
e^(alpha*x), alpha is a simple rootA*x*e^(alpha*x)Multiply by x
cos(beta*x) or sin(beta*x)A*cos(beta*x) + B*sin(beta*x)+-i*beta is not a root
e^(alpha*x)*Pn(x)e^(alpha*x)*Qn(x)Combination

**Resonance.** If $\alpha$ is a root of the characteristic equation with multiplicity $k$, multiply the trial solution by $x^k$. Without this, the coefficients cancel and the system has no solution. Physically: the external force hits the natural frequency of the system - energy accumulates without dissipation. This destroyed Tacoma Narrows Bridge.

**Superposition principle:** if $y_1$ is a particular solution for $f_1(x)$ and $y_2$ for $f_2(x)$, then $y_1 + y_2$ is a particular solution for $f_1 + f_2$. Linearity allows splitting complex right-hand sides into simpler ones. In ML the analogy is direct: loss = reconstruction_loss + KL_divergence uses the same superposition for gradients.

y'' + y = sin(x). Characteristic equation: r^2 + 1 = 0, roots r = +-i. What trial solution y_p should be chosen?

Variation of Parameters

Undetermined coefficients handles only guessable right-hand sides. Physics-Informed Neural Networks (PINNs) - DeepMind GraphCast, climate models, turbulence simulation - require solving ODEs with arbitrary data on the right-hand side. A universal method is needed.

**Variation of parameters (Lagrange, 1774).** Take the homogeneous general solution $C_1 y_1 + C_2 y_2$ and un-fix the constants - allow $C_1(x)$, $C_2(x)$ to depend on $x$. Substitute $y = C_1(x)y_1 + C_2(x)y_2$ and impose the constraint $C_1'y_1 + C_2'y_2 = 0$ (otherwise the system is overdetermined). This yields two equations for $C_1'$ and $C_2'$, solved via the Wronskian determinant.

StepFormulaDescription
1. WronskianW = y1*y2' - y2*y1'Determinant of the fundamental system
2. Find C1'C1' = -y2*f(x)/WFrom the system of equations
3. Find C2'C2' = y1*f(x)/WFrom the system of equations
4. IntegrateC1 = integral(C1' dx), C2 = integral(C2' dx)May require special functions
5. Assembley_p = C1(x)*y1 + C2(x)*y2Particular solution

**Wronskian W != 0** guarantees that $y_1$ and $y_2$ are linearly independent and the system for $C_1'$, $C_2'$ has a unique solution. If W = 0, the solutions are linearly dependent. Wroński (1778-1853) introduced this determinant half a century before its application to ODEs became standard.

The complete picture: $y'' + ay' + by = f(x)$ is "$F = ma$" in mathematical form. The characteristic equation reveals whether the system will oscillate or decay. Undetermined coefficients give a particular solution for standard forcing. Variation of parameters handles arbitrary $f(x)$. The same principle appears in the adjoint method - the way gradients flow through a Neural ODE, as described in Chen et al. 2018.

Resonance is a purely physical phenomenon

Resonance is a mathematical phenomenon: the right-hand side matches a solution of the homogeneous part. The amplitude grows linearly over time.

When f(x) = e^(rx) and r is a root of the characteristic equation, the standard substitution yields 0 = f(x) - a contradiction. Multiplying by x is mathematically necessary. Physically: the external force locks onto the natural frequency, and energy accumulates without dissipation. Tacoma Narrows Bridge (1940) and shattering a glass with sound are both resonance.

For y'' - y = e^x/x the method of undetermined coefficients does not work. Why?

Key Ideas

  • **The characteristic equation** $r^2 + ar + b = 0$ reduces the ODE to an algebraic problem - the seed of the Laplace transform
  • **Three regimes:** transient processes (D>0), critical damping (D=0) - door closers and shock absorbers, oscillations (D<0) - from the quantum harmonic oscillator to RLC circuits
  • **Undetermined coefficients:** guess y_p from the right-hand side; if alpha is a root - multiply by x^k (resonance)
  • **Variation of parameters:** universal method via the Wronskian for any right-hand side. The adjoint method in Neural ODE is its continuous counterpart

Related Topics

Second-order ODEs are the central topic in the analysis of oscillatory systems:

  • First-Order ODEs — The foundation: first-order methods are used when solving second-order equations
  • Systems of ODEs — A second-order ODE is equivalent to a system of two first-order ODEs
  • Dynamical Systems — Phase portraits and attractors - the geometry of ODE solutions

Вопросы для размышления

  • Critical damping (D = 0) gives the fastest return without oscillations. Why does an overdamped system (D > 0, two negative roots) return more slowly, even though it also avoids oscillation?
  • Neural ODE replaces discrete ResNet layers with a continuous ODE. Which of the three regimes (D>0, D=0, D<0) describes the behavior of the hidden state in a stably trained network?
  • Tacoma Narrows collapsed in 64 km/h winds. Engineers now compute natural frequencies of structures before construction. How does the characteristic equation make this possible analytically?

Связанные уроки

  • de-01 — First-order methods are the foundation before second-order
  • de-03 — A second-order ODE is equivalent to a system of two first-order ODEs
  • dyn-01 — Phase portraits and attractors for oscillatory systems
  • nm-01 — Numerical methods for when analytical solutions are unavailable
  • calc-01-sequences — Limits and convergence - foundation for stability analysis
  • la-13-eigenvectors
Second-Order ODEs