Calculus

Computing Limits

Цели урока

Master the method of algebraic transformations
Apply conjugate multiplication
Use remarkable limits
Apply L'Hopital's rule
Work with equivalent infinitesimals

Предварительные знания

The concept of a limit and the ε-δ definition
Remarkable limits
Basics of differentiation (for L'Hopital's rule)

The Concept of a Limit

Numerical differentiation in PyTorch: `torch.autograd.gradcheck` verifies analytical gradients via finite differences $(f(x+h) - f(x-h)) / (2h)$ at $h = 10^{-6}$. This is a limit - computed at a specific $h$. The whole idea is that $\frac{0}{0}$ at $h = 0$ can be resolved to a finite number. That is what this lesson does: five techniques that turn $\frac{0}{0}$ into an answer.

**`torch.autograd.gradcheck`**: compares analytical gradient with numerical $(f(x+h) - f(x-h)) / (2h)$. The numerical error is $O(h^2)$, and understanding this requires knowledge of limits and equivalent infinitesimals
**Taylor approximation in ML**: $\log(1 + \sigma(x)) \approx \sigma(x)$ for small $\sigma(x)$ - this is the equivalent infinitesimal $\ln(1+t) \sim t$ as $t \to 0$. Used for numerical stability in loss functions
**Batch Normalization**: analyzing BN behavior as $\sigma^2 \to 0$ requires computing a limit - a typical 0/0 problem
**Asymptotic complexity**: $O(n \log n)$ vs $O(n^2)$ is a statement about the limit $n \log n / n^2 \to 0$. L'Hopital's rule proves such relationships in seconds
**Softmax stability**: $\text{softmax}(x_i) = e^{x_i} / \sum e^{x_j}$ at extreme $x$ - the limit $e^x / e^x = 1$. The log-sum-exp numerical trick follows from this limit

The marquis who bought a theorem

The rule bearing the name of **Marquis de L'Hopital** was actually discovered by his teacher - the Swiss mathematician **Johann Bernoulli**. L'Hopital paid Bernoulli for lessons and obtained the right to publish his results. So the first calculus textbook "Analyse des infiniment petits" (1696) appeared under L'Hopital's name. Bernoulli stayed silent for many years, until their correspondence was published after both had died.

Method 1: Algebraic transformations

The cleanest method: **factor, cancel, substitute**. The indeterminate form $\frac{0}{0}$ arises because of a common factor that vanishes. Cancel it - and the problem is gone.

Difference of squares

Classic 0/0 case

$\lim_{x \to 2} \frac{x^2 - 4}{x - 2}$ At $x = 2$: $\frac{0}{0}$ - indeterminate! **Factor** the numerator using $a^2 - b^2 = (a-b)(a+b)$: $$\frac{x^2 - 4}{x - 2} = \frac{(x-2)(x+2)}{x-2} = x + 2 \quad (x \neq 2)$$ **Now** substitute: $\lim_{x \to 2} (x + 2) = \boxed{4}$

Difference of cubes

Factoring cubics

$\lim_{x \to 1} \frac{x^3 - 1}{x - 1}$ Formula: $a^3 - b^3 = (a-b)(a^2 + ab + b^2)$ $$\frac{x^3 - 1}{x - 1} = \frac{(x-1)(x^2 + x + 1)}{x-1} = x^2 + x + 1$$ $\lim_{x \to 1} (x^2 + x + 1) = 1 + 1 + 1 = \boxed{3}$

Compute $\lim_{x \to 3} \frac{x^2 - 9}{x - 3}$

$x^2 - 9 = (x-3)(x+3)$. After canceling: $\lim_{x \to 3}(x+3) = 6$.

Method 2: Conjugate multiplication

When the expression contains **radicals**, multiplying by the conjugate converts the difference of roots into a difference of squares:

Root in the numerator

Classic technique - and the connection to autograd

$\lim_{x \to 0} \frac{\sqrt{1+x} - 1}{x}$ At $x = 0$: $\frac{0}{0}$ **Multiply** by the conjugate $\frac{\sqrt{1+x} + 1}{\sqrt{1+x} + 1}$: $$\frac{\sqrt{1+x} - 1}{x} \cdot \frac{\sqrt{1+x} + 1}{\sqrt{1+x} + 1} = \frac{(1+x) - 1}{x(\sqrt{1+x} + 1)} = \frac{x}{x(\sqrt{1+x} + 1)}$$ $$= \frac{1}{\sqrt{1+x} + 1} \xrightarrow{x \to 0} \frac{1}{1 + 1} = \boxed{\frac{1}{2}}$$ **This is not just an exercise**: this limit is the derivative of $\sqrt{x}$ at $x = 1$. Numerical differentiation $(\sqrt{1+h} - 1)/h$ at $h \to 0$ gives exactly $1/2$.

Multiply only the numerator by the conjugate

BOTH numerator AND denominator must be multiplied (i.e., by 1)

This is a transformation, not a change to the expression. Multiplying by $\frac{\text{conj.}}{\text{conj.}} = 1$ does not change the value of the function - only the form.

What is the conjugate of the expression $\sqrt{x+4} - 2$?

The conjugate of $\sqrt{a} - b$ is $\sqrt{a} + b$. Their product gives $a - b^2$ without radicals.

Method 3: Remarkable limits

Two fundamental limits allow computing entire families of expressions. The first explains why $\sin x \approx x$ in linearization. The second - why $e$ is the base of the natural logarithm:

First remarkable limit

Reducing to standard form

$\lim_{x \to 0} \frac{\sin 5x}{3x}$ **Trick**: the argument of sine must match the denominator. $$\frac{\sin 5x}{3x} = \frac{5}{3} \cdot \frac{\sin 5x}{5x}$$ As $x \to 0$ we have $5x \to 0$, so $\frac{\sin 5x}{5x} \to 1$ $$\lim = \frac{5}{3} \cdot 1 = \boxed{\frac{5}{3}}$$ ML application: linearizing loss functions near a minimum uses $\sin \theta \approx \theta$, $\cos \theta \approx 1 - \theta^2/2$ - direct consequences of this limit.

Second remarkable limit

Indeterminate form 1^∞

$\lim_{x \to \infty} \left(1 + \frac{3}{x}\right)^x$ This is the $1^\infty$ indeterminate form! **Technique**: substitute $t = x/3$, then $x = 3t$ and as $x \to \infty$ we have $t \to \infty$: $$\left(1 + \frac{3}{x}\right)^x = \left(1 + \frac{1}{t}\right)^{3t} = \left[\left(1 + \frac{1}{t}\right)^{t}\right]^3$$ The inner bracket $\to e$ as $t \to \infty$ $$\lim = e^3 \approx \boxed{20.09}$$

What is $\lim_{x \to 0} \frac{\tan x}{x}$?

$\tan x = \frac{\sin x}{\cos x}$. So $\frac{\tan x}{x} = \frac{\sin x}{x} \cdot \frac{1}{\cos x} \to 1 \cdot 1 = 1$.

Method 4: L'Hopital's rule

A powerful tool for the $\frac{0}{0}$ and $\frac{\infty}{\infty}$ indeterminate forms. Bernoulli discovered it, L'Hopital bought and published it:

L'Hopital's rule is applicable **only** for $0/0$ or $\infty/\infty$ indeterminate forms. **Always check** the type of indeterminate form before applying!

Simple application

Exponential - and the link to autograd

$\lim_{x \to 0} \frac{e^x - 1}{x}$ Check: at $x = 0$ we get $\frac{e^0 - 1}{0} = \frac{0}{0}$ ✓ **Apply** L'Hopital's rule: $$\lim_{x \to 0} \frac{(e^x - 1)'}{(x)'} = \lim_{x \to 0} \frac{e^x}{1} = e^0 = \boxed{1}$$ This is the definition of the derivative of $e^x$ at 0. In PyTorch: `torch.exp(torch.tensor(0.0)).backward()` returns gradient=1, and this is why.

Applying twice

When once is not enough

$\lim_{x \to 0} \frac{1 - \cos x}{x^2}$ At $x = 0$: $\frac{0}{0}$ - apply L'Hopital: $$\lim_{x \to 0} \frac{\sin x}{2x} = \frac{0}{0}$$ - indeterminate again! Apply **again**: $$\lim_{x \to 0} \frac{\cos x}{2} = \frac{1}{2} = \boxed{0.5}$$ This result appears in the expansion $\cos x \approx 1 - x^2/2$ - an equivalent infinitesimal used when approximating cosine similarity at small angles.

L'Hopital's rule means differentiating the fraction using the quotient rule

It means differentiating numerator and denominator separately

Quotient rule: $(f/g)' = \frac{f'g - fg'}{g^2}$. L'Hopital's rule: $\lim \frac{f}{g} = \lim \frac{f'}{g'}$. Different things! L'Hopital is not about differentiating a fraction - it is about replacing a limit.

Compute $\lim_{x \to 0} \frac{e^x - 1}{x}$ via L'Hopital.

Form $0/0$. Differentiate numerator and denominator: $\lim (e^x / 1) = e^0 = 1$.

Method 5: Equivalent infinitesimals

As $x \to 0$ many functions behave "like $x$" or "like $x^2$". This allows fast substitutions in limits - exactly what is used for numerical stability in ML code:

Function	Equivalent as $x \to 0$
$\sin x$	$x$
$\tan x$	$x$
$\arcsin x$	$x$
$\arctan x$	$x$
$\ln(1+x)$	$x$
$e^x - 1$	$x$
$1 - \cos x$	$\frac{x^2}{2}$
$(1+x)^\alpha - 1$	$\alpha x$

Equivalence $f \sim g$ as $x \to 0$ means $\lim_{x \to 0} \frac{f(x)}{g(x)} = 1$. Substitution is valid in **products**, but NOT in sums! PyTorch uses $\log(1 + e^x) \approx x$ for large $x$ for numerical stability - this is the same table of equivalents.

Fast computation

Substituting equivalents

$\lim_{x \to 0} \frac{\sin 3x \cdot \ln(1+2x)}{x^2}$ **Substitutions**: $\sin 3x \sim 3x$, $\ln(1+2x) \sim 2x$ as $x \to 0$ $$\lim_{x \to 0} \frac{3x \cdot 2x}{x^2} = \lim_{x \to 0} \frac{6x^2}{x^2} = \boxed{6}$$ Numerical check: at $x = 0.001$: numerator $= \sin(0.003) \cdot \ln(1.002) \approx 0.003 \cdot 0.002 = 6 \cdot 10^{-6}$, denominator $= 10^{-6}$. Ratio $\approx 6$ ✓

What is $e^{2x} - 1$ equivalent to as $x \to 0$?

From the table: $e^t - 1 \sim t$ as $t \to 0$. Here $t = 2x$, so $e^{2x} - 1 \sim 2x$.

Limits at infinity

For rational functions as $x \to \infty$, divide by the **highest power** - this gives a quick answer without extra computation:

Equal degrees

Ratio of leading coefficients

$\lim_{x \to \infty} \frac{3x^2 + 2x - 1}{5x^2 - x + 4}$ Divide numerator and denominator by $x^2$: $$\lim_{x \to \infty} \frac{3 + 2/x - 1/x^2}{5 - 1/x + 4/x^2}$$ As $x \to \infty$ all fractions with $x$ → 0: $$= \frac{3 + 0 - 0}{5 - 0 + 0} = \boxed{\frac{3}{5}}$$ **Fast way**: with equal degrees, the limit is the ratio of leading coefficients. This is exactly what complexity analysis does: $T(n) = 3n^2 + 2n \sim 3n^2$, algorithm is $O(n^2)$.

**Quick rule** for rational functions as $x \to \infty$: if degrees are equal - ratio of leading coefficients; if numerator degree is higher - $\infty$; if lower - 0. This is exactly what algorithm comparison does in big-O analysis.

What is $\lim_{x \to \infty} (1 + 2/x)^x$?

Standard form $\lim (1 + a/x)^x = e^a$. Here $a = 2$.

Practice

Compute $\lim_{x \to 0} \frac{\sin 2x}{\sin 5x}$

$$\frac{\sin 2x}{\sin 5x} = \frac{\sin 2x}{x} \cdot \frac{x}{\sin 5x} = \frac{2 \cdot \frac{\sin 2x}{2x}}{5 \cdot \frac{\sin 5x}{5x}} \cdot \frac{2}{5}$$ Or faster with equivalents: $\sin 2x \sim 2x$, $\sin 5x \sim 5x$ $$\lim = \frac{2x}{5x} = \frac{2}{5}$$

Compute $\lim_{x \to 4} \frac{\sqrt{x} - 2}{x - 4}$

Method 1 (conjugate): $$\frac{\sqrt{x} - 2}{x - 4} \cdot \frac{\sqrt{x} + 2}{\sqrt{x} + 2} = \frac{x - 4}{(x-4)(\sqrt{x}+2)} = \frac{1}{\sqrt{x}+2}$$ $$\lim_{x \to 4} \frac{1}{\sqrt{x}+2} = \frac{1}{2+2} = \frac{1}{4}$$ Method 2 (factoring): $x - 4 = (\sqrt{x})^2 - 4 = (\sqrt{x}-2)(\sqrt{x}+2)$ Answer: $\boxed{\frac{1}{4}}$

Compute $\lim_{x \to 0} \frac{e^{3x} - 1 - 3x}{x^2}$ using L'Hopital's rule

At $x = 0$: $\frac{e^0 - 1 - 0}{0} = \frac{0}{0}$ ✓ L'Hopital (1st time): $$\lim_{x \to 0} \frac{3e^{3x} - 3}{2x} = \frac{3 - 3}{0} = \frac{0}{0}$$ - again! L'Hopital (2nd time): $$\lim_{x \to 0} \frac{9e^{3x}}{2} = \frac{9e^0}{2} = \frac{9}{2} = \boxed{4.5}$$ Verification via infinitesimals: $e^{3x} - 1 \sim 3x$ as $x \to 0$, so $e^{3x} - 1 - 3x \sim (3x)^2/2 = 9x^2/2$. Ratio $9x^2/2x^2 = 9/2$ ✓

Compute $\lim_{x \to 0} \frac{1 - \cos(x)}{x^2}$.

By Taylor: $\cos(x) = 1 - x^2/2 + O(x^4)$, so $(1 - \cos x)/x^2 \to 1/2$. L'Hopital twice gives the same answer.

Connection with other topics

Limit computation techniques are needed throughout analysis

Derivative — The definition of the derivative requires computing a limit of a difference quotient
Taylor series — Series expansions provide equivalent infinitesimals
Integrals — Improper integrals are limits
Differential equations — Asymptotic behavior of solutions

Итоги

**Algebraic transformations**: factor and cancel the common factor - the cleanest method
**Conjugate**: multiplying by $(\sqrt{a} + \sqrt{b})$ eliminates radicals - the foundation of numerical differentiation
**Remarkable limits**: $\frac{\sin x}{x} \to 1$, $(1+x)^{1/x} \to e$ - explain linearization and the base $e$
**L'Hopital's rule**: differentiate numerator and denominator separately. Only for 0/0 and ∞/∞. This is not the quotient rule
**Equivalent infinitesimals**: $\sin x \sim x$, $\ln(1+x) \sim x$, $e^x - 1 \sim x$ as $x \to 0$ - the table ML engineers use for numerical stability analysis

Вопросы для размышления

Which method to try first for a 0/0 indeterminate form? When does algebra beat L'Hopital?
Why are equivalent substitutions not valid in sums?
What is the difference between L'Hopital's rule and the quotient rule for differentiation?
How does tanh(x)/x → 1 as x → 0 relate to the first remarkable limit?

Связанные уроки

nm-01