What is Probability?

In 1654, Pascal and Fermat exchanged letters about gambling. From that correspondence, probability theory was born. 370 years later, the same formulas choose ChatGPT's next token, flag fraud at Stripe, and govern the loss function of every AI lab.

• ChatGPT: every token is a sample from a distribution over ~200,000 words. The temperature parameter in every API is raw probability theory.
• Gmail spam filter: Naive Bayes computes $P(spam|words)$ in microseconds, billions of times per day.
• Stripe fraud detection: $P(fraud|transaction)$ - a single probability number blocks or approves the payment.

Kolmogorov Axioms: Three Rules for Everything

In 1654, Pascal and Fermat exchanged letters about gambling. Chevalier de Méré asked Pascal: «How is a pot fairly split if the game is stopped?» From that correspondence, probability theory was born. 370 years later, the same formulas choose ChatGPT's next token, flag fraud at Stripe, and govern the loss function of every AI lab.

Before computing any probability, the **set of all possible outcomes** is required. We call it $\Omega$ (omega) - the sample space. An **event** $A$ is any subset of $\Omega$.

In 1933 Andrey Kolmogorov stopped the philosophical debate about the nature of probability with one move: «Let's stop arguing. Let's agree on the **rules** that any probability must obey - and derive everything from those rules.»

**Kolmogorov's three axioms:** **1. Non-negativity:** $P(A) \geq 0$ **2. Normalization:** $P(\Omega) = 1$ - something in $\Omega$ must happen **3. Additivity:** for mutually exclusive events $A \cap B = \emptyset$: $$P(A \cup B) = P(A) + P(B)$$ All of probability theory - Bayes' theorem, expectation, information theory, neural-network cross-entropy loss - follows from these three lines.

**Classical probability** (works when outcomes are equally likely): $P(A) = |A| / |\Omega|$. Sum = 7 with two dice: 6 pairs out of 36, $P = 6/36 \approx 16.7\%$. Sum = 12: only $(6,6)$, $P = 1/36 \approx 2.8\%$. That is where craps rules come from.

Frequentist vs Bayesian Probability

Kolmogorov's axioms define the **rules** for probability. But what does it **mean**? Physicists, statisticians, and engineers have disagreed on this for 300 years.

**In ML both views coexist:** - Classifier accuracy on a test set - frequentist probability - Posterior $P(\theta \mid \text{data})$ in Bayesian learning - Bayesian - Naive Bayes spam filter - Bayesian name, often trained frequentistically - Dropout at inference as Bayesian approximation - Bayesian interpretation of a frequentist method

Conditional Probability and Independence

Conditional probability $P(A \mid B)$ is the probability of event $A$ given that $B$ has already occurred. It is an update of information.

**Independence**: events $A$ and $B$ are independent if $P(A \mid B) = P(A)$, meaning knowledge of $B$ does not change the probability of $A$. Equivalently: $P(A \cap B) = P(A) \cdot P(B)$.

**Naive Bayes** - the fastest spam filter in existence - makes one strong assumption: words in an email are independent of each other (they are not, but it works remarkably well). $P(\text{spam} \mid w_1, w_2, \ldots) \propto P(\text{spam}) \prod_i P(w_i \mid \text{spam})$. Computed in microseconds, billions of times per day.

**The base rate fallacy**: a test with 99% accuracy sounds excellent. But if the disease is rare (1%), a positive test means actual disease in only 16% of cases. This is not an error - it is conditional probability. That is why medical screenings require a confirmatory test.

Summary

• $\Omega$ - sample space, events $A \subseteq \Omega$ are subsets of interest
• Kolmogorov's three axioms: non-negativity, normalization, additivity - all of probability from these
• Complement: $P(\bar{A}) = 1 - P(A)$ - the most used trick in the field
• Classical probability: $P(A) = |A| / |\Omega|$ for equally likely outcomes
• Frequentist vs Bayesian: objective limiting frequency vs subjective degree of belief
• Conditional probability: $P(A|B) = P(A \cap B) / P(B)$ - information update

What this unlocks

Three axioms and what they build:

• Combinatorics — Counting outcomes systematically - needed for anything beyond 36 dice pairs
• Conditional probability and Bayes' theorem — Update beliefs with evidence - spam filter, medical test, Bayesian A/B test
• Random variables — Probability with numbers: expectation, variance - the workhorses of statistics

Вопросы для размышления

• Why does intuition about randomness fail so often? Hint: we think in stories, not in sample spaces.
• «30% chance of rain» - what does it mean operationally for a frequentist and for a Bayesian?
• At what temperature does an LLM become deterministic? At what temperature does it become random noise?

Связанные уроки

• dm-01