Information Theory
Channel Capacity
Цели урока
- Define channel capacity C = max I(X;Y) and compute it for AWGN
- State the channel coding theorem in both directions
- Apply water-filling to parallel channels and OFDM systems
- Distinguish ergodic capacity from outage capacity for fading channels
Предварительные знания
- Rate-Distortion Theory
- Mutual Information
How many bits per second can be reliably transmitted over a channel with a given SNR? How should power be optimally distributed across OFDM subcarriers?
- 5G NR at 100 MHz and 30 dB SNR: C ~ 1 Gbps - all modern standards operate near this bound
- Wi-Fi 6 with 1024-QAM: adaptive modulation implements water-filling in real time
- Optical links at 400 Gbps: within 2 dB of Shannon's limit for fiber
- Polar codes in 5G: first construction theoretically achieving C on any symmetric channel
From Telephone to 5G: 75 Years of One Formula
1948 - Shannon publishes 'A Mathematical Theory of Communication'. Hartley (1928) defined information as log n but ignored noise. Shannon added noise and got C = 1/2 * log(1 + SNR). Hamming that same year built the first error-correcting code. The next 60 years: engineers build codes approaching the bound - turbo codes (Berrou 1993), LDPC (Gallager 1960, rediscovered 1995), polar codes (Arikan 2008). In 2016 polar codes entered the 5G NR standard - the first technology theoretically achieving capacity.
Channel Capacity and Shannon's Theorem
1948. Bell Labs. Shannon publishes 'A Mathematical Theory of Communication'. In 55 pages - the entire foundation of digital communications. The central result: for every noisy channel there exists a limit C below which reliable transmission is possible and above which it is not. No tradeoff. A hard boundary.
5G NR uses OFDM with 3300 subcarriers at 30 kHz each - each is a separate AWGN channel. Total capacity is the sum of C_i across all subcarriers. This is why 5G adapts modulation from QPSK to 256-QAM depending on the per-subcarrier SNR - water-pouring over frequency in real time.
The factor 1/2 in the real AWGN formula is not accidental. A real Gaussian channel uses only one quadrature. Complex baseband: C = log2(1 + P/N). This is why modern standards use I/Q modulation to achieve both quadratures.
What is the capacity of an AWGN channel with signal power P and noise power N?
Shannon-Hartley formula: C = 1/2 * log2(1 + SNR) bits per channel use. At SNR=10 (10 dB): C = 1/2 * log2(11) ~ 1.73 bits/use. Achieved by a Gaussian input.
The Channel Coding Theorem
Before Shannon, engineers believed reliability and speed were in direct conflict: add redundancy to improve reliability, but sacrifice rate. Shannon proved otherwise. Up to rate C, transmission with arbitrarily small error probability is possible. The paradox resolves through block length: large n is required.
Polar Codes: the First Achievement of C
Erdal Arikan (2008) proved that polar codes achieve the capacity of any symmetric binary-input channel as n grows to infinity. This is the first explicit (non-random) construction that provably achieves C. In 2016, 3GPP adopted polar codes for 5G NR control channels. SCL (Successive Cancellation List) decoding provides practical performance at block lengths of 512-1024 bits.
What does the converse of the channel coding theorem state at R > C?
Converse: at R > C, every code has P_e >= 1 - C/R - 1/n > 0. This is a hard limit - increasing n does not help.
Water-Filling Over Parallel Channels
Real channels are not flat-spectrum AWGN. Wi-Fi operates in the ISM band with fading and interference. Optical fiber has dispersion. Mobile communications has multipath propagation. The general principle remains: water-filling over frequency or spatial modes.
In water-filling, what happens to a channel with noise level N_k > mu?
p_k* = max(0, mu - N_k). When N_k > mu: p_k* = 0. There is no benefit spending power on a very noisy channel - better to allocate it elsewhere. This is why OFDM systems turn off subcarriers with poor SNR.
Capacity Under Fading
AWGN is an idealization. A real 5G channel has Rayleigh fading, Doppler shift, and inter-cell interference. For random H, capacity needs two concepts depending on how fast the channel varies relative to the codeword length.
LTE and 5G: C in a Real Network
LTE Category 20 with 4x4 MIMO, 5 CA, 256-QAM: theoretical peak 2 Gbps. Actual speed in-network: 50-300 Mbps. The gap: code rate 0.93, control channel overhead, fading, and inter-cell interference. 5G mmWave at 400 MHz and SNR 40 dB: theoretical C ~ 5 Gbps, indoor practical ~ 1-2 Gbps accounting for all losses.
What is the key difference between ergodic capacity and outage capacity for a fading channel?
Fast fading: codeword covers many fading states - average. Slow fading: H is fixed over the block - cannot average. Need C_out: rate guaranteed in (1-epsilon) fraction of cases. These are different operating regimes.
Connection to Other Topics
Channel capacity connects information theory to engineering practice. Mathematically C = max I(X;Y) is the dual of R(D) = min I(X;X-hat). The proof uses AEP and the method of typical sequences. In engineering: the formula C = 1/2 * log(1 + SNR) sets the limit for all standards from telephony to 5G. Water-filling is the optimal algorithm for OFDM, analogous to water-filling in R(D). Polar and LDPC codes are practical realizations of the channel coding theorem.
- Rate-Distortion Theory — Dual minimization: C = max I(X;Y) over channels mirrors R(D) = min I(X;X-hat) over sources
- Error-Correcting Codes — Capacity sets the achievable rate that LDPC, turbo, and polar codes approach
- Polar Codes: Arikan's Construction — First explicit capacity-achieving construction for binary memoryless channels
Итоги
- C = max_{p(x)} I(X;Y): capacity is a channel property, not a codec property
- AWGN: C = 1/2 * log2(1 + P/N), achieved by Gaussian input
- Coding theorem: R < C => P_e -> 0; R > C => P_e bounded below
- Water-filling: optimal power p_k* = max(0, mu - N_k) for parallel channels
Вопросы для размышления
- Why does the channel coding theorem not contradict the intuition that noise causes information loss?
- How does frequency-domain water-filling relate to water-filling distribution in R(D) theory?
- What prevents real 5G systems from operating exactly at the Shannon bound?