Vector Operations: The Arithmetic ML Runs On

A vector is not just a list of numbers - it is a point in space, a direction, a word embedding, a pixel. Every ML model, from linear regression to GPT, is written in the language of vector operations. Understanding them is understanding the grammar of modern AI.

• ML: every training example is a feature vector
• NLP: the word 'king' is a vector of 768 numbers in BERT's embedding space
• Computer graphics: position, normal, and color of a vertex are all vectors
• Physics: velocity, acceleration, force are all vector quantities
• Finance: a portfolio is a weight vector over assets

Vector Operations: The Arithmetic ML Runs On

**A 4-layer neural network is 4 rounds of addition and matrix multiplication.** Style transfer in Stable Diffusion is a linear combination of latent vectors. Batch normalization in every ResNet layer subtracts the mean vector and divides by the standard deviation vector. Vector operations look like dull math, but they are the entire forward pass of any neural network. There is not one extra line of code beyond them.

**What this lesson actually teaches**: not "how to add components", but why the same four operations produce school geometry, word2vec, and the linear layers of neural networks. By the end, "add vectors" and "run a forward pass" will turn out to be the same thing.

Addition: component-wise, always

Addition: component-wise, always

Two vectors are added strictly component-by-component: the i-th with the i-th. No mixing across positions.

FORMULA: a + b = (a₁ + b₁, a₂ + b₂, ..., aₙ + bₙ) EXAMPLE: a = (3, 1, 4) b = (1, 5, 2) ───────────── a + b = (4, 6, 6) <- each component separately In numpy: a + b <- broadcasting, one operation across the whole array

**Geometric meaning**: place the tail of the second vector at the tip of the first. The sum is the arrow from the start of the first to the end of the second. This is exactly how bias addition works in a linear layer: the same vector b is added to every vector in the batch.

The properties of addition are not an academic list - they are tools for simplifying computations:

• **Commutativity**: a + b = b + a (order does not matter)
• **Associativity**: (a + b) + c = a + (b + c) (brackets can be moved freely)
• **Zero vector**: a + 0 = a (adding zero changes nothing)
• **Additive inverse**: a + (-a) = 0 (subtracting a vector from itself)

Subtraction: difference as displacement

Subtraction: difference as displacement

Subtraction is adding the opposite vector. Geometrically **a - b** is the arrow **from the tip of b to the tip of a** (when both start at the origin). This is exactly the operation used to find the "direction" from one object to another in embedding space.

FORMULA: a - b = (a₁ - b₁, a₂ - b₂, ..., aₙ - bₙ) EXAMPLE: a = (5, 3, 1) b = (2, 1, 4) ───────────── a - b = (3, 2, -3) INTUITION (word2vec): v(king) - v(man) = "royalty vector without the male attribute" v(Paris) - v(France) = "capital-city direction vector" This is literally vector subtraction - nothing more.

Scalar multiplication: the volume knob

Scalar multiplication: the volume knob

Multiplying by scalar k scales every component. The direction stays - only the "intensity" changes.

FORMULA: k · a = (k·a₁, k·a₂, ..., k·aₙ) WHAT HAPPENS FOR DIFFERENT k: k = 2: same direction, twice as long k = 0.5: same direction, half as long k = -1: 180° reversal, same length k = 0: vector collapses to zero EXAMPLE: v = (2, -1, 3) 3v = (6, -3, 9) -v = (-2, 1, -3) 0.5v = (1, -0.5, 1.5)

**ML application - style steering**: researchers found that scaling an activation vector by a scalar shifts the "tone" of an LLM's response. The "Activation Addition" paper (2023): simply adding 1.5 * emb("happy") to the model's hidden state reliably makes responses more positive. No fine-tuning, no RLHF - just scalar multiplication and vector addition.

Linear combination: the foundation of neural networks

Linear combination: the foundation of neural networks

A **linear combination** is a weighted sum of vectors. Literally every neuron rests on this operation.

GENERAL FORM: α₁v₁ + α₂v₂ + ... + αₙvₙ EXAMPLE - decomposition through the standard basis of R²: e₁ = (1, 0), e₂ = (0, 1) (3, 5) = 3·e₁ + 5·e₂ <- linear combination of basis vectors ONE NEURON = LINEAR COMBINATION + NONLINEARITY: output = σ(w₁x₁ + w₂x₂ + ... + wₙxₙ + b) = σ(w · x + b) where w · x is exactly a linear combination of the components of x with weights w. Matrix × vector = a set of linear combinations = the forward pass of one linear layer.

**Critical fact**: a linear combination of linear combinations is itself a linear combination. Therefore a network with no nonlinear activations is equivalent to a single linear layer, regardless of depth. Nonlinearities (ReLU, GELU, sigmoid) are what turns a stack of matrix multiplications into a powerful function approximator.

Normalization: unit length as a standard

Normalization: unit length as a standard

Normalizing a vector means dividing it by its length. The result is a vector with the same direction but unit length. Normalized vectors are the foundation of cosine similarity and vector databases.

FORMULA: v̂ = v / ||v|| where ||v|| = sqrt(v₁² + v₂² + ... + vₙ²) CHECK: ||v̂|| = ||v/||v|||| = ||v||/||v|| = 1 OK EXAMPLE: v = (3, 4) ||v|| = sqrt(9 + 16) = 5 v̂ = (3/5, 4/5) = (0.6, 0.8) ||v̂|| = sqrt(0.36 + 0.64) = 1 OK SPECIAL CASE: The zero vector cannot be normalized (division by zero). In numpy: np.linalg.norm([0,0,0]) = 0.0

The zero vector cannot be normalized - it has no direction. In production code always guard: if np.linalg.norm(v) > 1e-8 before normalizing.

Batch Normalization: subtraction and scaling in neural networks

Batch Normalization: subtraction and scaling in neural networks

**Batch Normalization** (BatchNorm) is one of the key techniques in deep learning. Inside it is vector subtraction and division by a vector - literally the operations from this lesson.

**Why BatchNorm works**: without normalization, activations in deep networks "drift" - the mean and variance shift gradually, gradients explode or vanish. BatchNorm pins the distribution after each layer. Result: ResNet-50 converges in hours, not days. All operations are vector subtraction and vector division.

Where matrix operations actually run

Where matrix operations actually run

Vector operations in ML systems

From basic operation to real application

Practice: L2 normalization

Practice: L2 normalization

Interview questions

What to take from this lesson

• **Addition/subtraction** is component-wise; a + b in a neural network = adding bias b to every element of the batch
• **Scalar multiplication** scales without changing direction; the foundation of style steering in LLMs
• **Linear combination** = one neuron = one row of weight matrix W dotted with input x
• **Normalization** v/||v|| - the standard before vector search; makes cosine_sim equal to the dot product
• **BatchNorm** - subtracting the mean vector and dividing by the std vector; not magic, just vector operations
• Two linear layers without activation = one linear layer; nonlinearity is not optional

What's next

Vector operations are the alphabet. Matrices and what to do with them come next.

• Dot product — The dot product - foundation of cosine similarity, attention, and projections
• Matrices and operations — Matrix times vector = a batch of linear combinations = one neural network layer
• Eigenvectors and SVD — PCA projects data onto principal components through the same dot products

Связанные уроки

• calc-18-partial