DSL compilers

PostgreSQL ran a query in 185 seconds. DuckDB ran the same query in 3 seconds. The difference: SQL compiled to native code through LLVM instead of an interpreted plan. In games, shader compilation stutter comes from compiling GLSL to GPU ISA on first render. In DevOps, Dhall compiles typed configuration to JSON and catches errors before deployment. DSL compilers are specialized solutions that change what is possible inside their domain.

• **DuckDB** (in-process OLAP): compiles SQL expressions through LLVM and generates SIMD-vectorized code. 60x faster than PostgreSQL for analytical queries on a laptop
• **Apple Metal**: shaders are compiled to AIR during Xcode build time, not at runtime. That removes the shader compilation stutter present in Vulkan/OpenGL games
• **ClickHouse GROUP BY JIT**: the first query compiles the aggregation code through LLVM. Subsequent queries with the same pattern reuse the compiled code with 5-10x speedup

Query engines and compilation

Traditional query engines (Volcano/Iterator model) interpret the query plan: each operator (Scan, Join, Aggregate) implements a `next()` method, and data flows through the operator chain. Per-tuple overhead: virtual calls, branch mispredictions, cache misses. Compiling engines (Hyper, DuckDB, LLVM-based) generate native code for the entire query pipeline.

DuckDB uses push-based execution and compiles by inlining operator code into a shared pipeline. Velox (Meta, 2022) is a vectorized execution engine used in Presto and Spark that compiles expressions through an LLVM JIT. ClickHouse compiles GROUP BY aggregations through LLVM at runtime: the first query is slower, subsequent ones run up to 10x faster.

SQL compilation: from AST to native code

SQL compilation: SQL string -> lexer/parser -> AST -> semantic analysis (types, tables) -> logical plan -> optimizer (pushdown, join ordering) -> physical plan -> code generation (LLVM IR or code templates). The optimizer is the hardest part: finding the optimal plan for N joins is NP-hard.

PostgreSQL 11+ includes JIT compilation through LLVM for heavy queries: expressions, aggregation functions, tuple deforming. It triggers automatically when cost > jit_above_cost (100000). Apache Arrow Flight SQL is a protocol for transferring compiled query plans between engines. Calcite (Apache) is a framework for building SQL optimizers used by Hive, Flink, and Drill.

Shader compilers

A shader compiler translates GLSL/HLSL/Metal Shading Language into a GPU ISA (AMD GCN, NVIDIA SASS, Intel EU ISA). Pipeline: source -> frontend (Clang for Metal/HLSL) -> SPIR-V (standard intermediate format) -> vendor backend -> GPU binary. Shaders compile at first use, which causes stuttering.

Shader compilation stutter is a known problem in games: Cyberpunk 2077 at launch compiled shaders during gameplay. Fixes: Pipeline State Object cache (DirectX 12), Metal shader precompilation (Apple), Vulkan pipeline cache. Apple Metal through Xcode compiles Metal shaders to AIR (Apple Intermediate Representation) at application build time, avoiding runtime compilation.

Configuration languages and compilation

Configuration DSLs (Dhall, CUE, Jsonnet, Nix) are languages with type systems, compilation to JSON/YAML, and correctness guarantees. Dhall is fully deterministic and total: no IO, no infinite loops, always terminates. That makes it possible to statically verify configuration before deployment.

Dhall is used at Tweag (a functional consulting firm) to generate Kubernetes YAML. Its type system catches configuration errors before deployment. Pulumi (Infrastructure as Code) uses TypeScript/Python as a DSL with full type checking. HashiCorp Sentinel is a policy language compiled to bytecode for Terraform/Vault access policies.

Key ideas

• Query engines: a compiled pipeline (DuckDB, Hyper) removes virtual dispatch and enables SIMD. 10-100x faster than Volcano on analytical queries
• Shader compilers: GLSL/HLSL -> SPIR-V (vendor-neutral IR) -> GPU ISA. Shader stutter from runtime compilation is solved by pipeline caches and precompilation
• Config DSLs (Dhall, CUE): a type system plus totality guarantee allows static verification of configuration before deployment

Related topics

DSL compilers apply compiler techniques to specialized domains:

• MLIR — MLIR is used in ML compilers as a domain-specific IR. Similar approach to SPIR-V for GPUs
• JIT basics — ClickHouse and DuckDB apply JIT compilation to SQL during query execution
• LLVM infrastructure — PostgreSQL JIT, DuckDB, and ClickHouse use LLVM as the backend for SQL compilation

Вопросы для размышления

• DuckDB compiles SQL through LLVM. At what data size does the 100ms compilation overhead stop paying off?
• Shader compilation stutter can be removed with precompilation. But shaders are often parameterized (materials, lighting). How do game engines deal with the combinatorial explosion of shader variants?
• Dhall guarantees termination (totality). Which computations become impossible? What is inexpressible in Dhall that a Turing-complete config language allows?

Связанные уроки

• fl-01-intro