Vault and Secrets Management

The 2020 SolarWinds attack exposed a critical problem: static credentials in source code and CI/CD. 18,000 organizations were compromised through a single build pipeline. Dynamic secrets, automated rotation, and short-lived certificates are not optional - they are the minimum viable security posture for production systems.

• **Adobe** uses HashiCorp Vault to manage secrets for 30,000+ services - dynamic database credentials mean a single service compromise does not give access to other teams' databases.
• **Cloudflare** issues TLS certificates for internal services with 24-hour TTL via an internal CA - a compromised certificate becomes invalid within a day without manual revocation.
• **PagerDuty** fully automated rotation of all production credentials through Vault - the last manual secret rotation was in 2021.

HashiCorp Vault

HashiCorp Vault is a centralized secrets store with audit logging, fine-grained access control, and dynamic secret generation. Dynamic secrets are temporary credentials generated on-demand with a short TTL - a compromised service leaks only its own credentials, not shared ones.

Vault Audit Log records every secret access with timestamp, requester, and outcome. For SOC 2 and PCI-DSS compliance, this provides a complete access trail without storing the secret values themselves.

Sealed Secrets in Kubernetes

Kubernetes Secret in base64 is not encrypted - anyone who can read the Secret can decode it. Sealed Secrets (Bitnami) solves this: it encrypts Secrets with a cluster-specific key so the encrypted file can be safely stored in git.

Sealed Secrets: cluster-specific encryption (key rotates with cluster). External Secrets Operator: pull from existing Vault/AWS SM, no new encryption layer, better for multi-cluster environments.

Secret Rotation

Secret rotation regularly replaces credentials to limit the window of compromise. Automated rotation: AWS Secrets Manager rotates RDS passwords on a schedule. Zero-downtime rotation requires a two-phase approach: create new credential while keeping the old one valid, then cut over.

The hardest part of rotation is applications that cache credentials. Applications must support credential refresh without restart - reading from Secrets Manager on each connection attempt, not at startup.

PKI and TLS Certificates

PKI (Public Key Infrastructure) manages TLS certificates. cert-manager in Kubernetes automatically requests and renews certificates from Let's Encrypt or an internal CA. Vault PKI Engine acts as an internal CA for service-to-service mTLS certificates.

Short-lived certificates (24h) eliminate revocation complexity: a compromised certificate expires automatically. Long-lived certificates (1 year) require CRL or OCSP for revocation, which is complex to operate reliably.

Key Ideas

• **Vault Dynamic Secrets** - temporary credentials (1h TTL) per application; compromising one does not expose others.
• **Sealed Secrets / ESO** - safe git storage of K8s secrets (encrypted) or synchronization from AWS SM/Vault via External Secrets Operator.
• **Rotation + PKI** - two-phase zero-downtime password rotation; short-lived (24h) TLS certificates from Vault PKI for internal services.

Related Topics

Secrets management connects DevSecOps and service mesh security:

• DevSecOps — Secret scanning in CI/CD prevents secrets from entering git; Vault provides correct storage and rotation.
• Service Mesh: Istio, Linkerd — Vault PKI Engine issues certificates for mTLS between services; Istio Citadel is also a PKI CA.

Вопросы для размышления

• If 50 microservices share one database password - how do you migrate to Vault dynamic secrets without downtime?
• When are Sealed Secrets better than External Secrets Operator for Kubernetes, and vice versa?
• How do you set up PKI for 200 microservices with mTLS so that certificate renewal requires no manual intervention?

Связанные уроки

• sec-01
• sec-18