Skip to content

Secrets Management Runbook

This repo uses SOPS + age for secrets at rest. Private keys are stored in 1Password; encrypted files live in git alongside the code they configure.

Architecture

┌─────────────────────────────────────────────────────┐
│  .sops.yaml (in repo)                               │
│  Lists age public keys as recipients                │
└──────────┬──────────────────────────┬───────────────┘
           │                          │
    ┌──────▼──────┐           ┌───────▼──────┐
    │  Operator    │           │  CI (GitHub  │
    │  age key     │           │  Actions)    │
    │  (1Password) │           │  age key     │
    └──────┬──────┘           └───────┬──────┘
           │                          │
           ▼                          ▼
    sops -d secrets/foo.sops.yaml    sops -d (in workflow)

Bootstrap (new admin machine)

# 1. Sign in to 1Password CLI
op signin

# 2. Fetch the age key from 1Password and install it
./scripts/bootstrap-secrets.sh

# 3. Verify
sops -d secrets/example/demo.sops.yaml

The key is written to ~/.config/sops/age/keys.txt (the standard sops location). SOPS finds it automatically — no env vars needed.

On managed hosts

Managed hosts store the automation age key at /etc/homelab/age-key.txt. Non-interactive sessions (SSH, scripts, systemd) must set the key file path explicitly:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d <file>

For interactive shells, add this to the user's profile (.bashrc, .zshrc):

export SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt

The key file is owned by root:docker with mode 440, so users in the docker group can read it.

First-time setup (generating keys)

Only run this once, when creating the repo's encryption keys:

# Generate operator age key and store in 1Password
./scripts/setup-age-key.sh

# The script prints the public key. Add it to .sops.yaml:
#   OPERATOR_AGE_PUBLIC_KEY → replace with your actual public key

# Generate CI key (for GitHub Actions)
age-keygen -o /tmp/ci-age-key.txt
# Copy the public key to .sops.yaml as CI_AGE_PUBLIC_KEY
# Copy the private key to GitHub repo secret: SOPS_AGE_KEY
# Then delete /tmp/ci-age-key.txt

Encrypting a new secret

# Create and encrypt in one step (opens $EDITOR)
sops secrets/services/my-service.sops.yaml

# Or encrypt an existing plaintext file
sops -e -i my-plaintext.yaml

SOPS uses the creation rules in .sops.yaml to determine which keys to use based on the file path.

Decrypting

# Decrypt to stdout
sops -d secrets/services/my-service.sops.yaml

# Decrypt in place (careful — don't commit the plaintext!)
sops -d -i secrets/services/my-service.sops.yaml

Editing an encrypted file

# Opens in $EDITOR, re-encrypts on save
sops secrets/services/my-service.sops.yaml

Key rotation

If a key is compromised or an operator leaves:

  1. Generate a new age key: age-keygen
  2. Update .sops.yaml with the new public key
  3. Remove the compromised key from .sops.yaml
  4. Re-encrypt all files with the new key set:
# Find all SOPS files and update their recipients
find secrets/ -name '*.sops.yaml' -exec sops updatekeys {} \;
find services/ -name '.env.sops.yaml' -exec sops updatekeys {} \;
  1. Commit the re-encrypted files
  2. Store the new private key in 1Password (replace the old one)
  3. If the CI key was compromised, rotate the GitHub Actions secret too

Compromise response

If you suspect a key has been leaked:

  1. Immediately rotate the affected key (see above)
  2. Assume all secrets encrypted with that key are compromised
  3. Rotate all downstream credentials (API tokens, passwords, etc.)
  4. Check git history — if plaintext was ever committed, consider the repo's full history compromised. Use git filter-repo to scrub.
  5. Update the security register: docs/security-register.md

Decrypting SOPS YAML to dotenv

SOPS YAML files use key: value format, but Docker Compose .env files expect KEY=VALUE. When decrypting a SOPS file to a .env for a Compose stack, convert the format:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env

If the SOPS key name differs from the environment variable the service expects, chain an additional sed substitution. For example, Traefik expects CLOUDFLARE_DNS_API_TOKEN but the SOPS file uses the shorter CF_DNS_API_TOKEN:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/CF_DNS_API_TOKEN: /CLOUDFLARE_DNS_API_TOKEN=/' > .env

File conventions

Path pattern Purpose
secrets/ansible/*.sops.yaml Ansible vault replacements
secrets/services/<id>/*.sops.yaml Per-service credentials
secrets/appliances/<id>/*.sops.yaml Appliance configs (e.g., Saltbox)
services/<id>/.env.sops.yaml Compose stack environment variables

Saltbox integration

Saltbox expects two config files on disk:

File Source (encrypted) Destination on saltierpoop
accounts.yml secrets/appliances/saltierpoop/accounts.sops.yaml /srv/git/saltbox/accounts.yml
settings.yml secrets/appliances/saltierpoop/settings.sops.yaml /srv/git/saltbox/settings.yml

These are decrypted by Ansible during the saltbox-host role (Phase 3) and placed at the paths Saltbox expects. The role uses sops -d and writes the plaintext to the target host. The plaintext never exists in git.

CI integration

GitHub Actions uses a dedicated age key (separate from the operator key) so that CI can validate encrypted files without having access to the operator's 1Password.

GitHub Secret Value
SOPS_AGE_KEY The CI age private key (full content of the key file)

The CI workflow sets SOPS_AGE_KEY as an environment variable. SOPS reads it automatically (no key file needed).

Pre-commit hook

The sops-encryption-check hook in .pre-commit-config.yaml verifies that all *.sops.yaml files contain the sops metadata key — catching accidental commits of plaintext files that should have been encrypted.