Implementing SOPS - GitOps secrets management

The Secret Management Problem

With my entire infrastructure living in Git via FluxCD, I had one glaring gap: secrets. API tokens, passwords, SSH keys - everything that makes services actually work, couldn't be committed to a public repository.

The only approach fitting the "total reproducibility" goal is keeping the secrets encrypted inside the repository and decrypting them via Flux directly inside the cluster.

The solution: SOPS (Secrets OPerationS) with Age encryption.

⚙️

SOPS, Age, FluxCD, Ansible Vault

Why SOPS?

SOPS encrypts YAML files in place, leaving the structure readable but encrypting only the values. This means:

Encrypted secrets can live in Git alongside everything else
File structure remains visible (helpful for debugging)
FluxCD can decrypt automatically using a private key stored in the cluster
No external secret management service needed

Example encrypted secret:

apiVersion: v1
kind: Secret
metadata:
    name: grafana-admin
    namespace: monitoring
type: Opaque
stringData:
    admin-password: ENC[AES256_GCM,data:VnuVGrCk,iv:rhlUYsXMkg2RdLarqr3VLusPqAXW8q3day8MrJ90478=,tag:cQ35wrhayOcfECRGA1z1CA==,type:str]

The structure is readable, the value is encrypted. Perfect for GitOps.

Implementation

Step 1: Install SOPS and Generate Keys

On my workstation:

# Install SOPS via the official installation method

# Generate Age key pair
age-keygen

# Output looks like:
# Created: 2026-01-30T10:30:00Z
# Public key: age1t35sxk7cxkxk0rgfdmnkcnvky7z...
# Private key: AGE-SECRET-KEY-1XXXXXXXXXXXXXX...

The public key encrypts secrets. The private key decrypts them.

Step 2: Store Private Key Securely

I stored the private key in Ansible Vault alongside my GitHub PAT:

# ansible/inventory/group_vars/all/secrets.yml 
# (encrypted with ansible-vault)
vault_github_token: "github_pat_xxxxxxxxxxxx"
sops_age_public_key: "age1t35sxk7cxkxk0rgfdmnkcnvky7z..."
sops_age_private_key: "AGE-SECRET-KEY-1XXXXXXXXXXXXXX..."

This keeps the decryption key out of Git while making it available during cluster provisioning.

Step 3: Configure SOPS

Created .sops.yaml in the repository root:

creation_rules:
  - path_regex: .*\.sops\.yaml$
    age: age1t35sxk7cxkxk0rgfdmnkcnvky7z...

This tells SOPS: "Any file ending in .sops.yaml should be encrypted with this Age public key."

Usage:

# Create the secret.sops.yaml file in plain text, then:
apiVersion: v1
kind: Secret
metadata:
  name: example
stringData:
  password: supersecret

# Export the key, directly from Ansible Vault or as a string:
export SOPS_AGE_KEY_FILE="PRIVATE_KEY"

# Encrypt only stringData
sops --encrypt --in-place --encrypted-regex '^(data|stringData)$' secret.sops.yaml

Step 4: The FluxCD Bootstrap Problem

Here's where things got interesting.

The original approach (flux bootstrap):

flux bootstrap github \
  --owner=kristiangogov \
  --repository=homelab \
  --branch=staging \
  --path=clusters/staging

The problem:

I'd add SOPS decryption config to the cluster
FluxCD would detect the change in Git
Flux would re-bootstrap itself
The bootstrap process would wipe the SOPS configuration
Cluster couldn't decrypt secrets anymore

Classic bootstrapping paradox: The automation that manages secrets can't manage its own secret management configuration.

Step 5: Manual Flux Installation (Still Automated)

Instead of using flux bootstrap, I switched to applying Flux manifests directly from Git.

The new approach:

Install Flux CLI
Create the flux-system namespace manually
Create necessary secrets (GitHub credentials, SOPS decryption key)
Apply Flux components from Git
Flux takes over from there

The Ansible implementation:

- name: Check if Flux is already installed
  shell: kubectl get ns flux-system
  register: flux_check
  failed_when: false
  changed_when: false

- name: Install Flux CLI
  shell: curl -s https://fluxcd.io/install.sh | bash
  when: flux_check.rc != 0

- name: Create flux-system namespace
  shell: kubectl create namespace flux-system
  when: flux_check.rc != 0

- name: Create GitHub credentials secret
  shell: |
    kubectl create secret generic flux-system \
      --namespace=flux-system \
      --from-literal=username=kristiangogov \
      --from-literal=password='{{ vault_github_token }}' \
      --dry-run=client -o yaml | kubectl apply -f -
  when: flux_check.rc != 0

- name: Create sops-age secret for decryption
  shell: |
    kubectl create secret generic sops-age \
      --namespace=flux-system \
      --from-literal=age.agekey='{{ vault_sops_age_private_key }}' \
      --dry-run=client -o yaml | kubectl apply -f -
  when: flux_check.rc != 0

- name: Seed FluxCD from Git Manifests
  shell: |
    kubectl apply -f https://raw.githubusercontent.com/kristiangogov/homelab/staging/clusters/staging/flux-system/gotk-components.yaml
    kubectl apply -f https://raw.githubusercontent.com/kristiangogov/homelab/staging/clusters/staging/flux-system/gotk-sync.yaml
  when: flux_check.rc != 0

- name: Wait for Flux Controllers to be ready
  shell: |
    kubectl wait --for=condition=ready pod \
      -l app.kubernetes.io/part-of=flux \
      -n flux-system --timeout=300s
  when: flux_check.rc != 0

- name: Force final reconciliation
  shell: |
    flux reconcile source git flux-system
    flux reconcile kustomization flux-system
  when: flux_check.rc != 0
  ignore_errors: true

What this does:

Checks if Flux is already running (idempotent)
Installs Flux CLI if needed
Creates namespace and secrets with decryption key
Applies Flux components directly from GitHub
Waits for controllers to be ready
Forces reconciliation to sync everything from Git

The result: Flux installs itself from Git, never bootstraps again, and the SOPS configuration persists.

Testing Total Reproducibility

The real test: destroy everything and rebuild.

make destroy
make apply
make provision

What happens:

VMs provisioned (Terraform)
K3s installed (Ansible)
Flux installed with SOPS key (Ansible)
Flux syncs from Git (automatic)
All secrets decrypt successfully (automatic)
Services come up with correct credentials (automatic)

Time: ~15 minutes
Manual intervention: Still Zero

Everything that was manual is now automated. Everything that was in plain text is now encrypted.

What I Learned

SOPS is elegant but has sharp edges:

Encryption is straightforward (sops --encrypt --in-place --encrypted-regex '^(data|stringData)$' secret.sops.yaml)
Git integration works perfectly
The bootstrap problem isn't documented anywhere

FluxCD's bootstrap is opinionated:

Works great for most use cases
Assumes you control the entire bootstrap process
Doesn't play nice with pre-existing secret management

The manual installation approach is more flexible:

Full control over the installation process
Secrets can be seeded before Flux starts
Still fully GitOps - Flux manages itself after initial install

Ansible Vault + SOPS is a good pattern:

Ansible Vault protects the SOPS private key
SOPS protects Kubernetes secrets
Layered security at different stages

The Complete Flow

Development workflow:

Create a secret locally: cat > secret.sops.yaml
Encrypt it: sops --encrypt --in-place --encrypted-regex '^(data|stringData)$' secret.sops.yaml
Commit to Git: git add secret.sops.yaml && git commit
Push: git push
Flux detects change, pulls, decrypts, applies

Disaster recovery:

Run make provision
Ansible installs K3s and Flux with SOPS key
Flux syncs from Git
All secrets decrypt and services start
Cluster fully operational

No manual secret management. No plain text. Full reproducibility.

Up next

Networking overhaul with Cilium and Gateway API
If all goes according to plan, migrate the bare-metal production cluster to VMs

Series: Building a Production-Grade Lab

The repository is public and available at github.com/kristiangogov/homelab. Feel free to explore the manifests, open issues with suggestions, or reach out if you're building something similar!