Estimate project
← All posts

Storing Kubernetes Secrets securely with Mozilla SOPS and IaC

SOPS is an extremely flexible tool for handling encrypted secrets. Keep reading to learn how to store Kubernetes secrets securely using Mozilla SOPS and IaC.

4 min read
KubernetesMozilla SOPSIaCKubernetes SecretsCloud

Origin story

Anyone who has worked with Kubernetes for longer than a couple of weeks has most likely faced a case where a Secret had to be changed, for example - the password of a database user that your deployed application depends on. You have to start a client such as kubectl or Lens, locate the right Secret, decode its base64 value, modify it, encode it again and update the k8s Secret. In practice, however, such secrets offer no safety whatsoever. Base64 values are just a reversible, compact encoding of the decoded content. Should they leak or fall into the hands of an attacker, that person gets the data. What’s more, following best practices like infrastructure-as-code means the passwords also have to live alongside your config. It gets messy quickly, doesn’t it?

Normally you wouldn’t put a password in a Git repository – storing secrets as plaintext is, after all, a Bad Idea™. And since Git keeps every change in its history, every earlier password ends up stored there as well. That only makes matters worse… and, once more, Base64 offers no help whatsoever.

What we need is a way to encrypt sensitive data so that it stays secure while remaining usable by other tools and systems. This is where Mozilla SOPS comes in.

Introduction

SOPS, short for Secrets OPerationS, is a CLI tool that encrypts and decrypts YAML, JSON, BINARY, ENV, and INI files with the help of a range of backends, such as PGP, age, AWS KMS, GCP KMS, Azure Key Vault and HashiCorp Vault. Thanks to that, it doesn’t tie you to any cloud. It runs on Windows, Linux, and macOS.

In my view, SOPS really excels when you use it to store confidential data in the Git repo, so that your application config stays complete and in one place.

Say you’d like to deploy PostgreSQL next to your app with Helm on AWS – there’s no need to split sensitive data from non-sensitive data. Take a look at this sample values.yaml file:

  enabled: true
  postgresqlUsername: "postgres"
  postgresqlPassword: "pass"
  postgresqlDatabase: "pass"
  service:
    port: 5432

Suppose postgresqlPassword is the only thing that has to be encrypted. Everything else can stay untouched if we prepare a SOPS config telling it to encrypt only the values of keys matching a given regular expression (regex):

creation_rules:
  - path_regexp: \.(yml|yaml)$
    encrypted_regex: "^(.*[Pp]ass.*)$"
    key_groups:
      - kms:
          - arn: "arn:aws:kms:eu-central-1:999999999999:key/<key_uuid>"
            role: "arn:aws:iam::888888888888:role/sops-data-key-encryption"

This states that only files with the .yml and .yaml extensions should be encrypted. Within those files, all content should remain unchanged apart from the values of keys whose names contain “pass” or “Pass”. We also want to use an AWS KMS key together with an IAM role that grants permission to use it.

Running the sops -e values.yaml command would give us output along these lines:

  enabled: true
  postgresqlUsername: "postgres"
  postgresqlPassword: ENC[<redacted>]
  postgresqlDatabase: "db"
  service:
    port: 5432
sops:
    kms:
        - arn: arn:aws:kms:eu-central-1:999999999999:key/<key_uuid>
          role: arn:aws:iam::888888888888:role/sops-encryption
          created_at: "2023-01-09T10:25:12Z"
          enc: <redacted>
          aws_profile: ""
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-01-09T10:25:12Z"
    mac: <redacted>
    pgp: []
    encrypted_regex: ^(.*[Pp]ass.*)$
    version: 3.7.3

Clearly, the file has become considerably bigger. Let’s walk through it.

At the top there’s our postgresql section, which remains nearly identical – only the PostgreSQL password is now encrypted. That’s precisely what we were after. What we didn’t anticipate was an entirely new sops section. This is metadata, derived from the .sops.yaml config file, which SOPS appends to every encrypted file. Apart from the file’s creation and modification timestamps and the SOPS version, it also contains a copy of the encryption/decryption key that is used later for decryption. You can write it to a new file by redirecting the output the usual Unix way:

After encryption, the file can still be opened. All it takes is a valid AWS profile and the right permissions. Run the sops values.enc.yaml command to open your default text editor and view the decrypted contents of the file.

Comparison to Sealed Secrets

SOPS operates much like Sealed Secrets, another tool for encrypting Kubernetes Secrets. The key difference is where the encryption takes place. SOPS encrypts and decrypts files on the client side, whereas Sealed Secrets performs encryption/decryption on the server side, directly in the cluster. For SOPS this can turn into a security concern: to use it in CD pipelines (which you’ll most likely want sooner rather than later), you have to give the pipeline access to the key so it can deploy the app. CI/CD solutions such as Flux or ArgoCD support SOPS, which makes this less likely to become an actual problem. Sealed Secrets does not suffer from this issue.

Sealed Secrets, however, can only encrypt the entire file, which makes the result hard for humans to read. It also lacks support for any cloud key management system – the keys are managed by a dedicated Kubernetes controller.

Summary

SOPS is an extremely flexible tool for handling encrypted secrets. It works brilliantly with YAML and JSON files, yet it can handle other formats too, binaries included. CI/CD pipelines can use it without trouble, though it’s even better with a CD tool like Flux that integrates SOPS seamlessly.

This is only an introduction – in a follow-up article I’ll demonstrate how to use it in a GitOps environment built on Flux.