Use Sops for Secrets in Helm

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

Helm is an open source package manager for Kubernetes. It provides the ability to provide, share, and use software built for Kubernetes.

Using and storing secrets in Helm poses a problem. If we store secretes in plain text in Helm config files then we can not share and store our Helm configs in version control system. Storing plain text secrets is very bad security practice. To overcome this issue we can used sops with Helm secret plugin to store encrypted secrets in our version control. These secrets will be then be decrypted at install time minimizing the exposure of secret data.

Steps to use Sops for Secrets in Helm

To user sops first you need to install it in your environment. Download the latest release from here and place it in your path.

Next you’ll need to install Helm-secrets plugin. Use the following command to install Helm-secrets plugin.

helm plugin install https://github.com/jkroepke/helm-secrets --version v3.6.0

We’ll create a new chart to create out secret. Creating a chart will also allow us to version it using helm releases.

In this post we are going to encrypt our secret with out pgp key. But we can use a variety of other options including aws kms and gcp kms to encrypt our data.

First create a new Helm chart.

helm create my-secret
cd helm-testses

We will modify Helm chart a bit for our usecase.

Delete everything from except _helper.tpl file.

Update contents of values.yaml file as follwos.

# Default values for app-secrets.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Name of application that will use secrects and/or certificate
appName: "my-secret"

secrets: {}

Create a file name secrets.yaml in templates directory with following content.

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.appName }}-secrets
  labels:
    app: {{ .Values.appName }}-secrets
type: Opaque
data:
{{- range $key, $val := .Values.secrets }}
  {{ $key }}: {{ $val | b64enc | quote }}
{{- end }}

Run gpg --list-secret-keys to list your pgp keys.

➜  my-secret gpg --list-secret-keys
gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.3.1)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
/Users/dir/.gnupg/pubring.kbx
----------------------------------------
sec   ed25519 2021-05-26 [SC] [expires: 2023-05-26]
      7E6CBE978CCACDCBCC4E7F8006A2FX2FAX66X2XX
uid           [ultimate] john doe 
ssb   cv25519 2021-05-26 [E] [expires: 2023-05-26]
ssb   rsa2048 2021-05-26 [A] [expires: 2023-05-26]

Copy the key hash in bold above. We will use it in next step.

Create a new directory and create a file containing your secret ther.

mkdir secrets/my-secret.yaml

Add following content in that file.

secrets:
    MY-SECRET: secret-password

Now we are going to encrypt this file with out pgp key.

sops --encrypt --pgp 7E6CBE978CCACDCBCC4E7F8006A2FX2FAX66X2XX --in-place secrets/secret-one.yml

Now our file is encrypted. It will look like this.

secrets:
    MY-SECRET: ENC[AES256_GCM,data:A24t/SsvsH4=,iv:jefALk6/tRJFbdf0oN1uSYWKBhjU+eThWbGTdDtoBr8=,tag:gf8M+C3we63waHTBijBdYQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2021-06-11T19:12:12Z"
    mac: ENC[AES256_GCM,data:cMDft0vbpu4tTPDJewmPvVF1ij+PKbCXunJf67S5ODNMoiqIf9Qez6oxWmSzMk1tEF+nTakqgODmTs0LBLsHOUhUS+tC0siPWhaOLSRjzFB1QXDzo4SA/WoVqJ8b4cnpDwcX1yHfqRgMI8bT2Yg3Yb+GEChagEVOS+JEYmu/DSU=,iv:aPfcBnXGIZZ9Nd+3ZW9R1efXCBOyatcl5RHcKtiT86A=,tag:ZtczRHojSG27xAxzKIqlYA==,type:str]
    pgp:
        - created_at: "2021-06-11T19:12:11Z"
          enc: |
            -----BEGIN PGP MESSAGE-----

            hF4DJOQ8uoHoWuYSAQdAiqMYiO4AkRildvkQVKOiMxeGZxCX9mExlbdGHzx7fl0w
            co4VFit40cwo34S02b+FAX7JWxq7UB/MTvxEJyaOXmNysejJW1TutF/lwWRZm1sM
            1GgBCQIVk8xLHsgz0RKzc2ffBmO+smwruPOf9p07zVkGI+qJKFmmf3GlEsBlP30c
            V/wILgREeGj3aiHYFdXVDiZZoe0y3PAto9+W7Sy7NZ1xOWTXQvBbw+DwEg/XkoS0
            fVKJztGiPQGtUA==
            =NvjV
            -----END PGP MESSAGE-----
          fp: 6D6CBE978BEABDBBAC4E7F8006A2FC2FA066824C
    unencrypted_suffix: _unencrypted
    version: 3.7.1

It contains our encrypted secret with some metadata from sops command.

Now we can put this file in our version control system.

When its time to install this secret in out cluster we can do the following steps to install decrypted secret. We will need gpg key in order to decrypt it.

helm secrets install my-secret --atomic -f secrets/secret-one.yml .

In order to just view the rendered helm chart run following command.

helm secrets template my-secret --atomic -f secrets/secret-one.yml .

[helm-secrets] Decrypt: secrets/secret-one.yml
---
# Source: my-secret/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-secret-secrets
  labels:
    app: my-secret-secrets
type: Opaque
data:
  MY-SECRET: "cGFzc3dvcmQ="

[helm-secrets] Removed: secrets/secret-one.yml.yaml.dec

So now we have a complete workflow to keep our secret data secret and take full advantage of version control system to manage our helm charts.