Use EBS volume in EKS

Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, Kubernetes has two API resources: PersistentVolume and PersistentVolumeClaim.

Amazon Elastic Block Store (EBS) is an easy-to-use, scalable, high-performance block-storage service designed for Amazon Elastic Compute Cloud (EC2).

In traditional model EBS volume is directly attached to VM in AWS and processes on VM view it as a native disk drive. In Kubernetes Cluster (EKS) we can use the same EBS volumes and directly consume them inside the application pods. The volume is still attached to a specific node inside the cluster but PV and PVC make creating and consuming the volume inside the pod easier.

I will walk you through a scenario where we will dynamically create a EBS volume and use it inside our pod.

The first step is to create a persistent volume claim (PVC). PVC is the specification for a volume. In PVC we will define what type of persistent storage we want.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name:  my-app-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: fast
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "1Gi"

Save it as pvc.yaml

The annotation for storage class is not required but we can use it if we are using a custom storage class. By default EKS cluster already has a storage class that is used by PVC if no annotation is added.

We will need to apply this manifest to provision our persistent volume that will in turn create an EBS volume.

kubectl apply -f pvc.yaml

Run this command to get status of PVC and PV.

kubectl get pvc,pv

Next we will use this persistent volume. We will create a simple deployment to use this PVC.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  selector:
    matchLabels:
      app: wordpress
      tier: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mysql-pass
              key: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: my-app-pvc

In our deployment manifest inside volumes block we use persistentVolumeClaim key to link out earlier created PVC.

After applying the deployment we can see that PV is bound to newly created pod and application can write data to this volume.

If your cluster is spanning over multiple availability zones it is good idea to create a new storage class with specific availability zones. So that volumes are created in specific zone. Also we will need to use topology aware pod scheduling to avoid any problems when pod is restarted or rescheduled.

You can create a new storage class with zone topology as follows.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: gp2-east-1b
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
reclaimPolicy: Retain
allowedTopologies:
- matchLabelExpressions:
  - key: failure-domain.beta.kubernetes.io/zone
    values:
    - us-east-1b

You’ll also need to add affinity to your deployment spec.template.spec block.

      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                values:
                - us-east-1b
            topologyKey: "failure-domain.beta.kubernetes.io/zone"    

Enable Basic Authentication and SSL on a Mongo DB instance

creating ssl key and certificate for enabling ssl

Run the following command to generate ssl certificate and key file.

openssl req -newkey rsa:2048 -new -x509 -days 365 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key
cat mongodb-cert.key mongodb-cert.crt > mongodb.pem

add the following property to mongod.conf to enable ssl

# network interfaces
net:
  ssl:
    mode: requireSSL
    PEMKeyFile: <path to pem file created above>

Restart mongo db with new configuration.

enable basic authentication

Start MongoDB without access control and create the administrator user.

uses this command to commect to ssl enabled mongo using mong-shell

mongo --ssl --sslAllowInvalidCertificates 

then run the following script

use admin

db.createUser(
  {
    user: "<admin-user>",
    pwd: "<password>",
    roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
  }
)

or in One liner form

mongo admin --ssl --sslAllowInvalidCertificates  --eval "db.createUser( { user: "<admin-user>", pwd: "<password>", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ] } )"

add the following property in mongod.conf to enable autorization. (default location for mongod.conf is /etc/mongod.conf)

security:
  authorization: enabled

Restart mongo db with new configuration (with access control).

Connect to mongodb instance and authenticate as the user administrator. Add non previlaged users to manage and control access to different DB’s.

mongo --ssl --sslAllowInvalidCertificates --port 27017 -u "root" -p "pass" --authenticationDatabase "admin"
use admin

db.createUser(
    {
      user: "<user>",
      pwd: "<password>",
      roles: [
         { role: "readWrite", db: "test" }
      ]
    }
)

If you have followed the above steps, you have successfully added a new user to your db. Try logging in with the new user and try adding document to some db.

Use Sops for Secrets in Helm

black handled key on key hole

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault, age, and PGP.

Helm is an open source package manager for Kubernetes. It provides the ability to provide, share, and use software built for Kubernetes.

Using and storing secrets in Helm poses a problem. If we store secretes in plain text in Helm config files then we can not share and store our Helm configs in version control system. Storing plain text secrets is very bad security practice. To overcome this issue we can used sops with Helm secret plugin to store encrypted secrets in our version control. These secrets will be then be decrypted at install time minimizing the exposure of secret data.

Steps to use Sops for Secrets in Helm

To user sops first you need to install it in your environment. Download the latest release from here and place it in your path.

Next you’ll need to install Helm-secrets plugin. Use the following command to install Helm-secrets plugin.

helm plugin install https://github.com/jkroepke/helm-secrets --version v3.6.0

We’ll create a new chart to create out secret. Creating a chart will also allow us to version it using helm releases.

In this post we are going to encrypt our secret with out pgp key. But we can use a variety of other options including aws kms and gcp kms to encrypt our data.

First create a new Helm chart.

helm create my-secret
cd helm-testses

We will modify Helm chart a bit for our usecase.

Delete everything from except _helper.tpl file.

Update contents of values.yaml file as follwos.

# Default values for app-secrets.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

# Name of application that will use secrects and/or certificate
appName: "my-secret"

secrets: {}

Create a file name secrets.yaml in templates directory with following content.

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.appName }}-secrets
  labels:
    app: {{ .Values.appName }}-secrets
type: Opaque
data:
{{- range $key, $val := .Values.secrets }}
  {{ $key }}: {{ $val | b64enc | quote }}
{{- end }}

Run gpg --list-secret-keys to list your pgp keys.

➜  my-secret gpg --list-secret-keys
gpg: WARNING: server 'gpg-agent' is older than us (2.2.27 < 2.3.1)
gpg: Note: Outdated servers may lack important security fixes.
gpg: Note: Use the command "gpgconf --kill all" to restart them.
/Users/dir/.gnupg/pubring.kbx
----------------------------------------
sec   ed25519 2021-05-26 [SC] [expires: 2023-05-26]
      7E6CBE978CCACDCBCC4E7F8006A2FX2FAX66X2XX
uid           [ultimate] john doe 
ssb   cv25519 2021-05-26 [E] [expires: 2023-05-26]
ssb   rsa2048 2021-05-26 [A] [expires: 2023-05-26]

Copy the key hash in bold above. We will use it in next step.

Create a new directory and create a file containing your secret ther.

mkdir secrets/my-secret.yaml

Add following content in that file.

secrets:
    MY-SECRET: secret-password

Now we are going to encrypt this file with out pgp key.

sops --encrypt --pgp 7E6CBE978CCACDCBCC4E7F8006A2FX2FAX66X2XX --in-place secrets/secret-one.yml

Now our file is encrypted. It will look like this.

secrets:
    MY-SECRET: ENC[AES256_GCM,data:A24t/SsvsH4=,iv:jefALk6/tRJFbdf0oN1uSYWKBhjU+eThWbGTdDtoBr8=,tag:gf8M+C3we63waHTBijBdYQ==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2021-06-11T19:12:12Z"
    mac: ENC[AES256_GCM,data:cMDft0vbpu4tTPDJewmPvVF1ij+PKbCXunJf67S5ODNMoiqIf9Qez6oxWmSzMk1tEF+nTakqgODmTs0LBLsHOUhUS+tC0siPWhaOLSRjzFB1QXDzo4SA/WoVqJ8b4cnpDwcX1yHfqRgMI8bT2Yg3Yb+GEChagEVOS+JEYmu/DSU=,iv:aPfcBnXGIZZ9Nd+3ZW9R1efXCBOyatcl5RHcKtiT86A=,tag:ZtczRHojSG27xAxzKIqlYA==,type:str]
    pgp:
        - created_at: "2021-06-11T19:12:11Z"
          enc: |
            -----BEGIN PGP MESSAGE-----

            hF4DJOQ8uoHoWuYSAQdAiqMYiO4AkRildvkQVKOiMxeGZxCX9mExlbdGHzx7fl0w
            co4VFit40cwo34S02b+FAX7JWxq7UB/MTvxEJyaOXmNysejJW1TutF/lwWRZm1sM
            1GgBCQIVk8xLHsgz0RKzc2ffBmO+smwruPOf9p07zVkGI+qJKFmmf3GlEsBlP30c
            V/wILgREeGj3aiHYFdXVDiZZoe0y3PAto9+W7Sy7NZ1xOWTXQvBbw+DwEg/XkoS0
            fVKJztGiPQGtUA==
            =NvjV
            -----END PGP MESSAGE-----
          fp: 6D6CBE978BEABDBBAC4E7F8006A2FC2FA066824C
    unencrypted_suffix: _unencrypted
    version: 3.7.1

It contains our encrypted secret with some metadata from sops command.

Now we can put this file in our version control system.

When its time to install this secret in out cluster we can do the following steps to install decrypted secret. We will need gpg key in order to decrypt it.

helm secrets install my-secret --atomic -f secrets/secret-one.yml .

In order to just view the rendered helm chart run following command.

helm secrets template my-secret --atomic -f secrets/secret-one.yml .

[helm-secrets] Decrypt: secrets/secret-one.yml
---
# Source: my-secret/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: my-secret-secrets
  labels:
    app: my-secret-secrets
type: Opaque
data:
  MY-SECRET: "cGFzc3dvcmQ="

[helm-secrets] Removed: secrets/secret-one.yml.yaml.dec

So now we have a complete workflow to keep our secret data secret and take full advantage of version control system to manage our helm charts.