Secrets Manager

Kubernetes (alternative)

Keeper Secrets Manager integration into Kubernetes for dynamic secrets retrieval

We recommend using the Kubernetes External Secrets Operator integration for most use cases. This document describes an alternate method of integration which does not utilize the External Secrets Operator.

Features

  • Retrieve secrets from the Keeper Vault within the Kubernetes

  • Access secrets from the Keeper Vault in real-time across all pods

  • Copy secure file attachments from the Keeper Vault to the local filesystem

For a complete list of Keeper Secrets Manager features see the Overview

Prerequisites

This page documents the Secrets Manager Kubernetes integration. In order to utilize this integration, you will need:

  • Keeper Secrets Manager access (See the Quick Start Guide for more details)

    • Secrets Manager addon enabled for your Keeper account

    • Membership in a Role with the Secrets Manager enforcement policy enabled

  • A Keeper Secrets Manager Application with secrets shared to it

About

Keeper Secrets Manager can be integrated into your K8s cluster for accessing Keeper secrets in real-time across all pods.

Setup

Create a Secrets Manager Configuration

Using Commander, create a Secrets Manager device configuration for Kubernetes. Note that this configuration is not IP locked and it is pre-initialized.

Create the configuration in Commander using this command:

secrets-manager client add --app <APP NAME> --unlock-ip --config-init k8s

Example:

My Vault> sm client add --app MyAdd --unlock-ip --config-init k8s

Successfully generated Client Device
====================================

Initialized Config:

apiVersion: v1
data:
  config: ewog2N...ICIxMCIKfQ==
metadata:
  name: ksm-config
  namespace: default
type: Opaque

IP Lock: Disabled
Token Expires On: 2021-10-13 12:45:45
App Access Expires on: Never

In the example above, copy lines 8 through 14 and place them into a file called secret.yaml. Then, If you are using a machine with kubectl installed and you have access to your cluster, add the KSM SDK config to your Kubernetes secrets.

$ kubectl apply -f secret.yaml

For more information on creating Secrets Manger configurations, see the Configuration Documentation

Alternate Method: One Time Access Token and KSM CLI

Alternatively, you can create a configuration by generating a One Time Access Token with Commander (or the Vault UI) and then using the Keeper Secret Manager CLI to create a configuration as demonstrated below (replace XX:XXX) with the One Time Access Token.

$ ksm init k8s XX:XXX

apiVersion: v1
data:
  config: ewog2N[...]ICIxMCIKfQ==
kind: Secret
metadata:
  name: ksm-config
  namespace: default
type: Opaque

If you are using a machine with kubectl installed and you have access to your cluster, the parameter --apply can be set to automatically add the KSM SDK config to your Kubernetes secrets.

The output of redeeming the token can be piped to a file and then applied via kubectl. For example:

$ ksm init k8s XX:XXX > secret.yaml
$ kubectl apply -f secret.yaml
secret/ksm-config created

Using the KSM Config

The KSM config can be pulled into your K8s containers using secrets.

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
spec:
  containers:
  - name: mycontainer
    image: my_container:XXXXX
    env:
      - name: KSM_CONFIG
        valueFrom:
          secretKeyRef:
            name: ksm-config
            key: config
  restartPolicy: Never

At runtime, the Keeper Developer SDKs running in the K8s cluster will use the environment variable KSM_CONFIG to retrieve the device configuration and communicate with the Keeper Vault.

Examples

Example 1 - Custom App using SDK

Below is a simple example that will generate a deployment and service that displays database secrets via a web application. This example uses the Keeper Python Developer SDK for the web application. The SDK will get its configuration from a Kubernetes secret and then retrieve PostgreSQL database record information from the Keeper vault.

From the Keeper Vault, a "Database" record type is created using the following information.

Now let's create our web application. The web page can be created using any of the Developer SDKs. For this example, it will being using the Python SDK. A simple Flask application with one endpoint will display the HTML that contains the Vault record secrets. The secrets are retrieved using the Keeper Notation syntax.

from flask import Flask
from keeper_secrets_manager_core import SecretsManager
import os

app = Flask(__name__)

@app.route("/")
def hello_world():
    sm = SecretsManager()
    return """
<h1>Database</h1>
<ul>
    <li>Type: {}</li>
    <li>Host: {}</li>
    <li>Port: {}</li>
    <li>Login: {}</li>
    <li>Password: {}</li>
</ul>
""".format(
       sm.get_notation(os.environ.get("DB_TYPE")),
       sm.get_notation(os.environ.get("DB_HOST")),
       sm.get_notation(os.environ.get("DB_PORT")),
       sm.get_notation(os.environ.get("DB_LOGIN")),
       sm.get_notation(os.environ.get("DB_PASS")))

The next part is creating a Dockerfile. The Dockerfile below is based off of the Python Debian images from Docker Hub.

The Python SDK uses the cryptography module. This module requires the language Rust to be installed. Other Docker Hub images may provide Rust pre-installed.

FROM python:3.10.0-slim-bullseye
RUN apt-get update \
    && apt-get install -y gcc make libffi-dev curl libssl-dev \
    && apt-get clean
RUN pip3 install --upgrade pip wheel

# cryptology requires Rust to build
RUN curl https://sh.rustup.rs -sSf > /tmp/rust.sh \
    && chmod a+x /tmp/rust.sh \
    && /tmp/rust.sh -y
ENV PATH $PATH:/root/.cargo/bin

RUN pip3 install \
    flask \
    keeper-secrets-manager-core

RUN groupadd -g 5000 demo
RUN useradd -g demo demo

# Copy our application into the image
COPY demo.py /demo.py

USER demo

ENV FLASK_APP demo

EXPOSE 5000
CMD ["flask", "run"]

Next we will build a Docker image named ksm_demo.

$ docker build -t ksm_demo .

Assuming there is access to the Kubernetes cluster, the config can be generated from the One Time Access Token and automatically applied.

$ ksm init k8s --apply XX:XXXXXXXXXXX
secret/ksm-config created
Created secret for KSM config.

You can see the secret entry when you enter kubectl get secret.

$ kubectl get secret ksm-config
NAME         TYPE     DATA   AGE
ksm-config   Opaque   1      55s

You can then make a deployment and service for the ksm_demo Docker image. This example is going to name the file ksm_demo.yaml.

Defining which secrets you need, and the config for the SDK, happen in the env section in the list of containers. In this section the KSM_CONFIG environmental variable is defined to get it's value from the ksm-config Kubernetes secret, specifically the config key of the secret.

The other environmental variables are just list of name/values. The value is Keeper Notation which the web application will send to the notation retrieval method of the SDK.

The second record in the ksm_demo.yaml file is the Service definition. This can be changed to whatever works with your Kubernetes cluster. In our example, it uses an external IP address. It's safe to use one of your Kubernetes nodes IP address, 10.0.1.18 for the example.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ksm-demo-deployment
  labels:
    app: ksm-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ksm-demo
  template:
    metadata:
      labels:
        app: ksm-demo
    spec:
      nodeSelector:
        kubernetes.io/hostname: work
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 10.0.1.207
          - 1.1.1.1
      containers:
        - name: ksm-demo
          image: ksm_demo:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
              protocol: TCP
          env:
            - name: KSM_CONFIG
              valueFrom:
                secretKeyRef:
                  name: ksm-config
                  key: config
            - name: DB_TYPE
              value: "keeper://IUCvqyWcx7sG-BGIK1R9-g/field/Type"
            - name: DB_HOST
              value: "keeper://IUCvqyWcx7sG-BGIK1R9-g/field/host[hostName]"
            - name: DB_PORT
              value: "keeper://IUCvqyWcx7sG-BGIK1R9-g/field/host[port]"
            - name: DB_LOGIN
              value: "keeper://IUCvqyWcx7sG-BGIK1R9-g/field/login"
            - name: DB_PASS
              value: "keeper://IUCvqyWcx7sG-BGIK1R9-g/field/password"
---   
apiVersion: v1
kind: Service
metadata:
  name: ksm-demo-service
spec:
  ports:
    - name: http
      port: 5000
      targetPort: 5000
      protocol: TCP
  selector:
    app: ksm-demo
  externalIPs:
    - 10.0.1.18

At this point the deployment and service are ready to be applied.

$ kubectl apply -f ksm_demo.yaml
deployment/ksm-demo-deployment created
service/ksm-demo-service created

Wait until your deployment is ready. Either monitor it via the command line or Kubernetes Dashboard.

$ kubectl get deployment ksm-demo-deployment
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
ksm-demo-deployment   1/1     1            1           46m

$ kubectl get svc ksm-demo-service
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
ksm-demo-service   ClusterIP   10.107.91.89   10.0.1.18     5000/TCP   56m

Finally, use a web browser and go to the external IP address, port 5000, and you will see the Keeper vault record database secrets.

Example 2 - NGINX SSL certificates

In this example a pod will be created that contains the stock NGINX docker image and SSL certificates retrieved from the Keeper Vault.

A Login record is created in the vault to hold the SSL certificate, private key, and certificate password.

A one time token is generated and added to the Kubernetes ConfigMap.

$ ksm init k8s XX:XXX > secret.yaml
$ kubectl apply -f secret.yaml
secret/ksm-config created

The example website is simply an index HTML page. The HTML can be stored in the ConfigMap and mounted in to the document root directory.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-html-config
  namespace: default
  labels:
    app: nginx
data:
  index.html: |
    <html>
      <head>
         <title>Nginx Test Page</title>
      </head>
      <body>
        <h1>Hello From Keeper Secrets Manager!</h1>
      </body>
    </html>

The default.conf will be overwritten by our example. The certificate, key, and password will be placed in to the /etc/keys directory. For non-interactive startup, NGINX requires the certificate password be placed in a file and ssl_password_file to be included in the server configuration.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: default
  labels:
    app: nginx
data:
  default.conf: |
    server {
      listen 80 default;
      server_name localhost;
      location / {
        root /var/www/nginx-default;
        index index.html index.htm;
      }
    }
    server {
      listen 443 ssl;
      server_name localhost;
      ssl_certificate /etc/keys/example.com.crt;
      ssl_certificate_key /etc/keys/example.com.key;
      ssl_password_file /etc/keys/global.pass;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_ciphers HIGH:!aNULL:!MD5;
      location / {
        root /var/www/nginx-default;
        index index.html index.htm;
      }
    }

The deployment for this example looks like the following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      initContainers:
        - name: ksm
          image: keeper/keeper-secrets-manager-writer:latest
          env:
            - name: KSM_CONFIG
              valueFrom:
                secretKeyRef:
                  name: ksm-config
                  key: config
            - name: SECRETS
              value: |
                5x0v0VFwYj2VvuhamrJhzQ/field/password > file:/etc/keys/global.pass
                5x0v0VFwYj2VvuhamrJhzQ/file/example.com.crt > file:/etc/keys/example.com.crt
                5x0v0VFwYj2VvuhamrJhzQ/file/example.com.key > file:/etc/keys/example.com.key
          volumeMounts:
            - mountPath: "/etc/keys"
              name: keys-volume
      containers:
        - name: nginx
          image: nginx:1.21.4-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
              protocol: TCP
            - containerPort: 443
              protocol: TCP
          volumeMounts:
            - mountPath: "/etc/keys"
              name: keys-volume
            - mountPath: "/etc/nginx/conf.d/default.conf"
              name: nginx-config-file
              subPath: default.conf
            - mountPath: "/var/www/nginx-default/index.html"
              name: nginx-html-file
              subPath: index.html
      volumes:
        - name: keys-volume
          emptyDir: {}
        - name: nginx-config-file
          configMap:
            name: nginx-config
        - name: nginx-html-file
          configMap:
            name: nginx-html-config
      imagePullSecrets:
        - name: my-docker-hub-secrets

The docker image keeper/keeper-secrets-manager-writer is used for an initialization container. The container will retrieve the secrets and write them to disk so they can be used by NGINX. The secrets are written to the pod's emptyDir volume, mounted to /etc/keys. This directory will be removed when the pod is deleted.

Documentation for the Keeper Secrets Manager Writer can be found here.

The main container will also mount the pod's emptyDir volume to /etc/keys. And will also mount the default.conf into /etc/nginx/conf.d and the index.html file into the document root for the server.

The last part is to create a service to access NGINX as shown below.

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
    - name: http
      port: 80
      targetPort:  80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app: nginx
  externalIPs:
    - XXX.XXX.XXX.XX

The service can be tested by going to the external IP via https (ie https://XXX.XXX.XXX.XXX). Notice the lock in the address bar indicates the certificate is valid.

External Secrets

External Secrets is a Kubernetes operator that synchronizes secrets from External APIs and injects them into Kubernetes. For more information on how to setup External Secrets to synchronize secrets from your Keeper Vault into Kubernetes, visit the following:

Kubernetes External Secrets Operator

Next Steps

At this point, you can now integrate Keeper Secrets Manager into your K8s deployments using any of the Secrets Manager SDKs.

PreviousKubernetes External Secrets OperatorNextLinux Keyring

Last updated