Health Checks

Monitoring the Keeper Gateway using health checks

Overview

This document describes the health check functionality implemented for the KeeperPAM Gateway. Health checks provide essential monitoring capabilities that solve several common operational challenges.

Health Checks provide the following benefits:

Allow load balancers to automatically remove unhealthy instances from rotation and add them back when they recover.
Integrate with monitoring systems (Prometheus, Nagios, Datadog, etc.) to provide automated alerting and dashboards showing gateway health across your infrastructure.
Enable automated monitoring scripts and orchestration tools to detect failures and trigger recovery procedures without human intervention.

On native installs (Windows, Linux), the health check service is disabled by default. You must activate it as documented in the next sections.

Basic Configuration

The below configuration enables a basic health check service on the binary and Docker installation methods. More advanced configuration is also available as documented below.

Activating the Health Check

You can enable the health check service, add the following CLI property / environment variable on your gateway:

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: 'true'

See how to configure gateway environment variables.

In a Docker Compose file, you would also need to configure the healthcheck:

keeper-gateway:
    ...
    environment:
      ...
      KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: 'true'
    healthcheck:
      test:
      - CMD
      - /usr/local/bin/keeper-gateway
      - health-check
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    ...
    restart: unless-stopped

--health-check

See how to configure gateway CLI arguments.

Consulting the Health Check

Consult your gateway's health check once it has been activated.

CLI

On native Windows and Linux installations, you can consult the health check with the gateway CLI:

gateway health-check
>> OK: Gateway is running and connected

If you get an error like "Could not connect to health check server", it means you haven't enabled the health check properly.

If you see "Exception No such command 'keeper-gateway.exe'", you're using the wrong command syntax. Always use "gateway" as the command name.

Docker

On Docker installations, you can use Docker's inspect method to retrieve the health check.

If the container name is keeper-gateway, a one-line bash command to find the service status can be found like this:

docker inspect --format='{{.State.Health.Status}}' keeper-gateway
>> healthy

If you don't know the container name, this script will give it to you:

docker ps --filter "status=running" --format "{{.Names}} {{.Image}}" | grep keeper-gateway | awk '{print $1}'

Here's an example of checking the health with a bash command:

$ docker inspect --format='{{.State.Health.Status}}' my-gateway-container
>> healthy

The below complete bash script can be added to your watchdog services to check service status and automatically restart the container if it's unhealthy. Replace /path/to/ with the proper path.

watchdog.sh

#!/bin/bash

LOGFILE="/path/to/keeper-watchdog.log"

# Find running container matching keeper-gateway
CONTAINER_NAME=$(docker ps --filter "status=running" --format "{{.Names}} {{.Image}}" | grep keeper-gateway | awk '{print $1}')

if [ -z "$CONTAINER_NAME" ]; then
  echo "$(date): No running keeper-gateway container found." >> "$LOGFILE"
else
  # Get health status
  HEALTH=$(docker inspect --format='{{.State.Health.Status}}' "$CONTAINER_NAME")

  if [ "$HEALTH" != "healthy" ]; then
    echo "$(date): $CONTAINER_NAME is $HEALTH. Restarting..." >> "$LOGFILE"
    docker restart "$CONTAINER_NAME"
  else
    echo "$(date): $CONTAINER_NAME is healthy." >> "$LOGFILE"
  fi
fi

# Trim log file to last 100 lines
tail -n 100 "$LOGFILE" > "$LOGFILE.tmp" && mv "$LOGFILE.tmp" "$LOGFILE"

To schedule this health check on a Linux system, it can be added to the cron

crontab -e

Add to the crontab to watch every minute...

* * * * * /path/to/watchdog.sh

Advanced Configuration

The below section provides detailed configuration for customization of the health checks in different environments.

Activating the Health Check

Configuration

Start Command

Basic HTTP

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true

HTTP with Auth

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mytoken

HTTPS (SSL)

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem KEEPER_GATEWAY_SSL_KEY: /path/key.pem

HTTPS with Auth

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem KEEPER_GATEWAY_SSL_KEY: /path/key.pem KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mytoken

Custom Port

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true

KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8443

Custom Host

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true

KEEPER_GATEWAY_HEALTH_CHECK_HOST: 0.0.0.0

Production Setup

KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /path/cert.pem KEEPER_GATEWAY_SSL_KEY: /path/key.pem

KEEPER_GATEWAY_HEALTH_CHECK_HOST: 0.0.0.0

KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8443 KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: $(cat /etc/secrets/token)

Configuration

Start Command

Basic HTTP

gateway start \

--health-check

HTTP with Auth

gateway start \

--health-check \

--health-check-auth-token mytoken

HTTPS (SSL)

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem

HTTPS with Auth

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem \

--health-check-auth-token mytoken

Custom Port

gateway start \

--health-check \

--health-check-port 8443

Custom Host

gateway start \

--health-check \

--health-check-host 0.0.0.0

Production Setup

gateway start \

--health-check \

--health-check-ssl \

--health-check-ssl-cert /path/cert.pem \

--health-check-ssl-key /path/key.pem \

--health-check-host 0.0.0.0 \

--health-check-port 8443 \

--health-check-auth-token $(cat /etc/secrets/token)

You can consult our Gateway Environment Variables documentation to find more about default definitiions for these properties.

Consulting the Health Check

Configuration

CLI Health Check

Curl Health Check

Basic HTTP

gateway health-check

curl https://127.0.0.1:8099/health

HTTP with Auth

gateway health-check \

--token mytoken

curl https://127.0.0.1:8099/health \

-H "Authorization: Bearer mytoken"

HTTPS (SSL)

gateway health-check \

--ssl

curl https://127.0.0.1:8099/health \ -k

HTTPS with Auth

gateway health-check \

--ssl \

--token mytoken

curl https://127.0.0.1:8099/health \ -k

-H "Authorization: Bearer mytoken"

Custom Port

gateway health-check \

--port 8443

curl http://127.0.0.1:8443/health

Custom Host

gateway health-check \

--host 0.0.0.0

curl http://0.0.0.0:8099/health

Production Setup

gateway health-check \

--host 0.0.0.0 \

--port 8443 \

--ssl \

--token $(cat /etc/secrets/token)

curl -k https://0.0.0.0:8443/health \

-H "Authorization: Bearer $(cat /etc/secrets/token)"

Output Format Examples

Output Format

CLI Command

Description

Simple Status

gateway health-check \

--token mytoken

Returns:

OK: Gateway is running and connected

CRITICAL: ...

Detailed Info

gateway health-check \

--token mytoken \

--info

Returns:

Key=value pairs suitable for monitoring scripts

JSON Format

gateway health-check \

--token mytoken \

--json

Returns:

Full JSON response matching HTTP endpoint

HTTP Health Check

You can use the health check features detailed above to enable a HTTP health check for your gateway.

Usage

When enabled, the HTTP health check endpoint will be available at:

http://localhost:8099/health

Or with SSL:

https://localhost:8099/health

Response Format

The endpoint returns:

HTTP 200 if the gateway is healthy
HTTP 503 if the gateway is not healthy
JSON response with details

Sample Healthy Gateway

{
  "status": "healthy",
  "message": "Gateway is running and connected",
  "details": {
    "timestamp": 1784637354,
    "version": 2,
    "hostname": "localhost.localdomain",
    "instance_id": "IMJUOZ",
    "gateway_version": "1.8.6",
    "resources": {
      "memory": {
        "available_mb": 10200,
        "total_mb": 10939,
        "used_percent": 6.76,
        "headroom_percent": 93.24,
        "container_limit_mb": null
      },
      "connections": {
        "total": 0,
        "by_type": {}
      },
      "pressure": {
        "under_pressure": false,
        "can_accept_rbi": true
      }
    },
    "rotations": {
      "active": 0,
      "pool_active": 0,
      "pool_limit": 1,
      "pool_load_percentage": 0,
      "can_accept": true,
      "suggested_delay_ms": 0,
      "total": 0,
      "failed": 0,
      "failure_rate_percent": null,
      "failure_rate_10min_percent": null,
      "failure_rate_10min_count": 0,
      "failure_rate_jobs_percent": null,
      "failure_rate_jobs_count": 0
    },
    "discoveries": {
      "active": 0,
      "pool_active": 0,
      "pool_limit": 1,
      "pool_load_percentage": 0,
      "can_accept": true,
      "suggested_delay_ms": 0,
      "total": 0,
      "failed": 0,
      "failure_rate_percent": null,
      "failure_rate_10min_percent": null,
      "failure_rate_10min_count": 0,
      "failure_rate_jobs_percent": null,
      "failure_rate_jobs_count": 0
    },
    "tubes": {
      "can_accept_more": true,
      "active_creates": 0,
      "max_concurrent": 100,
      "queue_depth": 0,
      "avg_create_time_ms": 0,
      "total_creates": 0,
      "total_failures": 0,
      "load_percentage": 0,
      "suggested_delay_ms": 0
    },
    "overall_capacity": {
      "can_accept_work": true,
      "suggested_delay_ms": 0,
      "bottleneck": "none"
    },
    "connection_status": "connected",
    "websocket": {
      "uptime_seconds": 6014,
      "uptime_human": "1h 40m 14s",
      "last_ping_received_seconds_ago": 7,
      "latency_ms": 92,
      "last_ping_sent_timestamp": 1784637340,
      "last_pong_received_timestamp": 1784637340,
      "reconnect_count": 0
    }
  }
}

Sample Unhealthy Gateway:

{
  "status": "unhealthy",
  "message": "Gateway is not properly connected (status: disconnected)",
  "details": {
    "timestamp": 1784637802,
    "version": 2,
    "hostname": "localhost.localdomain",
    "instance_id": "HTLGDB",
    "gateway_version": "1.8.6",
    "resources": {
      "memory": {
        "available_mb": 10341,
        "total_mb": 10939,
        "used_percent": 5.47,
        "headroom_percent": 94.53,
        "container_limit_mb": null
      },
      "connections": {
        "total": 0,
        "by_type": {}
      },
      "pressure": {
        "under_pressure": false,
        "can_accept_rbi": true
      }
    },
    "rotations": {
      "active": 0,
      "pool_active": 0,
      "pool_limit": 1,
      "pool_load_percentage": 0,
      "can_accept": true,
      "suggested_delay_ms": 0,
      "total": 0,
      "failed": 0,
      "failure_rate_percent": null,
      "failure_rate_10min_percent": null,
      "failure_rate_10min_count": 0,
      "failure_rate_jobs_percent": null,
      "failure_rate_jobs_count": 0
    },
    "discoveries": {
      "active": 0,
      "pool_active": 0,
      "pool_limit": 1,
      "pool_load_percentage": 0,
      "can_accept": true,
      "suggested_delay_ms": 0,
      "total": 0,
      "failed": 0,
      "failure_rate_percent": null,
      "failure_rate_10min_percent": null,
      "failure_rate_10min_count": 0,
      "failure_rate_jobs_percent": null,
      "failure_rate_jobs_count": 0
    },
    "tubes": {
      "can_accept_more": true,
      "active_creates": 0,
      "max_concurrent": 100,
      "queue_depth": 0,
      "avg_create_time_ms": 0,
      "total_creates": 0,
      "total_failures": 0,
      "load_percentage": 0,
      "suggested_delay_ms": 0
    },
    "overall_capacity": {
      "can_accept_work": true,
      "suggested_delay_ms": 0,
      "bottleneck": "none"
    },
    "connection_status": "disconnected"
  }
}

Note that some metrics like latency_ms, last_ping_sent_timestamp, and last_pong_received_timestamp may not always be present in the response. The availability of these metrics depends on the current state of the WebSocket connection and the timing of ping/pong messages.

Status Update Delays

The health check reflects the current state of the WebSocket connection, but there may be a delay in status updates.

Delayed Status Updates

When connectivity is lost, it may take up to 2 minutes for the health check to report an "unhealthy" status, as the gateway attempts to reconnect. Similarly, when connectivity is restored, it may take up to 2 minutes for the health check to reflect a "healthy" status.

This latency is intentional and allows the gateway to attempt reconnection without immediately reporting failures for transient connectivity issues.

Security

The HTTP health check includes the following security features:

Authentication: When KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN is set, requests must include the token in the Authorization header:
```
Authorization: Bearer <token>
```
SSL/TLS: When SSL is enabled, all communication is encrypted. You must provide a valid certificate and private key.
Localhost binding: The server binds to localhost only by default, not exposing the endpoint over the network.
Security Headers: The health check server adds the following security headers to responses:
- X-Content-Type-Options: nosniff
- X-Frame-Options: DENY
- Content-Security-Policy: default-src 'none'
Rate Limiting: Automatic rate limiting is applied to non-localhost connections (60 requests per minute per IP).
Information Protection: When the server is bound to a non-localhost address, sensitive information is automatically redacted from the response.
Forced SSL: SSL is automatically enforced when binding to non-localhost interfaces.

TLS Compatibility

The health check server is configured to support a wide range of clients by:

Using secure TLS defaults (TLS 1.2+ minimum) for maximum security
Supporting modern cipher suites for strong encryption
Automatically handling protocol negotiation for HTTP and HTTPS

For clients that support modern TLS versions, use standard curl commands:

curl -k -H "Authorization: Bearer your_token" https://localhost:8099/health

Docker-Specific Configuration Requirements

When running Keeper Gateway inside Docker, special configuration may be required to expose the health check to the host or external systems:

Binding to 0.0.0.0

The health check server must bind to 0.0.0.0 to be reachable outside the container.
127.0.0.1 restricts access to within the container only.

SSL Enforcement

When using 0.0.0.0, Keeper Gateway forces SSL to protect health check data.
You must provide a valid certificate and key or the server will not start.

Authentication Requirement

If binding to 0.0.0.0, you must also specify an AUTH_TOKEN to secure the endpoint.

Docker Compose Example

services:
  keeper-gateway:
    image: keeper/gateway:latest
    ports:
      - "8099:8099"
    volumes:
      - ./certs:/certs:ro
    environment:
      KEEPER_GATEWAY_HEALTH_CHECK_ENABLED: true
      KEEPER_GATEWAY_HEALTH_CHECK_HOST: "0.0.0.0"
      KEEPER_GATEWAY_HEALTH_CHECK_PORT: 8099
      KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL: true
      KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT: /certs/healthcheck.crt
      KEEPER_GATEWAY_HEALTH_CHECK_SSL_KEY: /certs/healthcheck.key
      KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN: mysecrettoken

Generate a Self-Signed Certificate

mkdir -p certs
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout certs/healthcheck.key \
  -out certs/healthcheck.crt \
  -subj "/CN=localhost"

Test the Endpoint from the Host

curl -k -H "Authorization: Bearer mysecrettoken" https://localhost:8099/health

Example Linux Configuration

# Enable HTTP health check
export KEEPER_GATEWAY_HEALTH_CHECK_ENABLED=true
export KEEPER_GATEWAY_HEALTH_CHECK_PORT=8099
export KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN=mysecrettoken

# Start the gateway
gateway start

Or using command line arguments:

gateway start --health-check --health-check-port 8099 --health-check-auth-token mysecrettoken

Self-Signed SSL Certificates

For testing or internal use, you can generate self-signed certificates to enable SSL/TLS encryption:

# Generate a private key
openssl genrsa -out healthcheck.key 2048

# Generate a certificate signing request (CSR)
openssl req -new -key healthcheck.key -out healthcheck.csr -subj "/CN=localhost"

# Generate a self-signed certificate (valid for 365 days)
openssl x509 -req -days 365 -in healthcheck.csr -signkey healthcheck.key -out healthcheck.crt

# Set the environment variables
export KEEPER_GATEWAY_HEALTH_CHECK_ENABLED=true
export KEEPER_GATEWAY_HEALTH_CHECK_USE_SSL=true
export KEEPER_GATEWAY_HEALTH_CHECK_SSL_CERT=/path/to/healthcheck.crt
export KEEPER_GATEWAY_HEALTH_CHECK_SSL_KEY=/path/to/healthcheck.key
export KEEPER_GATEWAY_HEALTH_CHECK_PORT=8443  # Typical HTTPS port
export KEEPER_GATEWAY_HEALTH_CHECK_AUTH_TOKEN=mysecrettoken

# Start the gateway
gateway start

Or using command line arguments:

gateway --health-check --health-check-port 8443 --health-check-ssl --health-check-ssl-cert /path/to/healthcheck.crt --health-check-ssl-key /path/to/healthcheck.key --health-check-auth-token mysecrettoken start

When using self-signed certificates, your HTTP client will need to be configured to trust the certificate or ignore SSL verification (not recommended for production).

Monitoring Integration

This endpoint can be used with monitoring systems like:

Prometheus with blackbox exporter
Nagios/Icinga
Zabbix
Datadog
AWS CloudWatch
Any monitoring system that can perform HTTP checks

Troubleshooting

Issue

Test Command

Expected Result

Check if server is running

curl http://127.0.0.1:8099/health

Connection success or "Connection refused"

Test SSL connectivity

curl -k https://127.0.0.1:8099/health

SSL handshake success or SSL error

Test authentication

curl -k -H "Authorization: Bearer wrongtoken" https://127.0.0.1:8099/health

{"error": "Invalid authentication token"}

Check server binding

curl http://0.0.0.0:8099/health

Success if bound to 0.0.0.0, failure if bound to 127.0.0.1

Error Messages and Troubleshooting

The CLI health check provides detailed error messages to help diagnose issues:

Authentication Errors (HTTP 401)

CRITICAL: Authentication failed when connecting to https://127.0.0.1:8099/health
ERROR: Invalid or missing authentication token.

Possible fixes:
1. Check if auth token is required:
   curl -k https://127.0.0.1:8099/health
2. Provide the correct auth token:
   gateway health-check --ssl --token YOUR_TOKEN
3. Check gateway startup logs for the configured token

Connection Errors

CRITICAL: Could not connect to health check server at http://127.0.0.1:8099/health
ERROR: Connection failed.

Possible causes:
1. Health check server is not running
2. Wrong host/port combination
3. Network connectivity issues
4. SSL/non-SSL mismatch

Troubleshooting steps:
1. Verify gateway is running with health check enabled:
   gateway start --health-check
2. Check if server is using SSL:
   gateway health-check --ssl
3. Verify host and port:
   Current: 127.0.0.1:8099
4. Test with curl:
   curl http://127.0.0.1:8099/health

SSL Certificate Errors

CRITICAL: SSL error connecting to health check server at https://127.0.0.1:8099/health
ERROR: SSL certificate validation failed.

Possible causes:
- Self-signed certificate (try curl with -k flag)
- Invalid certificate path
- Certificate expired

Specifications

The Gateway health check is implemented using Bottle, a lightweight WSGI micro web-framework for Python. Bottle was chosen for the following advantages:

Minimal dependency (single file, ~60KB in size)
Enhanced security over the built-in Python HTTP server
Proper request routing and handling
Better error management
Thread safety
Production-ready with minimal overhead

PreviousAlerts and SIEM Integration

NextAdvanced Configuration

Last updated 2 months ago