Production Hardening

A remote signer holds your validator private keys. Every layer of your deployment should reflect that responsibility. This guide covers practical hardening steps you can apply today.

Network Exposure

Your signing port should never be reachable from the public internet. The right bind address depends on where the process runs:

• Bare metal / VM: bind to loopback for same-host validator clients, or to a specific private interface for remote validator clients.
• Docker / Kubernetes: bind inside the container or pod to 0.0.0.0 so the Docker port mapping or Kubernetes Service can reach it, then restrict exposure with host firewalls, private Services, NetworkPolicy, and cloud load-balancer settings.

Bare metal same-host example:

server:
  listen_address: "127.0.0.1"
  listen_port: 9000

metrics:
  listen_address: "127.0.0.1"
  listen_port: 3000

Kubernetes and Docker example:

server:
  listen_address: "0.0.0.0"
  listen_port: 9000

metrics:
  listen_address: "0.0.0.0"
  listen_port: 3000

Network controls to apply:

• Port 9000 (signing API): allow only your validator client IPs
• Port 3000 (metrics): allow only your monitoring system (Prometheus, Grafana, etc.)
• Block all other inbound traffic to these ports

Danger

Binding to 0.0.0.0 on a bare-metal host exposes the signing API on every host interface. In Kubernetes, 0.0.0.0 is expected inside the pod, but the Service must remain private and NetworkPolicy should restrict callers.

Token Security

If you’re running multiple clients against one signer, configure auth policies with per-client tokens.

Key rules for tokens:

• State-backed auth tokens are generated at runtime and persisted only as HMAC-SHA256 hashes in DynamoDB.
• State-backed token secrets are returned once at creation and never stored in plaintext.
• Stateless static_auth token secrets live in config, but should use env:VAR_NAME so the clear-text secret comes from your secrets manager or runtime environment.
• Static token secrets must be at least 16 characters. API-created tokens are prefixed (cc_token_... or cc_root_...) and generated by the server.
• Prefer short-lived client tokens over broad long-lived management tokens.
• Bind client tokens to source CIDRs with --token-bound-cidrs when validator-client egress IPs are stable.
• Store management tokens in a secrets manager.

See Auth Policies & Tokens for the policy model and API Reference for request schemas.

File Permissions

Tight file permissions prevent other users on the system from reading your keys or config.

# Config file: owner read/write only
sudo chmod 600 /etc/containment-chamber/config.yaml

# Keystores directory: owner only
sudo chmod 700 /var/lib/containment-chamber/keystores

# Individual keystore files
sudo chmod 600 /var/lib/containment-chamber/keystores/*.json

# Ensure correct ownership
sudo chown -R containment-chamber:containment-chamber \
  /etc/containment-chamber \
  /var/lib/containment-chamber

The SQLite slashing protection database is created with 0600 permissions automatically.

Tip

Run Containment Chamber as a dedicated unprivileged system user. The bare metal deployment guide walks through creating one with useradd --system.

systemd Hardening

On Linux with systemd, the service unit can enforce additional isolation. These directives are already included in the bare metal guide:

[Service]
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/
ReadWritePaths=/var/lib/containment-chamber

This prevents the process from gaining new privileges, restricts filesystem access to what it actually needs, and isolates its /tmp.

Docker Security

When running in Docker, apply the principle of least privilege:

docker run \
  --user 1000:1000 \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --cap-add IPC_LOCK \
  -v ./config.yaml:/config.yaml:ro \
  -v ./keystores:/keystores:ro \
  -v ./data:/data \
  ghcr.io/unforeseen-consequences/containment-chamber:latest \
  server -c /config.yaml

What each flag does:

• --user 1000:1000 runs as a non-root user
• --read-only makes the container filesystem immutable
• --tmpfs /tmp provides a writable scratch space
• --cap-drop ALL removes ambient Linux capabilities
• --cap-add IPC_LOCK lets the binary’s file capability activate mlockall, preventing key material from being paged to swap
• :ro mounts config and keystores as read-only

If your runtime forbids IPC_LOCK, the signer still starts, but logs a warning that memory locking could not be enabled.

Kubernetes Security

For Kubernetes, keep the application listener on 0.0.0.0 inside the pod and make the boundary private outside the pod:

config:
  server:
    listen_address: "0.0.0.0"
  metrics:
    listen_address: "0.0.0.0"

service:
  type: ClusterIP

netpolicies:
  ingress:
    enabled: true
    allowedNamespaces:
      - consensus-layer

The Helm chart is designed for a restricted container security context while preserving memory locking:

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
    add:
      - IPC_LOCK

Use an internal load balancer only when validator clients run outside the cluster. Avoid public Services or Ingress for the signing API.

Seccomp Syscall Filter

On Linux, you can restrict the signer to a minimal syscall allowlist using the kernel’s seccomp BPF filter. This limits what a code execution vulnerability can do — even if an attacker achieves arbitrary code execution, they can’t call execve, ptrace, or other dangerous syscalls.

server:
  seccomp: true  # opt-in, Linux only. Default: false

If the filter fails to apply (e.g., the kernel doesn’t support it or the process lacks CAP_SYS_ADMIN), the signer logs a warning and continues without the filter rather than refusing to start.

Tip

Seccomp is defense-in-depth. It doesn’t replace network isolation or file permission hardening — it reduces the blast radius if something else goes wrong.

Canary Keys

Canary keys are designated validator public keys that should never sign in normal operation. When a canary key signs, the signer logs a warning and increments the containment_canary_signing_total metric. Signing proceeds normally — canary keys don’t block requests.

canary_keys:
  - "0x1234..."
  - "0x5678..."

Use canary keys to detect unauthorized access. If an attacker can submit signing requests, they’ll likely try to sign with whatever keys are loaded. A canary key that suddenly appears in your metrics is a strong signal that something is wrong.

Caution

Canary keys must be loaded into the signer to trigger alerts. A key listed in canary_keys but not loaded will never fire.

Audit Logging

All security-relevant events are logged with target: "audit". This lets you route audit events to a separate sink — a SIEM, a write-once log store, or a separate file — without changing the rest of your logging configuration.

Events logged to the audit target include:

• State transitions — seal machine state changes (e.g., Sealed → KmsUnsealed → Unsealed)
• Signing requests — every signing attempt, including the key and operation type
• Unseal share submissions — when an operator submits a share, including the share index
• Seal operations — when the signer is sealed, and by whom

To capture audit events separately, configure your tracing subscriber to route the audit target:

# Include audit events at info level alongside normal logs
RUST_LOG=containment_chamber=info,audit=info

# Audit-only (suppress all other logs)
RUST_LOG=off,audit=info

In production, pipe JSON logs to a log aggregator and filter on "target":"audit" to build an audit trail.

Built-in Protections

Containment Chamber includes several protections that activate automatically:

• Memory zeroization: private keys are zeroed from memory when they’re no longer needed
• Core dump protection (Linux): the process marks itself as non-dumpable at startup, preventing key material from leaking into core dumps
• Memory locking (Linux): mlockall(MCL_CURRENT | MCL_FUTURE) is attempted at startup so resident pages are not swapped to disk; grant IPC_LOCK in container runtimes so this succeeds
• Token hashing: state-backed authentication tokens are HMAC-SHA256 hashed at creation time and persisted as hashes; stateless static_auth tokens are hashed into memory at boot
• Constant-time comparison: token validation uses constant-time comparison to prevent timing attacks

Core dump protection, zeroization, token hashing, and constant-time comparison require no configuration. Memory locking is automatic when the process has the required IPC_LOCK capability or equivalent OS limit.

Checklist

A quick reference for production deployments:

• Bare metal: signing API bound to loopback or a private interface
• Docker / Kubernetes: listener reachable inside the container or pod, with Service/firewall/NetworkPolicy restricting callers
• Ports 9000 and 3000 restricted to validator clients and monitoring systems
• Config file permissions set to 600
• Keystores directory permissions set to 700
• Running as dedicated unprivileged user
• State-backed auth policies and tokens created via the operator CLI
• Stateless static_auth token secrets injected with env:VAR_NAME
• Docker: non-root, read-only filesystem, capabilities dropped except IPC_LOCK
• Kubernetes: private Service plus NetworkPolicy for signing and metrics traffic
• systemd: NoNewPrivileges, ProtectSystem=strict, ReadOnlyPaths
• Seccomp filter enabled (server.seccomp: true) on Linux
• Canary keys configured for unauthorized-access detection
• Audit log target routed to a separate sink or SIEM

Next Steps

• Auth Policies & Tokens — create policies and tokens after hardening the network
• Observability — set up Prometheus metrics and audit-log routing
• Troubleshooting — diagnostic patterns for when hardening breaks something