Seal & Unseal Operations Guide

This runbook covers every ceremony you’ll perform as an operator: first boot, unsealing after restart, emergency seal, and all rotation procedures. For the concepts behind each state and mode, see Seal & Unseal Ceremony.

The seal/unseal system only applies when you’re using the DynamoDB key source. Filesystem keystores load directly at startup and aren’t affected.

Prerequisites

Before running any ceremony, confirm:

• DynamoDB table — provisioned via the terraform/dynamodb-keystore module
• KMS keys — created via the same Terraform module (one per AWS region or account for cross-region redundancy)
• IAM permissions — the signer’s role has kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey on each KMS key, plus DynamoDB read/write on the table
• Binary installed — containment-chamber is on your $PATH
• Config file ready — key_sources.dynamodb.table is set; see Configuration Reference
• Auth policies — configure auth_policies in your config file with scopes for seal operations. Without this, operator seal, operator status, and operator rotate commands return 401.

auth_policies:
  signing-client:
    token: "env:SIGNING_TOKEN"
    allowed_scopes:
      - sign
      - public_keys
  ops-team:
    token: "env:CC_AUTH_TOKEN"
    allowed_scopes:
      - chamber_seal
      - chamber_rotate
      - chamber_status

See Access Control for all available scopes.

Tip

The containment-chamber operator subcommands default to http://localhost:9000. Pass --signer-url if your signer listens elsewhere.

---

First Boot: KMS-Only Mode

Use this mode when you trust your AWS IAM posture and want fully automated restarts. No operator passphrases are required.

1. Start the signer.

containment-chamber --config config.yaml

On first boot, the signer logs a one-time setup token:

WARN Setup token generated (save securely, one-time use) setup_token=cc-setup-Xk9mP2...

Save this token from the logs. It’s generated once and never stored.

2. Verify the signer is ready for initialization.

containment-chamber operator status \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000

{"seal": {"status": "uninitialized"}, "keys": null}

3. Initialize with your KMS keys.

containment-chamber operator init \
  --setup-token "$SETUP_TOKEN" \
  --signer-url http://localhost:9000 \
  --kms-key-arns "arn:aws:kms:us-east-1:111111111111:key/aaa...,arn:aws:kms:eu-west-1:222222222222:key/bbb...,arn:aws:kms:ap-southeast-1:333333333333:key/ccc..." \
  --kms-threshold 2

The --kms-threshold sets how many KMS keys must be available to reconstruct the master key. A 2-of-3 setup tolerates one unavailable region.

4. Confirm the signer is unsealed.

{
  "status": "unsealed",
  "validators_loaded": 42
}

The signer is now operational. On every subsequent restart, KMS unseal happens automatically in the background — no human action needed.

---

First Boot: Unseal Mode

Use this mode when you want human-in-the-loop protection. Even a full KMS compromise can’t unseal the signer without operator passphrases.

1. Start the signer.

containment-chamber --config config.yaml

Save the setup token from the logs.

2. Verify the signer is in uninitialized state (same as KMS-only above).

3. Initialize with operators.

containment-chamber operator init \
  --setup-token "$SETUP_TOKEN" \
  --signer-url http://localhost:9000 \
  --kms-key-arns "arn:aws:kms:us-east-1:111111111111:key/aaa...,arn:aws:kms:eu-west-1:222222222222:key/bbb..." \
  --kms-threshold 2 \
  --unseal-threshold 2 \
  --operators "alice,bob,carol"

The --unseal-threshold sets how many operators must submit passphrases to unseal. A 2-of-3 setup tolerates one unavailable operator.

The response includes a registration token for each operator:

{
  "status": "awaiting_registration",
  "operator_tokens": {
    "alice": "cc-reg-A1b2C3d4...",
    "bob":   "cc-reg-E5f6G7h8...",
    "carol": "cc-reg-I9j0K1l2..."
  }
}

4. Distribute registration tokens to each operator.

The person running operator init is the ceremony coordinator. They receive all registration tokens in the response and must distribute each token to its corresponding operator through a secure, out-of-band channel.

Caution

The coordinator sees all registration tokens. This is an inherent trust point — choose your coordinator carefully. Tokens are single-use: once an operator registers, their token is consumed and cannot be reused.

For each operator, send them:

• Their operator name (e.g., alice)
• Their registration token (starts with cc-reg-)
• The signer URL

Use a secure channel: Signal, encrypted email, or your organization’s secrets manager. Never send tokens via plaintext email, Slack, or other unencrypted channels.

If a token is compromised before the operator registers, run operator init-reset to cancel the entire initialization and start over with fresh tokens.

5. Each operator registers their passphrase.

containment-chamber operator register \
  --operator alice \
  --registration-token "$ALICE_TOKEN" \
  --credential-id password-1 \
  --signer-url http://localhost:9000
# Prompts for passphrase interactively (minimum 16 characters)

The --credential-id names this credential. You can add more credentials later (see Registering Multiple Credentials). The passphrase is never stored — only the Argon2id-derived key is persisted in DynamoDB. If an operator loses all their credentials, their share is permanently inaccessible. Plan your threshold accordingly.

While operators are still registering:

{"registered": 1, "remaining": 2}

6. After all operators register, the signer transitions to sealed.

{"registered": 3, "remaining": 0, "status": "sealed"}

The setup token is now invalidated. Proceed to the unseal ceremony below.

---

Registering Multiple Credentials

After initial registration, each operator can add more credentials to their share. This lets them unseal with a password, a YubiKey, or a secrets manager entry — whichever is available at the time.

Adding a second password credential:

containment-chamber operator credentials add \
  --operator alice \
  --existing-credential-id password-1 \
  --new-credential-id password-2 \
  --signer-url http://localhost:9000
# Prompts for existing passphrase, then new passphrase

Adding a YubiKey credential (see YubiKey Setup for device configuration):

containment-chamber operator credentials add \
  --operator alice \
  --existing-credential-id password-1 \
  --new-credential-id yubikey-1 \
  --yubikey \
  --signer-url http://localhost:9000
# Prompts for existing passphrase; YubiKey performs challenge-response automatically

Tip

A practical setup for each operator: one password credential for day-to-day use, one YubiKey credential as a hardware backup. Either one can unseal independently.

---

Unsealing After Restart

After a restart in unseal mode, KMS unseal happens automatically in the background. The signer transitions to kms_unsealed — signing is still unavailable at this point. Operators then submit their passphrases to complete the ceremony.

1. Check the current state.

containment-chamber operator status \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000

You should see kms_unsealed. If you see sealed, KMS unseal is still in progress — wait a moment and check again.

2. Each operator submits their passphrase.

containment-chamber operator unseal \
  --operator alice \
  --signer-url http://localhost:9000
# Prompts for passphrase

No --credential-id is needed. The server tries all registered credentials for that operator and accepts the first one that decrypts the share. Operators can use whichever credential is convenient — password, YubiKey, or secrets manager.

Operators can submit in any order. While below threshold:

{"shares_received": 1, "shares_required": 2}

3. Once the threshold is reached, the signer unseals.

{"status": "unsealed", "validators_loaded": 42}

Signing is now available.

Tip

If a wrong passphrase was submitted and you want to clear accumulated shares and start over, run:

containment-chamber operator unseal-reset \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000

Tip

Running in a Nitro Enclave? After the first successful unseal, the master key is automatically TEE-sealed and stored in DynamoDB. On subsequent restarts within the same enclave (same binary, same IAM role), the signer auto-unseals without requiring operator passphrases. If the binary is updated (PCRs change), the TEE sealed blob is invalid and operators must run the full ceremony again.

Partial unseal timeout. By default, accumulated shares expire after 30 minutes if the ceremony isn’t completed. Configure this in your YAML:

key_sources:
  dynamodb:
    unseal_timeout_minutes: 30  # 0 = disabled

---

Managing Credentials

Operators can add, remove, and list their credentials at any time. All credential management operations require proving an existing credential — there’s no admin override.

Add a credential:

containment-chamber operator credentials add \
  --operator alice \
  --existing-credential-id password-1 \
  --new-credential-id yubikey-1 \
  --yubikey \
  --signer-url http://localhost:9000

Remove a credential:

containment-chamber operator credentials remove \
  --operator alice \
  --existing-credential-id password-1 \
  --remove-credential-id old-password \
  --signer-url http://localhost:9000

The server rejects removal of the last credential. An operator must always have at least one credential registered.

List credentials:

containment-chamber operator credentials list \
  --operator alice \
  --signer-url http://localhost:9000

{
  "operator": "alice",
  "credential_ids": ["password-1", "yubikey-1"]
}

---

Emergency Seal

Seal the signer immediately to stop all signing. Use this for suspected compromise, before maintenance, or before rotating credentials.

containment-chamber operator seal \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000

{"status": "sealed"}

The master key is zeroized from memory. All subsequent signing requests return 503. In-flight requests that already loaded the signer reference complete normally — the seal takes effect for new requests only.

Caution

After sealing, the signer won’t auto-unseal on the next restart in KMS-only mode unless the TEE sealed blob is still present. In unseal mode, operators must run the full ceremony again. Plan accordingly before sealing in production.

To resume signing, restart the signer (KMS unseal happens automatically) and, in unseal mode, run the operator ceremony.

---

KMS Key Rotation

Rotate to a new set of KMS keys without interrupting signing. Use this to add or remove keys, change the threshold, or move to keys in different AWS accounts.

containment-chamber operator rotate kms \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000 \
  --kms-key-arns "arn:aws:kms:us-east-1:111111111111:key/new-aaa...,arn:aws:kms:eu-west-1:222222222222:key/new-bbb..." \
  --kms-threshold 2

{"status": "ok", "shares": 2, "threshold": 2}

The rotation validates KMS access (encrypt + decrypt roundtrip) for each new key before committing. If any key fails validation, the rotation is rejected and the existing shares remain unchanged. Signing continues throughout.

Tip

Keep your old KMS keys accessible until you’ve confirmed the rotation succeeded and the signer has restarted at least once using the new keys.

---

Operator Rotation

Replace the operator set with new operators and/or a new threshold. Signing continues throughout — the signer stays in awaiting_rotation, which still serves signing requests.

1. Start the rotation.

containment-chamber operator rotate unseal \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000 \
  --unseal-threshold 2 \
  --operators "dave,eve,frank"

{
  "status": "awaiting_rotation",
  "operator_tokens": {
    "dave":  "cc-reg-M3n4O5p6...",
    "eve":   "cc-reg-Q7r8S9t0...",
    "frank": "cc-reg-U1v2W3x4..."
  }
}

2. Distribute tokens to each new operator via the same secure channels used during initial setup (see First Boot: Unseal Mode for distribution guidance). If a token is compromised, cancel the rotation with operator rotate unseal-cancel.

3. Each new operator registers their passphrase.

containment-chamber operator register \
  --operator dave \
  --registration-token "$DAVE_TOKEN" \
  --signer-url http://localhost:9000
# Prompts for passphrase

4. Once all new operators register, the rotation completes.

The signer returns to unsealed. The old operator set is replaced.

To cancel a rotation in progress and restore the previous operator set:

containment-chamber operator rotate unseal-cancel \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000

Registration during rotation

New operators can register while the signer is in awaiting_rotation — signing continues throughout. The same --credential-id flag applies, and operators can add multiple credentials immediately after their initial registration:

# Initial registration during rotation
containment-chamber operator register \
  --operator dave \
  --registration-token "$DAVE_TOKEN" \
  --credential-id password-1 \
  --signer-url http://localhost:9000

# Add a YubiKey credential right after
containment-chamber operator credentials add \
  --operator dave \
  --existing-credential-id password-1 \
  --new-credential-id yubikey-1 \
  --yubikey \
  --signer-url http://localhost:9000

---

Mode Switch

Switch between kms_only and unseal modes without restarting. Signing continues during the switch.

KMS-Only → Unseal

containment-chamber operator rotate mode \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000 \
  --mode unseal \
  --unseal-threshold 2 \
  --operators "alice,bob,carol"

The signer enters awaiting_rotation. New operators register using operator register with their tokens. Once all register, the signer returns to unsealed in unseal mode.

Unseal → KMS-Only

containment-chamber operator rotate mode \
  --auth-token "$AUTH_TOKEN" \
  --signer-url http://localhost:9000 \
  --mode kms_only

Takes effect immediately. All operator share rows are deleted from DynamoDB. Future restarts will auto-unseal via KMS only.

---

Binary Upgrades

When you deploy a new version of containment-chamber:

KMS-only mode: No action needed. The signer auto-unseals on restart using KMS.

Unseal mode without TEE: No action needed. KMS auto-unseal happens, then operators run the unseal ceremony as usual.

Unseal mode with TEE (Nitro Enclave): A binary update changes the enclave measurements (PCR0, PCR2). The TEE sealed blob from the previous binary will fail to decrypt. The signer falls back to the standard KMS + operator ceremony automatically. After operators unseal, a new TEE sealed blob is created for the updated binary.

Tip

Plan binary upgrades during a maintenance window when operators are available for the unseal ceremony, or switch to KMS-only mode before upgrading if you want zero-touch restarts.

---

Scripting: Passphrases via stdin

For automated workflows, pass passphrases via stdin instead of the interactive prompt:

echo "my-secure-passphrase-here" | containment-chamber operator unseal \
  --operator alice \
  --signer-url http://localhost:9000 \
  --passphrase-stdin

The same --passphrase-stdin flag works for operator register.

Caution

Avoid putting passphrases directly in shell commands or scripts — they appear in shell history and process listings (ps aux). Prefer reading from a file or a secrets manager:

# Read from a file (ensure 0600 permissions)
cat /run/secrets/alice-passphrase | containment-chamber operator unseal \
  --operator alice \
  --passphrase-stdin

# Or from a secrets manager
aws secretsmanager get-secret-value --secret-id alice-passphrase \
  --query SecretString --output text | \
  containment-chamber operator unseal --operator alice --passphrase-stdin

---

Token Management

Token type	Lifetime	How to get it
Setup token	One-time, logged on first boot	Restart the signer process — a new token is generated on every startup while in uninitialized state
Registration token	Single-use per operator	Printed in operator init or operator rotate unseal response
Auth token	Long-lived, configured by you	Set in auth_policies in your config file

Auth tokens in config — a typical production setup separates signing clients from operators:

auth_policies:
  signing-client:
    token: "env:SIGNING_TOKEN"
    allowed_scopes:
      - sign
      - public_keys
  ops-team:
    token: "env:CC_AUTH_TOKEN"
    allowed_scopes:
      - chamber_seal
      - chamber_rotate
      - chamber_status
  key-manager:
    token: "env:KEY_MANAGER_TOKEN"
    allowed_scopes:
      - chamber_keys_generate
      - chamber_keys_write
      - chamber_keys_list
      - chamber_keys_delete

Using env:VAR_NAME to avoid plaintext tokens in commands:

export CC_AUTH_TOKEN="my-operator-token-here"

containment-chamber operator status \
  --auth-token "env:CC_AUTH_TOKEN" \
  --signer-url http://localhost:9000

This works for all --auth-token, --setup-token, and --registration-token flags. The value is resolved from the environment at runtime and never appears in process listings.

---

Troubleshooting

503 on signing requests — “signer is sealed”

The signer is in sealed or kms_unsealed state. Check status and run the unseal ceremony:

containment-chamber operator status --auth-token "$AUTH_TOKEN" --signer-url http://localhost:9000

409 — “already initialized”

The signer has already been initialized. Use operator rotate kms, operator rotate unseal, or operator rotate mode instead of operator init.

Unseal timeout expired

Accumulated shares were cleared because the ceremony wasn’t completed within unseal_timeout_minutes. Start over — each operator submits their passphrase again from the beginning.

“invalid KMS ARN”

Check the ARN format: arn:aws:kms:<region>:<account-id>:key/<key-id>. The region is extracted from the ARN, so it must be correct. Verify the key exists and your IAM role has access.

“passphrase too short”

Passphrases must be at least 16 characters. There’s no maximum length.

“wrong passphrase” / share decryption fails

The passphrase doesn’t match what was registered. If you’re certain the passphrase is correct, check for encoding issues (trailing newlines, different character encoding). Run operator unseal-reset to clear partial shares and try again.

KMS unseal fails on restart

Check that the KMS keys are still accessible from the signer’s IAM role. If you rotated KMS keys, confirm the rotation completed successfully before the restart. Check CloudTrail for kms:Decrypt denials.

For issues not covered here, see the Troubleshooting guide or run with debug logging:

RUST_LOG=containment_chamber=debug containment-chamber --config config.yaml

---

API Reference

The credential management endpoints are available whenever the DynamoDB key source is configured. Authentication uses the existing passphrase as proof of identity — no separate auth token is required.

Add a credential

POST /api/v1/chamber/unseal/credentials

Request:

{
  "operator": "alice",
  "existing_passphrase": "my-current-passphrase",
  "existing_credential_id": "password-1",
  "new_passphrase": "my-new-passphrase",
  "new_credential_id": "password-2"
}

For a YubiKey credential, the new_passphrase field contains the HMAC-SHA1 challenge-response output (handled automatically by the CLI).

Response:

{
  "operator": "alice",
  "credential_id": "password-2"
}

Remove a credential

DELETE /api/v1/chamber/unseal/credentials

Request:

{
  "operator": "alice",
  "existing_passphrase": "my-current-passphrase",
  "existing_credential_id": "password-1",
  "remove_credential_id": "old-password"
}

Response:

{
  "operator": "alice",
  "removed_credential_id": "old-password"
}

Returns 409 Conflict if remove_credential_id is the last credential for that operator.

List credentials

GET /api/v1/chamber/unseal/credentials?operator=alice

Response:

{
  "operator": "alice",
  "credential_ids": ["password-1", "yubikey-1"]
}

This endpoint requires no authentication — credential IDs are not secret.

---

Resetting Initialization

If you need to start over during the initial setup (e.g., wrong KMS keys, wrong operator list), use init-reset while the signer is still in awaiting_registration:

containment-chamber operator init-reset \
  --setup-token "$SETUP_TOKEN" \
  --signer-url http://localhost:9000

This deletes the MASTER_KEY row from DynamoDB and returns the signer to uninitialized. A new setup token is generated and logged.

Caution

init-reset only works while the signer is in awaiting_registration — before all operators have completed registration. Once all operators register and the signer transitions to sealed, re-initialization is no longer possible. Instead, unseal the signer normally and use the rotation commands to change KMS keys (operator rotate kms), operators (operator rotate unseal), or mode (operator rotate mode).

Disaster Recovery

If you lose access to both KMS keys and operator passphrases (e.g., all KMS keys deleted and all operators unavailable), the master key is permanently unrecoverable. The BLS validator keys encrypted by that master key cannot be decrypted.

Your recovery option is the age-encrypted mnemonic backup — if backup.enabled: true was configured during key generation. This backup contains the BIP-39 mnemonic from which all BLS keys were derived. With it, you can:

1. Set up a fresh Containment Chamber instance
2. Run a new init ceremony with new KMS keys and operators
3. Re-import the validator keys from the mnemonic backup

See age-encrypted mnemonic backup for configuration and recovery procedures.

Caution

Without a mnemonic backup, losing access to both KMS and operator passphrases means permanent loss of your validator keys. Always configure and securely store backups before going to production.