Troubleshooting

This guide covers the most common errors you’ll encounter when operating Containment Chamber, with clear causes and fixes for each.

Diagnostic Commands

Before diving into specific errors, these commands help narrow down the problem:

# Is the signer running?
curl http://localhost:9000/upcheck

# Which keys are loaded?
curl http://localhost:9000/api/v1/eth2/publicKeys

# Verbose logging
RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

HTTP 412 — Slashing Protection Triggered

Cause: EIP-3076 anti-slashing refused the signing request because it would produce a slashable message (double vote or surround vote).

This is expected behavior. The signer is protecting your validators from being slashed.

Danger

Do not retry the same request, disable slashing protection, or delete the slashing database. The 412 means the protection system is working correctly.

Common triggers:

• Clock drift between your beacon node and the signer
• Restarting after a crash where the beacon node replays recent duties
• Importing a slashing protection database from the wrong source or validator

What to check:

• Ensure NTP is configured and clocks are synchronized
• If migrating validators, export the slashing DB from the previous signer and import it before starting
• See Anti-Slashing Protection for backend configuration

HTTP 404 — Key Not Found

Cause: The requested public key isn’t loaded. The signer doesn’t have the keystore for this validator.

Fix:

1. Verify the key is actually loaded:

   curl http://localhost:9000/api/v1/eth2/publicKeys

2. Check your key_sources.filesystem.paths configuration points to the correct directory
3. Verify file permissions: the keystore files must be readable by the service user
4. Check startup logs for load errors (wrong password, corrupt keystore, bad filename)

For encrypted keystores: each .json keystore file needs a matching YAML descriptor and .password file alongside it. The YAML descriptor’s keystorePasswordFile field must point to the correct password file. See the Key Formats guide for the expected directory structure.

HTTP 401 — Unauthorized

Cause: Auth policies are configured, but the request arrived without a valid authentication token. When policies exist and no unauthenticated_policy is defined, all unauthenticated requests are rejected.

Fix: Add the token to your request as an Authorization: Bearer <token> header:

curl -H "Authorization: Bearer your-token-here" http://signer:9000/api/v1/eth2/publicKeys

See Auth Policies for full configuration details.

HTTP 403 — Forbidden

Cause: The token is valid, but the associated policy denies this specific request. The key isn’t in the policy’s allowed keys, the operation is blocked by the policy rules, or the route’s scope is not permitted.

Fix:

1. Check which policy the token is bound to via GET /api/v1/auth/tokens/lookup-self
2. Review the policy rules via GET /api/v1/auth/policies/{name}
3. Verify the requested public key is permitted by the policy
4. Verify the operation and scope are allowed by the policy rules

An expired or invalid token also returns 403. Double-check the token value is correct.

See Auth Policies for the full policy behavior matrix.

HTTP 503 — Service Unavailable

Cause: Either the signer is sealed, or the signing request queue is full.

Fix: First check chamber status and unseal if needed. If the signer is already unsealed, increase the queue size or concurrency limit in your config:

signing:
  queue_buffer_size: 4000
  max_concurrent_jobs: 2000

If you’re consistently hitting 503s, you may be running too many validators for a single instance. Consider scaling horizontally with a shared PostgreSQL anti-slashing backend.

Keys Not Loading

Keystores can fail to load silently at startup. Run with debug logging to see what’s happening:

RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

Common causes:

• key_sources.filesystem.paths directory doesn’t exist or isn’t readable
• Encrypted keystores missing their password file
• Password file contains trailing whitespace or newlines
• Keystore JSON is malformed or uses an unsupported format
• File permissions too restrictive for the service user

Quick checklist:

1. Does the directory exist? ls -la /path/to/keystores/
2. Are keystore files readable? Check ownership and permissions
3. For encrypted keystores, is the password file present and correct?
4. Do the logs show any errors during key loading?

Startup Failures

Containment Chamber validates all configuration at startup and prints clear error messages to stderr.

Common validation errors:

• “cannot set both allowed_signing_operations and denied_signing_operations” — pick one approach per policy
• “cannot set both allowed_scopes and denied_scopes” — pick one approach per policy
• “allowed_signing_operations is meaningless when sign scope is denied” — remove signing op restrictions or allow the sign scope
• “invalid token” — tokens must be at least 16 characters, alphanumeric and dashes only
• “same token” — two policies share the same token value
• “unknown signing operation” — typo in an operation name (check the Auth guide)
• “invalid public key format” — allowed_keys entries must be 0x-prefixed, 96-character hex strings

Environment variable issues:

• Tokens using env:VAR_NAME syntax fail if the variable isn’t set
• Environment variables use CONTAINMENT_ prefix with __ for nesting (e.g., CONTAINMENT_ANTI_SLASHING__BACKEND)

Still Stuck?

If none of the above matches your issue:

1. Run with RUST_LOG=containment_chamber=debug and check the full log output
2. Verify your config file parses correctly by starting with a minimal configuration
3. Check the Configuration Reference for all available options