Troubleshooting

This guide covers the most common errors you’ll encounter when operating Containment Chamber, with clear causes and fixes for each.

Diagnostic Commands

Before diving into specific errors, these commands help narrow down the problem:

# Is the signer running?
curl http://localhost:9000/upcheck

# Which keys are loaded?
curl http://localhost:9000/api/v1/eth2/publicKeys

# Verbose logging
RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

HTTP 412 — Slashing Protection Triggered

Cause: EIP-3076 anti-slashing refused the signing request because it would produce a slashable message (double vote or surround vote).

This is expected behavior. The signer is protecting your validators from being slashed.

Danger

Do not retry the same request, disable slashing protection, or delete the slashing database. The 412 means the protection system is working correctly.

Common triggers:

• Clock drift between your beacon node and the signer
• Restarting after a crash where the beacon node replays recent duties
• Importing a slashing protection database from the wrong source or validator

What to check:

• Ensure NTP is configured and clocks are synchronized
• If migrating validators, export the slashing DB from the previous signer and import it before starting
• See Anti-Slashing Protection for backend configuration

HTTP 404 — Key Not Found

Cause: The requested public key isn’t loaded. The signer doesn’t have the keystore for this validator.

Fix:

1. Verify the key is actually loaded:

   curl http://localhost:9000/api/v1/eth2/publicKeys

2. Check your key_sources.filesystem.paths configuration points to the correct directory
3. Verify file permissions: the keystore files must be readable by the service user
4. Check startup logs for load errors (wrong password, corrupt keystore, bad filename)

For encrypted keystores: each .json keystore file needs a matching YAML descriptor and .password file alongside it. The YAML descriptor’s keystorePasswordFile field must point to the correct password file. See the Key Formats guide for the expected directory structure.

HTTP 400 — Bad Request

Cause: The signing request was rejected before any signature was attempted. Two cases produce a 400:

• Unsupported signing request — the request type is not one Containment Chamber implements, or the body does not match the expected schema for that type.
• Network mismatch — the validator client is configured for a different Ethereum network than the signer. The client-facing body is the generic network mismatch; the signer logs the expected vs received fork details server-side and intentionally does not disclose its configured network to the client.

Fix:

1. Confirm the validator client uses a supported Web3Signer operation — see Validator Clients.
2. Verify the signer and validator client target the same network — see Networks and Forks.

HTTP 401 — Unauthorized

Cause: The request arrived without a valid authentication token, and no anonymous policy matched it. In stateless mode, when no static_auth.anonymous policy allows the request, unauthenticated requests are rejected. State-backed deployments cannot use static_auth; create client tokens via the auth API instead.

Fix: Add the token to your request as an Authorization: Bearer <token> header:

curl -H "Authorization: Bearer your-token-here" http://signer:9000/api/v1/eth2/publicKeys

See Auth Policies for full configuration details.

HTTP 403 — Forbidden

Cause: The token is valid, but the associated policy denies this specific request. The key isn’t in the policy’s allowed keys, the operation is blocked by the policy rules, or the route’s scope is not permitted.

Fix:

1. Check which policy the token is bound to via GET /api/v1/auth/tokens/lookup-self
2. Review the policy rules via GET /api/v1/auth/policies/{name}
3. Verify the requested public key is permitted by the policy
4. Verify the operation and scope are allowed by the policy rules

An expired or invalid token also returns 403. Double-check the token value is correct.

See Auth Policies for the full policy behavior matrix.

HTTP 503 — Service Unavailable

Cause: Either the signer is sealed, or the signing request queue is full.

Fix: First check chamber status and unseal if needed. If the signer is already unsealed, increase the queue size or concurrency limit in your config:

signing:
  queue_buffer_size: 4000
  max_concurrent_jobs: 2000

If you’re consistently hitting 503s, you may be running too many validators for a single instance. Consider scaling horizontally with a shared PostgreSQL anti-slashing backend.

Keys Not Loading

Keystores can fail to load silently at startup. Run with debug logging to see what’s happening:

RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

Common causes:

• key_sources.filesystem.paths directory doesn’t exist or isn’t readable
• Encrypted keystores missing their password file
• Password file contains trailing whitespace or newlines
• Keystore JSON is malformed or uses an unsupported format
• File permissions too restrictive for the service user

Quick checklist:

1. Does the directory exist? ls -la /path/to/keystores/
2. Are keystore files readable? Check ownership and permissions
3. For encrypted keystores, is the password file present and correct?
4. Do the logs show any errors during key loading?

Startup Failures

Containment Chamber validates all configuration at startup and prints clear error messages to stderr.

Common startup-blocking misconfigurations:

• static_auth set together with signer_state — the two are mutually exclusive. Remove static_auth for a state-backed deployment, or drop signer_state for a stateless one.
• anti_slashing.backend: noop without --allow-unsafe-noop-anti-slashing — the noop backend is refused at startup unless you pass the explicit override flag. Never use noop in production.
• Unresolved env:VAR_NAME reference — a config value uses the env: prefix but the named environment variable is unset.

Policy and token validation does not happen at startup. Policies and tokens are created at runtime through the Auth API, and invalid input (bad scope, unknown operation, malformed public key, over-long identifier) is rejected there with HTTP 400, not at boot.

Environment variable issues:

• Tokens using env:VAR_NAME syntax fail if the variable isn’t set
• Environment variables use CONTAINMENT_ prefix with __ for nesting (e.g., CONTAINMENT_ANTI_SLASHING__BACKEND)

Next Steps

• Configuration Reference — every option, env var, and CLI flag
• Auth Policies & Tokens — fix policy syntax and token format errors
• Observability — set up structured logging before the next incident

Still Stuck?

If none of the above matches your issue:

1. Run with RUST_LOG=containment_chamber=debug and check the full log output
2. Verify your config file parses correctly by starting with a minimal configuration
3. Check the Configuration Reference for all available options