Troubleshooting

This guide covers the most common errors you’ll encounter when operating Containment Chamber, with clear causes and fixes for each.

Diagnostic Commands

Before diving into specific errors, these commands help narrow down the problem:

# Is the signer running?
curl http://localhost:9000/upcheck

# Which keys are loaded?
curl http://localhost:9000/api/v1/eth2/publicKeys

# Verbose logging
RUST_LOG=containment_chamber=debug containment-chamber -c config.yaml

HTTP 412 — Slashing Protection Triggered

Cause: EIP-3076 anti-slashing refused the signing request because it would produce a slashable message (double vote or surround vote).

This is expected behavior. The signer is protecting your validators from being slashed.

Danger

Do not retry the same request, disable slashing protection, or delete the slashing database. The 412 means the protection system is working correctly.

Common triggers:

• Clock drift between your beacon node and the signer
• Restarting after a crash where the beacon node replays recent duties
• Importing a slashing protection database from the wrong source or validator

What to check:

• Ensure NTP is configured and clocks are synchronized
• If migrating validators, export the slashing DB from the previous signer and import it before starting
• See Anti-Slashing Protection for backend configuration

HTTP 404 — Key Not Found

Cause: The requested public key isn’t loaded. The signer doesn’t have the keystore for this validator.

Fix:

1. Verify the key is actually loaded:

   curl http://localhost:9000/api/v1/eth2/publicKeys

2. Check your key_sources.filesystem.paths configuration points to the correct directory
3. Verify file permissions: the keystore files must be readable by the service user
4. Check startup logs for load errors (wrong password, corrupt keystore, bad filename)

For encrypted keystores: each .json keystore file needs a matching YAML descriptor and .password file alongside it. The YAML descriptor’s keystorePasswordFile field must point to the correct password file. See the Key Formats guide for the expected directory structure.

HTTP 401 — Unauthorized

Cause: auth_policies are configured, but the request arrived without an authentication token. When policies exist and no unauthenticated_policy is defined, all unauthenticated requests are rejected.

Fix: Add the token to your client’s signer URL using HTTP Basic auth:

http://x:your-token-here@signer:9000

The username (x) is ignored. Only the password portion matters.

See Auth Policies for full configuration details.

HTTP 403 — Forbidden

Cause: The token is valid, but the associated policy denies this specific request. The key isn’t in allowed_keys, the operation is blocked by allowed_signing_operations / denied_signing_operations, or the route’s scope is not in allowed_scopes.

Fix:

1. Check which policy the token maps to in your auth_policies config
2. Verify the requested public key is listed in allowed_keys (if set)
3. Verify the operation isn’t blocked by denied_signing_operations or missing from allowed_signing_operations
4. Verify the route’s scope is accessible via allowed_scopes / denied_scopes

An unknown token also returns 403. Double-check the token value matches your config exactly.

See Auth Policies for the full policy behavior matrix.

HTTP 503 — Service Unavailable

Cause: The signing request queue is full. Too many concurrent requests exceeded the buffer capacity.

Fix: Increase the queue size or concurrency limit in your config:

signing_queue_buffer_size: 5000       # default: 3000
max_concurrent_signing_jobs: 200      # default: 100

If you’re consistently hitting 503s, you may be running too many validators for a single instance. Consider scaling horizontally with a shared PostgreSQL anti-slashing backend.

Keys Not Loading

Keystores can fail to load silently at startup. Run with debug logging to see what’s happening:

RUST_LOG=containment_chamber=debug containment-chamber -c config.yaml

Common causes:

• key_sources.filesystem.paths directory doesn’t exist or isn’t readable
• Encrypted keystores missing their password file
• Password file contains trailing whitespace or newlines
• Keystore JSON is malformed or uses an unsupported format
• File permissions too restrictive for the service user

Quick checklist:

1. Does the directory exist? ls -la /path/to/keystores/
2. Are keystore files readable? Check ownership and permissions
3. For encrypted keystores, is the password file present and correct?
4. Do the logs show any errors during key loading?

Startup Failures

Containment Chamber validates all configuration at startup and prints clear error messages to stderr.

Common validation errors:

• “cannot set both allowed_signing_operations and denied_signing_operations” — pick one approach per policy
• “cannot set both allowed_scopes and denied_scopes” — pick one approach per policy
• “allowed_signing_operations is meaningless when sign scope is denied” — remove signing op restrictions or allow the sign scope
• “invalid token” — tokens must be at least 16 characters, alphanumeric and dashes only
• “same token” — two policies share the same token value
• “unknown signing operation” — typo in an operation name (check the valid operations list)
• “invalid public key format” — allowed_keys entries must be 0x-prefixed, 96-character hex strings

Environment variable issues:

• Tokens using env:VAR_NAME syntax fail if the variable isn’t set
• Environment variables use CONTAINMENT_ prefix with __ for nesting (e.g., CONTAINMENT_ANTISLASHING__BACKEND)

Still Stuck?

If none of the above matches your issue:

1. Run with RUST_LOG=containment_chamber=debug and check the full log output
2. Verify your config file parses correctly by starting with a minimal configuration
3. Check the Configuration Reference for all available options