Enclave Operations

PCR0 Rotation (Per Release)

PCR0 changes on every rebuild. With reproducible builds, each key holder can independently verify the new PCR0 before updating their KMS key policy.

1. Build the new EIF and extract PCR0:

   docker build -f Dockerfile.enclave -t containment-chamber:v1.5.0-enclave .
   nitro-cli build-enclave \
     --docker-uri containment-chamber:v1.5.0-enclave \
     --signing-certificate signing_cert.pem \
     --private-key signing_key.pem \
     --output-file enclave-v1.5.0.eif
   NEW_PCR0=$(nitro-cli describe-eif --eif-path enclave-v1.5.0.eif | jq -r '.Measurements.PCR0')
   echo "New PCR0: ${NEW_PCR0}"

2. Send PCR0 to all 3 key holders — each holder independently verifies by rebuilding:

   # Each key holder runs this to verify:
   docker build -f Dockerfile.enclave -t verify:local .
   nitro-cli build-enclave --docker-uri verify:local --output-file verify.eif
   nitro-cli describe-eif --eif-path verify.eif | jq -r '.Measurements.PCR0'
   # Must match the PCR0 you received

3. Each key holder updates their KMS key policy (add new PCR0, keep old for zero-downtime):

   terraform apply \
     -var 'enclave_pcr0_hashes=["NEW_PCR0", "OLD_PCR0"]'

4. Deploy the new enclave image:

    helm upgrade containment-chamber \
      oci://ghcr.io/unforeseen-consequences/charts/containment-chamber \
      --set enclave.enabled=true \
      --set image.tag=v1.5.0-nitro

5. Remove the old PCR0 after all instances are updated:

   terraform apply \
     -var 'enclave_pcr0_hashes=["NEW_PCR0"]'

PCR8 Rotation (Signing Certificate)

PCR8 only changes when you rotate the signing certificate. This is rare (years, not releases).

1. Generate a new signing keypair and certificate
2. Build and sign a new EIF with the new certificate
3. Extract the new PCR8 hash
4. Update all 3 KMS key policies with the new PCR8 (keep old PCR8 during transition)
5. Deploy the new enclave
6. Remove the old PCR8 from all key policies

Debug Mode

Debug mode breaks attestation

In debug mode, the NSM produces attestation documents with all PCR values set to zero. KMS will reject these documents if your key policy requires specific PCR values. Never use debug mode in production.

Debug mode is blocked by the entrypoint when PRODUCTION=true:

# This will fail with FATAL error:
kubectl set env deployment/containment-chamber \
  ENCLAVE_DEBUG=1 PRODUCTION=true

For development, omit PRODUCTION=true:

kubectl set env deployment/containment-chamber ENCLAVE_DEBUG=1
# Then attach to the enclave console:
kubectl exec -it <pod> -- nitro-cli console --enclave-name containment-chamber

Log Forwarding (vsock)

Tracing events emitted inside the enclave are forwarded over vsock to a listener in the parent pod, which writes them to the pod’s stdout. The parent and enclave logs share the same tracing-subscriber format — there is no [enclave] prefix; events are interleaved.

• Vsock port: 7000 (wire-protocol constant, not configurable).
• Parent listener: containment-chamber-enclave-proxy vsock-stdout is started by scripts/enclave-entrypoint.sh before nitro-cli run-enclave.
• Backpressure: the enclave-side writer is non-blocking with bounded buffering (tracing-appender). When the vsock connection is unavailable or backed off, individual events are dropped and counted by containment_enclave_log_events_dropped_total. Connection retries use exponential backoff (100 ms → 5 s) with ±20% jitter.
• Shutdown flush: a WorkerGuard is held for the lifetime of the server so that buffered events are flushed before exit.

kubectl logs <pod> -f
# Tracing events from both the parent process (proxies, init) and the enclave
# stream to stdout in the same format. Filter enclave-only spans by the fields
# emitted from inside (e.g. enclave_id, attestation_*).

# Verify drops are zero (or steady) under load:
curl -s http://<pod>:3000/metrics | grep enclave_log_events_dropped_total

If containment_enclave_log_events_dropped_total is climbing, investigate vsock saturation, the listener’s stdout consumer (kubelet), or the parent CPU budget. The metric is always exposed regardless of whether the nitro feature is built.

Troubleshooting

Enclave fails to start

kubectl exec -it <pod> -- nitro-cli describe-enclaves
# Check: is the enclave in "running" state?
# If empty: nitro-cli run-enclave failed — check pod logs
kubectl logs <pod>

KMS attestation rejected

Symptoms: enclave starts but fails to reconstruct master key. Check:

1. PCR0 in KMS key policy matches the deployed EIF
2. PCR8 in KMS key policy matches the signing certificate
3. The signer IAM role has permission to call KMS

# Extract PCR values from the running enclave
kubectl exec -it <pod> -- nitro-cli describe-enclaves | jq '.[0].Measurements'

vsock connectivity issues

# Check egress proxies are running
kubectl exec -it <pod> -- ps aux | grep vsock-proxy

# Test KMS connectivity from inside the pod
kubectl exec -it <pod> -- curl -k https://kms.us-east-1.amazonaws.com/

Entropy seeding failure

If the enclave fails with “NSM initialization failed”, the /dev/nsm device is not available. Verify:

• The node has Nitro Enclaves enabled in the launch template
• The device plugin DaemonSet is running on the node
• The pod has aws.ec2.nitro/nitro_enclaves: "1" in resource limits