Przejdź do treści

Sovereign agent operator playbook

Plan 118 — runbook for operators provisioning, monitoring, and revoking customer-hosted Secruna agents.

When to provision a sovereign agent

Use the sovereign agent for customers in any of these buckets:

  • No customer data may leave their secure environment (UK Defence supplier, regulated bank, etc.).
  • The customer wants annual zero-egress audit evidence.
  • The customer prefers to host inference on their own LLM stack (Ollama / Azure OpenAI in their subscription).

For everyone else, the SaaS connectors stay the right answer — less moving parts, no operator handoff.

Pre-provision checks

Before clicking "Provision" in /admin/agents:

  1. Confirm the tenant exists and has at least one org admin who can receive the deployment bundle.
  2. Confirm RULE_BOOK_SIGNING_PUBLIC_KEY_PEM is configured on cp-api (without it, the rendered template carries a placeholder and the agent refuses to apply rule books).
  3. Confirm SOVEREIGN_AGENT_IMAGE_TAG points at a tag the customer can pull (the customer's environment must reach ghcr.io/majest/sovereign-agent:{tag}).

Provision flow

  1. Navigate to /admin/agents in the dashboard.
  2. Click "Provision new agent". Fill in:
  3. Tenant UUID.
  4. Display name (operator-facing label; show "UK Defence supplier — prod", not just "agent 1").
  5. Deployment target — Docker Compose / Helm values / Bicep.
  6. The modal shows three blocks:
  7. Subject — record this in your handoff doc; it's the JWT sub the agent presents.
  8. Private key PEM — shown once. Copy to 1Password / Bitwarden / your customer's secure channel.
  9. Rendered template — paste-ready file for the customer's deployment shape.
  10. Once you've copied both, click "I've copied the secrets" to dismiss the dialog. The private key cannot be re-fetched.

The audit log records platform.sovereign_agent.provisioned with the actor email, subject, display name, and deployment target.

Monitoring

/admin/agents shows status per row, derived from last_heartbeat_at:

Status Meaning Action
pending Never heartbeated since provision Wait 60s; if still pending, ask customer to check container logs
online Heartbeat within last 5 min nothing
stale 5-15 min since heartbeat Customer's egress may be intermittent; not urgent
offline >15 min since heartbeat Investigate; customer container may be down
revoked revoked_at stamped Historical row

For deeper inspection, query the sovereign_agents table directly:

SELECT id, subject, display_name, last_heartbeat_at, last_version, last_rule_book_hash
FROM sovereign_agents
WHERE revoked_at IS NULL
ORDER BY last_heartbeat_at DESC NULLS LAST;

For audit forensics use the audit log:

SELECT created_at, action, context
FROM audit_log
WHERE action LIKE 'sovereign_agent.%' OR action LIKE 'platform.sovereign_agent.%'
ORDER BY created_at DESC LIMIT 50;

Rule book publishing

The sovereign agent only applies rule books with a verified signature. To publish a new version:

  1. Ensure RULE_BOOK_SIGNING_PRIVATE_KEY_PEM is configured on cp-api.
  2. Call POST /admin/rule-book/publish with the framework / version. The endpoint writes rule_book-{version}.tar.gz + .sig to dist/ and returns the SHA256.
  3. Upload both files to the rule book CDN:
az storage blob upload-batch \
  --source dist/ \
  --destination "$CONTAINER" \
  --account-name "$STORAGE_ACCOUNT"
  1. Bump SOVEREIGN_AGENT_RULE_BOOK_VERSION on any agents you want to fast-forward (default is "latest" — agents poll every 15 min and apply automatically).

For offline / air-gapped publishing run apps/cdn-builder/build_rule_book_artifact.py directly with a private key on a one-shot host.

Revoke

When a customer stops using their agent (container retired, key suspected of leaking, customer offboarded):

  1. Navigate to /admin/agents.
  2. Click "Revoke" on the row.
  3. Confirm.

After revoke: * Every JWT signed by the revoked private key returns 401 from cp-api. * The audit row platform.sovereign_agent.revoked carries the actor email + subject. * The agent container will keep running but every cp-api call fails — the customer should docker compose down.

To re-provision after revoke, the operator runs the provision flow again. The partial unique index on (tenant_id) WHERE revoked_at IS NULL permits this; the old row stays in the table for audit purposes.

Troubleshooting

Customer reports "401 unauthorized" in the agent logs

Check the audit log for sovereign_agent.invalid_jwt rows. The context.reason field discriminates:

  • unknown_subject — the agent's JWT carries a subject we don't have. Confirm the customer pasted the right SECRUNA_AGENT_SUBJECT into their env.
  • tenant_mismatch — agent JWT tenant_id ≠ path tenant_id. The customer may have copied a template for the wrong tenant.
  • revoked — the row was revoked; re-provision.
  • invalid_jwt — signature failure. The customer pasted only part of the private key, or pasted the public key by accident.

Agent shows last_rule_book_hash=NULL after 30 min

The agent is failing to verify the rule book artifact. Check:

  • SECRUNA_RULE_BOOK_PUBLIC_KEY_PEM on the customer side matches RULE_BOOK_SIGNING_PUBLIC_KEY_PEM on cp-api.
  • The CDN URL serves a valid tarball at {cdn}/v1/rule_book-v1.tar.gz + .sig.
  • If you recently rotated the rule book signing key, all running agents need their SECRUNA_RULE_BOOK_PUBLIC_KEY_PEM env var updated and restarted.

last_queue_depth stays >0 for a long time

The agent picked up a discovery task but can't complete it. Ask the customer to inspect the container logs; common causes:

  • LLM provider unreachable (Ollama down, Azure OpenAI key expired).
  • Connector credentials misconfigured.
  • Customer's network egress blocked.

Security notes

  • Treat every provision's private_key_pem like an API token. Once it leaves the modal, you cannot retrieve it.
  • The audit log NEVER contains the private key.
  • The sovereign_agents table stores ONLY the public key — losing the DB does not leak any agent's signing material.
  • Rule book signing key is held by the platform-admin team; rotate annually per the security review.