Production readiness checklist for agentic AI workloads

Model provider, model, and parameters (for example temperature and maximum tokens) set in configuration for each agent and recorded with the release. See AI model provider configuration

Default model configuration reviewed, so an agent without its own setting does not run on an unintended model

Location of each agent’s prompts known: in the code, or in a PromptTemplate entity

Access to change a PromptTemplate entity controlled, because a change takes effect at runtime without a deployment. The entity keeps the history of changes. See Prompts

Prompt in effect for each agent retrievable for any past release

Memory provider chosen for each agent, because session memory is enabled by default and an agent that must not keep history needs MemoryProvider.none(). See Session memory

History size bounded for each agent, with a limited window or compaction, because the history goes to the model as context in each request and adds to token usage

Session ID strategy defined: a new ID for each independent interaction, and a shared ID only where agents must share history

Function tools and MCP tools that each agent can call listed and reviewed

Each tool limited to the access that the agent needs, with side-effect tools identified

Retrieval sources and indexes that an agent reads, if it uses retrieval, versioned so you can tell which version an answer used

Guardrail, sanitizer, and evaluator entries in application.conf included in the release and reviewed with it

Control matrix of the service reviewed, with each guardrail, sanitizer, and evaluator naming the control that it implements. See Control ids

Tests with a mocked model run on every commit, covering structured output, tool failure, a guardrail denial, and escalation paths. See Testing

Suite against the real provider run on a schedule and before promotion, so a silent change in provider behavior shows up as a change in results, with assertions that allow for output that varies between runs

Evaluation suite covering a regression set, known-bad cases, tool use, and guardrails

Failing evaluation threshold blocks promotion in your pipeline, because an evaluator that records a verdict does not block anything by itself

Each production incident adds at least one case to the suite before the postmortem closes

Evaluators bound to the agents that need them in application.conf, with verdicts monitored through the akka.agent.evaluation.result metric. See Evaluators

Direct call to the judge agent where an agent or workflow must react to a verdict, such as retrying a step, because a bound evaluator cannot change the next step. See LLM evaluation

Guardrails covering the inputs, tool calls, and responses that matter for the workload, for example prompt injection, toxic output, and disclosure of personal data. See Guardrails

New guardrails run with report-only = true against realistic traffic before enforcement, with an acceptable false-positive rate

Enforced denial producing a response that the caller can act on: it reaches the agent’s onFailure handler as a GuardrailException, and an agent that returns a stream has no onFailure, so the stream fails

Own timeout set on any guardrail that calls a model or a classifier, because the runtime applies no timeout to a guardrail and an interaction waits for it

Sanitizers masking personal data in logs, model calls, and tool results where the workload handles it. See Data sanitization

Prompt-injection testing run against the current release, including injection through retrieved content and tool output. See Adversarial tests

akka.javasdk.agent.max-tool-call-steps set explicitly for each agent, with ToolCallLimitReachedException handled. The default is 100. See Tool call limits

Cost ceiling defined for each agent or for the service, with an alert that triggers before the ceiling is reached

Way to stop each agent documented, with a named person who has the authority and access to use it without a code change

Suspend and terminate rehearsed in the development region for each autonomous agent. A suspend takes effect at the next iteration boundary and does not interrupt an iteration in progress. See Suspend and resume

Longest iteration of each autonomous agent measured, because it is the longest time that a suspend can take to act

Stop path rehearsed in the development region for an agent that answers requests, using akka services pause. Pausing stops every component in the service, so an agent that must stop on its own runs in its own service. See akka services pause

Consumers of the agent’s output stop processing when the agent stops, and do not act on missing output

Tightened guardrail or reduced tool set deployed through your normal change path, with the time from decision to effect measured

Dashboards for token usage (gen_ai.client.token.usage), agent command duration, guardrail denials, and evaluation results. See the metrics reference

Token usage attributed to a team, a project, or an agent, with cost per interaction tracked

Alerts on a rise in guardrail denials, a drop in evaluation results, and an unusual level of token usage

One test interaction traced in your tracing backend, showing the agent command, model call, tool call, and guardrail spans. See the tracing reference

Ledger record retention confirmed with your Akka team against your audit retention

Past interaction rebuilt from the ledger and traces: the prompt, the model, the tool calls, the guardrail outcomes, and the final result

Human overrides of an agent’s output, where they exist, logged with a reason and reviewed for patterns

onFailure handler of each agent covering rate limits, timeouts, tool failures, and responses that cannot be parsed, ending in a default case. See Failure handling

Connection and response timeouts set for each model provider

Step timeout that fits a slow model call, and a recovery strategy, set on any workflow that calls an agent, so that a failing request does not retry indefinitely. See Agent invocation

Provider rate limits and quotas documented, with quota increases requested before launch

Provider outage rehearsed in the development region, for example with a mock model service that fails or responds slowly, with the fallback confirmed. See Testing

Capacity of a fallback provider or model, where you use one, sized for the full load of the primary

Deprecation dates of each model in use tracked, with a swap to another model rehearsed in the development region

Review and rollout steps defined for each kind of change: the model, a prompt, the tool set, retrieval data, and the runtime controls

Prompt rollback rehearsed, with the effect on sessions in progress understood

Model rollback rehearsed, with the evaluation suite confirming that behavior matches the earlier release

Rollback of a tool, a guardrail, or a parameter rehearsed

Rollback of retrieval data, if the agent uses it, to an earlier version rehearsed, with answers checked afterwards

Time to roll back each part measured

Playbook for each of these failures, with a detection signal and a containment step: prompt injection, disclosure of data through a tool or retrieval, an action outside the intended scope, runaway token usage, wrong answers, and a provider outage

Playbooks covering failures that do not raise an availability alert, such as a confident but wrong answer

Playbooks naming who can stop the agent and how

At least one playbook tested before go-live, for example in a tabletop exercise or with a simulated failure in the development region

Provider account, key, and quota for each region documented, with the data residency of prompts and responses stated

Session memory behavior tested in each region, including a failover. With request-region primary selection, session memory includes only the local region until another region reads it. See Session memory

Evaluation suite producing comparable results in each region

Cost ceiling defined for each region, not only for the whole service

Ability to stop the agent in one region while the other region keeps serving, for example with akka services pause --region