Guardrails

Overview

Guardrails are runtime-enforced checks that validate what goes into and comes out of your agents. They protect against harmful inputs, such as jailbreak attempts and prompt injection, and damaging outputs, such as toxic language, hallucinated claims, or accidental disclosure of a competitor’s product. A guardrail can also intercept a tool call and reject it before it runs.

Unlike external validation layers that sit outside your application, Akka guardrails are part of the runtime itself. You declare them in configuration, and the runtime enforces them on every interaction, whether the request targets a model, an MCP tool, or a function tool. This distinction matters: bolt-on governance fails because it depends on developers remembering to call the right library at the right time. When governance lives in the runtime, it cannot be bypassed, forgotten, or misconfigured by individual teams.

This architecture directly supports regulatory frameworks such as the EU AI Act, which requires that high-risk AI systems embed risk controls into their design rather than relying on after-the-fact auditing. Every guardrail decision is a record in the ledger and is also reported in logs, metrics, and traces, so you get an auditable compliance trail without extra instrumentation. See The ledger and evidence.

For protecting sensitive information like PII, see Sanitization.

Kinds of guardrail

A guardrail implements one of three interfaces. The interface sets where the runtime evaluates the guardrail:

  • ModelCallGuardrail evaluates the input to each model call. The runtime evaluates it before the first model call. It evaluates the guardrail again before each later call that sends tool results back to the model.

  • AgentResponseGuardrail evaluates the agent’s final reply, once per interaction.

  • ToolCallGuardrail evaluates a function tool call or an MCP tool call before the runtime runs the tool. It sees the tool name and the raw JSON arguments.

A guardrail class implements exactly one of these interfaces. To check several points, write one class per interface.

An agent that returns StreamEffect streams its reply. The caller receives the reply before the runtime evaluates an AgentResponseGuardrail, so the guardrail cannot block that reply.

The service fails to start when such an agent has an AgentResponseGuardrail entry without report-only = true. Fix the startup failure in one of these ways:

  • Change the agent’s command handler to return Agent.Effect.

  • Remove the agent from the agents or agent-roles list of the entry. If the entry binds agents through * or an agent role, list the other agents by name instead.

  • Remove the agent from the entry as above. Then add a separate entry for this agent with report-only = true. The guardrail then observes the reply without blocking it. The stream completes after the guardrail returns.

Do not set report-only = true on the existing entry. The guardrail would then stop blocking for every agent that the entry binds.

The runtime evaluates a ModelCallGuardrail before each model call of a streaming agent too. A deny fails the stream. See Handling a denied interaction.

All three return a Decision:

  • Decision.Allow: the guardrail found nothing wrong and the call proceeds.

  • Decision.Deny: the call must not proceed. The runtime returns the reason to the user as the failure message. The report-only setting decides whether the runtime aborts the interaction or only records the denial (see Configuring guardrails).

  • Decision.Fail: the evaluation itself failed and reached no verdict. The runtime passes the reason and the cause to the interaction’s failure handling, as the message and the cause of a Guardrail.GuardrailException. The runtime treats three cases as Decision.Fail: throwing from decide, returning a failed CompletionStage from decideAsync, and returning null.

The runtime reports a Deny or Fail with a null or blank reason as "no reason given". The runtime applies no timeout to a guardrail. An interaction waits forever if its guardrail never completes.

Implementing an agent response guardrail

An AgentResponseGuardrail receives a CallContext with the agentId(), sessionId(), and modelName() of the interaction, and returns a Decision. The reply() method returns the agent’s final reply as a Guardrail.Message.AiMessage:

ProfanityResponseGuard.java
import akka.javasdk.agent.AgentResponseGuardrail;
import akka.javasdk.agent.Decision;
import akka.javasdk.agent.GuardrailContext;

public class ProfanityResponseGuard implements AgentResponseGuardrail { (1)

  private final String searchFor;

  public ProfanityResponseGuard(GuardrailContext context) { (2)
    this.searchFor = context.config().getString("search-for");
  }

  @Override
  public Decision decide(CallContext ctx) { (3)
    if (ctx.reply().text().contains(searchFor)) {
      return new Decision.Deny("Response contained '%s'.".formatted(searchFor)); (4)
    }
    return new Decision.Allow(); (5)
  }
}
1 Implement AgentResponseGuardrail.
2 An optional GuardrailContext constructor parameter gives access to the guardrail’s configured name and its config section. Here the guardrail reads the search-for property from configuration.
3 decide receives the per-call CallContext.
4 Return Decision.Deny to reject the reply, with a reason surfaced to the user.
5 Return Decision.Allow to let the reply through.

decide may block, because the runtime evaluates guardrails on virtual threads. To compose futures instead, override decideAsync. The default decideAsync delegates to decide.

Implementing a model call guardrail

A ModelCallGuardrail receives the conversation that enters a model call. Its CallContext holds the systemMessage(), all messages() in the conversation, and the newMessages() after the last AiMessage. On the first model call, newMessages() holds the user message. On each later call, it holds the tool results that the runtime sends back to the model.

Each message is a Guardrail.Message that keeps its origin: a UserMessage, an AiMessage, or a ToolCallResponse. ToolCallResponse.isError() is true when the MCP server reported the result as an error, or when the runtime turned a failed tool call into an error result. When the model calls several tools at once, each result is a separate ToolCallResponse:

ToolResultInjectionGuard.java
import akka.javasdk.agent.Decision;
import akka.javasdk.agent.Guardrail.Message;
import akka.javasdk.agent.MessageContent;
import akka.javasdk.agent.ModelCallGuardrail;

public class ToolResultInjectionGuard implements ModelCallGuardrail { (1)

  @Override
  public Decision decide(CallContext ctx) {
    for (var message : ctx.newMessages()) { (2)
      if (message instanceof Message.ToolCallResponse result) { (3)
        for (var content : result.contents()) {
          if (
            content instanceof MessageContent.TextMessageContent text &&
            text.text().contains("ignore previous instructions")
          ) {
            return new Decision.Deny(
              "Tool '%s' returned an injection attempt.".formatted(result.name())
            ); (4)
          }
        }
      }
    }
    return new Decision.Allow();
  }
}
1 Implement ModelCallGuardrail.
2 Check only the messages that are new to this model call.
3 Match the tool results by type. Each one carries the tool name and its content parts.
4 Deny the model call when a tool result contains the injection text.

Implementing a tool call guardrail

A ToolCallGuardrail evaluates a function tool call or an MCP tool call before the runtime runs the tool. CallContext exposes agentId(), sessionId(), toolName(), toolCallId(), origin(), and arguments(). arguments() returns the raw JSON of the tool call:

PaymentToolGuard.java
import akka.javasdk.agent.Decision;
import akka.javasdk.agent.ToolCallGuardrail;

public class PaymentToolGuard implements ToolCallGuardrail { (1)

  @Override
  public Decision decide(CallContext ctx) { (2)
    if (!ctx.toolName().equals("transferFunds")) {
      return new Decision.Allow();
    }
    if (ctx.arguments().contains("\"amount\":10000")) { (3)
      return new Decision.Deny("transfers of 10000 or more require human approval"); (4)
    }
    return new Decision.Allow();
  }
}
1 Implement ToolCallGuardrail.
2 decide inspects the tool call described by CallContext.
3 The runtime delivers the arguments as a raw JSON string.
4 Return Decision.Deny to block the tool call before the runtime runs the tool.

The origin() method returns where the tool runs.

For a function tool, it returns a ToolOrigin.FunctionTool. For an MCP tool, it returns a ToolOrigin.RemoteMcp that holds the endpoint() of the MCP server. Some MCP servers take credentials in the URI, for example as a query parameter. In that case, endpoint() returns the credentials too.

For an MCP tool, toolName() returns the name of the tool on the MCP server. If the RemoteMcpTools has a ToolInterceptor, the runtime evaluates the guardrail after interceptRequest. The guardrail then sees the arguments that the interceptor returns.

Configuring guardrails

You enable guardrails in configuration, so you can enforce at deployment time that certain guardrails are always active.

src/main/resources/application.conf
akka.javasdk.agent.guardrails {
  "profanity guard" {                                       (1)
    class = "com.example.guardrail.ProfanityResponseGuard"  (2)
    agents = ["planner-agent"]                              (3)
    agent-roles = ["worker"]                                (4)
    category = TOXIC                                        (5)
    report-only = false                                     (6)
    search-for = "bad stuff"                                (7)
  }

  "tool result injection guard" {
    class = "com.example.guardrail.ToolResultInjectionGuard"
    agents = ["*"]
    category = PROMPT_INJECTION
  }

  "payment tool guard" {
    class = "com.example.guardrail.PaymentToolGuard"
    agents = ["*"]
    category = POLICY
    tools = ["transferFunds"]                               (8)
  }
}
1 Each configured guardrail has a unique name.
2 Implementation class of the guardrail.
3 Enable this guardrail for agents with these component ids.
4 Enable this guardrail for agents with these roles.
5 The type of validation, such as PII and TOXIC.
6 If the guardrail denies, the runtime either aborts the interaction or lets it continue. In both cases, the runtime records the result in the ledger and reports it in logs, metrics, and traces.
7 An application-specific property, read by the guardrail from its GuardrailContext.
8 Optional, for a ToolCallGuardrail only. Restricts the guardrail to the named tools. Without it, the guardrail sees every tool call on the selected agents. For a function tool, use ClassName_methodName, or the name set in @FunctionTool. For an MCP tool, use the name on the MCP server. The filter matches by name only, so it also matches a tool with the same name from another origin. Use origin() to tell them apart.

The class property sets the implementation class. The class must implement ModelCallGuardrail, AgentResponseGuardrail, or ToolCallGuardrail, be public, and have a public constructor. The constructor may take a GuardrailContext parameter, which holds the name and the config section of the guardrail. The ProfanityResponseGuard example reads its search-for property this way.

The runtime shares one guardrail instance across concurrent calls from different sessions and agents. The implementation must be thread safe.

The guardrail interface sets where the runtime evaluates the guardrail, so the configuration has no boundary setting.

Agent selection

You select which agents a guardrail applies to by matching agents or agent-roles configuration.

  • agents: enabled for agents with these component ids. If agents contains "*", the guardrail is enabled for all agents.

  • agent-roles: enabled for agents with these roles. If agent-roles contains "*", the guardrail is enabled for all agents that have a role, but not for agents without a role.

If both agents and agent-roles are defined, it is enough that one of them matches to enable the guardrail for an agent.

This role is defined in the @AgentRole annotation.

Names, categories, and observability

The name and the category are reported in logs, metrics, and traces. The category should classify the type of validation. It can be any value, but a few recommended categories are JAILBREAK, PROMPT_INJECTION, PII, TOXIC, HALLUCINATED, NSFW, FORMAT.

Set control-id to name the control that the guardrail implements. See control ids.

Classifier-backed guardrails

A guardrail can delegate its decision to a classifier, resolved by name through GuardrailContext.classifierClient(). The classifier computes the score, for example how toxic the text is, and the guardrail turns that score into an allow or a deny. Because the classifier name comes from config, you can configure the guardrail to use a different classifier without rebuilding. The blocking classify call is safe here, because guardrails are evaluated on virtual threads; to compose futures instead, use classifyAsync and override decideAsync:

ClassifierBackedGuard.java
import akka.javasdk.agent.AgentResponseGuardrail;
import akka.javasdk.agent.ClassifierClient;
import akka.javasdk.agent.Decision;
import akka.javasdk.agent.GuardrailContext;

public class ClassifierBackedGuard implements AgentResponseGuardrail {

  private final ClassifierClient classifierClient;
  private final String classifierName;

  public ClassifierBackedGuard(GuardrailContext context) {
    this.classifierClient = context.classifierClient(); (1)
    this.classifierName = context.config().getString("classifier"); (2)
  }

  @Override
  public Decision decide(CallContext ctx) {
    try {
      var classification = classifierClient.classify(classifierName, ctx.reply().text()); (3)
      if (classification.label().map("toxic"::equals).orElse(false)) {
        return new Decision.Deny("blocked by classifier " + classifierName);
      }
      return new Decision.Allow();
    } catch (RuntimeException e) {
      return new Decision.Fail("classifier invocation failed", e); (4)
    }
  }
}
1 Obtain a ClassifierClient from the GuardrailContext.
2 Read the classifier name from the guardrail’s own config, so it can be changed without rebuilding.
3 Invoke the classifier by name and turn its Classification into a Decision.
4 If the classifier invocation itself fails, return Decision.Fail rather than silently allowing the call.

The classifier is configured separately under akka.javasdk.agent.classifiers; see Classifiers.

Handling a denied interaction

When a ModelCallGuardrail, an AgentResponseGuardrail, or a ToolCallGuardrail returns Decision.Deny with report-only = false, the runtime aborts the interaction. The runtime surfaces the reason to the agent’s failure handling as a Guardrail.GuardrailException, which you can map in onFailure:

GuardedAgent.java
return effects()
  .systemMessage("You are a helpful...")
  .userMessage(question)
  .map(SomeResponse::new)
  .onFailure(cause ->
    switch (cause) {
      case Guardrail.GuardrailException e -> new SomeResponse(e.getMessage());
      default -> throw new RuntimeException(cause);
    })
  .thenReply();

An agent that returns StreamEffect has no onFailure. For such an agent, a deny fails the stream. The error message names the guardrail, its category, and the reason. The error type is not Guardrail.GuardrailException.

With report-only = true, a Deny is recorded in the ledger and reported in logs, metrics, and traces, but the interaction continues. A Decision.Fail with report-only = true lets the interaction continue. The runtime logs a warning and records the error.

Similarity guardrail

The built-in ModelCallSimilarityGuard is a ModelCallGuardrail. Before each model call, the runtime compares the user message and tool results with a dataset of "bad examples". If the similarity reaches the threshold, the runtime denies the call.

This is how to configure the ModelCallSimilarityGuard:

src/main/resources/application.conf
akka.javasdk.agent.guardrails {
  "jailbreak guard" {
    class = "akka.javasdk.agent.ModelCallSimilarityGuard"
    agents = ["planner-agent", "weather-agent"]
    category = JAILBREAK
    threshold = 0.75
    bad-examples-resource-dir = "guardrail/jailbreak"
  }
}

Here, it is using predefined examples of jailbreak prompts in guardrail/jailbreak. Those have been incorporated from https://github.com/verazuo/jailbreak_llms, but you can define your own examples and place in a subdirectory of src/main/resources/. All text files in the configured bad-examples-resource-dir are included in the similarity search.

You can use this for other things than jailbreak attempt detection.

Testing

Guardrails run in integration tests the same way they run in production. Configure the guardrail in src/test/resources/application.conf, run the agent with TestKitSupport, and use a TestModelProvider to produce text that triggers or passes the guardrail. A denied interaction surfaces as a GuardrailException, which the agent handles as shown in Handling a denied interaction.

See also