Routing Ruleset reference

Feature set: Inference Contact our support for access.

A routing ruleset is the JSON document applied by akka optimize rulesets apply. It declares the closed set of destinations the ruleset may emit, the destination for unclaimed traffic, the headers the gateway reports its decision in, and the rules themselves. A ruleset can also tag traffic with a value instead of routing it, and then states no headers. See A ruleset that tags traffic. See Routing on request attributes for the tasks these documents support.

A ruleset is a logical grouping of named rules. The ruleset is the unit of enforcement, not the individual rule. The evidence log emitted by the gateway records which rules were evaluated and their results, and rules cannot be enforced on requests outside a ruleset.

Documents sent with -f FILE are JSON. Passing -f - reads the document from standard input.

Predicates

A predicate is a boolean expression. The rule is enforced when the expression evaluates to true. An expression takes the form <attribute> <condition> <value>, for example protocol equals anthropic.messages.

Six predicates are available, and the set is closed.

Predicate JSON value Matches when

equals

string

The attribute equals the string.

prefix

string

The attribute starts with the string.

suffix

string

The attribute ends with the string.

contains

string

The attribute contains the string.

in

array of strings

The attribute equals one of the strings in the array.

exists

boolean

The attribute is present. Use false to match when it is absent.

String comparisons are case sensitive. equals, prefix, suffix, contains, and in all compare exactly as written, so a rule testing subject against GPT-4 does not match a request carrying gpt-4.

Attribute names are matched literally too. Header names are lower-cased when the attribute is populated, so a rule has to name one in lower case. header.x-tenant matches, and header.X-Tenant never does.

There is no regular expression predicate. Predicates are evaluated against prompt text and metadata that a caller can influence, and a regular expression failure would be a way to deny service to the gateway.

The exists predicate separates an absent attribute from an empty one. A request carrying x-tenant: with no value has the attribute present and empty. A request with no such header does not have the attribute at all.

Attributes

The attributes available to a rule are protocol agnostic, so one rule applies to every protocol the gateway accepts without naming which one, unless the rule is itself a predicate on protocol.

Attribute Value

protocol

The shape of the request, either openai.chat or anthropic.messages. Known once the headers arrive.

operation

What is being asked for, such as chat.completions. Known once the headers arrive. See Operation values for how the value is formed.

subject

The model or tool being asked for. Read from the body.

text.prompt

The system message and all message content, concatenated. Read from the body.

text.lastUserMessage

The most recent user turn. Read from the body.

conversation

The conversation the request belongs to. Absent when the caller opted out of being recognized across turns.

conversation.source

Which part of the request supplied the conversation identity.

header.<name>

A request header, with the name lower-cased. HTTP/2 pseudo-headers are excluded, so header.:path never resolves.

Operation values

The value of operation is derived from the protocol and the HTTP method, so it is not a fixed list. Each protocol contributes a base name. A request that uses any method other than POST prefixes that base name with the lower-cased method.

Protocol Method Value

openai.chat

POST

chat.completions

openai.chat

Any other method

The method, lower-cased, then .chat.completions. A GET produces get.chat.completions.

anthropic.messages

POST

messages

anthropic.messages

Any other method

The method, lower-cased, then .messages. A GET produces get.messages.

A request that arrives without a method is treated as a POST. The table covers the protocols the gateway accepts today, and a new protocol adds a base name to it.

Match blocks

A rule’s match is a block. A block takes one of four forms, and exactly one form may be present in any one block.

Form Meaning

all

Every block in the list must hold.

any

At least one block in the list must hold.

not

Inverts the single block it wraps.

attribute

A condition. Names an attribute and states exactly one predicate against it.

A condition is written flat. The attribute name and the predicate sit side by side in the same object:

{ "attribute": "text.prompt", "contains": "task:commit" }

Boolean structure is stated rather than inferred from field order, so a rule’s meaning does not depend on how it was written down. Nesting depth is bounded, and a block that states no conditions is rejected. Traffic that no rule claims falls through to the ruleset default destination.

Outcomes

A rule states what it decides when it matches. Exactly one outcome form is present.

Outcome Decision

toDestination

Send matching traffic to a destination the ruleset declares.

byClassifier

Defer the decision to a named classifier and map the classifier’s answers onto destinations.

byAttribute

Map a value that the request carries, such as a header, onto a destination.

canary

Send a share of the matching conversations to a candidate destination and the rest to a baseline destination.

deny

Refuse the exchange. Matching traffic never reaches a backend. status is bounded to 4xx and body is operator text. Nothing derived from the request is interpolated into the body.

byClassifier

A byClassifier outcome hands the decision to a model. This is the one case where attribute routing consults an LLM.

Field Value

classifier

The id of a registered classifier.

input

The attribute whose value is sent to the classifier, such as text.lastUserMessage.

destinations

A map from the classifier’s own answers to destination labels. The keys belong to the classifier and are not validated. Every value must be declared by the ruleset.

onFailure

default sends the request to the ruleset default destination when the classifier produces no usable answer.

affinity

none decides on every turn. conversation lets the first turn of a conversation decide and later turns reuse that decision. Omitting the field reads as none. Only a deferral can carry affinity.

Classifiers are registered separately from rulesets, and registration is immutable. See akka optimize routing-classifiers for the commands that manage them.

byAttribute

A byAttribute outcome maps a value that the request already carries onto a destination. The gateway router consults no classifier for it. The outcome works in a routing ruleset and in a tagging ruleset.

Field Value

input

The attribute whose value is mapped, such as header.x-optimize-team. It can be any attribute in Attributes.

values

A map from the values of the attribute to destination labels. Several values can map to one destination. Every destination must be declared by the ruleset.

onUnmapped

The destination when the request does not carry the attribute, or carries a value that values does not hold. It must be a destination the ruleset declares. Omitted or blank leaves the decision to the rules after this one, and then to the ruleset default destination.

The outcome applies only when the match of its rule holds. In the example of A ruleset that tags traffic, the match requires the header, so a request without it never reaches onUnmapped.

The gateway router matches the keys of values trimmed and without regard to case. A write with two keys that are the same after trimming and case folding is refused with ambiguous-label-mapping.

canary

A canary outcome sends a share of the conversations the rule matches to a candidate destination and the rest to a baseline destination. See Shifting traffic gradually for how to run a canary.

Field Value

baseline

The destination for the conversations outside the share. It must be a destination the ruleset declares.

candidate

The destination for the share of conversations. It must be a destination the ruleset declares.

candidatePercent

The share of conversations that go to the candidate, from 0 to 100 with up to two decimals.

All three fields are required. The service refuses a write with canary-same-destination when baseline and candidate name one destination, and with canary-percent-out-of-range when candidatePercent is outside 0 to 100.

Ruleset documents

Field Value

kind

route for a ruleset that routes traffic, or tag for a ruleset that tags it. Omitting the field reads as route.

defaultDestination

The destination for traffic that no rule claims. Must be one of the declared destinations.

outputHeader

The header the gateway reports the chosen destination label in. A tagging ruleset leaves it blank.

reasonHeader

The header the gateway reports the reason for the decision in. A tagging ruleset leaves it blank.

destinations

The closed set of labels this ruleset may emit, each with a description. Every outcome, including the default, must name one of these.

rules

The rules. Each carries an id, a priority, a match, and an outcome.

Rules are evaluated by priority, lower first, with id breaking ties. The ordering is total, so the order a client reads is the order any later evaluation uses.

A ruleset’s name, revision, and retired state are assigned by the service, and a submitted document cannot express them. The ruleset’s own priority is set through the CLI rather than through the document, so a reorder does not read as a policy change.

A ruleset that routes to a fixed destination

{
  "defaultDestination": "frontier",
  "outputHeader": "x-optimize-route",
  "reasonHeader": "x-optimize-reason",
  "destinations": [
    { "label": "finetune", "description": "the fine-tuned model" },
    { "label": "frontier", "description": "the frontier model" }
  ],
  "rules": [
    {
      "id": "commit-message",
      "priority": 100,
      "match": {
        "any": [
          { "attribute": "text.prompt", "contains": "task:commit" }
        ]
      },
      "outcome": { "toDestination": "finetune" }
    }
  ]
}

A ruleset that tags traffic

A ruleset of kind tag classifies traffic instead of routing it. The name that the ruleset is applied under is the tag key. Its declared destinations are the values that the key can take, and its default destination is the value for traffic that no rule claims. Applied under the name team, this ruleset gives each exchange a team tag:

{
  "kind": "tag",
  "defaultDestination": "unknown",
  "outputHeader": "",
  "reasonHeader": "",
  "destinations": [
    { "label": "payments", "description": "the payments team" },
    { "label": "search", "description": "the search team" },
    { "label": "unknown", "description": "traffic no team claims" }
  ],
  "rules": [
    {
      "id": "by-header",
      "priority": 100,
      "match": { "attribute": "header.x-optimize-team", "exists": true },
      "outcome": {
        "byAttribute": {
          "input": "header.x-optimize-team",
          "values": {
            "payments-api": "payments",
            "payments-batch": "payments",
            "search-api": "search"
          }
        }
      }
    }
  ]
}

A request that carries x-optimize-team: payments-batch gets the tag team=payments. A request without the header, or with a value that values does not hold, gets the default destination, team=unknown.

A tagging ruleset states no outputHeader and no reasonHeader, and no rule of it may deny a request. The service refuses a write that breaks one of these restrictions:

Refusal Cause

tag-writes-output-header

The tagging ruleset states an outputHeader.

tag-writes-reason-header

The tagging ruleset states a reasonHeader.

tag-denies

A rule of the tagging ruleset has a deny outcome.

A rule that defers to a classifier

A single rule is written with the same match and outcome shape, without the surrounding document:

{
  "id": "tenant-commit-routing",
  "priority": 100,
  "match": {
    "all": [
      { "attribute": "protocol", "equals": "openai.chat" },
      { "attribute": "header.x-tenant", "exists": true }
    ]
  },
  "outcome": {
    "byClassifier": {
      "classifier": "commit-router-v4",
      "input": "text.lastUserMessage",
      "destinations": { "slm": "finetune", "big": "frontier" },
      "onFailure": "default",
      "affinity": "conversation"
    }
  }
}

The two sides of the destinations map come from different places. slm and big are answers the classifier itself returns, and they belong to the classifier’s own vocabulary. finetune and frontier name backends registered with the gateway proxy, and each one has to be declared in the destinations list of the ruleset this rule belongs to.

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.

See also