Model routing, serving and training

The model routing, serving and training add-on is a set of infrastructure and service extensions for multi-model routing, inference for self-hosted models, training and testing small language models, and cost management. You add it to an existing AAO environment. It uses the same federation and application planes, command and control, communications, security, compliance, data protection, and monitoring infrastructure that AAO already installs. See How AAO works for that architecture.

What Akka installs

Akka installs open source infrastructure in the Kubernetes cluster that AAO uses. The installation is the same for BYOC and BYOK8s.

Component Role

vLLM

Hosts models and runs inference.

agentgateway

Routes requests across multiple models.

KubeRay

Runs training jobs and serves models.

NVIDIA device plugins

Schedules GPUs.

verl

Runs reinforcement learning for sharded and multi-device training runs. Also scores and grades training runs.

Akka also provisions the storage account (bucket) for training. The add-on stores logs and traces of AI interactions in your hyperscaler’s preferred object storage service.

What the add-on needs from you

  • A GPU capacity reserve that is integrated into the add-on. Your Akka team shows you how to configure it.

  • BYOK8s only: a node pool in your existing Kubernetes environment, dedicated to routing and inference.

Models and routing

You deploy a model onto GPU capacity in an Akka region. Any OpenAI-compatible client can call it, from inside an Akka project or from the internet. You can deploy three kinds of model:

  • A public open-weights model.

  • Your own small language model, private to you.

  • A public base model with your own low-rank adaptation (LoRA) adapter on top, served under its own name.

Akka installs and manages the routing and serving components separately from the training components. Initially, a deployment gives you direct access to manage routing and inference. For how-to guides, see Serving your first model and Routing requests to models. Akka’s team configures and manages the initial training loops with you, while Akka stabilizes the external interfaces that define and run long-running training loops.

The diagram shows how a request reaches a model. A client outside the cluster calls the route on a host name. An agent or service inside the project calls the model endpoint by its internal name. Both requests reach the gateway, which checks the bearer key and picks the engine from the model name. The engines accept only the gateway identity.

flowchart TD subgraph internet["internet"] client["OpenAI-compatible client"] end subgraph edge["contour edge"] route["KalixRoute
host + TLS + edge auth"] end subgraph project["project namespace"] agent["in-cluster agent or service"] endpoint["KalixModelEndpoint
models + apiKeySecret"] svc["Service (ExternalName)
CNAME to gateway envoy"] docs["docs-engine"] tiny["tiny-engine"] end subgraph gwsys["agentgateway-system"] gw["gateway envoy
listener per endpoint FQDN
Bearer key + request surface + model table"] end client -->|"https://host/v1"| route route -->|"Host rewritten to internal FQDN
l5d-dst-override to gateway svc"| svc agent -->|"http://endpoint.project.svc.cluster.local/v1
Authorization: Bearer"| svc endpoint -.->|"renders"| svc svc -.->|"DNS only (CNAME)"| gw endpoint -.->|"listener, table rows, key policy"| gw gw -->|"model: docs
mesh: only gateway identity admitted"| docs gw -->|"model: tiny
mesh: only gateway identity admitted"| tiny
Figure 1. How a request reaches a model

View the full-size model request diagram

Tools for evaluation and adversarial testing

Akka provides these tools with the add-on:

  • evalkit, a toolkit that you download separately and that integrates with your Akka project, runs evaluation and scoring campaigns locally.

  • redkit, a toolkit that is also part of your Akka project and downloadable separately, runs adversarial testing campaigns against models.

  • Extensions to the Akka CLI manage evalkit, redkit, and your hosted environment. See akka optimize.

For testing an agent, see Adversarial tests.

Next steps