Endpoints

An endpoint makes a model available for inference requests. Create one to test a selected base model or candidate model with your own requests.

Endpoints are for model testing rather than production traffic.

akka training endpoints info
akka training endpoints create --model triage/qwen3-0.6b:v3 --project triage
akka training endpoints get ENDPOINT_ID
akka training endpoints list
akka training endpoints send ENDPOINT_ID --input TEXT
akka training endpoints proxy ENDPOINT_ID
akka training endpoints detach ENDPOINT_ID MODEL
akka training endpoints delete ENDPOINT_ID

For --model, specify a selected base model or candidate model. For --project, specify the endpoint’s training project. The endpoint’s devices count against the training project’s GPU quota, and Optimize refuses a request that would exceed it. --project defaults to the training project recorded by the model. A candidate needs this flag only to confirm its training project, and Optimize refuses a different training project. A base model has no training project, so you must pass --project to serve one.

For ENDPOINT_ID, specify the complete endpoint ID or a unique prefix. endpoints list shows the beginning of each ID.

endpoints create returns immediately, while the endpoint loads the model. Add --wait to return when it responds. The wait ends after 15 minutes. To wait longer, use --timeout.

endpoints delete stops the entire endpoint. Optimize refuses this command when the endpoint serves more than one model because other people might be using those models. To remove one of several models, see Sharing an endpoint.

A training run can start an endpoint to serve its judge. endpoints get shows the run in the Held by row. Optimize refuses to stop the endpoint while the run is calling it. The run releases the endpoint when it finishes.

An endpoint reports one of the following phases:

Phase Meaning

Phase	Meaning
`AWAITING_CAPACITY`	The endpoint is queued, waiting for available devices.
`STARTING`	The endpoint has its devices and is loading the model.
`READY`	The model is loaded and the endpoint accepts requests.
`FAILED`	The endpoint could not start. To see why, run `endpoints get`.
`TERMINATED`	The endpoint has stopped and released its devices.

AWAITING_CAPACITY

The endpoint is queued, waiting for available devices.

STARTING

The endpoint has its devices and is loading the model.

READY

The model is loaded and the endpoint accepts requests.

FAILED

The endpoint could not start. To see why, run endpoints get.

TERMINATED

The endpoint has stopped and released its devices.

What an endpoint can serve

Run endpoints info to see what an endpoint in your deployment can serve before you create one:

akka training endpoints info

The output looks like this:

Engine                          vllm
Adapter ranks                   up to 64
Adapters loaded                 8 at once; more can attach
GPUs per endpoint               1 H200
Context                         the model's own
Idle                            stopped after 60m with no request
Gemma4ForConditionalGeneration  tool calls (gemma4); reasoning apart (gemma4)
Qwen3ForCausalLM                no tool calls: a request with tools is refused; reasoning stays in the answer

Adapter ranks is the largest LoRA rank an endpoint loads. An adapter of a lower rank also loads. Optimize refuses a run that trains an adapter above this rank.
Adapters loaded is how many adapters one endpoint has loaded at once, in addition to its base model. More can attach. The endpoint loads one when a request first uses it.
GPUs per endpoint is how many devices one endpoint reserves, and of which type. Where the deployment names no type, the row says the cluster places the endpoint on any type.
Context is the longest context an endpoint serves: a number of tokens where the deployment limits it on the vLLM engine, otherwise the model’s own.
On the vLLM engine, each architecture of a selected base model has a row that says what an endpoint returns for a model of that architecture beside its text. See Tool calls and reasoning.
Idle is how long an endpoint can go without a request before it stops. An endpoint that a running training run uses as its judge does not stop while the run uses it.

Your deployment sets each of these. endpoints create has no option to change them. Where an endpoint serves base models only, an Adapters row says so in place of the two adapter rows. Where the deployment serves nothing, a Serving row says why and the command shows nothing else.

An endpoint serves a base model and adapters trained over it. It does not serve a model registered as whole weights.

Send one input

Send one input to a model:

akka training endpoints send ENDPOINT_ID --input "Payment failed twice, card declined."

The command prints the answer as the model produces it. The training service authenticates the request, so you do not need separate credentials.

Use --file to read the input from a file, or --file - to read standard input.

An endpoint uses a trained model’s training instructions: the system message of the first row of its training dataset. endpoints get shows them on the model’s Instructions row. For a base model, an externally registered model, or a model trained on prepared token rows, the endpoint uses no instructions.

Use --instructions to provide your own instructions:

akka training endpoints send ENDPOINT_ID --file ticket.txt \
  --instructions "Reply with one label." --temperature 0

The same applies to a request sent through endpoints proxy: a request that states no system or developer message uses the training instructions, and a request that states one is sent as it is.

If the endpoint serves several models, select one with --model. Otherwise, the command uses the endpoint’s model.

Use --max-tokens to limit the length of the answer. Zero, the default, leaves it to the deployment.

With -o json, the command returns the model, the output, the reasoning, the tool calls and the token usage instead of streaming text.

Tool calls and reasoning

A model can be trained to call tools and to reason before it answers, each in its own format. An endpoint on the vLLM engine extracts tool calls and reasoning from the model’s output using a parser for each format. Each architecture has its own parsers, so an endpoint serves a model with the parsers of its base model’s architecture. endpoints info shows them for each architecture of a selected base model:

Where the architecture has a tool-call parser, a request can offer the model tools, and the endpoint returns each tool call the model asks for separately from the text. Where it has none, the endpoint refuses a request that offers tools.
Where the architecture has a reasoning parser, the endpoint returns the reasoning separately from the answer. Where it has none, any reasoning stays in the answer’s text.

Gemma 4 has both parsers, named gemma4. An endpoint on another engine lists no parsers, drops a request’s tools and answers in plain text. The endpoint formats the request with the chat template the model’s repository ships.

Offer tools with --tools, which takes a file holding an OpenAI tools array with at least one tool:

akka training endpoints send ENDPOINT_ID --input "What is the weather in Paris?" \
  --tools tools.json

The command prints the reasoning to standard error and the answer to standard output, followed by each tool call the model asks for, by its name and arguments. With -o json, the output has a reasoning field and a toolCalls list, each left out when the model wrote none.

A request sent through endpoints proxy can set tools and tool_choice too. The proxy passes them to the endpoint as they are, and returns the endpoint’s tool_calls and reasoning fields as it writes them.

Send requests from an application

endpoints send sends one input and retains no history. To use a model from an application or test script, open a local address that forwards to the endpoint:

akka training endpoints proxy ENDPOINT_ID

An endpoint responds inside the deployment, where your machine cannot reach it directly. The proxy forwards requests through the training service, which authenticates them like other commands. The command prints its local address and continues forwarding until you stop it with Ctrl-C. Use --port to choose the port.

The address is OpenAI-compatible. Configure an existing client to use it, or send requests directly:

curl -s ADDRESS/chat/completions -H 'content-type: application/json' \
  -d '{"model":"MODEL_NAME","messages":[{"role":"user","content":"Hello"}]}'

Replace the following:

ADDRESS: the address the proxy prints, which ends in /v1
MODEL_NAME: the name endpoints get shows beside the model

The proxy forwards output as it arrives, so a request that sets "stream": true produces tokens as the model writes them.

Testing and production

A training endpoint is for testing a model. You reach it through endpoints send and endpoints proxy, which authenticate with your akka login, and the proxy forwards requests only while its local process runs. An endpoint also stops after it goes without requests for the time endpoints info shows. Its address is not a stable address for an application to call.

A training endpoint is not for production traffic. The serving loop routes production traffic under a policy. See The loops.

How an endpoint is placed

endpoints create has no option for devices. Your deployment decides how an endpoint is placed, and endpoints info shows the result:

GPUs per endpoint is the number of devices one endpoint reserves. The default is one device.
The device type is the accelerator the deployment uses for every endpoint. If no type is configured, an endpoint can use any available device. For example, a deployment that sets one device of type H200 places each endpoint on one H200.
Models over one base model at one commit share one endpoint, so a candidate whose base model is already served adds no devices. See Sharing an endpoint.

An endpoint’s devices count against its training project’s GPU quota, as a run’s do. When the devices are not free, the endpoint waits in the AWAITING_CAPACITY phase, and endpoints get shows what it waits for. Your deployment uses either the same devices for endpoints, runs, and evaluations, or separate devices for endpoints. Ask your deployment’s operator to change how endpoints are placed.

Device usage

An endpoint reserves its devices from the time it starts until you delete it or it stops after going without requests, even when it is not handling requests.

Endpoints, runs, and evaluations use the same devices, unless your deployment gives endpoints devices of their own. A running endpoint then leaves fewer devices for training and evaluation. Delete the endpoint when you finish testing. Deleting it, or detaching the last model it serves, releases its devices without affecting other endpoints.

Models with the same base model at the same commit share one endpoint. The endpoint loads a candidate’s adapter when a request first uses that candidate. Eight candidates trained from one base model therefore share one endpoint and one set of devices.

endpoints create manages this sharing. If no endpoint serves the base model, the new endpoint receives its own devices and enters the STARTING phase. If an endpoint already serves the base model, the command adds your model to that endpoint without using more devices. The command does not reuse a deleted endpoint, even if you create an endpoint immediately after the deletion.

Each request specifies a model. A base model uses the repository as its name. A candidate uses the repository and adapter name. Run endpoints get to find the value to send.

The deployment determines how many adapters one endpoint can load.

To stop serving one model and leave the rest of the endpoint running:

akka training endpoints detach ENDPOINT_ID triage/qwen3-0.6b:v3

Specify the model’s registry version. endpoints get shows the version beside each served model. Use the version here, rather than the model name used in requests.

Detaching one model does not affect the others. Detaching the last one stops the endpoint and releases its devices.

Serve an adapter trained elsewhere

An endpoint can serve a LoRA adapter that was trained outside Optimize, such as a standard PEFT adapter: a directory that holds adapter_config.json and adapter_model.safetensors. Import it, then create an endpoint for it.

Select the base model the adapter was trained over:
```
akka training base-models select google/gemma-4-E4B-it --wait
```
base-models get shows the commit the deployment pins the model to. The adapter must have been trained over that commit. If it was trained over another, ask your deployment’s operator to allow the base model at the adapter’s commit.
Import the adapter as a version of a named model:
```
akka training adapters import triage/support-adapter --base google/gemma-4-E4B-it -d ./epoch-2
```
The command uploads the two files, follows the import while the trainer checks them, and prints the version it registered and the command that serves it.

Create an endpoint for the version:

akka training endpoints create --model triage/support-adapter:v1 --wait

Optimize refuses the import, registers nothing, and deletes what you uploaded when:

either file is missing;
either file is empty, adapter_config.json is larger than 1 MiB, or adapter_model.safetensors is larger than 16 GiB;
adapter_config.json is not a LoRA adapter’s configuration, its peft_type is not LORA, or it sets use_dora or modules_to_save, which an endpoint does not load;
its base_model_name_or_path is missing, names another base model, or is neither a repository such as google/gemma-4-E4B-it nor a path into the Hugging Face cache. A path into the cache, such as …/models—google—gemma-4-E4B-it/snapshots/COMMIT, names the repository and commit in it;
its revision is not the commit the base model pins, such as a branch name like main;
the named model already holds base model versions;
its rank, the largest of r and any value in rank_pattern, is above Adapter ranks in endpoints info;
the adapter does not load onto the base model, for example because it adapts a module the base model does not have. The refusal names the first such module. A deployment without a training cluster does not run this check;
the files are not all uploaded within an hour of the import’s start, or the load check does not finish within 30 minutes.

--base names a selected base model by its repository, or a base model version such as google/gemma-4-e4b-it:v1.

The import records the training project an endpoint for the adapter counts against. --project names it, and it defaults to the training project whose name is the model’s namespace, so endpoints create needs no --project for an imported adapter.

If you stop following an import, use training adapters get IMPORT to check its progress.

An imported adapter has no default instructions: a request that states no system message is sent as it is. Adapters over one base model at one commit share one endpoint, as Sharing an endpoint describes, and each request names its adapter by the name endpoints get shows.

Register an adapter with a descriptor

training adapters import writes a descriptor for you. To register an adapter whose files are already in your deployment’s adapters location, write the descriptor yourself and register it with registry models register:

servingModel:
  variant: ADAPTER
  baseModel: google/gemma-4-e4b-it:v1
  ref: s3://BUCKET/optimize-training/adapters/support-adapter-epoch-2/
  adapterRank: 64
lineage:
  derivedFrom:
    - artifactKind: model
      ref: google/gemma-4-e4b-it:v1
      role: base

The descriptor names the base model version twice: as baseModel, and as the one parent with the role base. base-models get REPOSITORY --jq .model prints that version. ref names the adapter’s directory in your deployment’s adapters location, and adapterRank is the largest of r and any value in rank_pattern in adapter_config.json. Leave out revision, because the base model version already pins the commit.

akka registry models register triage/support-adapter -f adapter.yaml
akka training endpoints create --model triage/support-adapter:v1 --project triage --wait

A version registered this way belongs to no training project, so endpoints create needs --project. When you create the endpoint, Optimize checks the adapter’s rank and location. It refuses an adapter whose ref names a bucket other than your deployment’s. It does not check that the adapter loads: an adapter whose tensors do not match its base model fails when a request first uses it.