Support triage: narrow classification graded by exact match

This example classifies a retail banking support message as one of 77 intents. Use it to try a narrow, high-volume task that can be checked without a human. The gold answer is a fixed label, so exact matching determines correctness without using one model to grade another.

The example directory contains the data preparation script, pipeline descriptor, base model descriptor, and walkthrough.sh. The walkthrough runs each step on this page against the project at AKKA_OPTIMIZE_URL. To download the directory, see the support triage archive.

Before you begin

  • A trainer and the artifact registry deployed in your Akka project, with the address of the project’s route in AKKA_OPTIMIZE_URL. akka route get <route> shows the route’s hostname. The walkthrough registers its dataset in the registry. See CLI commands for training.

  • An authenticated CLI session. Run akka auth login to authenticate akka.

  • Python 3. The data script uses the standard library only.

  • Network access to datasets-server.huggingface.co, where the data script reads the rows. The API limits the request rate, so fetching a large ROWS value takes several minutes. The script caches rows, so subsequent runs use the cached data.

  • A base model that your trainer allows. Run akka training base-models list to list the models, and akka training base-models select <repository> to select one. See Base models.

The walkthrough reads ROWS and EVAL_ROWS from the environment. Start with ROWS=500 EVAL_ROWS=100. This data is enough for a 1B model to learn the task, and training takes a few minutes on one GPU after the run receives a device. The defaults are 2,000 and 400.

Why this task

The task has the following properties:

Property How the task meets it

Gradeable without a judge

77 fixed labels, exact match.

Narrow and repetitive

One intent per message, no free text.

Small output

{"intent": "card_arrival"} rather than prose.

Openly available

mteb/banking77, MIT license.

The data

prepare_dataset.py fetches mteb/banking77 from the Hugging Face datasets-server API. It uses only the Python standard library. The script writes chat-format JSONL with the customer’s message as the user turn and a one-field JSON label as the assistant turn.

To prepare the data, run:

python3 prepare_dataset.py --rows "$ROWS" --eval-rows "$EVAL_ROWS"

The script writes three files:

File Purpose

data/train.jsonl

The rows that the model trains on.

data/eval-tuned.jsonl

Held-out rows under the same short system prompt that the training data uses. The tuned model is scored on this file.

data/eval-frontier.jsonl

The same rows and gold labels under a system prompt that lists all 77 intents. An untuned model needs this prompt on every request to answer at all. The baseline is scored on this file.

Both evaluation files contain identical rows and gold labels. Only the model and its prompt differ in the comparison. A tuned model avoids sending the approximately 1,900 extra prompt characters in every request.

The number of rows matters more than the number of epochs. With 500 rows across 77 intents, each class has about six examples. A classifier loses quality faster from too few examples per class than from fewer passes over those examples. To improve the result, increase ROWS before increasing epochs in the descriptor.

Register the datasets and the training project

To register the datasets, run:

akka registry datasets register support-triage/train -f data/train.jsonl
akka registry datasets register support-triage/eval -f data/eval-tuned.jsonl
akka registry datasets register support-triage/eval-frontier -f data/eval-frontier.jsonl

To create the training project, run:

akka training projects get support-triage >/dev/null 2>&1 ||
  akka training projects create support-triage
akka training projects set-evaluation support-triage --dataset support-triage/eval --grader json-label-match

The training project uses the tuned prompt file as its evaluation dataset and the default json-label-match grader. Optimize refuses creation when the project name already exists, so the walkthrough creates the project only when it is missing. It then sets the evaluation settings every time. As a result, set-evaluation applies this dataset and grader to a project left in any state by an earlier pass.

Before the first run, select the base model for the pipeline. A run can specify only a model that the deployment allows and that someone has selected:

akka training base-models select unsloth/Llama-3.2-1B-Instruct --wait

--wait returns when the selection is ready. If the selection did not cache the weights, the command says so, and the first run downloads them when it starts. Selecting an already selected model succeeds, so you can repeat the walkthrough.

Train and score

The descriptor defines a two-stage pipeline: a supervised run followed by an evaluation of its model.

{
  "kind": "pipeline",
  "stages": [
    {
      "type": "train",
      "name": "sft",
      "configName": "sft-r8-e2",
      "project": "support-triage",
      "baseModel": "unsloth/Llama-3.2-1B-Instruct",
      "config": {
        "dataset": "support-triage/train",
        "hyperparams": {"lora": {"rank": 8}, "optimizer": {"learningRate": 0.0001}, "training": {"epochs": 2}},
        "maxSeqLength": 2048
      },
      "execution": {"gpus": 1}
    },
    {
      "type": "evaluate",
      "name": "scored",
      "model": "@sft",
      "dataset": "support-triage/eval",
      "maxCompletionTokens": 32
    }
  ]
}

The base model must be one that your deployment allows and that someone has selected; see Base models. If the deployment does not allow unsloth/Llama-3.2-1B-Instruct, replace the model in pipeline.json and in the baseline commands below. The results will differ. maxCompletionTokens limits the length of each scored answer. A label is short, but a model that has not learned the format can otherwise continue generating.

To start the pipeline, run:

PIPELINE=$(akka training pipelines start -f pipeline.json -o json --jq .pipelineId)
akka training pipelines wait "$PIPELINE" --exit-status
akka training pipelines get "$PIPELINE"

pipelines get reports each stage, its run or evaluation, and the evaluation score.

The baseline

Score the untuned model on the same rows with the complete taxonomy in the prompt. Selecting the base model registered it as a version of a named model identified by its lowercase repository name. The version pins the commit resolved by the catalog. Use that version for the baseline.

To score the baseline, run:

# Selecting the base model registered it as a version of the named model its repository
# lowercases to, so the baseline names that version.
BASE=$(akka registry models get unsloth/llama-3.2-1b-instruct:latest -o json --jq .reference)
BASELINE=$(akka training evaluations start --model "$BASE" \
  --dataset support-triage/eval-frontier --project support-triage -o json --jq .evaluationId)
akka training evaluations wait "$BASELINE" --exit-status

Read the result

To read the scores, run:

akka training projects scores support-triage
akka training evaluations report "$BASELINE"

training projects scores tabulates the tuned run under the project’s evaluation settings. The baseline uses a different dataset, the complete prompt file, so read its evaluation report separately. Compare the two mean scores. Both models answer the same rows against the same gold labels. The tuned model stores the taxonomy in its weights, while the baseline receives it in every request.

Base model size affects the result more than rows or epochs. At 1B parameters, a short run learns the task. At 135M parameters, a short run completes the process but learns almost nothing.