Scoring bundles

A scoring bundle is a ZIP file of Python modules and their data files. The artifact registry holds bundles as named, versioned artifacts. Use a bundle to provide a custom function for a reward term or grader. The trainer runs the function inside a training job or serves it for an evaluation.

Register a bundle

Register a directory or ZIP file under a name in a namespace:

akka registry bundles register gsm8k/scoring -f scoring/
akka registry bundles register gsm8k/scoring -f scoring.zip --alias candidate

The CLI archives a directory consistently, so the same files produce the same archive. It sends a ZIP file unchanged. If the registry already has the archive, the CLI does not send it again. Registering an archive that the named bundle already holds returns the existing version. The command prints the version, such as gsm8k/scoring:v1.

A run names a bundle by registry reference. gsm8k/scoring resolves through latest, and gsm8k/scoring:v2, gsm8k/scoring:candidate and gsm8k/scoring@sha256:…​ name a version directly. The trainer resolves the reference when it admits a run and records the exact version, so moving an alias later does not change accepted work. A reference without a namespace names a bundle in the training project’s namespace, so in a run descriptor for gsm8k, scoring is gsm8k/scoring.

Layout

A bundle directory has the following layout:

scoring/
  reward.py
  helpers.py
  tables/
    schema.json

Modules use relative imports for sibling modules, such as from . import helpers. Locate data files through file.

For the function contract, see The bundle function.

Limits

The registry rejects an archive that contains an unsafe path, a symlink, a reserved top-level module name, or no Python code. It also rejects an archive that exceeds the configured archive size, unpacked size, or entry-count limits. The job verifies the archive’s hash before unpacking it.

Optimize refuses a run if the run cannot read the registered bundle from its storage location.

Your bundle code runs without a sandbox inside the training job.

Use a bundle

To use a bundle as a reward term, name it in the run descriptor:

{"kind": "rule", "weight": 1.0, "bundle": "gsm8k/scoring", "module": "reward"}

To use a bundle as a grader for an evaluation, name it with --grader. A # separates the bundle’s reference from the module, because a reference contains both a slash and a colon:

akka training evaluations start --model gsm8k/qwen3-0.6b:v1 --dataset gsm8k/eval \
  --grader bundle:gsm8k/scoring#reward:compute_score --wait

When the trainer serves the function as a grader, it starts a separate job for the evaluation and stops the job afterward.