Scoring bundles

A scoring bundle is a zip file of Python modules and their data files, identified by its content hash. Use a bundle to provide a custom function for a reward term or grader. The trainer runs the function inside a training job or serves it for an evaluation.

Push a bundle

Push a directory or zip file:

akka-optimize training bundles push scoring/ --name gsm8k-scoring
akka-optimize training bundles push scoring.zip

The CLI archives a directory consistently, so the same files produce the same hash. It sends a zip file without changing it. If the service already has the bundle, the CLI doesn’t send it again. The command prints the hash.

--name assigns a name to the bundle or moves an existing name to it. A run records the resolved hash when it is admitted, so moving the name later doesn’t change accepted work.

Layout

A bundle directory has the following layout:

scoring/
  reward.py
  helpers.py
  tables/
    schema.json

Modules use relative imports for sibling modules, such as from . import helpers. Locate data files through file. While the bundle runs, the TRAINER_BUNDLE_ROOT environment variable also specifies the bundle’s root.

For the function contract, see The bundle function.

Limits

The trainer rejects an upload that contains an unsafe path, a symlink, a reserved top-level module name, or no Python code. It also rejects an upload that exceeds the configured archive size, unpacked size, or entry-count limits. The job verifies the archive’s hash before unpacking it.

Your bundle code runs without a sandbox inside the training job.

Use a bundle

To use a bundle as a reward term, name it in the run descriptor:

{"kind": "rule", "weight": 1.0, "bundle": "gsm8k-scoring", "module": "reward"}

To use a bundle as a grader for an evaluation, name it with --grader:

akka-optimize training evaluations start --model MODEL_ID --dataset gsm8k-eval \
  --grader bundle:gsm8k-scoring/reward:compute_score --wait

When the trainer serves the function as a grader, it starts a separate job for the evaluation and stops the job afterward.