Allocating hardware to models

Feature set: Inference Contact our support for access.

An accelerator allocation claims whole devices from the pools described in Reviewing the available pools. A model is placed on a device of an allocation, never on a pool, so an allocation has to exist before a model can run.

Reviewing the available pools

Every inference-capable region declares pools of accelerator devices. Pools are inventory rather than something you create, so the first question before you deploy a model is which pools exist and what hardware each one holds.

Listing the pools

akka models pools list

The command lists the pools of every region that your organization can access. To list the pools of one region, pass --region. Repeat the flag to list several regions:

akka models pools list --region REGION
Column Value

REGION

The region that declares the pool.

NAME

The pool name. This is the value an allocation gives as acceleratorPool.

PRODUCT

The device that the pool holds.

MEMORY/DEVICE

Memory on one device. A model has to fit inside this at the context length that you configure.

MIG PROFILE

The partition of a device that the pool offers, or - when the pool offers whole devices.

When no region in the listing offers a pool, the command prints that no accessible region advertises one. Regions without the inference capability never do.

Comparing hardware across regions

A pool name is a regional identifier and nothing more. Two regions can name the same hardware differently, and one name can mean different hardware in different regions. Compare the PRODUCT and MIG PROFILE columns, not the name, when you compare hardware across regions.

A model is not placed on a pool directly. The listing is discovery only: how many devices your project can draw from a pool is decided by the region when it admits an allocation.

Declaring an allocation

Declare the allocation in the model descriptor, so that you can keep it under version control next to the models that use it:

resource: AcceleratorAllocation
resourceVersion: v1
metadata:
  name: tutorial
spec:
  allocations:
    - name: gpu (1)
      acceleratorPool: POOL_NAME (2)
      devices: 1 (3)
1 The entry name. A model places onto an entry by this name, so keep the same name in every region.
2 The pool to draw from. Replace POOL_NAME with a pool name from akka models pools list. A pool name is specific to a region, so this is the line that changes between regions.
3 The number of whole devices to claim from the pool. The minimum is 1.

An allocation can hold several entries. Two entries can draw on the same pool under different names, which is how a project holds a large and a small claim on the same hardware. Entry names must be unique across every allocation in the descriptor.

Each allocated device becomes one standing claim. A device is allocated to exactly one project, so projects cannot share a device. A model refers to a device by its index in the entry, starting at 0. See AcceleratorAllocation for every field.

Applying an allocation

Run the region’s admission checks without persisting anything:

akka models apply -f models.yaml --dry-run

The CLI checks the shape of the descriptor first and reports every problem at once. The region then checks whether the devices fit the pool. The region refuses an allocation that exceeds what the project can draw.

Apply the descriptor:

akka models apply -f models.yaml

apply creates or updates each resource by name, so it is safe to run again.

Reviewing the allocations you have

akka models allocations list
Column Value

NAME

The allocation name.

ENTRIES

The number of entries in the allocation.

DEVICES

The number of devices that the entries ask for.

BOUND

The number of devices that the region has bound to claims.

READY

True when every claim is bound to a device.

AGE

How long the allocation has existed.

To read one allocation entry by entry:

akka models allocations get NAME

The output lists, for each entry, the pool and the DEVICES and BOUND counts. An allocation is usable when Ready is True and DEVICES and BOUND agree. While a claim waits for a device, Ready is False with the reason WaitingForDevices and a message that says how many claims are bound.

Placing several models on one device

A model chooses an allocation entry and the index of a device in it. Models that give the same entry and index share the physical device. Each of those models sets placement.memoryGiB to bound its share of the device memory. See Sharing a device between models.

Releasing capacity

Deleting an allocation releases the devices that it held. Delete the models placed on it first. akka models delete -f does this in order: it deletes the endpoints, then the models, then the allocations that the descriptor names. See Removing models.

The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product.

See also