Reviewing available hardware
A device is a kind of GPU the platform has in a region. Devices are inventory rather than something you create, so the first question before deploying a model is which devices exist and how many cards of each are unused.
Listing the devices in a region
akka models devices listThe output names each kind of card, its memory, and how much of it is free:
NAME PRODUCT MEMORY ARCHITECTURE FP8 TOTAL FREE MAX/NODE
l4 NVIDIA L4 23034Mi Ada Lovelace yes 1 1 1| Column | Value |
|---|---|
|
The device name. This is the value a descriptor gives as |
|
The card the name refers to. |
|
Memory on one card. A model has to fit inside this, at the context length you configure. |
|
The GPU architecture, which determines the numeric formats the card supports. |
|
Whether the card supports 8-bit floating point. |
|
Cards of this kind in the region. |
|
Cards not currently claimed by an accelerator. |
|
The most cards of this kind on any single node. A deployment that splits one model across several cards has to fit inside one node, so this is the ceiling on |
Inspecting one kind of device
To see the individual cards behind a device name and what is running on each:
akka models devices get l4Use this when a card appears free in the list but a deployment will not place on it, because it shows the workload holding each card.
From devices to deployable capacity
A deployment is not placed on a device directly. It is placed on an accelerator, which is a claim on a slice of this inventory. See Allocating hardware to models.
|
The features described in this section are an add-on to Akka Automated Operations. They are not included in the base product. |