akka optimize training capacity

Report what holds the GPU pools

Synopsis

Report the devices this deployment schedules, what holds them, and what each workload holds against its quota.

POOLS is one row per pool. DECLARED is what the deployment says the pool holds, which is what work is admitted against, and OBSERVED is what its cluster last reported. The two disagree where a cluster has scaled to zero or has not grown a node yet, which is a fact about the cluster rather than about the queue. MAX CLAIM is the largest device claim the pool accepts. For a training pool, it is the most GPUs the deployment trains one run on, the same for every training pool unless the pool declares fewer. Run "training base-models get" to see the GPU counts admitted for a model. HOST RAM is what a node carrying the accelerator states it has, and is what a run’s own host memory figure is refused against.

DECLARED counts a capacity window while it is open. WINDOWS lists, for a pool whose accelerator exists for bounded times, every window that is open or still to come: the devices it adds to the pool’s fixed count, and when it opens and ends, in UTC. A claim for more devices than the pool always has waits for a window that can hold it, and a run no window left can hold is refused at submission. WARNINGS names each claim still holding devices of a window that has ended. Nothing stops such a claim, and its devices may already be gone.

CLAIMS is every run, evaluation and endpoint that holds devices or is waiting for them. An endpoint a run brought up as its judge names that run.

WORKLOADS is one row per workload that holds a claim or has a quota of its own. QUOTA is the devices it may hold at once, across every pool.

--workload narrows the claims and the workloads to one workload. The pools stay whole, because what a workload is waiting for is the state of the pool, whoever else is holding it.

akka optimize training capacity [flags]

Examples

  # What holds the pools
  akka-optimize training capacity

  # One workload's claims and quota
  akka-optimize training capacity --workload triage

  # Name the runs holding devices
  akka-optimize training capacity --jq '.claims[] | select(.state=="GRANTED") | .workId'

Options

  -h, --help              help for capacity
      --workload string   Narrow the claims and the workloads to one workload, by name, ID or the start of one

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default http://localhost:9001, or TRAINER_URL)
      --url string           Optimize service URL (default http://localhost:9010, or OPTIMIZE_URL)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO

  • akka optimize training - Base models, workloads, runs, snapshots, comparisons, pipelines, evaluations, endpoints and their data