akka optimize training capacity scan
Report what each cluster node holds of the jobs that ran on it
Synopsis
Run a short job on the training cluster that reads each live node, and report what it found: the devices the node can see, the space free on its disks, the directories jobs claimed in its temporary directory, the run directories under the scratch directory, and, where the trainer runs a runtime release, the releases installed on the node. The job claims nothing and changes nothing on a node.
DEVICES is measured on the node, and on Kubernetes lists what its pod claimed. "training capacity" shows what Ray was configured to offer instead. Where a node’s devices could not be read, DEVICES says why. Its directories were still read, so this does not change the exit status.
DISK FREE is what a job could still write on the disks of the temporary directory, the scratch directory and shared memory. A disk that could not be measured says why, and does not change the exit status either.
RELEASES is shown only where the trainer runs a runtime release. It lists each release directory on the node that holds a venv.
A directory a job claimed stays where a process was killed, until the next job of the same user claims one on that node. A run’s directory stays where its job was killed. Read a run’s own report with "training runs get".
This needs the control permission, since it submits a job and names every project’s runs. A node that has scaled down is not listed.
Exits non-zero when the scan fails, and when a node could not be read, so a node that was not read is not taken for an empty one.
akka optimize training capacity scan [flags]
Examples
# What the nodes hold
akka optimize training capacity scan
# Every directory jobs claimed, one a line
akka optimize training capacity scan --jq '.nodes[] | .node as $n | .claimed[] | "\($n) \(.)"'
Options inherited from parent commands
--cache-file string location of cache file (default "~/.akka/cache.yaml")
--config string location of config file (default "~/.akka/config.yaml")
--context string configuration context to use
--disable-prompt Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
--force Skip the confirmation prompt
--jq string Filter the JSON output with a jq expression
--json string Output JSON with only these fields; --json= lists the fields a command offers
--no-color Print without color or a redrawn progress line, as NO_COLOR does
-o, --output string set output format to one of [text,json,json-compact,go-template=] (default "text")
--page-mode string the mode for paging, either paged, buffered or auto. (default "auto")
-q, --quiet set quiet output (helpful when used as part of a script)
--region string Region to use if the project has more than one region
--timeout duration client command timeout (default 10s)
--trainer-url string Trainer service URL (default TRAINER_URL, then AKKA_OPTIMIZE_URL, then a local dev-mode service, then http://localhost:9001)
--use-grpc-web use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
--verbose set verbose output
SEE ALSO
-
akka optimize training capacity - Report what holds the GPU pools