0.8.1

This release fixes issues with resuming long runs, retaining saves, reporting validation loss and uploading large files.

Changes

  • akka training runs resume --step resumes from any save of the run that still holds its training state. Previously, Optimize refused a step whose save came after the run’s first 1000 saves.

  • execution.keepAdapters and execution.keepRecoverable apply to every save of a run. Previously, they did not reach the saves after a run’s first 1000, which were never removed.

  • A supervised run’s validation loss is a number when part of the validation set holds no supervised tokens. Previously, the validation loss could be NaN.

  • eval datasets create -f and training sessions create -f no longer time out while the file is still sending. They time out when an upload sends nothing for 60 seconds, or when Optimize takes more than ten minutes to answer once the file is sent, the same bounds as registry datasets register. Previously, they timed out after 30 and 60 seconds even while sending. --timeout still limits the whole command when you set it.

  • training runs get marks each start check ✓ passed, ✗ failed or – skipped, so you can distinguish the results with or without color.

  • training runs get --settings shows the validation batch size and how many of the newest saves keep their training state when execution.keepRecoverable is set.