akka optimize training endpoints proxy

Open a local address that forwards to an endpoint’s model

Synopsis

Forward a local address to a model served at an endpoint, named by its ID or an unambiguous prefix of it.

An endpoint answers inside the deployment, which a client on this machine cannot reach. This command listens on a local port and sends each request through the training service, which authenticates it and forwards it to the endpoint.

Only POST /v1/chat/completions is forwarded. Anything else is answered 404 here, so a client cannot reach the endpoint’s own resource through the proxy.

The command prints the address it listens on, and forwards until you stop it with Ctrl-C. Use --port to choose the port.

Answers are passed on as they arrive, so a request that sets "stream": true produces tokens as the model writes them.

akka optimize training endpoints proxy ENDPOINT [flags]

Examples

  # Forward a local port to an endpoint
  akka-optimize training endpoints proxy 970076e8

  # Choose the port yourself
  akka-optimize training endpoints proxy 970076e8 --port 8800

Options

  -h, --help       help for proxy
      --port int   Local port to listen on. Zero takes whichever port is free.

Options inherited from parent commands

      --cache-file string    location of cache file (default "~/.akka/cache.yaml")
      --config string        location of config file (default "~/.akka/config.yaml")
      --context string       configuration context to use
      --disable-prompt       Disable all interactive prompts when running akka commands. If input is required, defaults will be used, or an error will be raised.
                             This is equivalent to setting the environment variable AKKA_DISABLE_PROMPTS to true.
      --force                Skip the confirmation prompt
      --jq string            Filter the JSON output with a jq expression
      --json string          Output JSON with only these fields; --json= lists the fields a command offers
      --no-color             Print without color, as NO_COLOR does
  -o, --output string        set output format to one of [text,json,json-compact,go-template=] (default "text")
      --page-mode string     the mode for paging, either paged, buffered or auto. (default "auto")
  -q, --quiet                set quiet output (helpful when used as part of a script)
      --region string        Region to use if the project has more than one region
      --timeout duration     client command timeout (default 10s)
      --trainer-url string   Trainer service URL (default http://localhost:9001, or TRAINER_URL)
      --url string           Optimize service URL (default http://localhost:9010, or OPTIMIZE_URL)
      --use-grpc-web         use grpc-web when talking to Akka APIs. This is useful when behind corporate firewalls that decrypt traffic but don't support HTTP/2.
      --verbose              set verbose output

SEE ALSO