Manage external secrets

Akka allows integrating with various external secret managers. External secrets are provided to your services through filesystem mounts, however they will never be written to disk on your service.

Authentication with external secret managers is done using workload identity. When a service starts, the identity of that service is used to authentication with and be authorized by the secret manager, using OIDC.

Managing external secrets in a project

Listing external secrets

To list the external secrets in your Akka project, you can use the Akka CLI.

CLI

Use the akka secret external list command:

akka secret external list

Removing external secrets

To remove an external secret from your Akka project, you can use the Akka CLI.

CLI

akka secret external delete command:

akka secret external delete <secret-name>

Azure KeyVault

Akka services running on Azure can access external secrets from Azure KeyVault.

Setting up

Before you setting up Azure KeyVault, you will need the following information:

The name of the Azure KeyVault that you wish to access, which we will refer to in the scripts below using the environment variable KEYVAULT_NAME.
The ID of the Akka project that you wish to access to the secrets, which we will refer to in the scripts below using the environment variable AKKA_PROJECT_ID. This is a UUID, and can be obtained using the akka project get command.
The name of the service that you wish to access the secrets, which we will refer to in the scripts below using the environment variable AKKA_SERVICE_NAME.

The following script can set them:

export KEYVAULT_NAME=my-keyvault-name
export AKKA_PROJECT_ID=bc16cf0c-909f-402d-bbb0-88ea1d582854
export AKKA_SERVICE_NAME=my-service

Now, you will need to determine the OIDC issuer for your region. This can be determined by running:

akka secrets external info

Copy the issuer and place it in an environment variable called AKKA_OIDC_ISSUER, or if you only have a single region, you can do so using the following command:

export AKKA_OIDC_ISSUER=`akka secrets external info -o go-template='{{(index .Items 0).WorkloadIdentity.Azure.OidcIssuer}}'`

Now you need to create an application to access the secrets on behalf of your service. We’ll place the name of this application in an environment variable called APPLICATION_NAME, and then obtain the client ID for the application and place that in an environment variable called APPLICATION_CLIENT_ID:

export APPLICATION_NAME="my-akka-service-application"
az ad sp create-for-rbac --name "${APPLICATION_NAME}"
export APPLICATION_CLIENT_ID=$(az ad sp list --display-name ${APPLICATION_NAME} --query '[0].appId' -otsv)

Now we need to grant this application access to keys, secrets and certs in the KeyVault:

az keyvault set-policy -n $KEYVAULT_NAME --key-permissions get --spn ${APPLICATION_CLIENT_ID}
az keyvault set-policy -n $KEYVAULT_NAME --secret-permissions get --spn ${APPLICATION_CLIENT_ID}
az keyvault set-policy -n $KEYVAULT_NAME --certificate-permissions get --spn ${APPLICATION_CLIENT_ID}

Now to federate the credentials, we need the application object id of the application:

export APPLICATION_OBJECT_ID="$(az ad app show --id ${APPLICATION_CLIENT_ID} --query id -otsv)"

Now we’ll create a JSON parameters file for federating the credentials:

cat <<EOF > params.json
{
  "name": "akka-service-federated-credential",
  "issuer": "${AKKA_OIDC_ISSUER}",
  "subject": "system:serviceaccount:${AKKA_PROJECT_ID}:klx-${AKKA_SERVICE_NAME}",
  "description": "Akka service federated credential",
  "audiences": [
    "api://AzureADTokenExchange"
  ]
}
EOF

And finally federate the credentials:

az ad app federated-credential create --id "${APPLICATION_OBJECT_ID}" --parameters @params.json

Managing Azure KeyVault secrets using the project descriptor

The best way to manage Azure KeyVault secrets is using the project descriptor. Please refer to Project Descriptor reference for details.

Adding Azure KeyVault secrets

To add secrets to your Akka project, you can use the Akka CLI. You will need the following information:

The name of the KeyVault
The Tenant ID for the KeyVault
The Application Client ID of the application created above.

CLI

Use the akka secret external azure create command.

akka secret external azure create my-external-secret \ (1)
  --key-vault-name $KEYVAULT_NAME \
  --tenant-id $TENANT_ID \
  --client-id $APPLICATION_CLIENT_ID \
  --object-name some-secret \ (2)
  --object-type secret (3)

1	External secret name
2	The name of the object (secret) in the key store
3	The type of the secret, either secret, key or cert

Adding multiple objects can be done by updating the secret after initial creation.

Updating Azure KeyVault secrets

CLI

Use the akka secret external azure update command.

akka secret external azure update my-external-secret \
  --object-name some-other-secret \
  --object-type secret

When updating, if the passed in object name exists, the object will be updated, otherwise a new object will be added to the secret.

GCP Secret Manager

Akka services running on GCP can access external secrets from GCP Secret Manager. Authentication uses Workload Identity Federation. Akka presents an identity token that GCP trusts via a pre-configured identity pool, so no service account keys are needed.

Setting up

Before setting up GCP Secret Manager, you will need:

A GCP project with billing enabled, which we will refer to below using the environment variable GCP_PROJECT_ID.
The Google Cloud CLI (gcloud) installed and authenticated.
The Secret Manager API enabled in your GCP project.

The following script can set up your environment:

export GCP_PROJECT_ID=my-gcp-project
gcloud auth login
gcloud config set project $GCP_PROJECT_ID

Enable the Secret Manager API if you haven’t already:

gcloud services enable secretmanager.googleapis.com

Create a secret in GCP Secret Manager

gcloud secrets create my-secret --replication-policy="automatic"
echo -n "my-secret-value" | gcloud secrets versions add my-secret --data-file=-

You can verify the secret was stored correctly:

gcloud secrets versions access latest --secret="my-secret"

Grant Akka access to your GCP secrets

First, retrieve the workload identity information for your Akka project:

akka secrets external info

This outputs a workload identity pool path, a principal (for a specific service), and a principalSet (for all services in the project).

To grant access to all services in your Akka project, use the principalSet value:

gcloud secrets add-iam-policy-binding my-secret \
    --project=$GCP_PROJECT_ID \
    --role="roles/secretmanager.secretAccessor" \
    --member="principalSet://iam.googleapis.com/projects/GCP_PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/namespace/NAMESPACE_ID" (1)

1	Replace the `--member` value with the `principalSet` from the `akka secrets external info` output

To grant access to a specific Akka service only, use the principal value instead:

gcloud secrets add-iam-policy-binding my-secret \
    --project=$GCP_PROJECT_ID \
    --role="roles/secretmanager.secretAccessor" \
    --member="principal://iam.googleapis.com/projects/GCP_PROJECT_NUMBER/locations/global/workloadIdentityPools/POOL_ID/subject/ns/NAMESPACE_ID/sa/kalix-SERVICE_NAME" (1)

1	Copy the exact `principal` value from the `akka secrets external info` output rather than constructing it manually. Note the `kalix-` prefix before the Akka service name. Akka uses this prefix internally when registering workload identities

Repeat the IAM binding for each secret that your service needs to access.

Managing GCP secrets using the project descriptor

The best way to manage GCP Secret Manager secrets is using the project descriptor. Please refer to Project Descriptor reference for details.

Adding GCP Secret Manager secrets

To add a GCP external secret to your Akka project, you can use the Akka CLI. You will need:

The GCP project ID
The name of the secret in GCP Secret Manager

CLI

Use the akka secrets external create gcp command.

akka secrets external create gcp my-external-secret \ (1)
  --project-id $GCP_PROJECT_ID \ (2)
  --object-name my-secret \ (3)
  --object-path my-secret (4)

1	External secret name in Akka
2	The GCP project ID containing the secret
3	The name of the secret in GCP Secret Manager
4	The path for the mounted file

Adding multiple objects can be done by updating the secret after initial creation.

Updating GCP Secret Manager secrets

CLI

Use the akka secrets external update gcp command.

akka secrets external update gcp my-external-secret \
  --object-name another-secret \
  --object-path another-secret

When updating, if the passed in object name exists, the object will be updated, otherwise a new object will be added to the secret.

Mount secrets to the filesystem of your service

External secrets are provided to your service through filesystem mounts. Unlike regular Akka secrets, external secrets cannot be injected as environment variables.

External secrets are never stored in Kubernetes Secrets or etcd, and are never read by the Kubernetes API server or the kubelet. Instead, a process on the node running on behalf of the pod projects the secret value directly into the pod’s filesystem. This is why external secrets can only be mounted as files. Environment variable injection would require the kubelet to read the secret value, which would defeat the purpose. Only the service itself ever accesses the secret.

To mount an external secret, declare a volumeMount in your service descriptor:

resource: Service
resourceVersion: v1
metadata:
  name: my-service
spec:
  image: my-container-registry/my-image:latest
  volumeMounts:
  - mountPath: /secrets/my-secret (1)
    externalSecret:
      provider: my-external-secret (2)

1	The path where the secret will be available inside the container
2	The name of the external secret created with `akka secret external create`

The mount path is a directory. The file within it is named after the object’s path (for GCP) or name/alias (for Azure), as configured when creating the external secret. For example, if you created an external secret with --object-path my-secret and mounted it at /secrets/my-secret, the secret value is readable at /secrets/my-secret/my-secret:

String secretValue = Files.readString(Path.of("/secrets/my-secret/my-secret")).trim();

Deploy the service descriptor using akka project apply:

akka project apply --file project.yaml

While akka service deploy is a convenience command for getting started, production deployments should use service descriptors checked into version control and deployed with akka project apply. Descriptors give you full control over volume mounts, environment variables, and other service configuration.