Data migration setup on AWS

This page describes how to prepare an AWS account for data migration. You create an S3 bucket, grant Akka access to it, and then run exports and imports against it.

You can give Akka access to the bucket in two ways:

Workload identity (recommended). Akka assumes an IAM role using a short-lived token. You do not create or store any long-lived credentials.
Access key. You create an IAM user and store its access key in an Akka secret.

Create an S3 bucket

Create the S3 bucket in the same region as the Akka region that your service runs in. If you already have a bucket that you want to use, you can skip this step. The examples on this page call the bucket my-akka-bucket.

Instructions for creating a bucket in AWS are in the AWS documentation.

IAM policy

Both authentication options need an IAM policy that grants access to the bucket. For imports, the policy must grant s3:ListBucket on the bucket and s3:GetObject on the objects. For exports, the policy must also grant s3:PutObject and s3:DeleteObject on the objects. Here is an example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::my-akka-bucket",
                "arn:aws:s3:::my-akka-bucket/*"
            ]
        }
    ]
}

Instructions for creating IAM policies in AWS are in the AWS documentation.

Workload identity

With workload identity, the data management tool presents an OIDC token issued by Akka to AWS. AWS exchanges the token for temporary credentials for an IAM role. For background, see Workload Identity.

The token identifies the data management tool with this subject:

system:serviceaccount:<project-id>:data-management

The <project-id> is the UUID of your Akka project. You can get it with akka project get. The subject is the same for every service in the project. It is not the subject of your service, which has the klx- prefix.

Set the environment variables

export AWS_ACCOUNT_ID=123456789012
export AKKA_PROJECT_ID=bc16cf0c-909f-402d-bbb0-88ea1d582854
export AKKA_OIDC_ISSUER=$(akka projects regions workload-identity-info -o go-template='{{(index .Items 0).WorkloadIdentity.Aws.OidcIssuer}}')
export AKKA_OIDC_PROVIDER=$(echo $AKKA_OIDC_ISSUER | sed -e "s/^https:\/\///")

If your project has more than one region, copy the issuer for the correct region from the output of akka projects regions workload-identity-info.

Register the Akka OIDC provider

Register the issuer as an OIDC identity provider in your AWS account. You need to do this once for each AWS account and Akka region. If you already registered the provider to let a service use workload identity, you can skip this step.

aws iam create-open-id-connect-provider --url $AKKA_OIDC_ISSUER \
  --thumbprint-list 06b25927c42a721631c1efd9431e648fa62e1e39 \
  --client-id-list sts.amazonaws.com

Create an IAM role

Create a role that the data management tool can assume. The trust policy restricts the role to the data-management identity of your project.

TRUST_POLICY_JSON=$(cat <<EOF
{
  "Version":"2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${AKKA_OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${AKKA_OIDC_PROVIDER}:aud": "sts.amazonaws.com",
          "${AKKA_OIDC_PROVIDER}:sub": "system:serviceaccount:${AKKA_PROJECT_ID}:data-management"
        }
      }
    }
  ]
}
EOF
)

aws iam create-role --role-name akka-data-management \
  --assume-role-policy-document "$TRUST_POLICY_JSON" \
  --description "Role for Akka data management"

Grant access to the bucket

Create an IAM policy using the example above, and attach it to the role:

POLICY_ARN=$(aws --query Policy.Arn --output text iam create-policy \
  --policy-name akka-data-management-policy \
  --policy-document file://policy.json)

aws iam attach-role-policy --role-name akka-data-management \
  --policy-arn "$POLICY_ARN"

Run an export or import

Pass --s3-workload-identity and the ARN of the role:

akka service data export assets \
  --s3-workload-identity \
  --s3-role-arn arn:aws:iam::123456789012:role/akka-data-management \
  --s3-bucket my-akka-bucket \
  --include-event-sourced-entities

If the trust policy uses an audience other than sts.amazonaws.com, set it with --s3-audience. If the bucket is in a different region from your Akka region, set the bucket region with --s3-region.

The akka service data import command takes the same flags.

Access key

With this option, you create an IAM user, create an access key for it, and store the key in an Akka secret.

Create a user

Create an IAM user for Akka to use. The examples call the user my-akka-user. Instructions are in the AWS documentation.

Attach the IAM policy

Create an IAM policy using the example above, and attach it to the user. Instructions are in the AWS documentation.

Create an access key

Create an access key for the user. Instructions are in the AWS documentation. Copy the access key ID and the secret access key for the next step.

Create an Akka secret

Store the access key in an Akka secret. The examples call the secret my-credentials:

akka secrets create generic my-credentials \
  --literal access-key-id=C2VJCMV0IHNLY3JLDCBZZWNYZXQK \
  --literal secret-access-key=dGhpcyBpcyB2ZXJ5IHNlY3JldCBpcyBpdCBub3QK

The keys access-key-id and secret-access-key are the defaults that the data management tool expects. You can use other names, and then set them with the --s3-access-key and --s3-secret-key flags.

Run an export or import

akka service data export assets \
  --s3-secret my-credentials \
  --s3-bucket my-akka-bucket \
  --include-event-sourced-entities

The akka service data import command takes the same flags.