Export and import data

Akka allows exporting and importing data to and from your services.

Migrating data between environments
Bootstrapping development environments with data
Migrating data from an existing system into Akka
Migrating data out of Akka
Disaster recovery - note, Akka already backs up all data for disaster recovery purposes

Akka exports/imports data to/from local cloud object stores. If your service is deployed to an AWS region, then you can export/import data to/from S3, if GCP, then to/from GCS, and if Azure, then to/from Azure Blob Storage. Event sourced entities are supported, however at present views are not supported.

The export format can either be protobuf or line separated JSON.

Prepare

Before you can perform an export or import, you need an object store that Akka can access, and you need to give Akka permission to use it. Set up the object store on the cloud provider that your Akka region runs on:

Authentication options

Akka supports these ways to authenticate to the object store:

Workload identity (recommended): Akka presents a short-lived OIDC token to the cloud provider, and the cloud provider exchanges it for temporary credentials. You do not create, store, or rotate any long-lived credentials. For background, see Workload Identity. Use this option unless you have a reason not to.
Credentials in an Akka secret: You create a user, service account, or application on the cloud provider, and store its credentials in an Akka secret. You have to rotate these credentials yourself.
Platform managed store: In a small number of regions, Akka can provision an object store for specific customers. This option is not generally available. If it does not apply to your organization, use one of the options above. See Platform managed store.

Workload identity subject

The data management tool authenticates with its own identity, which is separate from the identities of your services. The OIDC token that Akka issues has this sub claim:

system:serviceaccount:<project-id>:data-management

The <project-id> is the UUID of your Akka project, which you can get with akka project get. The name of the service account is data-management, without the klx- prefix that services use. The same identity is used for every export and import in the project. You grant this identity access to the object store in your cloud provider, using the steps on the setup page for your provider.

Data format

The format that Akka exports data to/from can either be JSON or protobuf. By default, the exports are gzipped.

Protobuf

The protobuf schema for the export can be found here. While an export can be parsed and formatted as a single protobuf message, for large exports, this may not be practical or possible. Instead, we recommend a streaming approach for reading and writing exports.

Writing exports as streams

For streaming exports, we can take advantage of the fact that writing a single protobuf message out to a stream with a repeating field many times will result in a single large message with the entries from each message written being concatenated into one large repeating field.

public void export(Iterable<EventSourcedEntityEvent> events) throws IOException {
    var fileStream = new FileOutputStream("my-export.binpb");
    var stream = CodedOutputStream.newInstance(fileStream);

    // First write the header
    Export.newBuilder()
            .setVersion("v1")
            .setExportHeader(ExportHeader.newBuilder()
                            .setId("my-export")
                            .setRecordType(ExportRecordType.EVENT_SOURCED_ENTITY_EVENT)
                    ).build()
            .writeTo(stream);

    for (var event : events) {
        // Now write each event as if it was a single export message with a single records field.
        // This has the effect of concatenating the event into the records field of the
        // previously written message.
        Export.newBuilder()
                .addRecords(event.toByteString())
                .build()
                .writeTo(stream);
    }

    // Flush and close.
    stream.flush();
    fileStream.close();
}

Reading exports as streams

Reading exports is a little more complex, it requires using the protobuf CodedInputStream class, but is nonetheless straight forward.

public void readExport(File file) throws IOException {
    var extensionRegistry = ExtensionRegistry.newInstance();
    var stream = CodedInputStream.newInstance(new FileInputStream(file));
    int tag = stream.readTag();
    while (tag != 0) {
        switch (WireFormat.getTagFieldNumber(tag)) {
            case 1:
                // Version field
                var version = stream.readString();
                if (!version.equals("v1")) {
                    throw new RuntimeException("Unknown export version: " + version);
                }
                break;

            case 2:
                // Header field
                var headerBuilder = ExportHeader.newBuilder();
                stream.readMessage(headerBuilder, extensionRegistry);
                var header = headerBuilder.build();
                if (header.getRecordType() != ExportRecordType.EVENT_SOURCED_ENTITY_EVENT) {
                    throw new RuntimeException("Expected event sourced entity events, but got: " + header.getRecordType().name());
                }
                break;

            case 3:
                // A record
                var builder = EventSourcedEntityEvent.newBuilder();
                stream.readMessage(builder, extensionRegistry);
                var event = builder.build();
                handleEvent(event);
                break;
        }
        tag = stream.readTag();
    }
}

JSON

JSON exports use the protobuf schema specified here, with the exception that the JSON is formatted as newline separated JSON.

The first line should be the Export object with no records fields set. Each subsequent line should be a record for the records field.

In addition to this, JSON payloads in the export are encoded specially. Rather than encoding the JSON bytes as base64 in the value field of the protobuf Any message, the JSON message is placed in a field called json, with any type associated with the message being encoded in a field called type. Otherwise, it gets encoded as a regular protobuf Any message, with the bytes base64 encoded in the value field, and the typeUrl field populated with the type of the protobuf message.

Here is an example JSON export:

{"version":"v1","exportHeader":{"recordType":"EVENT_SOURCED_ENTITY_EVENT"}}
{"entityType":"asset","entityId":"1","seqNr":1,"timestamp":"2024-01-12T00:19:14.006111Z","payload":{"type":"asset-moved","json":{"location":"workshop"}}}
{"entityType":"asset","entityId":"1","seqNr":2,"timestamp":"2024-01-12T00:20:29.782976Z","payload":{"type":"asset-moved","json":{"location":"toolshed"}}}

Running exports

Pause the source service before running an export. If the source keeps producing events during the export, events written to a journal partition after the export cursor has already read past that partition are missing from the export, with no warning at export time. The exported data is then inconsistent between the journal and snapshots. If it is later imported into a service, this can lead to problems such as a workflow appearing paused in the console while endpoint calls reject state transitions because the snapshot says it is already completed.

The akka service data export command can be used to run an export. For example:

akka service data export assets --s3-secret my-credentials \
  --s3-bucket my-bucket --include-event-sourced-entities

The format of the export is controlled using the --format flag, which can either be protobuf or json. The default is protobuf. Additionally, the compression algorithm that is used to compress/decompress the export can be specified using the --compression flag, which can either be gzip or none, with the default being gzip.

The --include-event-sourced-entities flag is required. The export command fails without exporting anything if it is omitted.

By default, the export will be run synchronously, displaying the progress of the export as it runs. Cancelling the export command will not stop the export, it will continue to run in the background. The export command can be run asynchronously by passing the --async flag.

Exporting to AWS S3

To export to S3 using workload identity, pass --s3-workload-identity, the ARN of the IAM role to assume with --s3-role-arn, and the bucket name with --s3-bucket:

akka service data export assets --s3-workload-identity \
  --s3-role-arn arn:aws:iam::123456789012:role/akka-data-management \
  --s3-bucket my-akka-bucket --include-event-sourced-entities

To use credentials in an Akka secret instead, pass the secret with --s3-secret in place of the workload identity flags. If the bucket does not live in the same region as your service, supply the region name with --s3-region.

An optional prefix can be prepended to the names of the files that get exported, this can be specified using the --s3-object-prefix argument.

See Data migration setup on AWS for how to set up the role and the bucket.

Exporting to GCS

To export to GCS using workload identity, pass --gcs-workload-identity, the audience with --gcs-audience, and the bucket name with --gcs-bucket:

akka service data export assets --gcs-workload-identity \
  --gcs-audience //iam.googleapis.com/projects/791645197608/locations/global/workloadIdentityPools/kalix-dev.svc.id.goog \
  --gcs-bucket my-akka-bucket --include-event-sourced-entities

To impersonate a Google service account after the token exchange, also pass --gcs-impersonate-service-account. To use a service account key in an Akka secret instead, pass the secret with --gcs-secret in place of the workload identity flags.

An optional prefix can be prepended to the names of the files that get exported, this can be specified using the --gcs-object-prefix argument.

The audience is the workload identity pool printed by akka projects regions workload-identity-info, or the provider in your own pool. See Data migration setup on GCP for how to set up access and the bucket.

Exporting to Azure Blob Storage

To export to Azure Blob Storage using workload identity, pass --abs-workload-identity, the Microsoft Entra tenant ID with --abs-tenant-id, the application client ID with --abs-client-id, the container name with --abs-container, and the storage account endpoint with --abs-endpoint-url (for example, https://myakkastorage.blob.core.windows.net):

akka service data export assets --abs-workload-identity \
  --abs-tenant-id <tenant-id> --abs-client-id <client-id> \
  --abs-container my-akka-container \
  --abs-endpoint-url https://myakkastorage.blob.core.windows.net \
  --include-event-sourced-entities

To use a client secret in an Akka secret instead, pass the secret with --abs-secret in place of the workload identity flags. If the keys inside the secret use names other than the defaults (tenant-id, client-id, client-secret), supply them with the --abs-tenant-id-key, --abs-client-id-key, and --abs-client-secret-key arguments respectively.

An optional prefix can be prepended to the names of the files that get exported, this can be specified using the --abs-object-prefix argument.

See Data migration setup on Azure for how to set up the application and the container.

Running imports

The akka service data import command can be used to run an import. For example:

akka service data import assets --s3-secret my-credentials \
  --s3-bucket my-bucket --include-event-sourced-entities

When importing to a service, the first thing Akka will do is pause that service. If the service is running while the import is happening, the services data may be corrupted. If the import succeeds, Akka will resume the service.

If you wish to clear the services database before importing, use the --truncate flag. This will delete all existing data in the service.

Importing from S3, GCS or Azure Blob Storage

The names of the files to import can either be specified explicitly, by passing --event-sourced-entity-object and --event-sourced-entity-snapshot-object flags, or Akka can use the default filenames that it selects when creating an export. In the latter case, the --include-event-sourced-entities flag is required, so Akka knows to look for the default event sourced entity object. --s3-object-prefix, --gcs-object-prefix, and --abs-object-prefix can be used to prepend a prefix to those default filenames.

The flags for specifying credentials and object names specific to S3, GCS, and Azure Blob Storage are the same as for exporting. When you use workload identity, the role or application needs read access only. See the setup page for your cloud provider.

Managing tasks

When an import or export task is submitted, before anything is done, Akka will validate the credentials and other aspects of the task to ensure that it can be issued. Once that validation is complete, the task will be submitted to a task queue for processing. The status of the task can be queried by running akka service data list-tasks.

If a task fails, Akka will attempt to re-run it, using an exponential back-off. To cancel a task permanently, akka service data cancel can be used.

If you wish to watch a task that is running, you can use akka service data watch.

Platform managed store

In a small number of regions, Akka can provision an object store for specific customers. If this applies to your organization, you do not need to set up a bucket or credentials, and you can pass the --platform-managed flag to the export and import commands:

akka service data export assets --platform-managed --include-event-sourced-entities

Use --platform-managed-object-prefix to prepend a prefix to the object names. If you export with a prefix, supply the same prefix when you import:

akka service data import assets --platform-managed \
  --platform-managed-object-prefix my-export/ \
  --include-event-sourced-entities