Akka Distributed Cluster

Version 1.5.4

Part 4: Deploying with Kubernetes

In this part we will deploy Service to Service eventing and Active-active parts of the guide to Kubernetes in two separate regions.

Setup Kubernetes cluster

Use instructions from your preferred cloud provider of how to start two Kubernetes clusters in two separate regions.

As an example, we use Amazon EKS in regions us-east-2 and eu-central-1:

eksctl create cluster \
  --name eks-akka-us-east-2 \
  --version 1.24 \
  --region us-east-2 \
  --nodegroup-name linux-nodes \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --with-oidc \
  --managed

eksctl create cluster \
  --name eks-akka-eu-central-1 \
  --version 1.24 \
  --region eu-central-1 \
  --nodegroup-name linux-nodes \
  --node-type m5.xlarge \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --with-oidc \
  --managed

Setup database

Use instructions from your preferred cloud provider of how to run a PostgreSQL database in each of the two regions.

As an example, we use Amazon RDS Postgres. For a trial PostgreSQL you can select the following aside from defaults:

  • Standard create
  • PostgreSQL
  • Free tier (some regions don’t offer free tier)
  • DB instance identifier: shopping-db-us-east-2
  • Master password: <a password>
  • Turn off storage autoscaling
  • VPC: Use the same as your EKS cluster is running in
  • Create new VPC security group: shopping-db-sg
  • Turn off Automatic Backups in the Additional Configuration section.

To allow the nodes in the EKS cluster to connect to the database you have to add a rule in the security group.

Go to the VPC console. Select “Security Groups”.

There are 3 security groups for the EKS cluster and you should select the one with description “EKS created security group …”. The one that has a name that doesn’t contain ControlPlaneSecurityGroup and doesn’t contain ClusterSharedNodeSecurityGroup . Make a note of this security group id for the EKS cluster.

Go back to the Amazon RDS console. Select the database that you created. Click on the “VPC security groups” in the tab “Connectivity & security”.

Edit inbound rules > add rule > Custom TCP > Port 5432 > Source custom. Add the security group for the EKS cluster. Save rules.

More details in A DB instance in a VPC accessed by an EC2 instance in the same VPC

Namespace

Define a Kubernetes namespace with a namespace.json file:

JSON
source{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "shopping-cart-namespace",
    "labels": {
      "name": "shopping-cart-namespace"
    }
  }
}

Create the namespace with:

kubectl apply -f kubernetes/namespace.json

For convenience, you can use this namespace by default:

kubectl config set-context --current --namespace=shopping-cart-namespace

Role based access control

For Akka Cluster bootstrap we need to define access control with a rbac.yml file:

YAML
sourcekind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
  namespace: shopping-cart-namespace
rules:
  - apiGroups: [""] # "" indicates the core API group
    resources: ["pods"]
    verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
  namespace: shopping-cart-namespace
subjects:
  # Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
  # For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
  - kind: User
    name: system:serviceaccount:shopping-cart-namespace:default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Apply the access control with:

kubectl apply -f kubernetes/rbac.yml

Database secret

Create a Kubernetes secret with the connection details and credentials to the database:

kubectl create secret generic \
    db-secret \
    --from-literal=DB_HOST=shopping-db-us-east-2.cgrtpi2lqrw8.us-east-2.rds.amazonaws.com \
    --from-literal=DB_USER=postgres \
    --from-literal=DB_PASSWORD=<the password>

Database schema

You can run psql from the Kubernetes cluster with 

kubectl run -i --tty db-mgmt --image=postgres --restart=Never --rm \
  --env=PGPASSWORD=<the password> -- \
  psql -h shopping-db-us-east-2.cgrtpi2lqrw8.us-east-2.rds.amazonaws.com -U postgres

Paste the DDL statements from the ddl-scripts/create-tables.sql to the psql prompt:

SQL
sourceCREATE TABLE IF NOT EXISTS event_journal(
  slice INT NOT NULL,
  entity_type VARCHAR(255) NOT NULL,
  persistence_id VARCHAR(255) NOT NULL,
  seq_nr BIGINT NOT NULL,
  db_timestamp timestamp with time zone NOT NULL,

  event_ser_id INTEGER NOT NULL,
  event_ser_manifest VARCHAR(255) NOT NULL,
  event_payload BYTEA NOT NULL,

  deleted BOOLEAN DEFAULT FALSE NOT NULL,
  writer VARCHAR(255) NOT NULL,
  adapter_manifest VARCHAR(255),
  tags TEXT ARRAY,

  meta_ser_id INTEGER,
  meta_ser_manifest VARCHAR(255),
  meta_payload BYTEA,

  PRIMARY KEY(persistence_id, seq_nr)
);

CREATE INDEX IF NOT EXISTS event_journal_slice_idx ON event_journal(slice, entity_type, db_timestamp, seq_nr);

CREATE TABLE IF NOT EXISTS snapshot(
  slice INT NOT NULL,
  entity_type VARCHAR(255) NOT NULL,
  persistence_id VARCHAR(255) NOT NULL,
  seq_nr BIGINT NOT NULL,
  db_timestamp timestamp with time zone,
  write_timestamp BIGINT NOT NULL,
  ser_id INTEGER NOT NULL,
  ser_manifest VARCHAR(255) NOT NULL,
  snapshot BYTEA NOT NULL,
  tags TEXT ARRAY,
  meta_ser_id INTEGER,
  meta_ser_manifest VARCHAR(255),
  meta_payload BYTEA,

  PRIMARY KEY(persistence_id)
);

CREATE TABLE IF NOT EXISTS durable_state (
  slice INT NOT NULL,
  entity_type VARCHAR(255) NOT NULL,
  persistence_id VARCHAR(255) NOT NULL,
  revision BIGINT NOT NULL,
  db_timestamp timestamp with time zone NOT NULL,

  state_ser_id INTEGER NOT NULL,
  state_ser_manifest VARCHAR(255),
  state_payload BYTEA NOT NULL,
  tags TEXT ARRAY,

  PRIMARY KEY(persistence_id)
);

-- `durable_state_slice_idx` is only needed if the slice based queries are used
CREATE INDEX IF NOT EXISTS durable_state_slice_idx ON durable_state(slice, entity_type, db_timestamp);

-- Primitive offset types are stored in this table.
-- If only timestamp based offsets are used this table is optional.
-- Configure akka.projection.r2dbc.offset-store.offset-table="" if the table is not created.
CREATE TABLE IF NOT EXISTS akka_projection_offset_store (
  projection_name VARCHAR(255) NOT NULL,
  projection_key VARCHAR(255) NOT NULL,
  current_offset VARCHAR(255) NOT NULL,
  manifest VARCHAR(32) NOT NULL,
  mergeable BOOLEAN NOT NULL,
  last_updated BIGINT NOT NULL,
  PRIMARY KEY(projection_name, projection_key)
);

-- Timestamp based offsets are stored in this table.
CREATE TABLE IF NOT EXISTS akka_projection_timestamp_offset_store (
  projection_name VARCHAR(255) NOT NULL,
  projection_key VARCHAR(255) NOT NULL,
  slice INT NOT NULL,
  persistence_id VARCHAR(255) NOT NULL,
  seq_nr BIGINT NOT NULL,
  -- timestamp_offset is the db_timestamp of the original event
  timestamp_offset timestamp with time zone NOT NULL,
  -- timestamp_consumed is when the offset was stored
  -- the consumer lag is timestamp_consumed - timestamp_offset
  timestamp_consumed timestamp with time zone NOT NULL,
  PRIMARY KEY(slice, projection_name, timestamp_offset, persistence_id, seq_nr)
);

CREATE TABLE IF NOT EXISTS akka_projection_management (
  projection_name VARCHAR(255) NOT NULL,
  projection_key VARCHAR(255) NOT NULL,
  paused BOOLEAN NOT NULL,
  last_updated BIGINT NOT NULL,
  PRIMARY KEY(projection_name, projection_key)
);

Repeat for the other region

If you haven’t already, repeat the steps for namespace, rbac, database secret, and database schema in the other region.

Deploy shopping-cart-service

We are going to deploy the shopping-cart-service in region us-east-2.

The complete sample can be downloaded:

Build the image in shopping-cart-service directory:

Scala
sbt -Ddocker.username=<username> -Ddocker.registry=docker.io docker:publish
Java
mvn -DskipTests -Ddocker.registry=<username> clean package docker:push

The Kubernetes Deployment in deployment.yml file:

YAML
sourceapiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: shopping-cart-service
  name: shopping-cart-service
  namespace: shopping-cart-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: shopping-cart-service
  template:
    metadata:
      labels:
        app: shopping-cart-service
        actorSystemName: shopping-cart-service
    spec:
      containers:
        - name: shopping-cart-service
          # use specific image version from docker publish
          image: shopping-cart-service:latest
          # these will need to be increased tuned for production environments!
          resources:
            limits:
              memory: "2Gi"
            requests:
              memory: "2Gi"
              cpu: "1000m"
          readinessProbe:
            httpGet:
              path: /ready
              port: management
            periodSeconds: 10
            failureThreshold: 3
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: "/alive"
              port: management
            periodSeconds: 10
            failureThreshold: 5
            initialDelaySeconds: 20
          ports:
            # akka-management and bootstrap
            - name: management
              containerPort: 8558
              protocol: TCP
            - name: grpc
              containerPort: 8101
              protocol: TCP
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: REQUIRED_CONTACT_POINT_NR
              value: "1"
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75 -Dakka.persistence.r2dbc.connection-factory.ssl.enabled=true -Dakka.persistence.r2dbc.connection-factory.ssl.mode=require"
            - name: DB_HOST
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_HOST
                  optional: true
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_USER
                  optional: true
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_PASSWORD
                  optional: true

Update the image: in the deployment.yml with the specific image version and location you published.

kubectl apply -f kubernetes/deployment.yml

Port forward

Create a Kubernetes Service and port forward to simplify access to the pods from your local machine:

YAML
sourceapiVersion: v1
kind: Service
metadata:
  name: shopping-cart-service-svc
spec:
  selector:
    app: shopping-cart-service
  type: NodePort
  ports:
    - protocol: TCP
      port: 8101
      targetPort: 8101
kubectl apply -f kubernetes/service.yml

Start port forward with:

kubectl port-forward svc/shopping-cart-service-svc 8101:8101

Exercise the shopping-cart-service

Use grpcurl to exercise the service.

Add 7 socks to a cart:

grpcurl -d '{"cartId":"cart1", "itemId":"socks", "quantity":7}' -plaintext 127.0.0.1:8101 shoppingcart.ShoppingCartService.AddItem

Add 3 t-shirts to the same cart:

grpcurl -d '{"cartId":"cart1", "itemId":"t-shirt", "quantity":3}' -plaintext 127.0.0.1:8101 shoppingcart.ShoppingCartService.AddItem

Load balancer

To access the shopping-cart-service from shopping-analytics-service running in another region we need an Internet facing load balancer.

There are many alternatives for secure access with a load balancer. An incomplete list of options:

Mutual authentication with TLS (mTLS) can be very useful where only other known services are allowed to interact with a service, and public access should be denied. See Akka gRPC documentation.

We are going to use a network load balancer for simplicity of this example, and we are not using TLS. Real applications would of course require TLS.

Follow the instructions of how to install the AWS Load Balancer Controller add-on.

Create a Kubernetes Service that is configured for the load balancer:

YAML
source# Network load balancer https://docs.aws.amazon.com/eks/latest/userguide/network-load-balancing.html
apiVersion: v1
kind: Service
metadata:
  name: shopping-cart-service-nlb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
spec:
  selector:
    app: shopping-cart-service
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8101

The external DNS name is shown by:

kubectl get services

Deploy shopping-analytics-service

We are going to deploy the shopping-analytics-service in region eu-central-1.

The complete sample can be downloaded:

Build the image in shopping-analytics-service directory:

Scala
sbt -Ddocker.username=<username> -Ddocker.registry=docker.io docker:publish
Java
mvn -DskipTests -Ddocker.registry=<username> clean package docker:push

The Kubernetes Deployment in deployment.yml file:

YAML
sourceapiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: shopping-analytics-service
  name: shopping-analytics-service
  namespace: shopping-cart-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: shopping-analytics-service
  template:
    metadata:
      labels:
        app: shopping-analytics-service
        actorSystemName: shopping-analytics-service
    spec:
      containers:
        - name: shopping-analytics-service
          # use specific image version from docker publish
          image: shopping-analytics-service:latest
          # these will need to be increased tuned for production environments!
          resources:
            limits:
              memory: "2Gi"
            requests:
              memory: "2Gi"
              cpu: "1000m"
          readinessProbe:
            httpGet:
              path: /ready
              port: management
            periodSeconds: 10
            failureThreshold: 3
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: "/alive"
              port: management
            periodSeconds: 10
            failureThreshold: 5
            initialDelaySeconds: 20
          ports:
            # akka-management and bootstrap
            - name: management
              containerPort: 8558
              protocol: TCP
            - name: grpc
              containerPort: 8101
              protocol: TCP
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: REQUIRED_CONTACT_POINT_NR
              value: "1"
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75 -Dakka.persistence.r2dbc.connection-factory.ssl.enabled=true -Dakka.persistence.r2dbc.connection-factory.ssl.mode=require"
            - name: SHOPPING_CART_SERVICE_GRPC_HOST
              value: "shopping-cart-service-svc"
            - name: SHOPPING_CART_SERVICE_GRPC_PORT
              value: "8101"
            - name: DB_HOST
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_HOST
                  optional: true
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_USER
                  optional: true
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_PASSWORD
                  optional: true

Update the image in the deployment.yml with the specific image version and location you published.

Update the SHOPPING_CART_SERVICE_GRPC_HOST and SHOPPING_CART_SERVICE_GRPC_PORT in deployment.yml to correspond to the load balancer that you created, for example:

            - name: SHOPPING_CART_SERVICE_GRPC_HOST
              value: "k8s-shopping-shopping-b4157add0d-54780992c148fa88.elb.us-east-2.amazonaws.com"
            - name: SHOPPING_CART_SERVICE_GRPC_PORT
              value: "80"

Run the service:

kubectl apply -f kubernetes/debployment.yml

Exercise the shopping-analytics-service

You can watch the logs of the shopping-analytics-service with

kubectl get pods
kubectl logs -f <pod name>

Add some more items to the cart or create more carts as described for the shopping-cart-service. In the logs of the shopping-analytics-service you should see entries such as:

Projection [cart-events-cart-768-1023] consumed ItemQuantityAdjusted for cart cart2, changed 3 socks. Total [8] events.

Deploy replicated shopping-cart-service

This step is for deploying replicated-shopping-cart-service:

The complete sample can be downloaded:

Now there will be connections in both directions, so install the load balancer in the other region too.

Create a Kubernetes Secret with the gRPC connection details for the load balancers:

Example for us-east-2:

kubectl create secret generic \
    replication-secret \
    --from-literal=SELF_REPLICA_ID=replica1 \
    --from-literal=REPLICA2_GRPC_HOST=k8s-shopping-shopping-19708e1324-24617530ddc6d2cb.elb.eu-central-1.amazonaws.com \
    --from-literal=REPLICA2_GRPC_PORT=80

Example for eu-central-1:

kubectl create secret generic \
replication-secret \
--from-literal=SELF_REPLICA_ID=replica2 \
--from-literal=REPLICA1_GRPC_HOST=k8s-shopping-shopping-b4157add0d-54780992c148fa88.elb.us-east-2.amazonaws.com \
--from-literal=REPLICA1_GRPC_PORT=80

This secret is referenced from configuration environment variables from the deployment.yml:

YAML
sourceapiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: shopping-cart-service
  name: shopping-cart-service
  namespace: shopping-cart-namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: shopping-cart-service
  template:
    metadata:
      labels:
        app: shopping-cart-service
        actorSystemName: shopping-cart-service
    spec:
      containers:
        - name: shopping-cart-service
          # use specific image version from docker publish
          image: shopping-cart-service:latest
          # these will need to be increased tuned for production environments!
          resources:
            limits:
              memory: "2Gi"
            requests:
              memory: "2Gi"
              cpu: "1000m"
          readinessProbe:
            httpGet:
              path: /ready
              port: management
            periodSeconds: 10
            failureThreshold: 3
            initialDelaySeconds: 10
          livenessProbe:
            httpGet:
              path: "/alive"
              port: management
            periodSeconds: 10
            failureThreshold: 5
            initialDelaySeconds: 20
          ports:
            # akka-management and bootstrap
            - name: management
              containerPort: 8558
              protocol: TCP
            - name: grpc
              containerPort: 8101
              protocol: TCP
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: REQUIRED_CONTACT_POINT_NR
              value: "1"
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75 -Dakka.persistence.r2dbc.connection-factory.ssl.enabled=true -Dakka.persistence.r2dbc.connection-factory.ssl.mode=require"
            - name: SELF_REPLICA_ID
              valueFrom:
                secretKeyRef:
                  name: replication-secret
                  key: SELF_REPLICA_ID
            - name: REPLICA1_GRPC_HOST
              valueFrom:
                secretKeyRef:
                  name: replication-secret
                  key: REPLICA1_GRPC_HOST
                  optional: true
            - name: REPLICA1_GRPC_PORT
              valueFrom:
                secretKeyRef:
                  name: replication-secret
                  key: REPLICA1_GRPC_PORT
                  optional: true
            - name: REPLICA2_GRPC_HOST
              valueFrom:
                secretKeyRef:
                  name: replication-secret
                  key: REPLICA2_GRPC_HOST
                  optional: true
            - name: REPLICA2_GRPC_PORT
              valueFrom:
                secretKeyRef:
                  name: replication-secret
                  key: REPLICA2_GRPC_PORT
                  optional: true
            - name: DB_HOST
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_HOST
                  optional: true
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_USER
                  optional: true
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: DB_PASSWORD
                  optional: true

Build the image in replicated-shopping-cart-service directory and deploy in the same way as described previously.

Scala
sbt -Ddocker.username=<username> -Ddocker.registry=docker.io docker:publish
Java
mvn -DskipTests -Ddocker.registry=<username> clean package docker:push

Update the image: in the deployment.yml with the specific image version and location you published.

kubectl apply -f kubernetes/debployment.yml

Port forwards

Create a Kubernetes Service and port forward to simplify access to the pods from your local machine:

YAML
sourceapiVersion: v1
kind: Service
metadata:
  name: shopping-cart-service-svc
spec:
  selector:
    app: shopping-cart-service
  type: NodePort
  ports:
    - protocol: TCP
      port: 8101
      targetPort: 8101
kubectl apply -f kubernetes/service.yml

Start port forward with:

kubectl port-forward svc/shopping-cart-service-svc 8101:8101

Switch kubectl context to the other region and apply the service.yml and start the port forward on another port:

kubectl port-forward svc/shopping-cart-service-svc 8201:8101

Exercise the replicated shopping-cart-service

Use grpcurl to exercise the service.

Add 7 socks to a cart:

grpcurl -d '{"cartId":"cart2", "itemId":"socks", "quantity":7}' -plaintext 127.0.0.1:8101 shoppingcart.ShoppingCartService.AddItem

Add 3 t-shirts to the same cart but in the other region (port 8201):

grpcurl -d '{"cartId":"cart1", "itemId":"t-shirt", "quantity":3}' -plaintext 127.0.0.1:8201 shoppingcart.ShoppingCartService.AddItem

Retrieve the cart:

grpcurl -d '{"cartId":"cart2"}' -plaintext 127.0.0.1:8101 shoppingcart.ShoppingCartService.GetCart
grpcurl -d '{"cartId":"cart2"}' -plaintext 127.0.0.1:8201 shoppingcart.ShoppingCartService.GetCart
Part 3: Active-active
Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.