Disaster recovery

Disaster recovery scenarios for Dedicated, BYOC (Bring Your Own Cloud), and BYOK8s (Bring Your Own Kubernetes) deployments. Each entry names the detection owner, notification flow, recovery procedure, and whether Akka can drill it live before launch. The production readiness checklist links here for its DR tabletop item. The Shared Responsibility Model covers backup, PITR (Point-in-Time Recovery), and multi-region failover ownership.

Most procedures are the same in every model. Where a phase differs, the procedure table has an Applies to column that names the models each row covers. The differences follow from who owns what:

Dedicated: Akka owns and operates the cloud account, the network, and the infrastructure inside it, so platform-side recovery is Akka’s, as in BYOC, and Akka also handles the network and DNS that the customer handles in BYOC.
BYOC: the customer owns the cloud account and the surrounding network; Akka operates the platform, the cluster, and the database inside it.
BYOK8s: the customer owns the Kubernetes cluster and the database, so several scenarios shift ownership toward the customer: database backups and PITR execution, cluster capacity, and private connectivity end to end.

The DR tabletop

Akka runs this catalog as a joint tabletop with the customer before launch, led by the customer’s FDE (Forward Deployed Engineer). The tabletop confirms paging entry points, incident channel, and decision authority for the customer’s account, and records which scenarios get a live drill before go-live.

In Dedicated, Akka owns and operates the environment end to end, so Akka periodically validates disaster recovery for the platform internally rather than as a joint customer-facing tabletop. Joint exercises with the customer can be arranged as agreed.

How to read this

Each scenario carries two labels: readiness (can Akka drill it today?) and recommendation (should the customer drill it before launch?).

Readiness	Meaning
Drillable today	Runbook and tooling are ready. Can drill against pre-production.
Procedure documented	Steps are known. A live drill needs more engineering.
Named in catalog	High-level notes only. Needs deeper analysis before drilling.

Readiness

Meaning

Drillable today

Runbook and tooling are ready. Can drill against pre-production.

Procedure documented

Steps are known. A live drill needs more engineering.

Named in catalog

High-level notes only. Needs deeper analysis before drilling.

Akka recommends	Meaning
Must drill	Required before launch.
Recommended	Drill when the capability is ready. Not required before launch.
Optional	Discuss based on use case and risk tolerance.
Defer	Documented only, not yet meaningful to drill live. Revisit as capabilities mature.

Akka recommends

Meaning

Must drill

Required before launch.

Recommended

Drill when the capability is ready. Not required before launch.

Optional

Discuss based on use case and risk tolerance.

Defer

Documented only, not yet meaningful to drill live. Revisit as capabilities mature.

Scenario selection is a joint decision at the tabletop.

Notification cells say Standard flow when they match the pattern in the Standard communication flow section below. Only exceptions are spelled out.

Scenario catalog

Scenario	Applies to	Readiness	Akka recommends
Container Registry Config restore	All	Drillable today	Recommended (customer-led)
Database corruption / data recovery	All	Drillable today	Must drill
Service deletion and restore	All	Drillable today	Must drill
Full region failure and rebuild	All	Procedure documented	Recommended
Single / multi-AZ failure	All	Procedure documented	Recommended
Private connectivity failure	Dedicated and BYOC	Procedure documented	Optional
Private connectivity failure	BYOK8s	Drillable today	Must drill (customer-led)
Load balancer / ingress failure	All	Procedure documented	Optional
Runtime / SDK version compatibility	All	Named in catalog	Defer
Additional chaos vectors	All	Named in catalog	Defer

Scenario

Applies to

Readiness

Akka recommends

Container Registry Config restore

All

Drillable today

Recommended (customer-led)

Database corruption / data recovery

All

Drillable today

Must drill

Service deletion and restore

All

Drillable today

Must drill

Full region failure and rebuild

All

Procedure documented

Recommended

Single / multi-AZ failure

All

Procedure documented

Recommended

Private connectivity failure

Dedicated and BYOC

Procedure documented

Optional

Private connectivity failure

BYOK8s

Drillable today

Must drill (customer-led)

Load balancer / ingress failure

All

Procedure documented

Optional

Runtime / SDK version compatibility

All

Named in catalog

Defer

Additional chaos vectors

All

Named in catalog

Defer

1. Container Registry Config restore

The ContainerRegistryConfig (CRC), the Kubernetes resource holding the customer’s container registry credentials, is missing or its tokens are invalid for a namespace. Pods cannot pull their images and services in that namespace stop coming up.

Customer-controlled. Akka does not intervene unless asked, since the CRC holds customer-owned credentials.

Recovery procedure

Phase	Owner	Action
Detection	Customer	Services do not come back up in the console. Pod events show image pull errors. Customer observability (minimum-instance alerts and similar) surfaces the failure.
Triage	Customer	The customer confirms the CRC is missing or its tokens are invalid in the affected namespace.
Resolution	Customer	The customer re-applies the CRC by re-issuing tokens, updating credentials, or restoring from their own configuration source. Pods retry image pulls and start.
Escalation (only if needed)	Customer to Akka	If the customer cannot restore the CRC themselves, for example a suspected Akka operator bug, the customer raises a severity 1 incident. Akka runs a targeted Velero restore of the CRC.

Phase

Owner

Action

Detection

Customer

Services do not come back up in the console. Pod events show image pull errors. Customer observability (minimum-instance alerts and similar) surfaces the failure.

Triage

Customer

The customer confirms the CRC is missing or its tokens are invalid in the affected namespace.

Resolution

Customer

The customer re-applies the CRC by re-issuing tokens, updating credentials, or restoring from their own configuration source. Pods retry image pulls and start.

Escalation (only if needed)

Customer to Akka

If the customer cannot restore the CRC themselves, for example a suspected Akka operator bug, the customer raises a severity 1 incident. Akka runs a targeted Velero restore of the CRC.

2. Database corruption / data recovery

Data in the customer’s region is corrupted, lost, or rolled to an invalid state. Recovery restores the regional database to a known-good point in time.

Feasibility

In Dedicated and BYOC, the drill depends on the cloud provider. Akka’s tooling and access path for whole-database PITR is ready for some providers and still being prepared for others. Confirm feasibility at the tabletop.

In BYOK8s, the customer owns automated database backups and PITR configuration. Akka does not run backups against the customer’s database. The drill depends on the customer having automated backups, retention, and PITR enabled and accessible. Confirm at the tabletop.

Scope of recovery

Recovery uses whole-database PITR: Akka offers it in Dedicated and BYOC, and in BYOK8s it runs against the customer’s database. The database is single per region and holds state for every project and service running there. A restore returns all services to the chosen timestamp at once.

Recovery procedure

Phase	Applies to	Owner	Action
Detection	All	Shared	The customer usually sees symptoms first (data inconsistencies, missing records, business-logic failures). Akka may also surface platform-side signals such as stalled projections (Akka runtime constructs that derive views from event streams).
Notification	All	Shared	Standard flow.
Triage	Dedicated and BYOC	Shared	Akka identifies the most recent uncorrupted PITR target and confirms backup integrity. The customer identifies which services and data are affected and which appear healthy. Together, they weigh how far back to restore (loses recent activity across every service in the region) against reconciliation cost afterward.
Triage	BYOK8s	Shared	The customer identifies the most recent uncorrupted PITR target from their backups, confirms backup integrity, and identifies which services and data are affected. Akka shares platform-side signals that help scope the impact. Together, they weigh how far back to restore (loses recent activity across every service in the region) against reconciliation cost afterward.
Decision	All	Customer	The customer authorizes the whole-database restore at the chosen timestamp, acknowledging that all services in the region revert.
Resolution	Dedicated and BYOC	Akka	Akka clones the database via PITR, makes the clone the active datastore for the region, updates every service in the region to use it, and verifies platform health.
Resolution	BYOK8s	Shared	The customer clones the database at the target timestamp via PITR and notifies Akka the clone is ready, with connection details. Akka pauses traffic to the affected services, makes the clone the active datastore, updates every service in the region to use it, and resumes traffic. Both jointly verify platform health end to end.
Customer-side recovery	All	Customer	The customer verifies data state in every affected service, reconciles activity from after the restore point, and resumes traffic.

Phase

Applies to

Owner

Action

Detection

All

Shared

The customer usually sees symptoms first (data inconsistencies, missing records, business-logic failures). Akka may also surface platform-side signals such as stalled projections (Akka runtime constructs that derive views from event streams).

Notification

All

Shared

Standard flow.

Triage

Dedicated and BYOC

Shared

Akka identifies the most recent uncorrupted PITR target and confirms backup integrity. The customer identifies which services and data are affected and which appear healthy. Together, they weigh how far back to restore (loses recent activity across every service in the region) against reconciliation cost afterward.

Triage

BYOK8s

Shared

The customer identifies the most recent uncorrupted PITR target from their backups, confirms backup integrity, and identifies which services and data are affected. Akka shares platform-side signals that help scope the impact. Together, they weigh how far back to restore (loses recent activity across every service in the region) against reconciliation cost afterward.

Decision

All

Customer

The customer authorizes the whole-database restore at the chosen timestamp, acknowledging that all services in the region revert.

Resolution

Dedicated and BYOC

Akka

Akka clones the database via PITR, makes the clone the active datastore for the region, updates every service in the region to use it, and verifies platform health.

Resolution

BYOK8s

Shared

The customer clones the database at the target timestamp via PITR and notifies Akka the clone is ready, with connection details. Akka pauses traffic to the affected services, makes the clone the active datastore, updates every service in the region to use it, and resumes traffic. Both jointly verify platform health end to end.

Customer-side recovery

All

Customer

The customer verifies data state in every affected service, reconciles activity from after the restore point, and resumes traffic.

3. Service deletion and restore

An Akka service is hard-deleted from the cluster, either by Akka at the customer’s request or by misconfiguration. The service resource is gone from the console.

Scope of recovery

Customer-initiated service deletion is disabled; deletions are coordinated with Akka. Akka restores the service’s Kubernetes resource via Velero, the backup-and-restore tool Akka uses for stateful Kubernetes resources. The data lives in the regional shared database and is reachable once the service is restored. Data corruption within a running service requires whole-database recovery; see 2. Database corruption / data recovery.

Recovery procedure

Phase	Owner	Action
Detection	Customer	The service is gone from the console, or endpoints become unreachable.
Notification	Customer	Customer raises a severity 1 incident with the service name, namespace, and approximate deletion time. Akka starts an incident bridge or joins the customer’s existing bridge.
Triage	Akka	Akka confirms the service’s Kubernetes resource is missing from cluster state, identifies the most recent Velero backup that contains it, and confirms backup integrity.
Decision	Shared	Akka confirms scope: the service resource is restored, and its data in the regional database is unaffected. The customer confirms restore intent.
Resolution	Akka	Akka runs a targeted Velero restore, verifies the service is recreated, and verifies it connects to its data in the regional database.
Customer-side recovery	Customer	The customer verifies the service in the console, validates data is reachable, and resumes traffic.

Phase

Owner

Action

Detection

Customer

The service is gone from the console, or endpoints become unreachable.

Notification

Customer

Customer raises a severity 1 incident with the service name, namespace, and approximate deletion time. Akka starts an incident bridge or joins the customer’s existing bridge.

Triage

Akka

Akka confirms the service’s Kubernetes resource is missing from cluster state, identifies the most recent Velero backup that contains it, and confirms backup integrity.

Decision

Shared

Akka confirms scope: the service resource is restored, and its data in the regional database is unaffected. The customer confirms restore intent.

Resolution

Akka

Akka runs a targeted Velero restore, verifies the service is recreated, and verifies it connects to its data in the regional database.

Customer-side recovery

Customer

The customer verifies the service in the console, validates data is reachable, and resumes traffic.

4. Full region failure and rebuild

The cloud region becomes unavailable and the application is unreachable. Single-region deployments have no automatic failover; recovery is a coordinated rebuild.

Single-region scope

Single-region only. For failover between regions, see Multi-region operations. Replication between regions is asynchronous, so a failover can lose the most recent writes that had not yet replicated to the surviving region. A rebuild from PITR loses the activity after the chosen restore timestamp.

Recovery procedure

Phase	Applies to	Owner	Action
Detection	All	Shared	The customer monitors the cloud provider status page and end-to-end probes; Akka monitors platform health signals. Akka recommends the customer subscribe to the cloud provider status feed.
Notification	All	Shared	Standard flow.
Triage	Dedicated and BYOC	Shared	Akka and the customer confirm scope (region-wide versus localized), coordinate with the cloud provider for an ETA, and assess the wait-versus-rebuild trade-off.
Triage	BYOK8s	Shared	Akka and the customer confirm scope (region-wide versus localized) and both track the cloud provider’s status feed for an ETA. They assess the wait-versus-rebuild trade-off, and if rebuild looks likely, agree the PITR target timestamp using the same trade-off as 2. Database corruption / data recovery.
Decision	All	Customer	The customer decides whether to wait for cloud recovery or proceed with a full rebuild. Akka provides estimates and trade-offs.
Resolution: rebuild path	Dedicated and BYOC	Akka	Akka provisions the new region via infrastructure as code, runs a Velero restore of Kubernetes state, runs a PITR database restore into the new region, and re-establishes networking (peering, certificates, federation).
Resolution: rebuild path	BYOK8s	Shared	The customer provisions the new Kubernetes cluster in the replacement region, re-establishes the customer-managed side of private connectivity, and creates the new regional database via PITR at the agreed timestamp. Akka installs platform machinery on the new cluster, runs a Velero restore of Kubernetes state, connects services to the restored datastore, and re-establishes platform-side networking (certificates, federation, network policies, port allowlists). Total time depends largely on the customer’s cluster and database provisioning.
Customer-side recovery	Dedicated	Shared	Akka updates its DNS delegation and the DNS records it maintains in the account it owns. Where the customer brings its own domain, the customer updates those records for the new load balancer. Both validate traffic end to end.
Customer-side recovery	BYOC	Shared	Akka updates its DNS delegation. The customer updates DNS records for the new load balancer. Both validate traffic end to end.
Customer-side recovery	BYOK8s	Shared	The customer updates DNS records for the new load balancer. Both validate traffic end to end.

Phase

Applies to

Owner

Action

Detection

All

Shared

The customer monitors the cloud provider status page and end-to-end probes; Akka monitors platform health signals. Akka recommends the customer subscribe to the cloud provider status feed.

Notification

All

Shared

Standard flow.

Triage

Dedicated and BYOC

Shared

Akka and the customer confirm scope (region-wide versus localized), coordinate with the cloud provider for an ETA, and assess the wait-versus-rebuild trade-off.

Triage

BYOK8s

Shared

Akka and the customer confirm scope (region-wide versus localized) and both track the cloud provider’s status feed for an ETA. They assess the wait-versus-rebuild trade-off, and if rebuild looks likely, agree the PITR target timestamp using the same trade-off as 2. Database corruption / data recovery.

Decision

All

Customer

The customer decides whether to wait for cloud recovery or proceed with a full rebuild. Akka provides estimates and trade-offs.

Resolution: rebuild path

Dedicated and BYOC

Akka

Akka provisions the new region via infrastructure as code, runs a Velero restore of Kubernetes state, runs a PITR database restore into the new region, and re-establishes networking (peering, certificates, federation).

Resolution: rebuild path

BYOK8s

Shared

The customer provisions the new Kubernetes cluster in the replacement region, re-establishes the customer-managed side of private connectivity, and creates the new regional database via PITR at the agreed timestamp. Akka installs platform machinery on the new cluster, runs a Velero restore of Kubernetes state, connects services to the restored datastore, and re-establishes platform-side networking (certificates, federation, network policies, port allowlists). Total time depends largely on the customer’s cluster and database provisioning.

Customer-side recovery

Dedicated

Shared

Akka updates its DNS delegation and the DNS records it maintains in the account it owns. Where the customer brings its own domain, the customer updates those records for the new load balancer. Both validate traffic end to end.

Customer-side recovery

BYOC

Shared

Akka updates its DNS delegation. The customer updates DNS records for the new load balancer. Both validate traffic end to end.

Customer-side recovery

BYOK8s

Shared

The customer updates DNS records for the new load balancer. Both validate traffic end to end.

A second region for active-passive or active-active failover turns this rebuild into a routing event; see Multi-region operations.

5. Single / multi-AZ failure

A cloud availability zone (AZ) becomes unavailable. Quorum-based services heal automatically in single-AZ cases; multi-AZ failure with quorum loss may need manual intervention.

Recovery procedure

Phase	Applies to	Owner	Action
Detection	Dedicated and BYOC	Shared	The customer monitors the cloud provider status page and end-to-end checks. Akka monitors platform alerts (Kubernetes node-not-ready, pod reschedules, split-brain resolver activity). The split-brain resolver is the Akka-runtime mechanism that decides the authoritative subset of a cluster after a network partition or AZ failure.
Detection	BYOK8s	Shared	The customer monitors the cloud provider status page, Kubernetes-level signals from their cluster (node-not-ready, pod reschedules), and end-to-end checks. Akka monitors Akka-runtime alerts (split-brain resolver activity, missed heartbeats). The split-brain resolver is the Akka-runtime mechanism that decides the authoritative subset of a cluster after a network partition or AZ failure.
Notification	All	Shared	Standard flow. Akka issues a proactive status update even if no customer action is required.
Triage	Dedicated and BYOC	Akka	Akka confirms split-brain resolver activity and verifies that quorum is maintained.
Triage	BYOK8s	Shared	The customer confirms Kubernetes node and pod state from their cluster. Akka confirms split-brain resolver activity and verifies quorum at the Akka-runtime level. Together they classify: single-AZ failure with quorum preserved, or multi-AZ failure with quorum loss.
Decision	BYOK8s	Shared	Akka and the customer choose: wait for cloud recovery, add cluster capacity, or proceed with Akka-runtime manual recovery.
Resolution	Dedicated and BYOC	Akka	Single-AZ failure with quorum preserved: the platform heals automatically as Kubernetes reschedules pods onto remaining nodes. Multi-AZ failure with quorum loss: Akka may perform manual cluster recovery, depending on cluster size, redundancy, and the failure profile.
Resolution	BYOK8s	Shared	Single-AZ failure with quorum preserved: the platform auto-heals if the customer-managed cluster has capacity headroom in remaining AZs; Kubernetes reschedules pods onto remaining nodes while Akka monitors recovery. Multi-AZ failure with quorum loss: Akka performs manual Akka-runtime recovery per the documented runbook; the customer adds node capacity if Akka requests it.
Customer-side recovery	All	Customer	The customer verifies the application is functioning normally.

Phase

Applies to

Owner

Action

Detection

Dedicated and BYOC

Shared

The customer monitors the cloud provider status page and end-to-end checks. Akka monitors platform alerts (Kubernetes node-not-ready, pod reschedules, split-brain resolver activity). The split-brain resolver is the Akka-runtime mechanism that decides the authoritative subset of a cluster after a network partition or AZ failure.

Detection

BYOK8s

Shared

The customer monitors the cloud provider status page, Kubernetes-level signals from their cluster (node-not-ready, pod reschedules), and end-to-end checks. Akka monitors Akka-runtime alerts (split-brain resolver activity, missed heartbeats). The split-brain resolver is the Akka-runtime mechanism that decides the authoritative subset of a cluster after a network partition or AZ failure.

Notification

All

Shared

Standard flow. Akka issues a proactive status update even if no customer action is required.

Triage

Dedicated and BYOC

Akka

Akka confirms split-brain resolver activity and verifies that quorum is maintained.

Triage

BYOK8s

Shared

The customer confirms Kubernetes node and pod state from their cluster. Akka confirms split-brain resolver activity and verifies quorum at the Akka-runtime level. Together they classify: single-AZ failure with quorum preserved, or multi-AZ failure with quorum loss.

Decision

BYOK8s

Shared

Akka and the customer choose: wait for cloud recovery, add cluster capacity, or proceed with Akka-runtime manual recovery.

Resolution

Dedicated and BYOC

Akka

Single-AZ failure with quorum preserved: the platform heals automatically as Kubernetes reschedules pods onto remaining nodes. Multi-AZ failure with quorum loss: Akka may perform manual cluster recovery, depending on cluster size, redundancy, and the failure profile.

Resolution

BYOK8s

Shared

Single-AZ failure with quorum preserved: the platform auto-heals if the customer-managed cluster has capacity headroom in remaining AZs; Kubernetes reschedules pods onto remaining nodes while Akka monitors recovery. Multi-AZ failure with quorum loss: Akka performs manual Akka-runtime recovery per the documented runbook; the customer adds node capacity if Akka requests it.

Customer-side recovery

All

Customer

The customer verifies the application is functioning normally.

6. Private connectivity failure

Private connectivity between the customer’s environment and an external system fails. Symptoms include timeouts to private systems, blocked ports, network policy mismatches, or peering link failures. The customer’s services lose access to whatever the affected path was reaching.

In Dedicated, Akka operates the network in the account it owns, including peering, so the platform side of the path is Akka’s and the customer side covers only the customer’s own network where it integrates with the Akka region (for example, a transit gateway). In BYOK8s, the drill requires the customer to have private connectivity in place: the customer breaks and re-establishes their connectivity path, and Akka exercises the port allowlist or network policy adjustment path if any non-standard ports are in scope.

Recovery procedure

Phase	Applies to	Owner	Action
Detection	Dedicated and BYOC	Customer	The customer observes connectivity issues (timeouts, blocked ports, network policy mismatches, peering link failures). Akka has no direct alarm on the peering link.
Detection	BYOK8s	Customer	The customer designs, implements, and operates private connectivity, so detection sits with the customer. Akka has no direct alarm on the customer’s connectivity path.
Notification	Dedicated and BYOC	Customer	Customer raises a severity 1 incident with the observed symptom.
Notification	BYOK8s	Customer	If Akka platform reachability or operation is affected, the customer raises a severity 1 incident with the observed symptom. Otherwise the customer handles this internally.
Triage	Dedicated and BYOC	Shared	The customer shares the symptom and customer-side network configuration. Akka shares platform-side networking configuration. Together they identify the fault (blocked port, network policy, peering misconfiguration, or other) and the owning side.
Triage	BYOK8s	Shared	The customer identifies the fault on the connectivity path (blocked port, network policy, peering misconfiguration, or other). Akka confirms platform-side network policies, ports, and configuration relevant to the connectivity.
Decision	Dedicated and BYOC	Shared	Akka and the customer agree on the fix and who executes each part.
Decision	BYOK8s	Shared	If the fault is customer-side, the customer decides the fix. If platform-side changes are needed (a new port allowlist or network policy adjustment), Akka and the customer agree on those.
Resolution	Dedicated and BYOC	Shared	The owning side executes. Customer-side changes (re-establishing peering, updating customer network policies, opening customer-side ports): customer. Platform-side changes (port allowlisting, network policy adjustments on the Akka-managed side): Akka. In Dedicated, peering on the Akka region’s side is Akka’s.
Resolution	BYOK8s	Shared	The customer executes connectivity-path fixes on the customer-managed side: re-establishing peering, updating network policies, opening ports, or rebuilding the affected connectivity component. Akka adjusts platform-side network policies or port allowlists via infrastructure as code as needed.
Customer-side recovery	All	Customer	The customer verifies connectivity is restored end to end.

Phase

Applies to

Owner

Action

Detection

Dedicated and BYOC

Customer

The customer observes connectivity issues (timeouts, blocked ports, network policy mismatches, peering link failures). Akka has no direct alarm on the peering link.

Detection

BYOK8s

Customer

The customer designs, implements, and operates private connectivity, so detection sits with the customer. Akka has no direct alarm on the customer’s connectivity path.

Notification

Dedicated and BYOC

Customer

Customer raises a severity 1 incident with the observed symptom.

Notification

BYOK8s

Customer

If Akka platform reachability or operation is affected, the customer raises a severity 1 incident with the observed symptom. Otherwise the customer handles this internally.

Triage

Dedicated and BYOC

Shared

The customer shares the symptom and customer-side network configuration. Akka shares platform-side networking configuration. Together they identify the fault (blocked port, network policy, peering misconfiguration, or other) and the owning side.

Triage

BYOK8s

Shared

The customer identifies the fault on the connectivity path (blocked port, network policy, peering misconfiguration, or other). Akka confirms platform-side network policies, ports, and configuration relevant to the connectivity.

Decision

Dedicated and BYOC

Shared

Akka and the customer agree on the fix and who executes each part.

Decision

BYOK8s

Shared

If the fault is customer-side, the customer decides the fix. If platform-side changes are needed (a new port allowlist or network policy adjustment), Akka and the customer agree on those.

Resolution

Dedicated and BYOC

Shared

The owning side executes. Customer-side changes (re-establishing peering, updating customer network policies, opening customer-side ports): customer. Platform-side changes (port allowlisting, network policy adjustments on the Akka-managed side): Akka. In Dedicated, peering on the Akka region’s side is Akka’s.

Resolution

BYOK8s

Shared

The customer executes connectivity-path fixes on the customer-managed side: re-establishing peering, updating network policies, opening ports, or rebuilding the affected connectivity component. Akka adjusts platform-side network policies or port allowlists via infrastructure as code as needed.

Customer-side recovery

All

Customer

The customer verifies connectivity is restored end to end.

7. Load balancer / ingress failure

Recovery procedure

Phase	Applies to	Owner	Action
Detection	Dedicated and BYOC	Shared	Akka monitors load balancer health checks; the customer monitors frontend behavior.
Detection	BYOK8s	Customer	Frontend behavior or customer monitoring shows failed requests, edge errors, or dropped traffic.
Notification	Dedicated and BYOC	Shared	Standard flow.
Notification	BYOK8s	Customer	Customer raises a severity 1 incident with the observed symptom.
Triage	All	Akka	Akka confirms the load balancer is the fault and decides whether to fix in place or recreate.
Decision	All	Shared	Fix in place, or recreate. Recreate requires a DNS update.
Resolution	Dedicated and BYOC	Akka	Akka fixes or recreates the load balancer. An in-cluster proxy fallback, where configured at install time, carries traffic while the load balancer is recreated.
Resolution	BYOK8s	Akka	Akka fixes or recreates the load balancer. If the in-cluster proxy fallback is configured, traffic continues via the proxy while the load balancer is recreated. Akka confirms whether this applies for the customer at the tabletop.
Customer-side recovery	Dedicated	Shared	If recreated, Akka updates the DNS records it maintains, and the customer updates DNS records for any domain it brings and verifies traffic.
Customer-side recovery	BYOC and BYOK8s	Customer	If recreated, the customer updates DNS records for the new endpoint and verifies traffic.

Phase

Applies to

Owner

Action

Detection

Dedicated and BYOC

Shared

Akka monitors load balancer health checks; the customer monitors frontend behavior.

Detection

BYOK8s

Customer

Frontend behavior or customer monitoring shows failed requests, edge errors, or dropped traffic.

Notification

Dedicated and BYOC

Shared

Standard flow.

Notification

BYOK8s

Customer

Customer raises a severity 1 incident with the observed symptom.

Triage

All

Akka

Akka confirms the load balancer is the fault and decides whether to fix in place or recreate.

Decision

All

Shared

Fix in place, or recreate. Recreate requires a DNS update.

Resolution

Dedicated and BYOC

Akka

Akka fixes or recreates the load balancer. An in-cluster proxy fallback, where configured at install time, carries traffic while the load balancer is recreated.

Resolution

BYOK8s

Akka

Akka fixes or recreates the load balancer. If the in-cluster proxy fallback is configured, traffic continues via the proxy while the load balancer is recreated. Akka confirms whether this applies for the customer at the tabletop.

Customer-side recovery

Dedicated

Shared

If recreated, Akka updates the DNS records it maintains, and the customer updates DNS records for any domain it brings and verifies traffic.

Customer-side recovery

BYOC and BYOK8s

Customer

If recreated, the customer updates DNS records for the new endpoint and verifies traffic.

8. Runtime / SDK version compatibility

Runtime-versus-SDK compatibility is a shared responsibility. Akka detects what it can on the platform side; the customer smoke-tests SDK-runtime combinations in pre-production before promotion.

Recovery procedure

Phase	Owner	Action
Detection	Shared	Akka runs preliminary compatibility checks and can surface known-bad combinations. The customer smoke-tests SDK-runtime combinations in pre-production before promotion.
Notification	Shared	Standard flow. Customer includes the runtime version, the SDK version, and the observed error.
Triage	Akka	Akka confirms it is a runtime/SDK mismatch and identifies the offending combination.
Decision	Shared	Akka recommends a path by blast radius: runtime rollback (smallest blast radius, default where viable), manual patching (where rollback is risky or not viable), or rolling the SDK forward (where rollback is not viable and the SDK change is small). The customer confirms.
Resolution	Akka	Akka rolls back the runtime per the documented procedure. If rollback fails, Akka performs manual patching.
Customer-side recovery	Customer	The customer verifies the application is functioning normally.

Phase

Owner

Action

Detection

Shared

Akka runs preliminary compatibility checks and can surface known-bad combinations. The customer smoke-tests SDK-runtime combinations in pre-production before promotion.

Notification

Shared

Standard flow. Customer includes the runtime version, the SDK version, and the observed error.

Triage

Akka

Akka confirms it is a runtime/SDK mismatch and identifies the offending combination.

Decision

Shared

Akka recommends a path by blast radius: runtime rollback (smallest blast radius, default where viable), manual patching (where rollback is risky or not viable), or rolling the SDK forward (where rollback is not viable and the SDK change is small). The customer confirms.

Resolution

Akka

Akka rolls back the runtime per the documented procedure. If rollback fails, Akka performs manual patching.

Customer-side recovery

Customer

The customer verifies the application is functioning normally.

9. Additional chaos vectors

Documented procedures. Not covered in detail at the tabletop. Drill readiness varies.

Scenario	Notes
Federation / control-plane connectivity loss	New project creation may be blocked. Existing workloads continue uninterrupted.
Certificate failure	Variants: Let’s Encrypt cluster issuer, customer-provided issuer, Akka mTLS rotation, customer-provisioned certificates.
In-cluster proxy / mesh failure	Component-level restart; fallback routing where configured.
Runtime issue requiring rollback	Documented rollback procedure; manual patching as last resort.
Observability stack failure	Both sides lose visibility; the procedure must specify out-of-band communication paths.

Scenario

Notes

Federation / control-plane connectivity loss

New project creation may be blocked. Existing workloads continue uninterrupted.

Certificate failure

Variants: Let’s Encrypt cluster issuer, customer-provided issuer, Akka mTLS rotation, customer-provisioned certificates.

In-cluster proxy / mesh failure

Component-level restart; fallback routing where configured.

Runtime issue requiring rollback

Documented rollback procedure; manual patching as last resort.

Observability stack failure

Both sides lose visibility; the procedure must specify out-of-band communication paths.

Standard communication flow

The detecting side notifies the other. In BYOK8s, Akka also shares guidance on what each side should monitor so the customer is equipped to detect issues early. The flow:

Detection.
The first responder identifies the scenario class.
The customer raises a severity 1 incident, or Akka notifies through the agreed channel.
A joint incident channel is opened.
Akka and the customer triage jointly.
The two sides decide which procedure applies.
The procedure executes: Akka on the platform side, the customer on the application side.
The customer is notified at the start, at the decision point, and at completion.
A post-incident review follows within the SLA (Service Level Agreement).

What to monitor

The customer should subscribe to the cloud provider status page (for example status.cloud.google.com for GCP, or status.aws.amazon.com for AWS), configure status feeds for the region in use, and watch end-to-end probes against their application. In BYOK8s, that includes Kubernetes-level signals from their cluster. Akka shares which platform signals it emits so the customer can layer additional alerts.

Follow-up commitments

Before scenarios chosen for a live drill are exercised:

The customer wires up production logging, metrics, and alerting sufficient to surface the documented failure modes, including, in BYOK8s, Kubernetes-level signals from the customer’s cluster.
The customer defines the on-call rotation and severity 1 entry point.
The customer identifies the decision authority for declaring an incident and authorizing recovery.
In Dedicated and BYOC, Akka verifies backup retention, PITR enablement, and deletion protection for the customer’s region.
In BYOK8s, the customer configures and verifies automated database backups, PITR enablement, retention window, and deletion protection on the regional database, and Akka and the customer schedule periodic backup-and-restore testing.

These outputs, with the scenarios selected for a live drill, feed the production readiness checklist's disaster recovery drill item.