Google Cloud Storage

Google Cloud Storage allows world-wide storage and retrieval of any amount of data at any time.

Further information at the official Google Cloud Storage documentation website. This connector communicates to Cloud Storage via HTTP requests.

Artifacts

The Akka dependencies are available from Akka’s library repository. To access them there, you need to configure the URL for this repository.

sbt

resolvers += "Akka library repository".at("https://repo.akka.io/maven")

Maven

<project>
  ...
  <repositories>
    <repository>
      <id>akka-repository</id>
      <name>Akka library repository</name>
      <url>https://repo.akka.io/maven</url>
    </repository>
  </repositories>
</project>

Gradle

repositories {
    mavenCentral()
    maven {
        url "https://repo.akka.io/maven"
    }
}

Additionally, add the dependencies as below.

sbt

val AkkaVersion = "2.10.0"
val AkkaHttpVersion = "10.7.0"
libraryDependencies ++= Seq(
  "com.lightbend.akka" %% "akka-stream-alpakka-google-cloud-storage" % "9.0.0",
  "com.typesafe.akka" %% "akka-stream" % AkkaVersion,
  "com.typesafe.akka" %% "akka-http" % AkkaHttpVersion,
  "com.typesafe.akka" %% "akka-http-spray-json" % AkkaHttpVersion
)

Maven

<properties>
  <akka.version>2.10.0</akka.version>
  <akka.http.version>10.7.0</akka.http.version>
  <scala.binary.version>2.13</scala.binary.version>
</properties>
<dependencies>
  <dependency>
    <groupId>com.lightbend.akka</groupId>
    <artifactId>akka-stream-alpakka-google-cloud-storage_${scala.binary.version}</artifactId>
    <version>9.0.0</version>
  </dependency>
  <dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-stream_${scala.binary.version}</artifactId>
    <version>${akka.version}</version>
  </dependency>
  <dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-http_${scala.binary.version}</artifactId>
    <version>${akka.http.version}</version>
  </dependency>
  <dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-http-spray-json_${scala.binary.version}</artifactId>
    <version>${akka.http.version}</version>
  </dependency>
</dependencies>

Gradle

def versions = [
  AkkaVersion: "2.10.0",
  AkkaHttpVersion: "10.7.0",
  ScalaBinary: "2.13"
]
dependencies {
  implementation "com.lightbend.akka:akka-stream-alpakka-google-cloud-storage_${versions.ScalaBinary}:9.0.0"
  implementation "com.typesafe.akka:akka-stream_${versions.ScalaBinary}:${versions.AkkaVersion}"
  implementation "com.typesafe.akka:akka-http_${versions.ScalaBinary}:${versions.AkkaHttpVersion}"
  implementation "com.typesafe.akka:akka-http-spray-json_${versions.ScalaBinary}:${versions.AkkaHttpVersion}"
}

The table below shows direct dependencies of this module and the second tab shows all libraries it depends on transitively.

Direct dependencies

Organization	Artifact	Version
com.lightbend.akka	akka-stream-alpakka-google-common_2.13	9.0.0
com.typesafe.akka	akka-http-spray-json_2.13	10.7.0
com.typesafe.akka	akka-http_2.13	10.7.0
com.typesafe.akka	akka-stream_2.13	2.10.0
org.scala-lang	scala-library	2.13.12

Dependency tree

com.lightbend.akka    akka-stream-alpakka-google-common_2.13    9.0.0
    com.github.jwt-scala    jwt-json-common_2.13    9.4.6    Apache-2.0
        com.github.jwt-scala    jwt-core_2.13    9.4.6    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.google.auth    google-auth-library-credentials    1.24.1
    com.typesafe.akka    akka-http-spray-json_2.13    10.7.0    BUSL-1.1
        com.typesafe.akka    akka-http_2.13    10.7.0    BUSL-1.1
            com.typesafe.akka    akka-http-core_2.13    10.7.0    BUSL-1.1
                com.typesafe.akka    akka-parsing_2.13    10.7.0    BUSL-1.1
                    org.scala-lang    scala-library    2.13.12    Apache-2.0
                org.scala-lang    scala-library    2.13.12    Apache-2.0
            com.typesafe.akka    akka-pki_2.13    2.10.0    BUSL-1.1
                com.hierynomus    asn-one    0.6.0    The Apache License, Version 2.0
                com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
                    com.typesafe    config    1.4.3    Apache-2.0
                    org.scala-lang    scala-library    2.13.12    Apache-2.0
                org.scala-lang    scala-library    2.13.12    Apache-2.0
                org.slf4j    slf4j-api    2.0.16
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        io.spray    spray-json_2.13    1.3.6    Apache 2
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.typesafe.akka    akka-http_2.13    10.7.0    BUSL-1.1
        com.typesafe.akka    akka-http-core_2.13    10.7.0    BUSL-1.1
            com.typesafe.akka    akka-parsing_2.13    10.7.0    BUSL-1.1
                org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        com.typesafe.akka    akka-pki_2.13    2.10.0    BUSL-1.1
            com.hierynomus    asn-one    0.6.0    The Apache License, Version 2.0
            com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
                com.typesafe    config    1.4.3    Apache-2.0
                org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.slf4j    slf4j-api    2.0.16
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.typesafe.akka    akka-stream_2.13    2.10.0    BUSL-1.1
        com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
            com.typesafe    config    1.4.3    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        com.typesafe.akka    akka-protobuf-v3_2.13    2.10.0    BUSL-1.1
        org.reactivestreams    reactive-streams    1.0.4    MIT-0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    org.scala-lang    scala-library    2.13.12    Apache-2.0
com.typesafe.akka    akka-http-spray-json_2.13    10.7.0    BUSL-1.1
    com.typesafe.akka    akka-http_2.13    10.7.0    BUSL-1.1
        com.typesafe.akka    akka-http-core_2.13    10.7.0    BUSL-1.1
            com.typesafe.akka    akka-parsing_2.13    10.7.0    BUSL-1.1
                org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        com.typesafe.akka    akka-pki_2.13    2.10.0    BUSL-1.1
            com.hierynomus    asn-one    0.6.0    The Apache License, Version 2.0
            com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
                com.typesafe    config    1.4.3    Apache-2.0
                org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
            org.slf4j    slf4j-api    2.0.16
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    io.spray    spray-json_2.13    1.3.6    Apache 2
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    org.scala-lang    scala-library    2.13.12    Apache-2.0
com.typesafe.akka    akka-http_2.13    10.7.0    BUSL-1.1
    com.typesafe.akka    akka-http-core_2.13    10.7.0    BUSL-1.1
        com.typesafe.akka    akka-parsing_2.13    10.7.0    BUSL-1.1
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.typesafe.akka    akka-pki_2.13    2.10.0    BUSL-1.1
        com.hierynomus    asn-one    0.6.0    The Apache License, Version 2.0
        com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
            com.typesafe    config    1.4.3    Apache-2.0
            org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
        org.slf4j    slf4j-api    2.0.16
    org.scala-lang    scala-library    2.13.12    Apache-2.0
com.typesafe.akka    akka-stream_2.13    2.10.0    BUSL-1.1
    com.typesafe.akka    akka-actor_2.13    2.10.0    BUSL-1.1
        com.typesafe    config    1.4.3    Apache-2.0
        org.scala-lang    scala-library    2.13.12    Apache-2.0
    com.typesafe.akka    akka-protobuf-v3_2.13    2.10.0    BUSL-1.1
    org.reactivestreams    reactive-streams    1.0.4    MIT-0
    org.scala-lang    scala-library    2.13.12    Apache-2.0
org.scala-lang    scala-library    2.13.12    Apache-2.0

Configuration

The Storage connector shares its basic configuration with all the Google connectors in Alpakka. Additional Storage-specific configuration settings can be found in its own reference.conf.

Store a file in Google Cloud Storage

A file can be uploaded to Google Cloud Storage by creating a source of ByteStringByteString and running that with a sink created from GCStorage.resumableUploadGCStorage.resumableUpload.

Scala

copysourceval sink =
  GCStorage.resumableUpload(bucketName, fileName, ContentTypes.`text/plain(UTF-8)`, chunkSize, metadata)

val source = Source(
  List(ByteString(firstChunkContent), ByteString(secondChunkContent))
)

val result: Future[StorageObject] = source.runWith(sink)

Java

copysource
final Sink<ByteString, CompletionStage<StorageObject>> sink =
    GCStorage.resumableUpload(
        bucketName(), fileName(), ContentTypes.TEXT_PLAIN_UTF8, chunkSize);

final Source<ByteString, NotUsed> source =
    Source.from(
        Arrays.asList(
            ByteString.fromString(firstChunkContent),
            ByteString.fromString(secondChunkContent)));

final CompletionStage<StorageObject> result = source.runWith(sink, system());

Download a file from Google Cloud Storage

A source for downloading a file can be created by calling GCStorage.downloadGCStorage.download. It will emit an OptionOptional that will hold file’s data or will be empty if no such file can be found.

If you need to download the specific version of the object in a bucket where object versioning is enabled, you can specify the generation.

Scala

copysource
val downloadSource: Source[Option[Source[ByteString, NotUsed]], NotUsed] =
  GCStorage.download(bucketName, fileName)

val downloadGenerationSource: Source[Option[Source[ByteString, NotUsed]], NotUsed] =
  GCStorage.download(bucketName, fileName, Some(generation))

Java

copysource
final Source<Optional<Source<ByteString, NotUsed>>, NotUsed> downloadSource =
    GCStorage.download(bucketName(), fileName());

final Source<Optional<Source<ByteString, NotUsed>>, NotUsed> downloadGenerationSource =
    GCStorage.download(bucketName(), fileName(), generation());

Access object metadata without downloading object from Google Cloud Storage

If you do not need object itself, you can query for only object metadata using a source from GCStorage.getObjectGCStorage.getObject.

If you need the specific version of the object metadata in a bucket where object versioning is enabled, you can specify the generation.

Scala

copysource
val getObjectSource: Source[Option[StorageObject], NotUsed] = GCStorage.getObject(bucketName, fileName)

val getObjectGenerationSource: Source[Option[StorageObject], NotUsed] =
  GCStorage.getObject(bucketName, fileName, Some(generation))

Java

copysource
final Source<Optional<StorageObject>, NotUsed> getObjectSource =
    GCStorage.getObject(bucketName(), fileName());

final Source<Optional<StorageObject>, NotUsed> getObjectGenerationSource =
    GCStorage.getObject(bucketName(), fileName(), generation());

List bucket contents

To get a list of all objects in a bucket, use GCStorage.listBucketGCStorage.listBucket. When run, this will give a stream of StorageObject.

To get a list of both live and archived versions of all objects in a bucket where object versioning is enabled, the versions has to be set to true

Scala

copysource
val listSource: Source[StorageObject, NotUsed] = GCStorage.listBucket(bucketName, Some(folder))
val listVersionsSource: Source[StorageObject, NotUsed] = GCStorage.listBucket(bucketName, Some(folder), versions)

Java

copysource
final Source<StorageObject, NotUsed> listSource = GCStorage.listBucket(bucketName(), folder);

final Source<StorageObject, NotUsed> listVersionsSource =
    GCStorage.listBucket(bucketName(), folder, versions);

Rewrite (multi part)

Copy an Google Clouds Storage object from source bucket to target bucket using GCStorage.rewriteGCStorage.rewrite. When run, this will emit a single StorageObject with the information about the copied object.

Scala

copysource
val result: Future[StorageObject] = GCStorage.rewrite(bucketName, fileName, rewriteBucketName, fileName).run()

Java

copysource
final CompletionStage<StorageObject> result =
    GCStorage.rewrite(bucketName(), fileName(), rewriteBucketName, fileName()).run(system());

Apply Google Cloud Storage settings to a part of the stream

It is possible to make one part of the stream use different GoogleSettings from the rest of the graph. This can be useful, when one stream is used to copy files across regions with different service accounts. You can attach a custom GoogleSettings instance or a custom config path to a graph using attributes from GoogleAttributes:

Scala

copysource
val newSettings = GoogleSettings(system).withProjectId("projectId")

val listSource: Source[StorageObject, NotUsed] =
  GCStorage.listBucket(bucketName, None).withAttributes(GoogleAttributes.settings(newSettings))

Java

copysource
final GoogleSettings newSettings = GoogleSettings.create(system()).withProjectId("projectId");

final Source<StorageObject, NotUsed> listSource =
    GCStorage.listBucket(bucketName()).withAttributes(GoogleAttributes.settings(newSettings));

Bucket management

Bucket management API provides functionality for both Sources and Futures / CompletionStages. In case of the Future API user can specify attributes to the request in the method itself and as for Sources it can be done via method .withAttributes. For more information about attributes see: GCStorageAttributes and Attributes

Make bucket

In order to create a bucket in Google Cloud Storage you need to specify it’s unique name. This value has to be set accordingly to the requirements. The bucket will be created in the given location.

Scala

copysource
implicit val sampleAttributes: Attributes = GoogleAttributes.settings(sampleSettings)

val createBucketResponse: Future[Bucket] = GCStorage.createBucket(bucketName, location)
val createBucketSourceResponse: Source[Bucket, NotUsed] = GCStorage.createBucketSource(bucketName, location)

Java

copysource
final Attributes sampleAttributes = GoogleAttributes.settings(sampleSettings);

final CompletionStage<Bucket> createBucketResponse =
    GCStorage.createBucket(bucketName(), location, actorSystem, sampleAttributes);
final Source<Bucket, NotUsed> createBucketSourceResponse =
    GCStorage.createBucketSource(bucketName(), location);

Delete bucket

To delete a bucket you need to specify its name and the bucket needs to be empty.

Scala

copysource
implicit val sampleAttributes: Attributes = GoogleAttributes.settings(sampleSettings)

val deleteBucketResponse: Future[Done] = GCStorage.deleteBucket(bucketName)
val deleteBucketSourceResponse: Source[Done, NotUsed] = GCStorage.deleteBucketSource(bucketName)

Java

copysource
final Attributes sampleAttributes = GoogleAttributes.settings(sampleSettings);

final CompletionStage<Done> deleteBucketResponse =
    GCStorage.deleteBucket(this.bucketName(), actorSystem, sampleAttributes);
final Source<Done, NotUsed> deleteBucketSourceResponse =
    GCStorage.deleteBucketSource(this.bucketName());

Get bucket

To get a bucket you need to specify its name.

Scala

copysource
implicit val sampleAttributes: Attributes = GoogleAttributes.settings(sampleSettings)

val getBucketResponse: Future[Option[Bucket]] = GCStorage.getBucket(bucketName)
val getBucketSourceResponse: Source[Option[Bucket], NotUsed] = GCStorage.getBucketSource(bucketName)

Java

copysource
final Attributes sampleAttributes = GoogleAttributes.settings(sampleSettings);

final CompletionStage<Optional<Bucket>> getBucketResponse =
    GCStorage.getBucket(this.bucketName(), actorSystem, sampleAttributes);
final Source<Optional<Bucket>, NotUsed> getBucketSourceResponse =
    GCStorage.getBucketSource(this.bucketName());

Running the example code

The code in this guide is part of runnable tests of this project. You are welcome to edit the code and run it in sbt.

Scala

sbt
> google-cloud-storage/test

Java

sbt
> google-cloud-storage/test

Some test code requires access to Google cloud storage, to run them you will need to configure a project and pub/sub in google cloud and provide your own credentials.