Google Cloud BigQuery Storage
The BigQuery Storage API offers fast access to BigQuery-managed storage using an rpc-based protocol. It is seen as an improvement over the REST API, and bulk data extract
jobs for accessing BigQuery-managed table data, but doesn’t offer any functionality around managing BigQuery resources. Further information at the official Google Cloud documentation website .
This connector communicates to the BigQuery Storage API via the gRPC protocol. The integration between Akka Stream and gRPC is handled by the Akka gRPC library . Currently, this connector only supports returning each row as an Avro GenericRecord.
[+] Show project info
Artifacts
The Akka dependencies are available from Akka’s library repository. To access them there, you need to configure the URL for this repository.
sbt resolvers += "Akka library repository" . at ( "https://repo.akka.io/maven" )
Maven <project>
...
<repositories>
<repository>
<id> akka-repository </id>
<name> Akka library repository </name>
<url> https://repo.akka.io/maven </url>
</repository>
</repositories>
</project>
Gradle repositories {
mavenCentral ()
maven {
url "https://repo.akka.io/maven"
}
}
Additionally, add the dependencies as below. Since Akka gRPC uses Akka Discovery internally. Make sure to add Akka Discovery with the same Akka version that the application uses.
sbt val AkkaVersion = "2.10.0"
libraryDependencies ++= Seq (
"com.lightbend.akka" %% "akka-stream-alpakka-google-cloud-bigquery-storage" % "9.0.1" ,
"com.typesafe.akka" %% "akka-stream" % AkkaVersion ,
"com.typesafe.akka" %% "akka-discovery" % AkkaVersion
)
Maven <properties>
<akka.version> 2.10.0 </akka.version>
<scala.binary.version> 2.13 </scala.binary.version>
</properties>
<dependencies>
<dependency>
<groupId> com.lightbend.akka </groupId>
<artifactId> akka-stream-alpakka-google-cloud-bigquery-storage_${scala.binary.version} </artifactId>
<version> 9.0.1 </version>
</dependency>
<dependency>
<groupId> com.typesafe.akka </groupId>
<artifactId> akka-stream_${scala.binary.version} </artifactId>
<version> ${akka.version} </version>
</dependency>
<dependency>
<groupId> com.typesafe.akka </groupId>
<artifactId> akka-discovery_${scala.binary.version} </artifactId>
<version> ${akka.version} </version>
</dependency>
</dependencies>
Gradle def versions = [
AkkaVersion : "2.10.0" ,
ScalaBinary : "2.13"
]
dependencies {
implementation "com.lightbend.akka:akka-stream-alpakka-google-cloud-bigquery-storage_${versions.ScalaBinary}:9.0.1"
implementation "com.typesafe.akka:akka-stream_${versions.ScalaBinary}:${versions.AkkaVersion}"
implementation "com.typesafe.akka:akka-discovery_${versions.ScalaBinary}:${versions.AkkaVersion}"
}
The table below shows direct dependencies of this module and the second tab shows all libraries it depends on transitively.
Direct dependencies
Organization Artifact Version
com.google.protobuf protobuf-java 3.25.5
com.lightbend.akka.grpc akka-grpc-runtime_2.13 2.5.0
com.lightbend.akka akka-stream-alpakka-google-common_2.13 9.0.1
com.thesamet.scalapb scalapb-runtime_2.13 0.11.17
com.typesafe.akka akka-discovery_2.13 2.10.0
com.typesafe.akka akka-http-spray-json_2.13 10.7.0
com.typesafe.akka akka-stream_2.13 2.10.0
io.grpc grpc-auth 1.63.2
io.grpc grpc-stub 1.63.2
org.scala-lang scala-library 2.13.12
Dependency tree com.google.protobuf protobuf-java 3.25.5
com.lightbend.akka.grpc akka-grpc-runtime_2.13 2.5.0 BUSL-1.1
com.google.protobuf protobuf-java 3.25.5
com.thesamet.scalapb scalapb-runtime_2.13 0.11.17 Apache 2
com.google.protobuf protobuf-java 3.25.5
com.thesamet.scalapb lenses_2.13 0.11.17 Apache 2
org.scala-lang.modules scala-collection-compat_2.13 2.12.0 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang.modules scala-collection-compat_2.13 2.12.0 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-discovery_2.13 2.10.0 BUSL-1.1
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-http-core_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-parsing_2.13 10.7.0 BUSL-1.1
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-http_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http-core_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-parsing_2.13 10.7.0 BUSL-1.1
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-pki_2.13 2.10.0 BUSL-1.1
com.hierynomus asn-one 0.6.0 The Apache License, Version 2.0
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.slf4j slf4j-api 2.0.16
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-pki_2.13 2.10.0 BUSL-1.1
com.hierynomus asn-one 0.6.0 The Apache License, Version 2.0
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.slf4j slf4j-api 2.0.16
com.typesafe.akka akka-stream_2.13 2.10.0 BUSL-1.1
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-protobuf-v3_2.13 2.10.0 BUSL-1.1
org.reactivestreams reactive-streams 1.0.4 MIT-0
org.scala-lang scala-library 2.13.12 Apache-2.0
io.grpc grpc-core 1.63.2 Apache 2.0
com.google.android annotations 4.1.1.4 Apache 2.0
com.google.code.gson gson 2.10.1 Apache-2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-context 1.63.2 Apache 2.0
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.perfmark perfmark-api 0.26.0 Apache 2.0
org.codehaus.mojo animal-sniffer-annotations 1.23
io.grpc grpc-netty-shaded 1.63.2 Apache 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-core 1.63.2 Apache 2.0
com.google.android annotations 4.1.1.4 Apache 2.0
com.google.code.gson gson 2.10.1 Apache-2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-context 1.63.2 Apache 2.0
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.perfmark perfmark-api 0.26.0 Apache 2.0
org.codehaus.mojo animal-sniffer-annotations 1.23
io.grpc grpc-util 1.63.2 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-core 1.63.2 Apache 2.0
com.google.android annotations 4.1.1.4 Apache 2.0
com.google.code.gson gson 2.10.1 Apache-2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-context 1.63.2 Apache 2.0
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.perfmark perfmark-api 0.26.0 Apache 2.0
org.codehaus.mojo animal-sniffer-annotations 1.23
org.codehaus.mojo animal-sniffer-annotations 1.23
io.perfmark perfmark-api 0.26.0 Apache 2.0
io.grpc grpc-protobuf 1.63.2 Apache 2.0
com.google.api.grpc proto-google-common-protos 2.29.0 Apache-2.0
com.google.protobuf protobuf-java 3.25.5
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
com.google.protobuf protobuf-java 3.25.5
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-protobuf-lite 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
org.scala-lang scala-library 2.13.12 Apache-2.0
com.lightbend.akka akka-stream-alpakka-google-common_2.13 9.0.1
com.github.jwt-scala jwt-json-common_2.13 9.4.6 Apache-2.0
com.github.jwt-scala jwt-core_2.13 9.4.6 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.google.auth google-auth-library-credentials 1.24.1
com.typesafe.akka akka-http-spray-json_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http-core_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-parsing_2.13 10.7.0 BUSL-1.1
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-pki_2.13 2.10.0 BUSL-1.1
com.hierynomus asn-one 0.6.0 The Apache License, Version 2.0
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.slf4j slf4j-api 2.0.16
org.scala-lang scala-library 2.13.12 Apache-2.0
io.spray spray-json_2.13 1.3.6 Apache 2
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-http_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http-core_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-parsing_2.13 10.7.0 BUSL-1.1
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-pki_2.13 2.10.0 BUSL-1.1
com.hierynomus asn-one 0.6.0 The Apache License, Version 2.0
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.slf4j slf4j-api 2.0.16
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-stream_2.13 2.10.0 BUSL-1.1
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-protobuf-v3_2.13 2.10.0 BUSL-1.1
org.reactivestreams reactive-streams 1.0.4 MIT-0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.thesamet.scalapb scalapb-runtime_2.13 0.11.17 Apache 2
com.google.protobuf protobuf-java 3.25.5
com.thesamet.scalapb lenses_2.13 0.11.17 Apache 2
org.scala-lang.modules scala-collection-compat_2.13 2.12.0 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang.modules scala-collection-compat_2.13 2.12.0 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-discovery_2.13 2.10.0 BUSL-1.1
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-http-spray-json_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-http-core_2.13 10.7.0 BUSL-1.1
com.typesafe.akka akka-parsing_2.13 10.7.0 BUSL-1.1
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-pki_2.13 2.10.0 BUSL-1.1
com.hierynomus asn-one 0.6.0 The Apache License, Version 2.0
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
org.slf4j slf4j-api 2.0.16
org.scala-lang scala-library 2.13.12 Apache-2.0
io.spray spray-json_2.13 1.3.6 Apache 2
org.scala-lang scala-library 2.13.12 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-stream_2.13 2.10.0 BUSL-1.1
com.typesafe.akka akka-actor_2.13 2.10.0 BUSL-1.1
com.typesafe config 1.4.3 Apache-2.0
org.scala-lang scala-library 2.13.12 Apache-2.0
com.typesafe.akka akka-protobuf-v3_2.13 2.10.0 BUSL-1.1
org.reactivestreams reactive-streams 1.0.4 MIT-0
org.scala-lang scala-library 2.13.12 Apache-2.0
io.grpc grpc-auth 1.63.2 Apache 2.0
com.google.auth google-auth-library-credentials 1.24.1
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-stub 1.63.2 Apache 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
io.grpc grpc-api 1.63.2 Apache 2.0
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava guava 32.1.3-android
com.google.code.findbugs jsr305 3.0.2 The Apache Software License, Version 2.0
com.google.errorprone error_prone_annotations 2.23.0 Apache 2.0
com.google.guava failureaccess 1.0.1
com.google.guava listenablefuture 9999.0-empty-to-avoid-conflict-with-guava
com.google.j2objc j2objc-annotations 2.8 Apache License, Version 2.0
org.checkerframework checker-qual 3.37.0 The MIT License
org.scala-lang scala-library 2.13.12 Apache-2.0
Build setup
The Alpakka Google Cloud BigQuery Storage library contains the classes generated from Google’s protobuf specification .
Java 17
When running on Java 17 and newer you have to add JVM flag --add-opens=java.base/java.nio=ALL-UNNAMED
.
Configuration
The BigQuery Storage connector shares its basic configuration with all the Google connectors in Alpakka.
Example Test Configuration
alpakka.google.cloud.bigquery.grpc {
host = "localhost"
port = 21000
rootCa = "none"
callCredentials = "none"
}
For more configuration details consider the underlying configuration for Akka gRPC .
A manually initialized akka.stream.alpakka.googlecloud.bigquery.storage.scaladsl.GrpcBigQueryStorageReader
akka.stream.alpakka.googlecloud.bigquery.storage.javadsl.GrpcBigQueryStorageReader
can be used by providing it as an attribute to the stream:
Scala
copy source val reader : GrpcBigQueryStorageReader = GrpcBigQueryStorageReader ( BigQueryStorageSettings ( "localhost" , 8000 ))
val sourceForReader : Source [( ReadSession . Schema , Seq [ Source [ ReadRowsResponse . Rows , NotUsed ]]), Future [ NotUsed ]] =
BigQueryStorage
. create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO )
. withAttributes (
BigQueryStorageAttributes . reader ( reader )
)
Java
copy source GrpcBigQueryStorageReader reader =
GrpcBigQueryStorageReader . apply ( BigQueryStorageSettings . apply ( "localhost" , 8000 ), sys );
Source <
Tuple2 <
com . google . cloud . bigquery . storage . v1 . stream . ReadSession . Schema ,
List < Source < ReadRowsResponse . Rows , NotUsed >>>,
CompletionStage < NotUsed >>
sourceForReader =
BigQueryStorage . create (
"projectId" , "datasetId" , "tableId" , DataFormat . AVRO , readOptions , 1 )
. withAttributes ( BigQueryStorageAttributes . reader ( reader ));
Reading
We can read in a number of ways. To read data from a table a read session needs to be created. On the session creation we can specify the number of streams to be used in order to transfer the data, this makes it feasible to achieve parallelism while ingesting the data, thus achieving better performance. To create a session the data format needs to be specified. The options provided are Avro and Arrow.
If no TableReadOptions
are specified all the table’s columns shall be retrieved as a Source
containing a Source
for each stream, which will each deliver a section of the rows:
Scala
copy source import akka . NotUsed
import com . google . cloud . bigquery . storage . v1 . storage . ReadRowsResponse
import com . google . cloud . bigquery . storage . v1 . DataFormat
import com . google . cloud . bigquery . storage . v1 . stream . ReadSession
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . BigQueryStorage
import akka . stream . scaladsl . Source
import com . google . cloud . bigquery . storage . v1 . stream . ReadSession . TableReadOptions
import scala . concurrent . Future
val sourceOfSources : Source [( ReadSession . Schema , Seq [ Source [ ReadRowsResponse . Rows , NotUsed ]]), Future [ NotUsed ]] =
BigQueryStorage . create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO )
Java
copy source import akka . stream . alpakka . googlecloud . bigquery . storage . BigQueryRecord ;
import akka . stream . alpakka . googlecloud . bigquery . storage . BigQueryStorageSettings ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryArrowStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryAvroStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . BigQueryStorageAttributes ;
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . GrpcBigQueryStorageReader ;
import akka . stream . javadsl . Source ;
import akka . util . ByteString ;
import scala . Tuple2 ;
import com . google . cloud . bigquery . storage . v1 . DataFormat ;
import com . google . cloud . bigquery . storage . v1 . ReadSession ;
import com . google . cloud . bigquery . storage . v1 . storage . ReadRowsResponse ;
import akka . http . javadsl . unmarshalling . Unmarshaller ;
Source <
Tuple2 <
com . google . cloud . bigquery . storage . v1 . stream . ReadSession . Schema ,
List < Source < ReadRowsResponse . Rows , NotUsed >>>,
CompletionStage < NotUsed >>
sourceOfSources =
BigQueryStorage . create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO );
Secondly, by specifying TableReadOptions
, we can narrow down the amount of data returned, filtering down the columns returned, and/or a row_restriction
. This is defined as:
SQL text filtering statement, similar to a WHERE clause in a query. Currently, only a single predicate that is a comparison between a column and a constant value is supported. Aggregates are not supported.
Scala
copy source val readOptions = TableReadOptions ( selectedFields = Seq ( "stringField" , "intField" ), rowRestriction = "intField >= 5" )
val sourceOfSourcesFiltered
: Source [( ReadSession . Schema , Seq [ Source [ ReadRowsResponse . Rows , NotUsed ]]), Future [ NotUsed ]] =
BigQueryStorage . create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO , Some ( readOptions ))
Java
copy source ReadSession . TableReadOptions readOptions =
ReadSession . TableReadOptions . newBuilder ()
. setSelectedFields ( 0 , "stringField" )
. setSelectedFields ( 1 , "intField" )
. setRowRestriction ( "intField >= 5" )
. build ();
Source <
Tuple2 <
com . google . cloud . bigquery . storage . v1 . stream . ReadSession . Schema ,
List < Source < ReadRowsResponse . Rows , NotUsed >>>,
CompletionStage < NotUsed >>
sourceOfSourcesFiltered =
BigQueryStorage . create (
"projectId" , "datasetId" , "tableId" , DataFormat . AVRO , readOptions , 1 );
You can then choose to read and process these streams as is or merged. You can process the streams merged in rows. You need to provide a ByteString
Unmarshaller
based on the format requested.
Scala
copy source implicit val unmarshaller : FromByteStringUnmarshaller [ List [ BigQueryRecord ]] =
mock ( classOf [ FromByteStringUnmarshaller [ List [ BigQueryRecord ]]])
val sequentialSource : Source [ List [ BigQueryRecord ], Future [ NotUsed ]] =
BigQueryStorage . createMergedStreams ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO )
Java
copy source Unmarshaller < ByteString , List < BigQueryRecord >> unmarshaller = null ;
Source < List < BigQueryRecord >, CompletionStage < NotUsed >> sequentialSource =
BigQueryStorage .< List < BigQueryRecord >> createMergedStreams (
"projectId" , "datasetId" , "tableId" , DataFormat . AVRO , unmarshaller );
Or process the stream of rows individually:
Scala
copy source import akka . NotUsed
import com . google . cloud . bigquery . storage . v1 . storage . ReadRowsResponse
import com . google . cloud . bigquery . storage . v1 . DataFormat
import com . google . cloud . bigquery . storage . v1 . stream . ReadSession
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . BigQueryStorage
import akka . stream . scaladsl . Source
import com . google . cloud . bigquery . storage . v1 . stream . ReadSession . TableReadOptions
import scala . concurrent . Future
val sourceOfSources : Source [( ReadSession . Schema , Seq [ Source [ ReadRowsResponse . Rows , NotUsed ]]), Future [ NotUsed ]] =
BigQueryStorage . create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO )
Java
copy source import akka . stream . alpakka . googlecloud . bigquery . storage . BigQueryRecord ;
import akka . stream . alpakka . googlecloud . bigquery . storage . BigQueryStorageSettings ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryArrowStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryAvroStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . javadsl . BigQueryStorage ;
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . BigQueryStorageAttributes ;
import akka . stream . alpakka . googlecloud . bigquery . storage . scaladsl . GrpcBigQueryStorageReader ;
import akka . stream . javadsl . Source ;
import akka . util . ByteString ;
import scala . Tuple2 ;
import com . google . cloud . bigquery . storage . v1 . DataFormat ;
import com . google . cloud . bigquery . storage . v1 . ReadSession ;
import com . google . cloud . bigquery . storage . v1 . storage . ReadRowsResponse ;
import akka . http . javadsl . unmarshalling . Unmarshaller ;
Source <
Tuple2 <
com . google . cloud . bigquery . storage . v1 . stream . ReadSession . Schema ,
List < Source < ReadRowsResponse . Rows , NotUsed >>>,
CompletionStage < NotUsed >>
sourceOfSources =
BigQueryStorage . create ( "projectId" , "datasetId" , "tableId" , DataFormat . AVRO );
Since Avro and Arrow are the formats available, streams for those specific formats can be created.
You can read Arrow Record streams merged
Scala
copy source val arrowSequentialSource : Source [ Seq [ BigQueryRecord ], Future [ NotUsed ]] =
BigQueryArrowStorage . readRecordsMerged ( "projectId" , "datasetId" , "tableId" )
Java
copy source Source < List < BigQueryRecord >, CompletionStage < NotUsed >> arrowSequentialSource =
BigQueryArrowStorage . readRecordsMerged ( "projectId" , "datasetId" , "tableId" );
You can read Arrow Record streams individually
Scala
copy source val arrowParallelSource : Source [ Seq [ Source [ BigQueryRecord , NotUsed ]], Future [ NotUsed ]] =
BigQueryArrowStorage . readRecords ( "projectId" , "datasetId" , "tableId" )
Java
copy source Source < List < Source < BigQueryRecord , NotUsed >>, CompletionStage < NotUsed >> arrowParallelSource =
BigQueryArrowStorage . readRecords ( "projectId" , "datasetId" , "tableId" );
You can read Avro Record streams merged
Scala
copy source val avroSequentialSource : Source [ Seq [ BigQueryRecord ], Future [ NotUsed ]] =
BigQueryAvroStorage . readRecordsMerged ( "projectId" , "datasetId" , "tableId" )
Java
copy source Source < List < BigQueryRecord >, CompletionStage < NotUsed >> avroSequentialSource =
BigQueryAvroStorage . readRecordsMerged ( "projectId" , "datasetId" , "tableId" );
You can read Avro Record streams individually
Scala
copy source val avroParallelSource : Source [ Seq [ Source [ BigQueryRecord , NotUsed ]], Future [ NotUsed ]] =
BigQueryAvroStorage . readRecords ( "projectId" , "datasetId" , "tableId" )
Java
copy source Source < List < Source < BigQueryRecord , NotUsed >>, CompletionStage < NotUsed >> avroParallelSource =
BigQueryAvroStorage . readRecords ( "projectId" , "datasetId" , "tableId" );
Running the test code
The tests use a BigQueryMockServer
that implements the server defined in the protobuf for the Storage API. It essentially provides a mock table on which to query. Tests can be started from sbt by running:
sbt
> google - cloud - bigquery - storage / test