Hadoop Distributed File System - HDFS

The connector offers Flows and Sources that interact with HDFS file systems.

For more information about Hadoop, please visit the Hadoop documentation.

Project Info: Alpakka HDFS
Artifact
com.lightbend.akka
akka-stream-alpakka-hdfs
1.1.2
JDK versions
OpenJDK 8
Scala versions2.12.7, 2.11.12
JPMS module nameakka.stream.alpakka.hdfs
License
Readiness level
Since 0.20, 2018-07-04
Home pagehttps://doc.akka.io/docs/alpakka/current
API documentation
Forums
Release notesIn the documentation
IssuesGithub issues
Sourceshttps://github.com/akka/alpakka

Artifacts

sbt
libraryDependencies += "com.lightbend.akka" %% "akka-stream-alpakka-hdfs" % "1.1.2"
Maven
<dependency>
  <groupId>com.lightbend.akka</groupId>
  <artifactId>akka-stream-alpakka-hdfs_2.12</artifactId>
  <version>1.1.2</version>
</dependency>
Gradle
dependencies {
  compile group: 'com.lightbend.akka', name: 'akka-stream-alpakka-hdfs_2.12', version: '1.1.2'
}

The table below shows direct dependencies of this module and the second tab shows all libraries it depends on transitively.

Direct dependencies
OrganizationArtifactVersionLicense
com.typesafe.akkaakka-stream_2.122.5.23Apache License, Version 2.0
org.apache.hadoophadoop-client3.1.1Apache License, Version 2.0
org.scala-langscala-library2.12.7BSD 3-Clause
org.typelevelcats-core_2.121.6.0MIT
Dependency tree
com.typesafe.akka    akka-stream_2.12    2.5.23    Apache License, Version 2.0
    com.typesafe.akka    akka-actor_2.12    2.5.23    Apache License, Version 2.0
        com.typesafe    config    1.3.3    Apache License, Version 2.0
        org.scala-lang.modules    scala-java8-compat_2.12    0.8.0    BSD 3-clause
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    com.typesafe.akka    akka-protobuf_2.12    2.5.23    Apache License, Version 2.0
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    com.typesafe    ssl-config-core_2.12    0.3.7    Apache-2.0
        com.typesafe    config    1.3.3    Apache License, Version 2.0
        org.scala-lang.modules    scala-parser-combinators_2.12    1.1.1    BSD 3-clause
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.reactivestreams    reactive-streams    1.0.2    CC0
    org.scala-lang    scala-library    2.12.7    BSD 3-Clause
org.apache.hadoop    hadoop-client    3.1.1    Apache License, Version 2.0
    org.apache.hadoop    hadoop-annotations    3.1.1    Apache License, Version 2.0
    org.apache.hadoop    hadoop-common    3.1.1    Apache License, Version 2.0
        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
        com.fasterxml.woodstox    woodstox-core    5.0.3    The Apache License, Version 2.0
            org.codehaus.woodstox    stax2-api    3.1.4    The BSD License
        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
        com.google.code.gson    gson    2.2.4    The Apache Software License, Version 2.0
        com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
            com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
        com.google.protobuf    protobuf-java    2.5.0    New BSD license
        com.google.re2j    re2j    1.1    The Go license
        com.sun.jersey    jersey-servlet    1.19    CDDL 1.1
        commons-beanutils    commons-beanutils    1.9.3    Apache License, Version 2.0
            commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
        commons-cli    commons-cli    1.2    The Apache Software License, Version 2.0
        commons-codec    commons-codec    1.11    Apache License, Version 2.0
        commons-collections    commons-collections    3.2.2    Apache License, Version 2.0
        commons-io    commons-io    2.5    Apache License, Version 2.0
        commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
        commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
        commons-net    commons-net    3.6    Apache License, Version 2.0
        javax.servlet.jsp    jsp-api    2.1
        org.apache.avro    avro    1.7.7    The Apache Software License, Version 2.0
            com.thoughtworks.paranamer    paranamer    2.3    BSD
            org.apache.commons    commons-compress    1.4.1    The Apache Software License, Version 2.0
                org.tukaani    xz    1.0    Public Domain
            org.codehaus.jackson    jackson-core-asl    1.9.13    The Apache Software License, Version 2.0
            org.codehaus.jackson    jackson-mapper-asl    1.9.13    The Apache Software License, Version 2.0
                org.codehaus.jackson    jackson-core-asl    1.9.13    The Apache Software License, Version 2.0
            org.slf4j    slf4j-api    1.7.25    MIT License
            org.xerial.snappy    snappy-java    1.0.5    The Apache Software License, Version 2.0
        org.apache.commons    commons-compress    1.4.1    The Apache Software License, Version 2.0
            org.tukaani    xz    1.0    Public Domain
        org.apache.commons    commons-configuration2    2.1.1    Apache License, Version 2.0
            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
        org.apache.commons    commons-lang3    3.4    Apache License, Version 2.0
        org.apache.commons    commons-math3    3.1.1    The Apache Software License, Version 2.0
        org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
            com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
            org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.curator    curator-recipes    2.12.0    The Apache Software License, Version 2.0
            org.apache.curator    curator-framework    2.12.0    The Apache Software License, Version 2.0
                org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
                    com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.hadoop    hadoop-annotations    3.1.1    Apache License, Version 2.0
        org.apache.hadoop    hadoop-auth    3.1.1    Apache License, Version 2.0
            com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
            com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                    net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                        org.ow2.asm    asm    5.0.4    BSD
            commons-codec    commons-codec    1.11    Apache License, Version 2.0
            net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                    org.ow2.asm    asm    5.0.4    BSD
            org.apache.curator    curator-framework    2.12.0    The Apache Software License, Version 2.0
                org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
                    com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.httpcomponents    httpclient    4.5.2    Apache License, Version 2.0
                commons-codec    commons-codec    1.11    Apache License, Version 2.0
                commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                org.apache.httpcomponents    httpcore    4.4.4    Apache License, Version 2.0
            org.apache.kerby    kerb-simplekdc    1.0.1    Apache License, Version 2.0
                org.apache.kerby    kerb-admin    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-server    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                            commons-io    commons-io    2.5    Apache License, Version 2.0
                            org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-identity    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerby-xdr    1.0.1    Apache License, Version 2.0
                org.apache.kerby    kerb-client    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                        commons-io    commons-io    2.5    Apache License, Version 2.0
                        org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    token-provider    1.0.1    Apache License, Version 2.0
                        com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                            com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                            net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                                net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                    org.ow2.asm    asm    5.0.4    BSD
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
            org.slf4j    slf4j-api    1.7.25    MIT License
        org.apache.htrace    htrace-core4    4.1.0-incubating    The Apache Software License, Version 2.0
        org.apache.httpcomponents    httpclient    4.5.2    Apache License, Version 2.0
            commons-codec    commons-codec    1.11    Apache License, Version 2.0
            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
            org.apache.httpcomponents    httpcore    4.4.4    Apache License, Version 2.0
        org.apache.kerby    kerb-simplekdc    1.0.1    Apache License, Version 2.0
            org.apache.kerby    kerb-admin    1.0.1    Apache License, Version 2.0
                org.apache.kerby    kerb-server    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                        commons-io    commons-io    2.5    Apache License, Version 2.0
                        org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-identity    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    kerby-xdr    1.0.1    Apache License, Version 2.0
            org.apache.kerby    kerb-client    1.0.1    Apache License, Version 2.0
                org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                    commons-io    commons-io    2.5    Apache License, Version 2.0
                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.kerby    token-provider    1.0.1    Apache License, Version 2.0
                    com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                        com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                        net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                            net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                org.ow2.asm    asm    5.0.4    BSD
                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
        org.codehaus.woodstox    stax2-api    3.1.4    The BSD License
        org.eclipse.jetty    jetty-servlet    9.3.19.v20170502    Apache Software License - Version 2.0
            org.eclipse.jetty    jetty-security    9.3.19.v20170502    Apache Software License - Version 2.0
        org.eclipse.jetty    jetty-webapp    9.3.19.v20170502    Apache Software License - Version 2.0
            org.eclipse.jetty    jetty-servlet    9.3.19.v20170502    Apache Software License - Version 2.0
                org.eclipse.jetty    jetty-security    9.3.19.v20170502    Apache Software License - Version 2.0
            org.eclipse.jetty    jetty-xml    9.3.19.v20170502    Apache Software License - Version 2.0
        org.slf4j    slf4j-api    1.7.25    MIT License
    org.apache.hadoop    hadoop-hdfs-client    3.1.1    Apache License, Version 2.0
        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
        com.squareup.okhttp    okhttp    2.7.5    Apache 2.0
            com.squareup.okio    okio    1.6.0    Apache 2.0
    org.apache.hadoop    hadoop-mapreduce-client-core    3.1.1    Apache License, Version 2.0
        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
        com.google.protobuf    protobuf-java    2.5.0    New BSD license
        org.apache.hadoop    hadoop-hdfs-client    3.1.1    Apache License, Version 2.0
            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
            com.squareup.okhttp    okhttp    2.7.5    Apache 2.0
                com.squareup.okio    okio    1.6.0    Apache 2.0
        org.apache.hadoop    hadoop-yarn-client    3.1.1    Apache License, Version 2.0
            commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
        org.apache.hadoop    hadoop-yarn-common    3.1.1    Apache License, Version 2.0
            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.jaxrs    jackson-jaxrs-json-provider    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.jaxrs    jackson-jaxrs-base    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
            com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
            com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
            com.google.protobuf    protobuf-java    2.5.0    New BSD license
            com.sun.jersey    jersey-client    1.19    CDDL 1.1
            com.sun.jersey    jersey-core    1.19    CDDL 1.1
                javax.ws.rs    jsr311-api    1.1.1                     CDDL License             
            commons-cli    commons-cli    1.2    The Apache Software License, Version 2.0
            commons-codec    commons-codec    1.11    Apache License, Version 2.0
            commons-io    commons-io    2.5    Apache License, Version 2.0
            commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
            javax.servlet    javax.servlet-api    3.1.0    CDDL + GPLv2 with classpath exception
            javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
            org.apache.commons    commons-compress    1.4.1    The Apache Software License, Version 2.0
                org.tukaani    xz    1.0    Public Domain
            org.apache.hadoop    hadoop-auth    3.1.1    Apache License, Version 2.0
                com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                    com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                    com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                    net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                        net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                            org.ow2.asm    asm    5.0.4    BSD
                commons-codec    commons-codec    1.11    Apache License, Version 2.0
                net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                    net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                        org.ow2.asm    asm    5.0.4    BSD
                org.apache.curator    curator-framework    2.12.0    The Apache Software License, Version 2.0
                    org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
                        com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                            com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                        org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.httpcomponents    httpclient    4.5.2    Apache License, Version 2.0
                    commons-codec    commons-codec    1.11    Apache License, Version 2.0
                    commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                    org.apache.httpcomponents    httpcore    4.4.4    Apache License, Version 2.0
                org.apache.kerby    kerb-simplekdc    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-admin    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-server    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                                commons-io    commons-io    2.5    Apache License, Version 2.0
                                org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-identity    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-xdr    1.0.1    Apache License, Version 2.0
                    org.apache.kerby    kerb-client    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                            commons-io    commons-io    2.5    Apache License, Version 2.0
                            org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.kerby    token-provider    1.0.1    Apache License, Version 2.0
                            com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                                com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                                net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                                    net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                        org.ow2.asm    asm    5.0.4    BSD
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.hadoop    hadoop-yarn-api    3.1.1    Apache License, Version 2.0
                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                    com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                com.google.protobuf    protobuf-java    2.5.0    New BSD license
                commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
            org.eclipse.jetty    jetty-util    9.3.19.v20170502    Apache Software License - Version 2.0
            org.slf4j    slf4j-api    1.7.25    MIT License
        org.slf4j    slf4j-api    1.7.25    MIT License
    org.apache.hadoop    hadoop-mapreduce-client-jobclient    3.1.1    Apache License, Version 2.0
        com.google.protobuf    protobuf-java    2.5.0    New BSD license
        org.apache.hadoop    hadoop-mapreduce-client-common    3.1.1    Apache License, Version 2.0
            com.google.protobuf    protobuf-java    2.5.0    New BSD license
            org.apache.hadoop    hadoop-mapreduce-client-core    3.1.1    Apache License, Version 2.0
                com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.google.protobuf    protobuf-java    2.5.0    New BSD license
                org.apache.hadoop    hadoop-hdfs-client    3.1.1    Apache License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.squareup.okhttp    okhttp    2.7.5    Apache 2.0
                        com.squareup.okio    okio    1.6.0    Apache 2.0
                org.apache.hadoop    hadoop-yarn-client    3.1.1    Apache License, Version 2.0
                    commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                    commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                org.apache.hadoop    hadoop-yarn-common    3.1.1    Apache License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.jaxrs    jackson-jaxrs-json-provider    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.jaxrs    jackson-jaxrs-base    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                                com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                    com.google.protobuf    protobuf-java    2.5.0    New BSD license
                    com.sun.jersey    jersey-client    1.19    CDDL 1.1
                    com.sun.jersey    jersey-core    1.19    CDDL 1.1
                        javax.ws.rs    jsr311-api    1.1.1                     CDDL License             
                    commons-cli    commons-cli    1.2    The Apache Software License, Version 2.0
                    commons-codec    commons-codec    1.11    Apache License, Version 2.0
                    commons-io    commons-io    2.5    Apache License, Version 2.0
                    commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                    commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                    javax.servlet    javax.servlet-api    3.1.0    CDDL + GPLv2 with classpath exception
                    javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
                    org.apache.commons    commons-compress    1.4.1    The Apache Software License, Version 2.0
                        org.tukaani    xz    1.0    Public Domain
                    org.apache.hadoop    hadoop-auth    3.1.1    Apache License, Version 2.0
                        com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                            com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                        com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                            com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                            net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                                net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                    org.ow2.asm    asm    5.0.4    BSD
                        commons-codec    commons-codec    1.11    Apache License, Version 2.0
                        net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                            net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                org.ow2.asm    asm    5.0.4    BSD
                        org.apache.curator    curator-framework    2.12.0    The Apache Software License, Version 2.0
                            org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
                                com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                                    com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                        org.apache.httpcomponents    httpclient    4.5.2    Apache License, Version 2.0
                            commons-codec    commons-codec    1.11    Apache License, Version 2.0
                            commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                            org.apache.httpcomponents    httpcore    4.4.4    Apache License, Version 2.0
                        org.apache.kerby    kerb-simplekdc    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-admin    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-server    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                                        commons-io    commons-io    2.5    Apache License, Version 2.0
                                        org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                    org.slf4j    slf4j-api    1.7.25    MIT License
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerb-identity    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                org.slf4j    slf4j-api    1.7.25    MIT License
                                        org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                org.slf4j    slf4j-api    1.7.25    MIT License
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-xdr    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-client    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                                    commons-io    commons-io    2.5    Apache License, Version 2.0
                                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                org.slf4j    slf4j-api    1.7.25    MIT License
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                org.slf4j    slf4j-api    1.7.25    MIT License
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    token-provider    1.0.1    Apache License, Version 2.0
                                    com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                                        com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                                        net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                                            net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                                org.ow2.asm    asm    5.0.4    BSD
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                        org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.hadoop    hadoop-yarn-api    3.1.1    Apache License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                            com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                        com.google.protobuf    protobuf-java    2.5.0    New BSD license
                        commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                        commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                        javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
                    org.eclipse.jetty    jetty-util    9.3.19.v20170502    Apache Software License - Version 2.0
                    org.slf4j    slf4j-api    1.7.25    MIT License
                org.slf4j    slf4j-api    1.7.25    MIT License
            org.apache.hadoop    hadoop-yarn-client    3.1.1    Apache License, Version 2.0
                commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
            org.apache.hadoop    hadoop-yarn-common    3.1.1    Apache License, Version 2.0
                com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.jaxrs    jackson-jaxrs-json-provider    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.jaxrs    jackson-jaxrs-base    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                            com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.fasterxml.jackson.module    jackson-module-jaxb-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                    com.fasterxml.jackson.core    jackson-databind    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                        com.fasterxml.jackson.core    jackson-core    2.7.8    The Apache Software License, Version 2.0
                com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                    com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                com.google.protobuf    protobuf-java    2.5.0    New BSD license
                com.sun.jersey    jersey-client    1.19    CDDL 1.1
                com.sun.jersey    jersey-core    1.19    CDDL 1.1
                    javax.ws.rs    jsr311-api    1.1.1                     CDDL License             
                commons-cli    commons-cli    1.2    The Apache Software License, Version 2.0
                commons-codec    commons-codec    1.11    Apache License, Version 2.0
                commons-io    commons-io    2.5    Apache License, Version 2.0
                commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                javax.servlet    javax.servlet-api    3.1.0    CDDL + GPLv2 with classpath exception
                javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
                org.apache.commons    commons-compress    1.4.1    The Apache Software License, Version 2.0
                    org.tukaani    xz    1.0    Public Domain
                org.apache.hadoop    hadoop-auth    3.1.1    Apache License, Version 2.0
                    com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                    com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                        com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                        net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                            net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                org.ow2.asm    asm    5.0.4    BSD
                    commons-codec    commons-codec    1.11    Apache License, Version 2.0
                    net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                        net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                            org.ow2.asm    asm    5.0.4    BSD
                    org.apache.curator    curator-framework    2.12.0    The Apache Software License, Version 2.0
                        org.apache.curator    curator-client    2.12.0    The Apache Software License, Version 2.0
                            com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                                com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                            org.slf4j    slf4j-api    1.7.25    MIT License
                    org.apache.httpcomponents    httpclient    4.5.2    Apache License, Version 2.0
                        commons-codec    commons-codec    1.11    Apache License, Version 2.0
                        commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                        org.apache.httpcomponents    httpcore    4.4.4    Apache License, Version 2.0
                    org.apache.kerby    kerb-simplekdc    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-admin    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-server    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                                    commons-io    commons-io    2.5    Apache License, Version 2.0
                                    org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                                org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                                org.slf4j    slf4j-api    1.7.25    MIT License
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-identity    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-xdr    1.0.1    Apache License, Version 2.0
                        org.apache.kerby    kerb-client    1.0.1    Apache License, Version 2.0
                            org.apache.kerby    kerb-common    1.0.1    Apache License, Version 2.0
                                commons-io    commons-io    2.5    Apache License, Version 2.0
                                org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerb-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                                org.apache.kerby    kerb-crypto    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                            org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                            org.slf4j    slf4j-api    1.7.25    MIT License
                                    org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                    org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    kerby-config    1.0.1    Apache License, Version 2.0
                                org.slf4j    slf4j-api    1.7.25    MIT License
                            org.apache.kerby    token-provider    1.0.1    Apache License, Version 2.0
                                com.nimbusds    nimbus-jose-jwt    4.41.1    The Apache Software License, Version 2.0
                                    com.github.stephenc.jcip    jcip-annotations    1.0-1    Apache License, Version 2.0
                                    net.minidev    json-smart    2.3    The Apache Software License, Version 2.0
                                        net.minidev    accessors-smart    1.2    The Apache Software License, Version 2.0
                                            org.ow2.asm    asm    5.0.4    BSD
                                org.apache.kerby    kerb-core    1.0.1    Apache License, Version 2.0
                                    org.apache.kerby    kerby-pkix    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-asn1    1.0.1    Apache License, Version 2.0
                                        org.apache.kerby    kerby-util    1.0.1    Apache License, Version 2.0
                                        org.slf4j    slf4j-api    1.7.25    MIT License
                    org.slf4j    slf4j-api    1.7.25    MIT License
                org.apache.hadoop    hadoop-yarn-api    3.1.1    Apache License, Version 2.0
                    com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
                    com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
                        com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
                    com.google.protobuf    protobuf-java    2.5.0    New BSD license
                    commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
                    commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
                    javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
                org.eclipse.jetty    jetty-util    9.3.19.v20170502    Apache Software License - Version 2.0
                org.slf4j    slf4j-api    1.7.25    MIT License
            org.slf4j    slf4j-api    1.7.25    MIT License
        org.slf4j    slf4j-api    1.7.25    MIT License
    org.apache.hadoop    hadoop-yarn-api    3.1.1    Apache License, Version 2.0
        com.fasterxml.jackson.core    jackson-annotations    2.7.8    The Apache Software License, Version 2.0
        com.google.guava    guava    11.0.2    The Apache Software License, Version 2.0
            com.google.code.findbugs    jsr305    3.0.0    The Apache Software License, Version 2.0
        com.google.protobuf    protobuf-java    2.5.0    New BSD license
        commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
        commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
        javax.xml.bind    jaxb-api    2.2.11    CDDL 1.1
    org.apache.hadoop    hadoop-yarn-client    3.1.1    Apache License, Version 2.0
        commons-lang    commons-lang    2.6    The Apache Software License, Version 2.0
        commons-logging    commons-logging    1.1.3    The Apache Software License, Version 2.0
org.scala-lang    scala-library    2.12.7    BSD 3-Clause
org.typelevel    cats-core_2.12    1.6.0    MIT
    org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.typelevel    cats-kernel_2.12    1.6.0    MIT
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.typelevel    cats-macros_2.12    1.6.0    MIT
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.typelevel    machinist_2.12    0.6.6    MIT
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause
            org.scala-lang    scala-reflect    2.12.7    BSD 3-Clause
                org.scala-lang    scala-library    2.12.7    BSD 3-Clause
    org.typelevel    machinist_2.12    0.6.6    MIT
        org.scala-lang    scala-library    2.12.7    BSD 3-Clause
        org.scala-lang    scala-reflect    2.12.7    BSD 3-Clause
            org.scala-lang    scala-library    2.12.7    BSD 3-Clause

Specifying a Hadoop Version

By default, HDFS connector uses Hadoop 3.1.1. If you are using a different version of Hadoop, you should exclude the Hadoop libraries from the connector dependency and add the dependency for your preferred version.

Set up client

Flows provided by this connector need a prepared org.apache.hadoop.fs.FileSystem to interact with HDFS.

Scala
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem

val conf = new Configuration()
conf.set("fs.default.name", "hdfs://localhost:54310")

val fs: FileSystem = FileSystem.get(conf)
Java
Configuration conf = new Configuration();
conf.set("fs.default.name", "hdfs://localhost:54310");

fs = FileSystem.get(conf);

Writing

The connector provides three Flows. Each flow requires RotationStrategy and SyncStrategy to run. HdfsFlow. HdfsFlow.

The flows push OutgoingMessage to a downstream.

Data Writer

Use HdfsFlow.data to stream with FSDataOutputStream without any compression.

Scala
val flow = HdfsFlow.data(
  fs,
  SyncStrategy.count(500),
  RotationStrategy.size(1, FileUnit.GB),
  HdfsWritingSettings()
)
Java
Flow<HdfsWriteMessage<ByteString, NotUsed>, RotationMessage, NotUsed> flow =
    HdfsFlow.data(
        fs, SyncStrategy.count(500), RotationStrategy.size(1, FileUnit.GB()), settings);

Compressed Data Writer

First, create CompressionCodec.

Scala
val codec = new DefaultCodec()
codec.setConf(fs.getConf)
Java
DefaultCodec codec = new DefaultCodec();
codec.setConf(fs.getConf());

Then, use HdfsFlow.compress to stream with CompressionOutputStream and CompressionCodec.

Scala
val flow = HdfsFlow.compressed(
  fs,
  SyncStrategy.count(1),
  RotationStrategy.size(0.1, FileUnit.MB),
  codec,
  settings
)
Java
Flow<HdfsWriteMessage<ByteString, NotUsed>, RotationMessage, NotUsed> flow =
    HdfsFlow.compressed(
        fs, SyncStrategy.count(50), RotationStrategy.size(0.1, FileUnit.MB()), codec, settings);

Sequence Writer

Use HdfsFlow.sequence to stream a flat file consisting of binary key/value pairs.

Without Compression

Scala
val flow = HdfsFlow.sequence(
  fs,
  SyncStrategy.none,
  RotationStrategy.size(1, FileUnit.MB),
  settings,
  classOf[Text],
  classOf[Text]
)
Java
Flow<HdfsWriteMessage<Pair<Text, Text>, NotUsed>, RotationMessage, NotUsed> flow =
    HdfsFlow.sequence(
        fs,
        SyncStrategy.none(),
        RotationStrategy.size(1, FileUnit.MB()),
        SequenceFile.CompressionType.BLOCK,
        codec,
        settings,
        Text.class,
        Text.class);

With Compression

First, define a codec.

Scala
val codec = new DefaultCodec()
codec.setConf(fs.getConf)
Java
DefaultCodec codec = new DefaultCodec();
codec.setConf(fs.getConf());

Then, create a flow.

Scala
val flow = HdfsFlow.sequence(
  fs,
  SyncStrategy.none,
  RotationStrategy.size(1, FileUnit.MB),
  CompressionType.BLOCK,
  codec,
  settings,
  classOf[Text],
  classOf[Text]
)
Java
Flow<HdfsWriteMessage<Pair<Text, Text>, NotUsed>, RotationMessage, NotUsed> flow =
    HdfsFlow.sequence(
        fs,
        SyncStrategy.none(),
        RotationStrategy.size(1, FileUnit.MB()),
        settings,
        Text.class,
        Text.class);

Passing data through HdfsFlow

Use HdfsFlow.dataWithPassThrough, HdfsFlow.compressedWithPassThrough or HdfsFlow.sequenceWithPassThrough.

When streaming documents from Kafka, you might want to commit to Kafka. The flow will emit two messages. For every input, it will produce WrittenMessage and when it rotates, RotationMessage.

Let’s say that we have these classes.

Scala
case class Book(title: String)
case class KafkaOffset(offset: Int)
case class KafkaMessage(book: Book, offset: KafkaOffset)
Java
public static class Book {
  final String title;

  Book(String title) {
    this.title = title;
  }
}

static class KafkaCommitter {
  List<Integer> committedOffsets = new ArrayList<>();

  void commit(KafkaOffset offset) {
    committedOffsets.add(offset.offset);
  }
}

static class KafkaOffset {
  final int offset;

  KafkaOffset(int offset) {
    this.offset = offset;
  }
}

static class KafkaMessage {
  final Book book;
  final KafkaOffset offset;

  KafkaMessage(Book book, KafkaOffset offset) {
    this.book = book;
    this.offset = offset;
  }
}

Then, we can stream with passThrough.

Scala
// We're going to pretend we got messages from kafka.
// After we've written them to HDFS, we want
// to commit the offset to Kafka
val messagesFromKafka = List(
  KafkaMessage(Book("Akka Concurrency"), KafkaOffset(0)),
  KafkaMessage(Book("Akka in Action"), KafkaOffset(1)),
  KafkaMessage(Book("Effective Akka"), KafkaOffset(2)),
  KafkaMessage(Book("Learning Scala"), KafkaOffset(3)),
  KafkaMessage(Book("Scala Puzzlers"), KafkaOffset(4)),
  KafkaMessage(Book("Scala for Spark in Production"), KafkaOffset(5))
)

var committedOffsets = List[KafkaOffset]()

def commitToKafka(offset: KafkaOffset): Unit =
  committedOffsets = committedOffsets :+ offset

val resF = Source(messagesFromKafka)
  .map { kafkaMessage: KafkaMessage =>
    val book = kafkaMessage.book
    // Transform message so that we can write to hdfs
    HdfsWriteMessage(ByteString(book.title), kafkaMessage.offset)
  }
  .via(
    HdfsFlow.dataWithPassThrough[KafkaOffset](
      fs,
      SyncStrategy.count(50),
      RotationStrategy.count(4),
      HdfsWritingSettings().withNewLine(true)
    )
  )
  .map { message =>
    message match {
      case WrittenMessage(passThrough, _) =>
        commitToKafka(passThrough)
      case _ => ()
    }
    message
  }
  .collect {
    case rm: RotationMessage => rm
  }
  .runWith(Sink.seq)
Java
// We're going to pretend we got messages from kafka.
// After we've written them to HDFS, we want
// to commit the offset to Kafka
List<KafkaMessage> messagesFromKafka =
    Arrays.asList(
        new KafkaMessage(new Book("Akka Concurrency"), new KafkaOffset(0)),
        new KafkaMessage(new Book("Akka in Action"), new KafkaOffset(1)),
        new KafkaMessage(new Book("Effective Akka"), new KafkaOffset(2)),
        new KafkaMessage(new Book("Learning Scala"), new KafkaOffset(3)),
        new KafkaMessage(new Book("Scala Puzzlers"), new KafkaOffset(4)),
        new KafkaMessage(new Book("Scala for Spark in Production"), new KafkaOffset(5)));

final KafkaCommitter kafkaCommitter = new KafkaCommitter();

Flow<HdfsWriteMessage<ByteString, KafkaOffset>, OutgoingMessage<KafkaOffset>, NotUsed> flow =
    HdfsFlow.dataWithPassThrough(
        fs,
        SyncStrategy.count(50),
        RotationStrategy.count(4),
        HdfsWritingSettings.create().withNewLine(true));

CompletionStage<List<RotationMessage>> resF =
    Source.from(messagesFromKafka)
        .map(
            kafkaMessage -> {
              Book book = kafkaMessage.book;
              // Transform message so that we can write to hdfs\
              return HdfsWriteMessage.create(
                  ByteString.fromString(book.title), kafkaMessage.offset);
            })
        .via(flow)
        .map(
            message -> {
              if (message instanceof WrittenMessage) {
                kafkaCommitter.commit(((WrittenMessage<KafkaOffset>) message).passThrough());
                return message;
              } else {
                return message;
              }
            })
        .collectType(RotationMessage.class) // Collect only rotation messages
        .runWith(Sink.seq(), materializer);

Configuration

We can configure the sink by HdfsWritingSettings.

Scala
val settings =
  HdfsWritingSettings()
    .withOverwrite(true)
    .withNewLine(false)
    .withLineSeparator(System.getProperty("line.separator"))
    .withPathGenerator(pathGenerator)
Java
HdfsWritingSettings.create()
    .withOverwrite(true)
    .withNewLine(false)
    .withLineSeparator(System.getProperty("line.separator"))
    .withPathGenerator(pathGenerator);

File path generator

FilePathGenerator provides a functionality to generate rotation path in HDFS.

Scala
val pathGenerator =
  FilePathGenerator(
    (rotationCount: Long, timestamp: Long) => s"/tmp/alpakka/$rotationCount-$timestamp"
  )
Java
BiFunction<Long, Long, String> func =
    (rotationCount, timestamp) -> "/tmp/alpakka/" + rotationCount + "-" + timestamp;
FilePathGenerator pathGenerator = FilePathGenerator.create(func);

Rotation Strategy

RotationStrategy provides a functionality to decide when to rotate files.

Sync Strategy

SyncStrategy provides a functionality to decide when to synchronize the output.

Reading

Use HdfsSource to read from HDFS. HdfsSource. HdfsSource.

Data Reader

Scala
val source = HdfsSource.data(fs, path)
Java
Source<ByteString, CompletionStage<IOResult>> source = HdfsSource.data(fs, path);

Compressed Data Reader

Scala
val source = HdfsSource.compressed(fs, path, codec)
Java
Source<ByteString, CompletionStage<IOResult>> source = HdfsSource.compressed(fs, path, codec);

Sequence Reader

Scala
val source = HdfsSource.sequence(fs, path, classOf[Text], classOf[Text])
Java
Source<Pair<Text, Text>, NotUsed> source =
    HdfsSource.sequence(fs, path, Text.class, Text.class);

Running the example code

The code in this guide is part of runnable tests of this project. You are welcome to edit the code and run it in sbt.

Scala
sbt
> hdfs/testOnly *.HdfsWriterSpec
> hdfs/testOnly *.HdfsReaderSpec
Java
sbt
> hdfs/testOnly *.HdfsWriterTest
> hdfs/testOnly *.HdfsReaderTest
Found an error in this documentation? The source code for this page can be found here. Please feel free to edit and contribute a pull request.