To those users it comes as a surprise that Avro actually ships with exactly such command line tools but apparently they are not prominently advertised or documented as such. In this short article I will show a few hands-on examples on how to read, write, compress and convert data from and to binary Avro using Avro Tools 1.7.4.

What we want to do

Here is an overview of what we want to do:

• We will start with an example Avro schema and a corresponding data file in plain-text JSON format.
• We will use Avro Tools to convert the JSON file into binary Avro, without and with compression (Snappy), and from binary Avro back to JSON.

Getting Avro Tools

You can get a copy of the latest stable Avro Tools jar file from the Avro Releases page. The actual file is in the java subdirectory of a given Avro release version. Here is a direct link to avro-tools-1.7.4.jar (11 MB) on the US Apache mirror site.

Save avro-tools-1.7.4.jar to a directory of your choice. I will use ~/avro-tools-1.7.4.jar for the examples shown below.

Tools included in Avro Tools

Just run Avro Tools without any parameters to see what’s included:

$ java -jar ~/avro-tools-1.7.4.jar
[...snip...]
Available tools:
      compile  Generates Java code for the given schema.
       concat  Concatenates avro files without re-compressing.
   fragtojson  Renders a binary-encoded Avro datum as JSON.
     fromjson  Reads JSON records and writes an Avro data file.
     fromtext  Imports a text file into an avro data file.
      getmeta  Prints out the metadata of an Avro data file.
    getschema  Prints out schema of an Avro data file.
          idl  Generates a JSON schema from an Avro IDL file
       induce  Induce schema/protocol from Java class/interface via reflection.
   jsontofrag  Renders a JSON-encoded Avro datum as binary.
      recodec  Alters the codec of a data file.
  rpcprotocol  Output the protocol of a RPC service
   rpcreceive  Opens an RPC Server and listens for one message.
      rpcsend  Sends a single RPC message.
       tether  Run a tethered mapreduce job.
       tojson  Dumps an Avro data file as JSON, one record per line.
       totext  Converts an Avro data file to a text file.
  trevni_meta  Dumps a Trevni file's metadata as JSON.
trevni_random  Create a Trevni file filled with random instances of a schema.
trevni_tojson  Dumps a Trevni file as JSON.

Likewise run any particular tool without parameters to see its usage/help output. For example, here is the help of the fromjson tool:

$ java -jar ~/avro-tools-1.7.4.jar fromjson
Expected 1 arg: input_file
Option                                  Description
------                                  -----------
--codec                                 Compression codec (default: null)
--schema                                Schema
--schema-file                           Schema File

Note that most of the tools write to STDOUT, so normally you would like to pipe their output to a file via the Bash > redirection operator (particularly when working with large files).

Example data

In the next sections I will use the following example data to demonstrate Avro Tools. You can also download the example files from https://github.com/miguno/avro-cli-examples.

Avro schema

The schema below defines a tuple of (username, tweet, timestamp) as the format of our example data records.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

{
  "type" : "record",
  "name" : "twitter_schema",
  "namespace" : "com.miguno.avro",
  "fields" : [ {
    "name" : "username",
    "type" : "string",
    "doc"  : "Name of the user account on Twitter.com"
  }, {
    "name" : "tweet",
    "type" : "string",
    "doc"  : "The content of the user's Twitter message"
  }, {
    "name" : "timestamp",
    "type" : "long",
    "doc"  : "Unix epoch time in seconds"
  } ],
  "doc:" : "A basic schema for storing Twitter messages"
}

Data records in JSON format

And here is some corresponding example data with two records that follow the schema defined in the previous section. We store this data in the file twitter.json.

1
2

{"username":"miguno","tweet":"Rock: Nerf paper, scissors is fine.","timestamp": 1366150681 }
{"username":"BlizzardCS","tweet":"Works as intended.  Terran is IMBA.","timestamp": 1366154481 }

Converting to and from binary Avro

JSON to binary Avro

Without compression:

$ java -jar ~/avro-tools-1.7.4.jar fromjson --schema-file twitter.avsc twitter.json > twitter.avro

With Snappy compression:

$ java -jar ~/avro-tools-1.7.4.jar fromjson --codec snappy --schema-file twitter.avsc twitter.json > twitter.snappy.avro

Binary Avro to JSON

The same command will work on both uncompressed and compressed data.

$ java -jar ~/avro-tools-1.7.4.jar tojson twitter.avro > twitter.json
$ java -jar ~/avro-tools-1.7.4.jar tojson twitter.snappy.avro > twitter.json

Retrieve Avro schema from binary Avro

The same command will work on both uncompressed and compressed data.

$ java -jar ~/avro-tools-1.7.4.jar getschema twitter.avro > twitter.avsc
$ java -jar ~/avro-tools-1.7.4.jar getschema twitter.snappy.avro > twitter.avsc

Known Issues of Snappy with JDK 7 on Mac OS X

If you happen to use JDK 7 on Mac OS X 10.8 you will run into the error below when trying to run the Snappy related commands above. In that case make sure to explicitly use JDK 6. On Mac OS 10.8 the JDK 6 java binary is by default available at /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java.

The cause of this problem is documented in the bug report Native (Snappy) library loading fails on openjdk7u4 for mac. This bug is already fixed in the latest Snappy-Java 1.5 milestone releases, but Avro 1.7.4 still depends on the latest stable release of Snappy-Java which is 1.0.4.1 (see lang/java/pom.xml in the Avro source code).

I also found that one way to fix this problem when writing your own Java code is to explicitly require Snappy 1.5.x. Here is the relevant dependency declaration for build.gradle in case you are using Gradle. This seems to solve the problem, but I have yet to confirm whether this is a safe way for production scenarios.

// Required to fix a Snappy native library error on OS X when trying to compress Avro files with Snappy;
// Avro 1.7.4 uses the latest stable release of Snappy, 1.0.4.1 (see avro/lang/java/pom.xml) that still contains
// the original bug described at https://github.com/xerial/snappy-java/issues/6.
//
// Note that in a production setting we do not care about OS X, so we could use Snappy 1.0.4.1 as required by
// Avro 1.7.4 as is.
//
compile group: 'org.xerial.snappy', name: 'snappy-java', version: '1.0.5-M4'

Detailed error message:

$ uname -a
Darwin mac.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan  6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64

$ java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

$ java -jar ~/avro-tools-1.7.4.jar fromjson --codec snappy --schema-file twitter.avsc twitter.json > twitter.snappy.avro

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
	at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
	at org.apache.avro.file.SnappyCodec.compress(SnappyCodec.java:43)
	at org.apache.avro.file.DataFileStream$DataBlock.compressUsing(DataFileStream.java:349)
	at org.apache.avro.file.DataFileWriter.writeBlock(DataFileWriter.java:348)
	at org.apache.avro.file.DataFileWriter.writeIfBlockFull(DataFileWriter.java:295)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:266)
	at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:109)
	at org.apache.avro.tool.Main.run(Main.java:80)
	at org.apache.avro.tool.Main.main(Main.java:69)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1860)
	at java.lang.Runtime.loadLibrary0(Runtime.java:845)
	at java.lang.System.loadLibrary(System.java:1084)
	at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
	... 15 more
Exception in thread "main" org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
	at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
	at org.apache.avro.file.SnappyCodec.compress(SnappyCodec.java:43)
	at org.apache.avro.file.DataFileStream$DataBlock.compressUsing(DataFileStream.java:349)
	at org.apache.avro.file.DataFileWriter.writeBlock(DataFileWriter.java:348)
	at org.apache.avro.file.DataFileWriter.writeIfBlockFull(DataFileWriter.java:295)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:266)
	at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:109)
	at org.apache.avro.tool.Main.run(Main.java:80)
	at org.apache.avro.tool.Main.main(Main.java:69)

Where to go from here

The example commands above show just a few variants of how to use Avro Tools to read, write and convert Avro files. The Avro Tools library is documented at:

• Java API docs of org.apache.avro.tool

That said I found those docs not that helpful (the sources are however). I’d recommend to just try running the command line tools without parameters and have a look at their usage instructions which they will print to STDOUT. Normally this is enough to understand how they should be used.

What we want to do

Here is an overview of what we want to do:

• Build Kafka 0.8-trunk for Scala 2.9.2.
  • I also provide instructions for the default 2.8.0, just in case.
• Use a single machine for this Kafka setup.
• Run 1 ZooKeeper instance on that machine.
• Run 3 Kafka brokers on that machine.
• Create a Kafka topic called “zerg.hydra” and send/receive messages for that topic via the console. The topic will be configured to use 3 partitions and 2 replicas per partition.

The purpose of this article is not to present a production-ready configuration of a Kafka cluster. However it should get you started with using Kafka as a distributed messaging system in your own infrastructure.

Installing Kafka

Background: Why Kafka and Scala 2.9?

Personally I’d like to use Scala 2.9.2 for Kafka – which is still built for Scala 2.8.0 by default as of today – because many related software packages that are of interest to me (such as Finagle, Kestrel) are based on Scala 2.9. Also, the current versions of many development and build tools (e.g. IDEs, sbt) for Scala require at least version 2.9. If you are working in a similar environment you may want build Kafka for Scala 2.9 just like I did – otherwise you can expect to run into issues such as Scala version conflicts.

Option 1 (preferred): Kafka 0.8-trunk with Scala 2.9.2

Unfortunately the current trunk of Kafka has problems to build against Scala 2.9.2 out of the box. I created a fork of Kafka 0.8-trunk that includes the required fix (a change of one file) in the branch “scala-2.9.2”. The fix ties the Scala version used by Kafka’s shell scripts to 2.9.2 instead of 2.8.0.

The following instructions will use this fork to download, build and install Kafka for Scala 2.9.2:

$ cd $HOME
$ git clone git@github.com:miguno/kafka.git
$ cd kafka
# this branch of includes a patched bin/kafka-run-class.sh for Scala 2.9.2
$ git checkout -b scala-2.9.2 remotes/origin/scala-2.9.2
$ ./sbt update
$ ./sbt "++2.9.2 package"

Option 2: Kafka 0.8-trunk with Scala 2.8.0

If you are fine with Scala 2.8 you need to build and install Kafka as follows.

$ cd $HOME
$ git clone git@github.com:apache/kafka.git
$ cd kafka
$ ./sbt update
$ ./sbt package

Configuring and running Kafka

Unless noted otherwise all commands below assume that you are in the top level directory of your Kafka installation. If you followed the instructions above, this directory is $HOME/kafka/.

Start ZooKeeper

Kafka ships with a reasonable default ZooKeeper configuration for our simple use case. The following command launches a local ZooKeeper instance.

1

$ bin/zookeeper-server-start.sh config/zookeeper.properties

By default the ZooKeeper server will listen on *:2181/tcp.

Configure and start the Kafka brokers

We will create 3 Kafka brokers, whose configurations are based on the default config/server.properties. Apart from the settings below the configurations of the brokers are identical.

The first broker:

1

$ cp config/server.properties config/server1.properties

Edit config/server1.properties and replace the existing config values as follows:

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1

The second broker:

1

$ cp config/server.properties config/server2.properties

Edit config/server2.properties and replace the existing config values as follows:

broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2

The third broker:

1

$ cp config/server.properties config/server3.properties

Edit config/server3.properties and replace the existing config values as follows:

broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3

Now you can start each Kafka broker in a separate console:

1

$ env JMX_PORT=9999  bin/kafka-server-start.sh config/server1.properties

1

$ env JMX_PORT=10000 bin/kafka-server-start.sh config/server2.properties

1

$ env JMX_PORT=10001 bin/kafka-server-start.sh config/server3.properties

Here is a summary of the configured network interfaces and ports that the brokers will listen on:

        Broker 1     Broker 2      Broker 3
----------------------------------------------
Kafka   *:9092/tcp   *:9093/tcp    *:9094/tcp
JMX     *:9999/tcp   *:10000/tcp   *:10001/tcp

Excursus: Topics, partitions and replication in Kafka

In a nutshell Kafka partitions incoming messages for a topic, and assigns those partitions to the available Kafka brokers. The number of partitions is configurable and can be set per-topic and per-broker.

First the stream [of messages] is partitioned on the brokers into a set of distinct partitions. The semantic meaning of these partitions is left up to the producer and the producer specifies which partition a message belongs to. Within a partition messages are stored in the order in which they arrive at the broker, and will be given out to consumers in that same order.

A new feature of Kafka 0.8 is that those partitions will be now be replicated across Kafka brokers to make the cluster more resilient against host failures:

Partitions are now replicated. Previously the topic would remain available in the case of server failure, but individual partitions within that topic could disappear when the server hosting them stopped. If a broker failed permanently any unconsumed data it hosted would be lost. Starting with 0.8 all partitions have a replication factor and we get the prior behavior as the special case where replication factor = 1. Replicas have a notion of committed messages and guarantee that committed messages won’t be lost as long as at least one replica survives. Replica are byte-for-byte identical across replicas.

Producer and consumer are replication aware. When running in sync mode, by default, the producer send() request blocks until the messages sent is committed to the active replicas. As a result the sender can depend on the guarantee that a message sent will not be lost. Latency sensitive producers have the option to tune this to block only on the write to the leader broker or to run completely async if they are willing to forsake this guarantee. The consumer will only see messages that have been committed.

The following diagram illustrates the relationship between topics, partitions and replicas.

Logically this relationship is very similar to how Hadoop manages blocks and replication in HDFS.

When a topic is created in Kafka 0.8, Kafka determines how each replica of a partition is mapped to a broker. In general Kafka tries to spread the replicas across all brokers (source). Messages are first sent to the first replica of a partition (i.e. to the current “leader” broker of that partition) before they are replicated to the remaining brokers. Message producers may choose from different strategies for sending messages (e.g. synchronous mode, asynchronous mode). Producers discover the available brokers in a cluster and the number of partitions on each, by registering watchers in ZooKeeper.

If you wonder how to configure the number of partitions per topic/broker, here’s feedback from LinkedIn developers:

At LinkedIn, some of the high volume topics are configured with more than 1 partition per broker. Having more partitions increases I/O parallelism for writes and also increases the degree of parallelism for consumers (since partition is the unit for distributing data to consumers). On the other hand, more partitions adds some overhead: (a) there will be more files and thus more open file handlers; (b) there are more offsets to be checkpointed by consumers which can increase the load of ZooKeeper. So, you want to balance these tradeoffs.

Create a Kafka topic

In Kafka 0.8, there are 2 ways of creating a new topic:

1. Turn on auto.create.topics.enable option on the broker. When the broker receives the first message for a new topic, it creates that topic with num.partitions and default.replication.factor.
2. Use the admin command bin/kafka-topics.sh.

We will use the latter approach. The following command creates a new topic “zerg.hydra”. The topic is configured to use 3 partitions and a replication factor of 2. Note that in a production setting we’d rather set the replication factor to 3, but a value of 2 is better for illustrative purposes (i.e. we intentionally use different values for the number of partitions and replications to better see the effects of each setting).

1
2

$ bin/kafka-topics.sh --zookeeper localhost:2181 \
    --create --topic zerg.hydra --partitions 3 --replication-factor 2

This has the following effects:

• Kafka will create 3 logical partitions for the topic.
• Kafka will create a total of two replicas (copies) per partition. For each partition it will pick two brokers that will host those replicas. For each partition Kafka will elect a “leader” broker.

Ask Kafka for a list of available topics. The list should include the new zerg.hydra topic:

1
2
3
4

$ bin/kafka-topics.sh --zookeeper localhost:2181 --list
<snipp>
zerg.hydra
</snipp>

You can also inspect the configuration of the topic as well as the currently assigned brokers per partition and replica. Because a broker can only host a single replica per partition, Kafka has opted to use a broker’s ID also as the corresponding replica’s ID.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic zerg.hydra
<snipp>
zerg.hydra
    configs:
    partitions: 3
        partition 0
        leader: 1 (192.168.0.153:9092)
        replicas: 1 (192.168.0.153:9092), 2 (192.168.0.153:9093)
        isr: 1 (192.168.0.153:9092), 2 (192.168.0.153:9093)
        partition 1
        leader: 2 (192.168.0.153:9093)
        replicas: 2 (192.168.0.153:9093), 3 (192.168.0.153:9094)
        isr: 2 (192.168.0.153:9093), 3 (192.168.0.153:9094)
        partition 2
        leader: 3 (192.168.0.153:9094)
        replicas: 3 (192.168.0.153:9094), 1 (192.168.0.153:9092)
        isr: 3 (192.168.0.153:9094), 1 (192.168.0.153:9092)
<snipp>

In this example output the first broker (with broker.id = 1) happens to be the designated leader for partition 0 at the moment. Similarly, the second and third brokers are the leaders for partitions 1 and 2, respectively.

The following diagram illustrates the setup (and also includes the producer and consumer that we will run shortly).

You can also inspect the local filesystem to see how the --describe output above matches actual files. By default Kafka persists topics as “log files” (Kafka terminology) in the log.dir directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

$ tree /tmp/kafka-logs-{1,2,3}
/tmp/kafka-logs-1                   # first broker (broker.id = 1)
├── zerg.hydra-0                    # replica of partition 0 of topic "zerg.hydra" (this broker is leader)
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
├── zerg.hydra-2                    # replica of partition 2 of topic "zerg.hydra"
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
└── replication-offset-checkpoint

/tmp/kafka-logs-2                   # second broker (broker.id = 2)
├── zerg.hydra-0                    # replica of partition 0 of topic "zerg.hydra"
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
├── zerg.hydra-1                    # replica of partition 1 of topic "zerg.hydra" (this broker is leader)
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
└── replication-offset-checkpoint

/tmp/kafka-logs-3                   # third broker (broker.id = 3)
├── zerg.hydra-1                    # replica of partition 1 of topic "zerg.hydra"
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
├── zerg.hydra-2                    # replica of partition 2 of topic "zerg.hydra" (this broker is leader)
│   ├── 00000000000000000000.index
│   └── 00000000000000000000.log
└── replication-offset-checkpoint

6 directories, 15 files

Caveat: Deleting a topic via bin/kafka-topics.sh --delete will apparently not delete the corresponding local files for that topic. I am not sure whether this behavior is expected or not.

Start a producer

Start a console producer in sync mode:

1
2

$ bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --sync \
    --topic zerg.hydra

Example producer output:

[...] INFO Verifying properties (kafka.utils.VerifiableProperties)
[...] INFO Property broker.list is overridden to localhost:9092,localhost:9093,localhost:9094 (...)
[...] INFO Property compression.codec is overridden to 0 (kafka.utils.VerifiableProperties)
[...] INFO Property key.serializer.class is overridden to kafka.serializer.StringEncoder (...)
[...] INFO Property producer.type is overridden to sync (kafka.utils.VerifiableProperties)
[...] INFO Property queue.buffering.max.messages is overridden to 10000 (...)
[...] INFO Property queue.buffering.max.ms is overridden to 1000 (kafka.utils.VerifiableProperties)
[...] INFO Property queue.enqueue.timeout.ms is overridden to 0 (kafka.utils.VerifiableProperties)
[...] INFO Property request.required.acks is overridden to 0 (kafka.utils.VerifiableProperties)
[...] INFO Property request.timeout.ms is overridden to 1500 (kafka.utils.VerifiableProperties)
[...] INFO Property send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
[...] INFO Property serializer.class is overridden to kafka.serializer.StringEncoder (...)

You can now enter new messages, one per line. Here we enter two messages “Hello, world!” and “Rock: Nerf Paper. Scissors is fine.”:

Hello, world!
Rock: Nerf Paper. Scissors is fine.

After the messages are produced, you should see the data being replicated to the three log directories for each of the broker instances, i.e. /tmp/kafka-logs-{1,2,3}/zerg.hydra-*/.

Start a consumer

Start a console consumer that reads messages in zerg.hydra from the beginning (in a production setting you would usually NOT want to add the --from-beginning option):

1

$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic zerg.hydra --from-beginning

The consumer will see a new message whenever you enter a message in the producer above.

Example consumer output:

<snipp>
[...] INFO [console-consumer-28434_panama.local-1363174829799-954ed29e], Connecting to zookeeper instance at localhost:2181 ...
[...] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[...] INFO Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT ...
[...] INFO Client environment:host.name=192.168.0.153 (org.apache.zookeeper.ZooKeeper)
<snipp>
[...] INFO Fetching metadata with correlation id 0 for 1 topic(s) Set(zerg.hydra) (kafka.client.ClientUtils$)
[...] INFO Connected to 192.168.0.153:9092 for producing (kafka.producer.SyncProducer)
[...] INFO Disconnecting from 192.168.0.153:9092 (kafka.producer.SyncProducer)
[...] INFO [ConsumerFetcherThread-console-consumer-28434_panama.local-1363174829799-954ed29e-0-3], Starting ...
[...] INFO [ConsumerFetcherManager-1363174829916] adding fetcher on topic zerg.hydra, partion 2, initOffset -1 to broker 3 with fetcherId 0 ...
[...] INFO [ConsumerFetcherThread-console-consumer-28434_panama.local-1363174829799-954ed29e-0-2], Starting ...
[...] INFO [ConsumerFetcherManager-1363174829916] adding fetcher on topic zerg.hydra, partion 1, initOffset -1 to broker 2 with fetcherId 0 ...
[...] INFO [ConsumerFetcherThread-console-consumer-28434_panama.local-1363174829799-954ed29e-0-1], Starting ...
[...] INFO [ConsumerFetcherManager-1363174829916] adding fetcher on topic zerg.hydra, partion 0, initOffset -1 to broker 1 with fetcherId 0 ...

And at the end of the output you will see the following messages:

Hello, world!
Rock: Nerf Paper. Scissors is fine.

That’s it!

A note when using Kafka with Storm

The maximum parallelism you can have on a KafkaSpout is the number of partitions of the corresponding Kafka topic. The following question-answer thread (I slightly modified the original text for clarification purposes) is from the Storm user mailing list, but supposedly refers to Kafka pre-0.8 and thereby before the replication feature was added:

Question: Suppose the number of Kafka partitions per broker is configured as 1 and the number of hosts is 2. If we set the spout parallelism as 10, then how does Storm handle the difference between the number of Kafka partitions and the number of spout tasks? Since there are only 2 partitions, does every other spout task (greater than first 2) not read the data or do they read the same data?

Answer (by Nathan Marz): The remaining 8 (= 10 - 2) spout tasks wouldn’t read any data from the Kafka topic.

My current understanding is that the number of partitions (i.e. regardless of replicas) is still the limiting factor for the parallelism of a KafkaSpout. Why? Because Kafka is not allowing consumers to read from replicas other than the (replica of the) leader of a partition to simplify concurrent access to data in Kafka.

A note when using Kafka with Hadoop

LinkedIn has published their Kafka->HDFS pipeline named Camus. It is a MapReduce job that does distributed data loads out of Kafka.

Where to go from here

The following documents provide plenty of information about Kafka that goes way beyond what I covered in this article:

• Kafka Design
• Kafka Configuration
• Kafka 0.8 Quickstart
• Kafka Operations – how to run Kafka in production
• Kafka Replication including Detailed Replication Design v3 for Kafka 0.8
• Writing a Driver for Kafka – includes information about topics, partitions, Kafka’s “log” files and message objects

Software used in the bootstrap project

• Gradle version 1.3 – build tool
• TestNG version 6.8 – unit testing framework
• Mockito version 1.9.0 – mocking framework
• FEST-Assert 2 version 2.0M8 – fluent interface for assertions that allows you to write assertions that read more like natural language (unfortunately Java lacks something like the awesome ScalaTest framework)
• Cobertura plugin for Gradle version 1.0 – allows Gradle to generate Cobertura compatible test reports (mostly used for integrating test results with Jenkins)

Packages only used for showcasing the functionality:

• Google Guava version 13 – solely used to show how compile-time dependencies are configured in Gradle

The latest dependency information is always available in build.gradle on GitHub.

What we want to do

We have two complimentary goals:

1. You should be able to use Eclipse as the IDE of choice to work with the code (e.g. run the build and tests locally on your machine).
2. You should be able to integrate the code with the Jenkins continuous integration server (e.g. to let it run the build and tests for your team and publish the test results).

The first goal covers your personal workflow as a software engineer with the code. The second goal covers integrating the code with the your engineering team as a whole.

How to use the bootstrap project

1. Download the bootstrap project as described below.
2. Configure Eclipse (optional).
3. Configure Jenkins CI server (optional).
4. Hack away!

Download

You have the following two options to start your own project with the bootstrap project.

Option 1: You do not have a GitHub account – clone the bootstrap project

If you don’t have a GitHub account, the simplest way is to just clone (i.e. checkout) the original bootstrap project. The only requirement is a local installation of git on your machine.

1

$ git clone git://github.com/miguno/gradle-testng-mockito-bootstrap.git

Now you can start hacking away!

1
2

$ cd gradle-testng-mockito-bootstrap
# ...start coding...

Option 2: You do have a GitHub account – fork the bootstrap project

If you do have a GitHub account, I recommend that you fork the bootstrap project. Then start writing your own code against your personal fork.

First, open the bootstrap project on GitHub and fork it.

Then clone (i.e. checkout) your personal fork.

1

$ git clone git@github.com:YOURUSERNAME/gradle-testng-mockito-bootstrap.git

Now you can start hacking away!

1
2

$ cd gradle-testng-mockito-bootstrap
# ...start coding...

About the actual Java code in the bootstrap project

The bootstrap project ships with only two classes:

• BobRoss.java – A simple class that implements a few features that we can write unit tests for. We pretend to be the late painting instructor Bob Ross who, well, is painting a picture with us.
• BobRossTest.java – This class tests the former class. It illustrates the use of TestNG, Mockito and FEST-Assert 2 to write these unit tests. Don’t pay too much attention to the semantics of the actual tests, we’re just showcasing here.

Using Gradle on the command line

Installing Gradle

Before continuing you must download and install Gradle if you haven’t done so already.

If you are on a Mac and have the Homebrew package manager installed, you just need to run:

1

$ brew install gradle

Command Examples

The bootstrap project is a normal gradle project. Have a look at the gradle documentation what this allows you to do. I will only list the most important commands here. If you want to see what gradle tasks are available out of the box in the bootstrap project, run gradle tasks.

1
2
3
4
5
6
7
8

# General commands
$ gradle clean          # deletes the build directory
$ gradle clean test     # runs the unit tests (and compile before if needed)
$ gradle clean build    # assembles and tests this project

# Eclipse related
$ gradle cleanEclipse   # cleans all Eclipse files
$ gradle eclipse        # generates all Eclipse files

By default, executing the commands above will create output in the following locations:

• build/ – this sub-directory is used by Gradle
• build/reports/cobertura/main/coverage.xml – Cobertura test coverage report in XML format
• build/reports/tests/testng-results.xml – TestNG Results in XML format
• bin/ – this sub-directory is used by Eclipse

Feel free to browse the directory tree to find additional files that you might need.

Configuring Eclipse

Importing the bootstrap project

The following steps will import your local clone of the bootstrap project into Eclipse.

First, you must generate the required project files for Eclipse via gradle:

1

$ gradle cleanEclipse eclipse

Then you import the bootstrap project into Eclipse as follows.

Open File > Import...:

Select General > Existing Projects into Workspace:

Navigate to your local copy of the bootstrap project. In the dialogue window you can leave the other values at their defaults and just click Finish. Of course feel free to modify the default values as you see fit (e.g. to add the project to a working set of your choice).

Now you should see the bootstrap project in the Package Explorer of Eclipse:

Installing the TestNG plugin for Eclipse

You will need the TestNG plugin for Eclipse so that you can conveniently run the included unit tests from within Eclipse.

To install the plugin open Help > Eclipse Marketplace... in Eclipse. Search for “gradle” and then install the “TestNG for Eclipse” plugin by Cédric Beust. Make sure to restart Eclipse after installing the plugin.

Now you can run the TestNG unit tests, for instance, by right-clicking on the BobRossTest class in the Package Explorer and selecting Run as... > TestNG Test.

If you give it a go, you should see the following results in the TestNG view in Eclipse. If you don’t find the TestNG view in your Eclipse, make sure it is enabled via Window > Show View > TestNG.

Configuring Jenkins

This section assumes that you are familiar with Jenkins and have a Jenkins instance installed and running.

Installing required Jenkins plugins

Install the following Jenkins plugins via Jenkins > Manage Jenkins > Manage Plugins:

• Cobertura plugin for Jenkins
• Gradle plugin for Jenkins
• TestNG plugin for Jenkins
• Git plugin for Jenkins

Creating a new Jenkins build job

Now you can add a new Jenkins build job via Jenkins > New Job.

• Select “Build a free-style software project” and give your project a name. Click Ok.
• In section “Source Code Management” select Git and enter your repository URL.
• In section “Build” click on “Add build step” and select “Invoke Gradle script”.
  • Enter clean build javadoc in the “Tasks” field.
• In section “Post-build Actions” click on “Add post-build action” and select “Publish Cobertura Coverage Report”.
  • Enter **/build/reports/cobertura/main/coverage.xml in the “Cobertura xml report pattern” field.
• In section “Post-build Actions” click again on “Add post-build action” and select “Publish TestNG Reports”.
  • Enter ./build/reports/tests/testng-results.xml in the “TestNG XML report pattern” field.
• Make sure you click on the Save button at the very bottom to actually save your new Jenkins build job!

Now you can start using Jenkins to execute the build and run the tests of your bootstrap project!

Here are some screenshots how it will look like:

What else for Jenkins?

You can also configure Jenkins to fetch the latest contents of your GitHub code repository when you run a new build (or even trigger a build automatically when a new commit is pushed to the repository). I will not cover this setup in this article though.

Summary

This bootstrap project should get you started quickly with your own Java code project backed by Gradle, TestNG & Co. If you come up with any improvements, feel free to write a comment here or to send me a pull request on the bootstrap GitHub repository.

About Trending Topics and Sliding Windows

First, let me explain what I mean by “trending topics” so that we have a common understanding. Here is an explanation taken from Wikipedia:

Trending topics

A word, phrase or topic that is tagged at a greater rate than other tags is said to be a trending topic. Trending topics become popular either through a concerted effort by users or because of an event that prompts people to talk about one specific topic. These topics help Twitter and their users to understand what is happening in the world.

In other words, it is a measure of “What’s hot?” in a user community. Typically, you are interested in trending topics for a given time span; for instance, the most popular topics in the past five minutes or the current day. So the question “What’s hot?” is more precisely stated as “What’s hot today?” or “What’s hot this week?”.

In this article we assume we have a system that uses the Twitter API to pull the latest tweets from the live Twitter stream. We assume further that we have a mechanism in place that extracts topical information in the form of words from those tweets. For instance, we could opt to use a simple pattern matching algorithm that treats #hashtags in tweets as topics. Here, we would consider a tweet such as

1

@miguno The #Storm project rocks for real-time distributed #data processing!

to “mention” the topics

1
2

storm
data

We design our system so that it considers topic A more popular than topic B (for a given time span) if topic A has been mentioned more often in tweets than topic B. This means we only need to count the number of occurrences of topics in tweets.

For the context of this article we do not care how the topics are actually derived from user content or user activities as long as the derived topics are represented as textual words. Then, the Storm topology described in this article will be able to identify in real-time the trending topics in this input data using a time-sensitive rolling count algorithm (rolling counts are also known as sliding windows) coupled with a ranking step. The former aspect takes care of filtering user input by time span, the latter of ranking the most trendy topics at the top the list.

Eventually we want our Storm topology to periodically produce the top N of trending topics similar to the following example output, where t0 to t2 are different points in time:

1
2
3
4
5
6
7

Rank @ t0   ----->   t1   ----->   t2
---------------------------------------------
1.    java   (33)   ruby   (41)   scala  (32)
2.    php    (30)   scala  (28)   python (29)
3.    scala  (21)   java   (27)   ruby   (24)
4.    ruby   (16)   python (21)   java   (21)
5.    python (15)   php    (14)   erlang (18)

In this example we can see that over time “scala” has become the hottest trending topic.

Sliding Windows

The last background aspect I want to cover are sliding windows aka rolling counts. A picture is worth a thousand words:

A formula might also be worth a bunch of words – ok, ok, maybe not a full thousand of them – so mathematically speaking we could formalize such a sliding-window sum algorithm as follows:

where t continually advances (most often with time) and m is the window size.

From size to time: If the window is advanced with time, say every N minutes, then the individual elements in the input represent data collected over the same interval of time (here: N minutes). In that case the window size is equivalent to N x m minutes. Simply speaking, if N=1 and m=5, then our sliding window algorithm emits the latest five-minute aggregates every one minute.

Now that we have introduced trending topics and sliding windows we can finally start talking about writing code for Storm that implements all this in practice – large-scale, distributed, in real time.

Before We Start

About storm-starter

The storm-starter project on GitHub provides example implementations of various Storm real-time data processing topologies such as a simple streaming WordCount algorithm. It also includes a Rolling Top Words topology that can be used for computing trending topics, the purpose of which is exactly what I want to cover in this article.

When I began to tackle trending topic analysis with Storm I expected that I could re-use most if not all of the Rolling Top Words code in storm-starter. But I soon realized that the old code would need some serious redesigning and refactoring before one could actually use it in a real-world environment – including being able to efficiently maintain and augment the code in a team of engineers across release cycles.

In the next section I will briefly summarize the state of the Rolling Top Words topology before and after my refactoring to highlight some important changes and things to consider when writing your own Storm code. Then I will continue with covering the most important aspects of the new implementation in further detail. And of course I contributed the new implementation back to the Storm project.

The Old Code and My Goals for the New Code

My initial reaction to the old code was that, frankly speaking, I had no idea what and how it was doing its job. The various logical responsibilities of the code were mixed together in the existing classes, clearly not abiding by the Single Responsibility Principle. And I am not talking about academic treatments of SRP and such – I was hands-down struggling to wrap my head around the old code because of this.

Also, I noticed a few synchronized statements and threads being launched manually, hinting at additional parallel operations beyond what the Storm framework natively provides you with. Here, I was particularly concerned with those functionalities that interacted with the system time (calls to System.currentTimeMillis()). I couldn’t help the feeling that they looked prone to concurrency issues. And my suspicions were eventually confirmed when I discovered a dirty-write bug in the RollingCountObjects bolt code for the slot-based counting (using long[]) of object occurrences. In practice this dirty-write bug in the old rolling count implementation caused data corruption, i.e. the code was not carrying out its main responsibility correctly – that of counting objects. That said I’d argue that it would not have been trivial to spot this error in the old code prior to refactoring (where it was eventually plain to see), so please don’t think it was just negligence on the part of the original authors. With the new tick tuple feature in Storm 0.8 I was feeling confident that this part of the code could be significantly simplified and fixed.

In general I figured that completely refactoring the code and untangling these responsibilities would not only make the code more approachable and readable for me and others – after all the storm-starter code’s main purpose is to jumpstart Storm beginners – but it would also allow me to write meaningful unit tests, which would have been very difficult to do with the old code.

The design and implementation that I will describe in the following sections are the result of a number of refactoring iterations. I started with smaller code changes that served me primarily to understand the existing code better (e.g. more meaningful variable names, splitting long methods into smaller logical units). The more I felt comfortable the more I started to introduce substantial changes. Unfortunately the existing code was not accompanied by any unit tests, so while refactoring I was in the dark, risking to break something that I was not even aware of breaking. I considered writing unit tests for the existing code first and then go back to refactoring but I figured that this would not be the best approach given the state of the code and the time I had available.

In summary my goals for the new trending topics implementation were:

1. The new code should be clean and easy to understand, both for the benefit of other developers when adapting or maintaining the code and for reasoning about its correctness. Notably, the code should decouple its data structures from the Storm sub-system and, if possible, favor native Storm features for concurrency instead of custom approaches.
2. The new code should be covered by meaningful unit tests.
3. The new code should be good enough to contribute it back to the Storm project to help its community.

Implementing the Data Structures

Eventually I settled down to the following core data structures for the new distributed Rolling Count algorithm. As you will see, an interesting characteristic is that these data structures are completely decoupled from any Storm internals. Our Storm bolts will make use of them, of course, but there is no dependency in the opposite direction from the data structures to Storm.

• Classes used for counting objects: SlotBasedCounter, SlidingWindowCounter
• Classes used for ranking objects by their count: Rankings, Rankable, RankableObjectWithFields

Another notable improvement is that the new code removes any need and use of concurrency-related code such as synchronized statements or manually started background threads. Also, none of the data structures are interacting with the system time. Eliminating direct calls to system time and manually started background threads makes the new code much simpler and testable than before.

1
2
3
4

// such code from the old RollingCountObjects bolt is not needed anymore
long delta = millisPerBucket(_numBuckets)
               - (System.currentTimeMillis() % millisPerBucket(_numBuckets));
Utils.sleep(delta);

SlotBasedCounter

The SlotBasedCounter class provides per-slot counts of the occurrences of objects. The number of slots of a given counter instance is fixed. The class provides four public methods:

1
2
3
4
5

public void incrementCount(T obj, int slot);
public void wipeSlot(int slot):
public long getCount(T obj, int slot)
// get the *total* counts of all objects across all slots
public Map<T, Long> getCounts();

Here is a usage example:

1
2
3
4
5
6
7
8
9
10
11
12
13

// we want to count Object's using five slots
SlotBasedCounter counter = new SlotBasedCounter<Object>(5);

// counting
Object trackMe = ...;
int currentSlot = 0;
counter.incrementCount(trackMe, currentSlot);

// the counts of an object for a given slot
long counts = counter.getCount(trackMe, currentSlot);

// the total counts (across all slots) of all objects
Map<Object, Long> counts = counter.getCounts();

Internally SlotBasedCounter is backed by a Map<T, long[]> for the actual count state. You might be surprised to see the low-level long[] array here – wouldn’t it be better OO style to introduce a new, separate class that is just used for the counting of a single slot, and then we use a couple of these single-slot counters to form the SlotBasedCounter? Well, yes we could. But for performance reasons and for not deviating too far from the old code I decided not to go down this route. Apart from updating the counter – which is a WRITE operation – the most common operation in our use case is a READ operation to get the total counts of tracked objects. Here, we must calculate the sum of an object’s counts across all slots. And for this it is preferable to have the individual data points for an object close to each other (kind of data locality), which the long[] array allows us to do. Your mileage may vary though.

The SlotBasedCounter is a primitive class that can be used, for instance, as a building block for implementing sliding window counting of objects. And this is exactly what I will describe in the next section.

SlidingWindowCounter

The SlidingWindowCounter class provides rolling counts of the occurrences of “things”, i.e. a sliding window count for each tracked object. Its counting functionality is based on the previously described SlotBasedCounter. The size of the sliding window is equivalent to the (fixed) number of slots number of a given SlidingWindowCounter instance. It is used by RollingCountBolt for counting incoming data tuples.

The class provides two public methods:

1
2

public void incrementCount(T obj);
Map<T, Long> getCountsThenAdvanceWindow();

What might be surprising to some readers is that this class does not have any notion of time even though “sliding window” normally means a time-based window of some kind. In our case however the window does not advance with time but whenever (and only when) the method getCountsThenAdvanceWindow() is called. This means SlidingWindowCounter behaves just like a normal ring buffer in terms of advancing from one window to the next.

Here is an illustration showing the behavior of SlidingWindowCounter over multiple iterations:

Rankings and Rankable

The Rankings class represents fixed-size rankings of objects, for instance to implement “Top 10” rankings. It ranks its objects descendingly according to their natural order, i.e. from largest to smallest. This class is used by AbstractRankerBolt and its derived bolts to track the current rankings of incoming objects over time.

The class provides five public methods:

1
2
3
4
5

public void updateWith(Rankable r);
public void updateWith(Rankings other);
public List<Rankable> getRankings();
public int maxSize(); // as supplied to constructor
public int size(); // current size, might be less than maximum size

Whenever you update Rankings with new data, it will discard any elements that are smaller than the updated top N, where N is the maximum size of the Rankings instance (e.g. 10 for a top 10 ranking).

Now the sorting aspect of the ranking is driven by the natural order of the ranked objects. In my specific case, I created a Rankable interface that in turn implements the Comparable interface. In practice, you simply pass a Rankable object to the Rankings class, and the latter will update its rankings accordingly.

1
2
3
4
5

Rankings topTen = new Rankings(10);
Rankable C = ...;
topTen.updateWith(r);

List<Rankable> rankings = topTen.getRankings();

As you can see it is really straight-forward and intuitive in its use.

The concrete class implementing Rankable is RankableObjectWithFields. The bolt IntermediateRankingsBolt, for instance, creates Rankables from incoming data tuples via a factory method of this class:

1
2
3
4
5

    @Override
    void updateRankingsWithTuple(Tuple tuple) {
        Rankable rankable = RankableObjectWithFields.from(tuple);
        super.getRankings().updateWith(rankable);
    }

Have a look at Rankings, Rankable and RankableObjectWithFields for details. If you run into a situation where you have to implement classes like these yourself, make sure you follow good engineering practice and add standard methods such as equals() and hashCode() as well to your data structures.

Implementing the Rolling Top Words Topology

So where are we? In the sections above we have already discussed a number of Java classes but not even a single one of them has been directly related to Storm. It’s about time that we start writing some Storm code!

In the following sections I will describe the Storm components that make up the Rolling Top Words topology. When reading the sections keep in mind that the “words” in this topology represent the topics that are currently being mentioned by the users in our imaginary system.

Overview of the Topology

The high-level view of the Rolling Top Words topology is shown in the figure below.

The main responsibilities are split as follows:

1. In the first layer the topology runs many TestWordSpout instances in parallel to simulate the load of incoming data – in our case this would be the names of the topics (represented as words) that are currently being mentioned by our users.
2. The second layer comprises multiple instances of RollingCountBolt, which perform a rolling count of incoming words/topics.
3. The third layer uses multiple instances of IntermediateRankingsBolt (“I.R. Bolt” in the figure) to distribute the load of pre-aggregating the various incoming rolling counts into intermediate rankings. Hadoop users will see a strong similarity here to the functionality of a combiner in Hadoop.
4. Lastly, there is the final step in the topology. Here, a single instance of TotalRankingsBolt aggregates the incoming intermediate rankings into a global, consolidated total ranking. The output of this bolt are the currently trending topics in the system. These trending topics can then be used by downstream data consumers to provide all the cool user-facing and backend features you want to have in your platform.

In code the topology wiring looks as follows in RollingTopWords:

1
2
3
4
5
6
7

builder.setSpout(spoutId, new TestWordSpout(), 2);
builder.setBolt(counterId, new RollingCountBolt(9, 3), 3)
            .fieldsGrouping(spoutId, new Fields("word"));
builder.setBolt(intermediateRankerId, new IntermediateRankingsBolt(TOP_N), 2)
            .fieldsGrouping(counterId, new Fields("obj"));
builder.setBolt(totalRankerId, new TotalRankingsBolt(TOP_N))
            .globalGrouping(intermediateRankerId);

TestWordSpout

The only spout we will be using is the TestWordSpout that is part of backtype.storm.testing package of Storm itself. I will not cover the spout in detail because it is a trivial class. The only thing it does is to select a random word from a fixed list of five words (“nathan”, “mike”, “jackson”, “golda”, “bertels”) and emit that word to the downstream topology every 100ms. For the sake of this article, we consider these words to be our “topics”, of which we want to identify the trending ones.

The spout’s output can be visualized as follows. Note that the @XXXms milliseconds timeline is not part of the actual output.

1
2
3
4
5
6
7
8

@100ms: nathan
@200ms: golda
@300ms: golda
@400ms: jackson
@500ms: mike
@600ms: nathan
@700ms: bertels
...

Excursus: Tick Tuples in Storm 0.8+

A new and very helpful (read: awesome) feature of Storm 0.8 is the so-called tick tuple. Whenever you want a spout or bolt execute a task at periodic intervals – in other words, you want to trigger an event or activity – using a tick tuple is normally the best practice.

Nathan Marz described tick tuples in the Storm 0.8 announcement as follows:

Tick tuples: It’s common to require a bolt to “do something” at a fixed interval, like flush writes to a database. Many people have been using variants of a ClockSpout to send these ticks. The problem with a ClockSpout is that you can’t internalize the need for ticks within your bolt, so if you forget to set up your bolt correctly within your topology it won’t work correctly. 0.8.0 introduces a new “tick tuple” config that lets you specify the frequency at which you want to receive tick tuples via the “topology.tick.tuple.freq.secs” component-specific config, and then your bolt will receive a tuple from the __system component and __tick stream at that frequency.

Here is how you configure a bolt/spout to receive tick tuples every 10 seconds:

1
2
3
4
5
6
7

    @Override
    public Map<String, Object> getComponentConfiguration() {
        Config conf = new Config();
        int tickFrequencyInSeconds = 10;
        conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, tickFrequencyInSeconds);
        return conf;
    }

Usually you will want to add a conditional switch to the component’s execute method to tell tick tuples and “normal” tuples apart:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

    @Override
    public void execute(Tuple tuple) {
        if (isTickTuple(tuple)) {
            // now you can trigger e.g. a periodic activity
        }
        else {
            // do something with the normal tuple
        }
    }

    private static boolean isTickTuple(Tuple tuple) {
        return tuple.getSourceComponent().equals(Constants.SYSTEM_COMPONENT_ID)
            && tuple.getSourceStreamId().equals(Constants.SYSTEM_TICK_STREAM_ID);
    }

I hope that, like me, you can appreciate the elegance of solely using Storm’s existing primitives to implement the new tick tuple feature. :-)

RollingCountBolt

This bolt performs rolling counts of incoming objects, i.e. sliding window based counting. Accordingly it uses the SlidingWindowCounter class described above to achieve this. In contrast to the old implementation only this bolt (more correctly: the instances of this bolt that run as Storm tasks) is interacting with the SlidingWindowCounter data structure. Each instance of the bolt has its own private SlidingWindowCounter field, which eliminates the need for any custom inter-thread communication and synchronization.

The bolt combines the previously described tick tuples (that trigger at fix intervals in time) with the time-agnostic behavior of SlidingWindowCounter to achieve time-based sliding window counting. Whenever the bolt receives a tick tuple, it will advance the window of its private SlidingWindowCounter instance and emit its latest rolling counts. In the case of normal tuples it will simply count the object and ack the tuple.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

@Override
public void execute(Tuple tuple) {
    if (TupleHelpers.isTickTuple(tuple)) {
        LOG.info("Received tick tuple, triggering emit of current window counts");
        emitCurrentWindowCounts();
    }
    else {
        countObjAndAck(tuple);
    }
}

private void emitCurrentWindowCounts() {
    Map<Object, Long> counts = counter.getCountsThenAdvanceWindow();
    ...
    emit(counts, actualWindowLengthInSeconds);
}

private void emit(Map<Object, Long> counts) {
    for (Entry<Object, Long> entry : counts.entrySet()) {
        Object obj = entry.getKey();
        Long count = entry.getValue();
        collector.emit(new Values(obj, count));
    }
}

private void countObjAndAck(Tuple tuple) {
    Object obj = tuple.getValue(0);
    counter.incrementCount(obj);
    collector.ack(tuple);
}

That’s all there is to it! The new tick tuples in Storm 0.8 and the cleaned code of the bolt and its collaborators also make the code much more testable (the new code of this bolt has 98% test coverage). Compare the code above to the old implementation of the bolt and decide for yourself which one you’d prefer adapting or maintaining:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
    _collector = collector;
    cleaner = new Thread(new Runnable() {
        public void run() {
            Integer lastBucket = currentBucket(_numBuckets);

            while(true) {
              int currBucket = currentBucket(_numBuckets);
              if(currBucket!=lastBucket) {
                  int bucketToWipe = (currBucket + 1) % _numBuckets;
                  synchronized(_objectCounts) {
                      Set objs = new HashSet(_objectCounts.keySet());
                      for (Object obj: objs) {
                        long[] counts = _objectCounts.get(obj);
                        long currBucketVal = counts[bucketToWipe];
                        counts[bucketToWipe] = 0;
                        long total = totalObjects(obj);
                        if(currBucketVal!=0) {
                            _collector.emit(new Values(obj, total));
                        }
                        if(total==0) {
                            _objectCounts.remove(obj);
                        }
                      }
                  }
                  lastBucket = currBucket;
              }
              long delta = millisPerBucket(_numBuckets) - (System.currentTimeMillis() % millisPerBucket(_numBuckets));
              Utils.sleep(delta);
            }
        }
    });
    cleaner.start();
}

public void execute(Tuple tuple) {
    Object obj = tuple.getValue(0);
    int bucket = currentBucket(_numBuckets);
    synchronized(_objectCounts) {
        long[] curr = _objectCounts.get(obj);
        if(curr==null) {
            curr = new long[_numBuckets];
            _objectCounts.put(obj, curr);
        }
        curr[bucket]++;
        _collector.emit(new Values(obj, totalObjects(obj)));
        _collector.ack(tuple);
    }
}

Unit Test Example

Since I mentioned unit testing a couple of times in the previous section, let me briefly discuss this point in further detail. I implemented the unit tests with TestNG, Mockito and FEST-Assert. Here is an example unit test for RollingCountBolt, taken from RollingCountBoltTest.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

@Test
public void shouldEmitNothingIfNoObjectHasBeenCountedYetAndTickTupleIsReceived() {
    // given
    Tuple tickTuple = MockTupleHelpers.mockTickTuple();
    RollingCountBolt bolt = new RollingCountBolt();
    Map conf = mock(Map.class);
    TopologyContext context = mock(TopologyContext.class);
    OutputCollector collector = mock(OutputCollector.class);
    bolt.prepare(conf, context, collector);

    // when
    bolt.execute(tickTuple);

    // then
    verifyZeroInteractions(collector);
}

AbstractRankerBolt

This abstract bolt provides the basic behavior of bolts that rank objects according to their natural order. It uses the template method design pattern for its execute() method to allow actual bolt implementations to specify how incoming tuples are processed, i.e. how the objects embedded within those tuples are retrieved and counted.

This bolt has a private Rankings field to rank incoming tuples (those must contain Rankable objects, of course) according to their natural order.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

/**
 * This method functions as a template method (design pattern).
 */
@Override
public final void execute(Tuple tuple, BasicOutputCollector collector) {
    if (TupleHelpers.isTickTuple(tuple)) {
        getLogger().info("Received tick tuple, triggering emit of current rankings");
        emitRankings(collector);
    }
    else {
        updateRankingsWithTuple(tuple);
    }

    abstract void updateRankingsWithTuple(Tuple tuple);

}

The two actual implementations used in the Rolling Top Words topology, IntermediateRankingsBolt and TotalRankingsBolt, only need to implement the updateRankingsWithTuple() method.

IntermediateRankingsBolt

This bolt extends AbstractRankerBolt and ranks incoming objects by their count in order to produce intermediate rankings. This type of aggregation is similar to the functionality of a combiner in Hadoop. The topology runs many of such intermediate ranking bolts in parallel to distribute the load of processing the incoming rolling counts from the RollingCountBolt instances.

This bolt only needs to override updateRankingsWithTuple() of AbstractRankerBolt:

1
2
3
4
5

@Override
void updateRankingsWithTuple(Tuple tuple) {
    Rankable rankable = RankableObjectWithFields.from(tuple);
    super.getRankings().updateWith(rankable);
}

TotalRankingsBolt

This bolt extends AbstractRankerBolt and merges incoming intermediate Rankings emitted by the IntermediateRankingsBolt instances.

Like IntermediateRankingsBolt, this bolt only needs to override the updateRankingsWithTuple() method:

1
2
3
4
5

@Override
void updateRankingsWithTuple(Tuple tuple) {
    Rankings rankingsToBeMerged = (Rankings) tuple.getValue(0);
    super.getRankings().updateWith(rankingsToBeMerged);
}

Since this bolt is responsible for creating a global, consolidated ranking of currently trending topics, the topology must run only a single instance of TotalRankingsBolt. In other words, it must be a singleton in the topology.

The bolt’s current code in storm-starter does not enforce this behavior though – instead it relies on the RollingTopWords class to configure the bolt’s parallelism correctly (if you ask yourself why it doesn’t: that was simply oversight on my part, oops). If you want to improve that, you can provide a so-called per-component Storm configuration for this bolt that sets its maximum task parallelism to 1:

1
2
3
4
5
6
7
8

@Override
public Map<String, Object> getComponentConfiguration() {
    Map<String, Object> conf = new HashMap<String, Object>();
    conf.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, emitFrequencyInSeconds);
    // run only a single instance of this bolt in the Storm topology
    conf.setMaxTaskParallelism(1);
    return conf;
}

RollingTopWords

The class RollingTopWords ties all the previously discussed code pieces together. It implements the actual Storm topology, configures spouts and bolts, wires them together and launches the topology in local mode (Storm’s local mode is similar to a pseudo-distributed, single-node Hadoop cluster).

By default, it will produce the top 5 rolling words (our trending topics) and run for one minute before terminating. If you want to twiddle with the topology’s configuration settings, here are the most important:

• Configure the number of generated trending topics by setting the TOP_N constant in RollingTopWords.
• Configure the length and emit frequencies (both in seconds) for the sliding window counting in the constructor of RollingCountBolt in RollingTopWords#wireTopology().
• Similarly, configure the emit frequencies (in seconds) of the ranking bolts by using their corresponding constructors.
• Configure the parallelism of the topology by setting the parallelism_hint parameter of each bolt and spout accordingly.

Apart from this there is nothing special about this class. And because we have already seen the most important code snippet from this class in the section Overview of the Topology I will not describe it any further here.

Running the Rolling Top Words topology

Now that you know how the trending topics Storm code works it is about time we actually launch the topology! The topology is configured to run in local mode, which means you can just grab the code to your development box and launch it right away. You do not need any special Storm cluster installation or similar setup.

First you must checkout the latest code of the storm-starter project from GitHub:

1

$ git clone git://github.com/nathanmarz/storm-starter.git

Then you compile and run the Rolling Top Words topology:

1
2
3
4

# make sure you are in the top-level directory of the storm-starter repo
$ cd storm-starter

$ mvn -f m2-pom.xml compile exec:java -Dexec.classpathScope=compile -Dexec.mainClass=storm.starter.RollingTopWords

By default the topology will run for one minute and then terminate automatically.

Example Logging Output

Here is some example logging output of the topology. The first colum is the current time in milliseconds since the topology was started (i.e. it is 0 at the very beginning). The second colum is the ID of the thread that logged the message. I deliberately removed some entries in the log flow to make the output easier to read. For this reason please take a close look on the timestamps (first column) when you want to compare the various example outputs below.

Also, the Rolling Top Words topology has debugging output enabled. This means that Storm itself will by default log information such as what data a bolt/spout has emitted. For that reason you will see seemingly duplicate lines in the logs below.

Lastly, to make the logging output easier to read here is some information about the various thread IDs in this example run:

The topology has just started to run. The spouts generate their first output messages:

1
2
3
4
5
6

2056 [Thread-37] INFO  backtype.storm.daemon.task  - Emitting: wordGenerator default [golda]
2057 [Thread-19] INFO  backtype.storm.daemon.executor  - Processing received message source: wordGenerator:11, stream: default, id: {}, [golda]
2063 [Thread-39] INFO  backtype.storm.daemon.task  - Emitting: wordGenerator default [nathan]
2064 [Thread-25] INFO  backtype.storm.daemon.executor  - Processing received message source: wordGenerator:12, stream: default, id: {}, [nathan]
2069 [Thread-37] INFO  backtype.storm.daemon.task  - Emitting: wordGenerator default [mike]
2069 [Thread-21] INFO  backtype.storm.daemon.executor  - Processing received message source: wordGenerator:13, stream: default, id: {}, [mike]

The three RollingCountBolt instances start to emit their first sliding window counts:

1
2
3
4
5
6
7
8
9

4765 [Thread-19] INFO  backtype.storm.daemon.executor  - Processing received message source: __system:-1, stream: __tick, id: {}, [3]
4765 [Thread-19] INFO  storm.starter.bolt.RollingCountBolt  - Received tick tuple, triggering emit of current window counts
4765 [Thread-25] INFO  backtype.storm.daemon.executor  - Processing received message source: __system:-1, stream: __tick, id: {}, [3]
4765 [Thread-25] INFO  storm.starter.bolt.RollingCountBolt  - Received tick tuple, triggering emit of current window counts
4766 [Thread-21] INFO  backtype.storm.daemon.executor  - Processing received message source: __system:-1, stream: __tick, id: {}, [3]
4766 [Thread-21] INFO  storm.starter.bolt.RollingCountBolt  - Received tick tuple, triggering emit of current window counts
4766 [Thread-19] INFO  backtype.storm.daemon.task  - Emitting: counter default [golda, 24, 2]
4766 [Thread-25] INFO  backtype.storm.daemon.task  - Emitting: counter default [nathan, 33, 2]
4766 [Thread-21] INFO  backtype.storm.daemon.task  - Emitting: counter default [mike, 27, 2]

The two IntermediateRankingsBolt instances emit their intermediate rankings:

1
2
3
4

5774 [Thread-31] INFO  backtype.storm.daemon.task  - Emitting: intermediateRanker default [[[mike|27|2], [golda|24|2]]]
5774 [Thread-33] INFO  backtype.storm.daemon.task  - Emitting: intermediateRanker default [[[bertels|31|2], [jackson|19|2]]]
5774 [Thread-31] INFO  storm.starter.bolt.IntermediateRankingsBolt  - Rankings: [[mike|27|2], [golda|24|2]]
5774 [Thread-33] INFO  storm.starter.bolt.IntermediateRankingsBolt  - Rankings: [[bertels|31|2], [jackson|19|2]]

The single TotalRankingsBolt instance emits its global rankings:

1
2
3
4
5
6

3765 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: []
5767 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: []
7768 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: [[nathan|33|2], [bertels|31|2], [mike|27|2], [golda|24|2], [jackson|19|2]]
9770 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: [[bertels|76|5], [nathan|58|5], [mike|49|5], [golda|24|2], [jackson|19|2]]
11771 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: [[bertels|76|5], [nathan|58|5], [jackson|52|5], [mike|49|5], [golda|49|5]]
13772 [Thread-27] INFO  storm.starter.bolt.TotalRankingsBolt  - Rankings: [[bertels|110|8], [nathan|85|8], [golda|85|8], [jackson|83|8], [mike|71|8]]

What I Did Not Cover

I introduced a new feature to the Rolling Top Words code that I contributed back to storm-starter. This feature is a metric that tracks the difference between the configured length of the sliding window (in seconds) and the actual window length as seen in the emitted output data.

1

4763 [Thread-25] WARN  storm.starter.bolt.RollingCountBolt  - Actual window length is 2 seconds when it should be 9 seconds (you can safely ignore this warning during the startup phase)

This metric provides downstream data consumers with additional meta data, namely the time range that a data tuple actually covers. It is a nifty addition that will make the life of your fellow data scientists easier. Typically, you will see a difference between configured and actual window length a) during startup for the reasons mentioned above and b) when your machines are under high load and therefore not respond perfectly in time. I omitted the discussion of this new feature to prevent this article from getting too long.

Also, there are some minor changes in my own code that I did not contribute back to storm-starter because I did not want to introduce too many changes at once (such as a refactored TestWordSpout class).

Summary

In this article I described how to implement a distributed, real-time trending topics algorithm in Storm. It uses the latest features available in Storm 0.8 (namely tick tuples) and should be a good starting point for anyone trying to implement such an algorithm for their own application. The new code is now available in the official storm-starter repository, so feel free to take a deeper look.

You might ask whether there is a use of a distributed sliding window analysis beyond the use case I presented in this article. And for sure there is. The sliding window analysis described here applies to a broader range of problems than computing trending topics. Another typical area of application is real-time infrastructure monitoring, for instance to identify broken servers by detecting a surge of errors originating from problematic machines. A similar use case is identifying attacks against your technical infrastructure, notably flood-type DDoS attacks. All of these scenarios can benefit from sliding window analyses of incoming real-time data through tools such as Storm.

If you think the starter code can be improved further, please contribute your changes back to the storm-starter project on GitHub by forking the official repository, making your changes and sending a pull request.

Related Links

• Understanding the Parallelism of a Storm Topology

In this article I want to share my own understanding of the parallelism of a Storm topology after reading the documentation and writing some first prototype code. More specifically, I describe the relationships of worker processes, executors (threads) and tasks, and how you can configure them according to your needs. This article is based on Storm release 0.8.1, the latest version as of October 2012.

What is Storm?

For those readers unfamiliar with Storm here is a brief description taken from its homepage:

Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

What makes a running topology: worker processes, executors and tasks

Storm distinguishes between the following three main entities that are used to actually run a topology in a Storm cluster:

• Worker processes
• Executors (threads)
• Tasks

Here is a simple illustration of their relationships:

A worker process executes a subset of a topology. A worker process belongs to a specific topology and may run one or more executors for one or more components (spouts or bolts) of this topology. A running topology consists of many such processes running on many machines within a Storm cluster.

An executor is a thread that is spawned by a worker process. It may run one or more tasks for the same component (spout or bolt).

A task performs the actual data processing – each spout or bolt that you implement in your code executes as many tasks across the cluster. The number of tasks for a component is always the same throughout the lifetime of a topology, but the number of executors (threads) for a component can change over time. This means that the following condition holds true: #threads <= #tasks. By default, the number of tasks is set to be the same as the number of executors, i.e. Storm will run one task per thread.

Configuring the parallelism of a topology

Note that in Storm’s terminology “parallelism” is specifically used to describe the so-called parallelism hint, which means the initial number of executors (threads) of a component. In this article though I use the term “parallelism” in a more general sense to describe how you can configure not only the number of executors but also the number of worker processes and the number of tasks of a Storm topology. I will specifically call out when “parallelism” is used in the narrow definition of Storm.

The following table gives an overview of the various configuration options and how to set them in your code. There is more than one way of setting these options though, and the table lists only some of them. Storm currently has the following order of precedence for configuration settings: defaults.yaml > storm.yaml > topology-specific configuration > internal component-specific configuration > external component-specific configuration. Please take a look at the Storm documentation for more details.

Here is an example code snippet to show these settings in practice:

1
2
3

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
               .setNumTasks(4)
               .shuffleGrouping("blue-spout");

In the above code we configured Storm to run the bolt GreenBolt with an initial number of two executors and four associated tasks. Storm will run two tasks per executor (thread). If you do not explicitly configure the number of tasks, Storm will run by default one task per executor.

Example of a running topology

The following illustration shows how a simple topology would look like in operation. The topology consists of three components: one spout called BlueSpout and two bolts called GreenBolt and YellowBolt. The components are linked such that BlueSpout sends its output to GreenBolt, which in turns sends its own output to YellowBolt.

The GreenBolt was configured as per the code snippet above whereas BlueSpout and YellowBolt only set the parallelism hint (number of executors). Here is the relevant code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

Config conf = new Config();
conf.setNumWorkers(2); // use two worker processes

topologyBuilder.setSpout("blue-spout", new BlueSpout(), 2); // parallelism hint

topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)
               .setNumTasks(4)
               .shuffleGrouping("blue-spout");

topologyBuilder.setBolt("yellow-bolt", new YellowBolt(), 6)
               .shuffleGrouping("green-bolt");

StormSubmitter.submitTopology(
        "mytopology",
        conf,
        topologyBuilder.createTopology()
    );

And of course Storm comes with additional configuration settings to control the parallelism of a topology, including:

• TOPOLOGY_MAX_TASK_PARALLELISM: This setting puts a ceiling on the number of executors that can be spawned for a single component. It is typically used during testing to limit the number of threads spawned when running a topology in local mode. You can set this option via e.g. Config#setMaxTaskParallelism().

Update Oct 18: Nathan Marz informed me that TOPOLOGY_OPTIMIZE will be removed in a future release. I have therefore removed its entry from the configuration list above.

How to change the parallelism of a running topology

A nifty feature of Storm is that you can increase or decrease the number of worker processes and/or executors without being required to restart the cluster or the topology. The act of doing so is called rebalancing.

You have two options to rebalance a topology:

1. Use the Storm web UI to rebalance the topology.
2. Use the CLI tool storm rebalance as described below.

Here is an example of using the CLI tool:

1
2
3
4
5

 # Reconfigure the topology "mytopology" to use 5 worker processes,
 # the spout "blue-spout" to use 3 executors and
 # the bolt "yellow-bolt" to use 10 executors.

 $ storm rebalance mytopology -n 5 -e blue-spout=3 -e yellow-bolt=10

References for this article

To compile this article (and to write my related test code) I used information primarily from the following sources:

• The Storm wiki, most notably the pages Concepts, Running topologies on a production cluster, Local mode, Tutorial
• Storm 0.8.1 API documentation, most notably the class Config
• The announcement of Storm 0.8.0 release on the storm-user mailing list.

Summary

My personal impression is that Storm is a very promising tool. On the one hand I like its clean and elegant design, and on the other hand I loved to find out that a young open source tool can still have an excellent documentation. In this article I tried to summarize my own understanding of the parallelism of topologies, which may or may not be 100% correct – feel free to let me know if there are any mistakes in the description above!

In this blog post we will look at three commands:

• hadoop fsck (docs)
• hadoop fs -dus (docs)
• hadoop fs -count -q (docs and more docs)

hadoop fsck and hadoop fs -dus

First, let’s start with hadoop fsck and hadoop fs -dus because they will report identical numbers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ hadoop fsck /path/to/directory
 Total size:    16565944775310 B    <=== see here
 Total dirs:    3922
 Total files:   418464
 Total blocks (validated):      502705 (avg. block size 32953610 B)
 Minimally replicated blocks:   502705 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          18
 Number of racks:               1
FSCK ended at Thu Oct 20 20:49:59 CET 2011 in 7516 milliseconds

The filesystem under path '/path/to/directory' is HEALTHY

$ hadoop fs -dus /path/to/directory
hdfs://master:54310/path/to/directory        16565944775310    <=== see here

As you can see, hadoop fsck and hadoop fs -dus report the effective HDFS storage space used, i.e. they show the “normal” file size (as you would see on a local filesystem) and do not account for replication in HDFS. In this case, the directory path/to/directory has stored data with a size of 16565944775310 bytes (15.1 TB). Now fsck tells us that the average replication factor for all files in path/to/directory is exactly 3.0 This means that the total raw HDFS storage space used by these files – i.e. factoring in replication – is actually:

1

3.0 x 16565944775310 (15.1 TB) = 49697834325930 Bytes (45.2 TB)

This is how much HDFS storage is consumed by files in path/to/directory

hadoop fs -count -q

Now, let us inspect the HDFS quota set for path/to/directory. If we run hadoop fs -count -q we get this result:

1
2
3

$ hadoop fs -count -q /path/to/directory
  QUOTA  REMAINING_QUOTA     SPACE_QUOTA  REMAINING_SPACE_QUOTA    DIR_COUNT  FILE_COUNT      CONTENT_SIZE FILE_NAME
   none              inf  54975581388800          5277747062870        3922       418464    16565944775310 hdfs://master:54310/path/to/directory

(I manually added the column headers like QUOTA to the output for making it easier to read.)

The seventh column CONTENT_SIZE is, again, the effective HDFS storage space used: 16565944775310 Bytes (15.1 TB).

The third column SPACE_QUOTA however, 54975581388800 is the raw HDFS space quota in bytes. The fourth column REMAINING_SPACE_QUOTA with 5277747062870 is the remaining raw HDFS space quota in bytes. Note that whereas hadoop fsck and hadoop fs -dus report the effective data size (= the same numbers you see on a local filesystem), the third and fourth columns of hadoop fs -count -q indirectly return how many bytes this data actually consumes across the hard disks of the distributed cluster nodes – and for this it is counting each of the 3.0 replicas of an HDFS block individually (here, the value 3.0 has been taken from the hadoop fsck output above and actually matches the default value of the replication count). So if we make the subtraction of these two quota-related numbers we get back the number from above:

1

54975581388800 (50 TB) - 5277747062870 (4.8 TB) = 49697834325930 (45.2 TB)

Now keep in mind that the Hadoop space quota always counts against the raw HDFS disk space consumed. So if you have a quota of 10MB, you can store only a single 1MB file if you set its replication to 10. Or you can store up to three 1MB files if their replication is set to 3. The reason why Hadoop’s quotas work like that is because the replication count of an HDFS file is a user-configurable setting. Though Hadoop ships with a default value of 3 it is up to the users to decide whether they want to keep this value or change it. And because Hadoop can’t anticipate how users might be playing around with the replication setting for their files, it was decided that the Hadoop quotas always operate on the raw HDFS disk space consumed.

TL;DR and Summary

If you never change the default value of 3 for the HDFS replication count of any files you store in your Hadoop cluster, this means in a nutshell that you should always multiply the numbers reported by hadoop fsck or hadoop fs -dus times 3 when you want to reason about HDFS space quotas.

I hope this clears things up a bit!

Related Articles

• Hadoop space quotas, HDFS block size, replication and small files (earlier blog post)

Before We Start

This guide should cover the majority of tasks and aspects of upgrading HDFS but it may not necessarily be complete. Feel free to provide your feedback and suggestions.

Apart from my own input, this article uses information from the Hadoop Upgrade guide, the HDFS User Guide and Tom White’s book Hadoop: The Definitive Guide (see References section at the bottom).

Background information on HDFS upgrades

An HDFS upgrade may be required when you upgrade from an older version of Hadoop to a newer version. For instance, an HDFS upgrade is required when you upgrade from Hadoop 0.20.2 to Hadoop 0.20.203.0.

An upgrade of HDFS makes a copy of the previous version’s metadata and data. Doing an upgrade does not double the storage requirements of the cluster, as the datanodes use hard links to keep two references (for the current and previous version) to the same block of data. This design makes it straightforward to roll back to the previous version of the filesystem, should you need to. You should understand that any changes made to the data on the upgraded system will be lost after the rollback completes.

You can keep only the previous version of the filesystem: you can’t roll back several versions. Therefore, to carry out another upgrade to HDFS data and metadata, you will need to delete the previous version, a process called finalizing the upgrade. Once an upgrade is finalized, there is no procedure for rolling back to a previous version.

How to check whether an HDFS upgrade is required

Unfortunately, the most reliable way of finding out whether you need to upgrade the HDFS filesystem is by performing a trial on a test cluster (doh!).

If you have installed a new version of Hadoop and it expects a different HDFS layout version, the NameNode will refuse to run. A message like the following will appear in the NameNode logs:

1
2
3

File system image contains an old layout version -18.
An upgrade to version -31 is required.
Please restart NameNode with -upgrade option.

HDFS upgrade instructions

There are several steps you have to perform for upgrading HDFS. First, you have to perform some preparation steps before installing the new (updated) Hadoop software. Then, after having updated Hadoop (e.g. from version 0.20.2 to 0.20.203.0), you will follow-up with another set of steps to actually start the HDFS upgrade and, after successful testing, finalize it.

Before you install the new Hadoop version

1. Make sure that any previous upgrade is finalized before proceeding with another upgrade. To find out whether the cluster needs to be finalized run the command:

   $ hadoop dfsadmin -upgradeProgress status
   

2. Stop all client applications running on the MapReduce cluster.

3. Stop the MapReduce cluster with

   $ stop-mapred.sh
   

   and kill any orphaned task process on the TaskTrackers.

4. Stop all client applications running on the HDFS cluster.
5. Perform some sanity checks on HDFS.

   • Perform a filesystem check:

       $ hadoop fsck / -files -blocks -locations > dfs-v-old-fsck-1.log
     

   • Fix HDFS to the point there are no errors. The resulting file will contain a complete block map of the file system. Note: Redirecting the fsck output is recommend for large clusters in order to avoid time consuming output to STDOUT.

   • Save a complete listing of the HDFS namespace to a local file:

       $ hadoop dfs -lsr / > dfs-v-old-lsr-1.log
     

   • Create a list of DataNodes participating in the cluster:

       $ hadoop dfsadmin -report > dfs-v-old-report-1.log
     

6. Optionally, copy all or unrecoverable only data stored in HDFS to a local file system or a backup instance of HDFS.

7. Optionally, stop and restart HDFS cluster, in order to create an up-to-date namespace checkpoint of the old version:

   $ stop-dfs.sh
   $ start-dfs.sh
   

8. Optionally, repeat 5, 6, 7 and compare the results with the previous run to ensure the state of the file system remained unchanged.
9. Create a backup copy of the dfs.name.dir directory on the NameNode (if you followed my Hadoop tutorials: /app/hadoop/tmp/dfs/name). Among other important files, the dfs.name.dir directory includes the checkpoint files edits and fsimage.

10. Stop the HDFS cluster.

    $ stop-dfs.sh
    

    Verify that HDFS has really stopped, and kill any orphaned DataNode processes on the DataNodes.

Install the new Hadoop version

Now you can install the new version of the Hadoop software.

Note: Make sure to update any symlinks you are using; e.g. if you have symlinks such as /usr/local/hadoop → /usr/local/hadoop-0.20.203.0.

After you have installed the new Hadoop version

1. Optionally, update the conf/slaves file before starting to reflect the current set of active nodes.
2. Optionally, change the configuration of the NameNode’s and the JobTracker’s port numbers (i.e. the fs.default.name property in conf/core-site.xml plus conf/hdfs-site.xml and the mapred.job.tracker property in conf/mapred-site.xml respectively) in order to ignore unreachable nodes that are still running the old version (with the old port numbers), preventing them from connecting and disrupting system operation.
3. Start the actual HDFS upgrade process.

   • Upgrade the NameNode by converting the checkpoint to the new version format

       $ hadoop-daemon.sh start namenode -upgrade
     

     Note: You need to add the -upgrade switch only once for actual upgrade process. Once it has successfully completed, you can start the NameNode via hadoop-daemon.sh start-dfs.sh and start-all.sh like you’d normally do. The un-finalized upgrade will be in effect until you either finalize the upgrade to make it permanent or until you perform a rollback of the upgrade (see below).

     The NameNode log will show messages like the following:

       # NameNode log file
       2011-06-21 14:40:32,579 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading image directory /path/to/nn_namespace_dir.
          old LV = -18; old CTime = 0.
          new LV = -31; new CTime = 1308660032579
       2011-06-21 14:40:32,581 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 8447 saved in 0 seconds.
       2011-06-21 14:40:32,639 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrade of /path/to/nn_namespace_dir is complete.
       2011-06-21 14:40:32,639 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading image directory /path/to/nn_namespace_dir_bk.
          old LV = -18; old CTime = 0.
          new LV = -31; new CTime = 1308660032579
       2011-06-21 14:40:32,644 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 8447 saved in 0 seconds.
       2011-06-21 14:40:32,650 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrade of /path/to/nn_namespace_dir_bk is complete.
       2011-06-21 14:40:32,651 INFO org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0 entries 0 lookups
       2011-06-21 14:40:32,652 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 441 msecs
       2011-06-21 14:40:32,660 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
     

   • The NameNode upgrade process can take a while depending on how many files you have. You can follow the process by inspecting the NameNode logs, by running “hadoop dfsadmin -upgradeProgress status and/or by accessing the NameNode Web UI. Once the upgrade process has completed, the NameNode Web UI will show a message similar to Upgrade for version -31 has been completed. Upgrade is not finalized You will finalize the upgrade in a later step. Right now, the NameNode is in Safe Mode waiting for the DataNodes to connect.

     

     An example status output of the upgradeProgress command at this stage:

       $ hadoop dfsadmin -upgradeProgress status
       Upgrade for version -31 has been completed.
       Upgrade is not finalized.
     

   • Optionally, save a complete listing of the new HDFS namespace to a local file:

       $ hadoop dfs -lsr / > dfs-v-new-lsr-0.log
     

     and compare it with dfs-v-old-lsr-1.log you created previously.

   • Start the HDFS cluster. Since the NameNode is already running, only the DataNodes and the SecondaryNameNode will actually be started.

       $ start-dfs.sh
     

     Note: You do not need to add the upgrade switch here because it is only passed to the NameNode anyways, and the NameNode has already been instructed to perform an upgrade.

     After your DataNodes have completed the upgrade process, you should see a message Safe mode will be turned off automatically in X seconds. in the NameNode Web UI. The NameNode should then automatically exit Safe Mode and HDFS will be in full operation. You can monitor the process via the NameNode Web UI and the NameNode/DataNode logs.

       # DataNode log file
       2011-06-21 14:50:56,103 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrading storage directory /app/hadoop/tmp/dfs/data.
          old LV = -18; old CTime = 0.
          new LV = -31; new CTime = 1308660032579
       2011-06-21 14:50:56,196 INFO org.apache.hadoop.hdfs.server.common.Storage: HardLinkStats: 1 Directories, including 0 Empty Directories, 0 single Link o
       perations, 1 multi-Link operations, linking 80 files, total 80 linkable files.  Also physically copied 1 other files.
       2011-06-21 14:50:56,196 INFO org.apache.hadoop.hdfs.server.common.Storage: Upgrade of /app/hadoop/tmp/dfs/data is complete.
     

     You can check the NameNode Web UI whether the NameNode has already exited Safe Mode (see screenshot below). Alternatively, you can run hadoop dfsadmin -safemode get.

     

     Note that the status output of the upgradeProgress command should not have changed at this point:

       $ hadoop dfsadmin -upgradeProgress status
       Upgrade for version -31 has been completed.
       Upgrade is not finalized.
     

4. Perform some sanity checks on the new HDFS

   • Create a list of DataNodes participating in the updated cluster.

       $ hadoop dfsadmin -report > dfs-v-new-report-1.log
     

     and compare it with dfs-v-old-report-1.log to ensure all DataNodes previously belonging to the cluster are up and running.

   • Save a complete listing of the new HDFS namespace to a local file:

       $ hadoop dfs -lsr / > dfs-v-new-lsr-1.log
     

     and compare it with dfs-v-old-lsr-1.log. These files should be identical unless the format of hadoop fs -lsr reporting or the data structures have changed in the new version.

   • Perform a filesystem check:

       $ hadoop fsck / -files -blocks -locations > dfs-v-new-fsck-1.log
     

     and compare with dfs-v-old-fsck-1.log. These files should be identical, unless the hadoop fsck reporting format has changed in the new version.

5. Start the MapReduce cluster

   $ start-mapred.sh
   

6. Let internal customers perform their own testing on the new HDFS filesystem version.
7. Roll back or finalize the upgrade (optional).

Finishing the HDFS upgrade process

After you have initiated the HDFS upgrade in the sections above, you will eventually have to decide – hopefully after thorough testing – whether you want to stick to the upgraded HDFS (= your tests were successful) or to revert the HDFS upgrade (= your tests failed). The following two sections describe how to finalize (i.e. stick to) an HDFS upgrade and how to perform a rollback of an HDFS upgrade.

How to finalize an HDFS upgrade

If the HDFS upgrade worked out fine and your subsequent testing was successful, you may want to finalize the HDFS upgrade.

You can check with the following command whether the cluster needs to be finalized:

1

$ hadoop dfsadmin -upgradeProgress status

Run the actual finalize command to make the HDFS upgrade permanent:

1

$ hadoop dfsadmin -finalizeUpgrade

The -finalizeUpgrade command removes the previous version of the NameNode’s and DataNodes’ storage directories.

In the NameNode Web UI you should see a message "Upgrades: There are no upgrades in progress".

The NameNode logs should contain entries similar to the following:

1
2
3
4

# NameNode log file
2011-06-22 20:21:22,333 INFO org.apache.hadoop.hdfs.server.common.Storage: Finalizing upgrade for storage directory /app/hadoop/tmp/dfs/name.
   cur LV = -31; cur CTime = 1308774082
2011-06-22 20:21:22,336 INFO org.apache.hadoop.hdfs.server.common.Storage: Finalize upgrade for /app/hadoop/tmp/dfs/name is complete.

Also, running hadoop dfsadmin will now report that there are no pending upgrades.

1
2

$ hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.

Note: The finalize upgrade procedure can run in the background without disrupting the performance of the Hadoop cluster.

It is worth mentioning that deleting files that existed before the upgrade does not free up real disk space on the DataNodes until the HDFS cluster is finalized.

The official HDFS User Guide in the Hadoop documentation provides the full instructions for HDFS Upgrade and Rollback.

How to roll back an HDFS upgrade

If the HDFS upgrade failed and/or your testing was not successful, you may want to revert the HDFS upgrade.

The official HDFS User Guide in the Hadoop documentation provides the full instructions for HDFS Upgrade and Rollback. This section only highlights the most important parts.

If there is a need to move back to the old version, you have to:

1. Stop all client applications on the cluster.
2. Stop the full cluster (MapReduce + HDFS).
3. Distribute the previous version of Hadoop, i.e. the Hadoop version used before the attempted upgrade.
4. Start the HDFS cluster with the rollback option. $ start-dfs.sh -rollback

The NameNode logs should contain entries similar to the following:

1
2
3
4
5
6

# NameNode log file
2011-06-22 19:31:54,039 INFO org.apache.hadoop.hdfs.server.common.Storage: Rolling back storage directory /app/hadoop/tmp/dfs/name.
   new LV = -18; new CTime = 0
2011-06-22 19:31:54,041 INFO org.apache.hadoop.hdfs.server.common.Storage: Rollback of /app/hadoop/tmp/dfs/name is complete.
2011-06-22 19:31:54,042 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1117
2011-06-22 19:31:54,191 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0

The DataNode logs should contain entries similar to the following:

1
2
3
4

# DataNode log file
2011-06-22 19:31:55,816 INFO org.apache.hadoop.hdfs.server.common.Storage: Rolling back storage directory /app/hadoop/tmp/dfs/data.
   target LV = -18; target CTime = 0
2011-06-22 19:32:02,363 INFO org.apache.hadoop.hdfs.server.common.Storage: Rollback of /app/hadoop/tmp/dfs/data is complete.

The rollback will take some time to complete. The NameNode will stay in Safe Mode until all DataNodes have finished their rollback work and have registered back at the NameNode.

Also, running dfsadmin will now also report that there are no pending upgrades:

1
2

$ hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.

The HDFS is now reverted back to its previous state, i.e. the state before you run -upgrade.

References

• Hadoop Upgrade on the Hadoop Wiki
• HDFS User Guide for Hadoop 0.20.203.0
• File System Shell Guide for Hadoop 0.20.203.0
• Commands Guide for Hadoop 0.20.203.0
• Tom White, Hadoop: The Definitive Guide (2nd ed.), O’Reilly, pp. 316-319 (plus some other pages)

Before we start

The examples below use git (not svn)

In the following sections, I will use git as the version control system to work on the Hadoop source code. Why? Because I am much more comfortable with git than svn, so please bear with me.

If you are using Subversion, feel free to adapt the git commands described below. You are invited to write a comment to this article about your SVN experience so that other SVN users can benefit, too!

Hadoop 0.20.2 versus 0.20.203.0

Hadoop is covered. What about HBase then?

I focus solely in this article on building a Hadoop 0.20.x version (see the Background section below) that is compatible with HBase 0.90.2. In a future article, I may describe how to actually install and set up HBase 0.90.2 on the Hadoop 0.20.x version that we created here.

Version of Hadoop 0.20-append used in this article

The instructions below use the latest version of branch-0.20-append. As of this writing, the latest commit to the append branch is git commit df0d79cc aka Subversion rev 1057313. For reference, the corresponding commit message is “HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.” from January 10, 2011.

1
2
3
4
5
6
7

commit df0d79cc2b09438c079fdf10b913936492117917
Author: Hairong Kuang <hairong@apache.org>
Date:   Mon Jan 10 19:01:36 2011 +0000

    HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.

    git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append@1057313 13f79535-47bb-0310-9956-ffa450edef68

That said, the steps should also work for newer versions of branch-0.20-append.

Background

Hadoop and HBase: Which versions to pick for production clusters?

Hadoop 0.20.2 is the latest stable release of Apache Hadoop that is marked ready for production. Unfortunately, the latest stable release of Apache HBase, i.e. HBase 0.90.2, is not compatible with Hadoop 0.20.2: If you try to run HBase 0.90.2 on an unmodified version of Hadoop 0.20.2 release, you might lose data!

The following lines are taken slightly modified from the HBase documentation:

This version of HBase [0.90.2] will only run on Hadoop 0.20.x. It will not run on Hadoop 0.21.x (nor 0.22.x). HBase will lose data unless it is running on an HDFS that has a durable sync. Currently only the branch-0.20-append branch has this attribute. No official releases have been made from this branch up to now so you will have to build your own Hadoop from the tip of this branch.

Here is a quick overview:

To be honest, it took me quite some time to get up to speed with the various requirements, dependencies, project statuses, etc. for marrying Hadoop 0.20.x and HBase 0.90.2. Hence I want to contribute back to the Hadoop and HBase communities by writing this article.

Alternatives to what we are doing here

Another option you have to get HBase up and running on Hadoop – rather than build Hadoop 0.20-append yourself – is using Cloudera’s CDH3 distribution. CDH3 has the Hadoop 0.20-append patches needed to add a durable sync, i.e. to make Hadoop 0.20.x compatible with HBase 0.90.2.

A word of caution and a Thank You

First, a warning: while I have taken great care to compile and describe the steps in the following sections, I still cannot give you any guarantees. If in doubt, join our discussions on the HBase mailing list.

Second, I am only stitching together the pieces of the puzzle here. The heavy lifting has done by others. Hence I would like to thank Michael Stack for his great feedback while preparing the information for this article, and to both him and the rest of the HBase developers for their help on the HBase mailing list. It’s much appreciated!

Building Hadoop 0.20-append from branch-0.20-append

Retrieve the Hadoop 0.20-append sources

Hadoop as of version 0.20.x is not separated into the Common, HDFS and MapReduce components as the versions >= 0.21.0 are. Hence you find all the required code in the Hadoop Common repository.

So the first step is to check out the Hadoop Common repository.

1
2

$ git clone http://git.apache.org/hadoop-common.git
$ cd hadoop-common

However, the previous git command only retrieved the latest version of Hadoop common, i.e. the tip aka HEAD of the development for Hadoop Common. We however are only interested in the code tree for Hadoop 0.20-append, i.e. the branch branch-0.20-append. Because git by default does not download remote branches from a cloned repository, we must instruct it to explicitly do so:

1
2
3
4
5

# Retrieve the (remote) Hadoop 0.20-append branch as git normally checks out
# only the ``master`` tree (``trunk`` in Subversion language).
$ git checkout -t origin/branch-0.20-append
Branch branch-0.20-append set up to track remote branch branch-0.20-append from origin.
Switched to a new branch 'branch-0.20-append'

Hadoop 0.20.2 release vs. Hadoop 0.20-append

Up to now, you might have asked yourself what the difference between the 0.20.2 release of Hadoop and its append branch actually is. Here’s the answer: The Hadoop 0.20-append branch is effectively a superset of Hadoop 0.20.2 release. In other words, there is not a single “real” commit in Hadoop 0.20.2 release that is not also in Hadoop 0.20-append. This means that Hadoop 0.20-append brings all the goodies that Hadoop 0.20.2 release has, great!

Run the following git command to verify this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ git show-branch release-0.20.2 branch-0.20-append
! [release-0.20.2] Hadoop 0.20.2 release
 * [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
--
 * [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
 * [branch-0.20-append^] HDFS-1555. Disallow pipelien recovery if a file is already being lease recovered. Contributed by Hairong Kuang.
 * [branch-0.20-append~2] Revert the change made to HDFS-1555: merge -c -1056483 https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 * [branch-0.20-append~3] HDFS-1555. Disallow pipeline recovery if a file is already being lease recovered. Contributed by Hairong Kuang.
[...]
 * [branch-0.20-append~50] JDiff output for release 0.20.2
 * [branch-0.20-append~51] HADOOP-1849. Merge -r 916528:916529 from trunk to branch-0.20.
+  [release-0.20.2] Hadoop 0.20.2 release
+  [release-0.20.2^] Hadoop 0.20.2-rc4
+* [branch-0.20-append~52] Prepare for 0.20.2-rc4

As you can see, there are only two commits in 0.20.2 release that are not in branch-0.20-append, namely the commits “Hadoop 0.20.2 release” and “Hadoop 0.20.2-rc4”. Both of these commits are simple tagging commits, i.e. they are just used for release management but do not introduce any changes to the content of the Hadoop source code.

Run the build process

Build commands

First, we have to create the build.properties file (see full instructions).

Here are the contents of mine:

1
2
3
4
5
6
7
8
9

#this is essential
resolvers=internal
#you can increment this number as you see fit
version=0.20-append-for-hbase
project.version=${version}
hadoop.version=${version}
hadoop-core.version=${version}
hadoop-hdfs.version=${version}
hadoop-mapred.version=${version}

The build.properties file should be placed (or available) in the hadoop-common top directory, i.e. hadoop-common/build.properties. You can either place the file there directly or you can follow the recommended approach, where you place the file in a parent directory and create a symlink to it. The latter approach is convenient if you also have checked out the repositories of the Hadoop sub-projects hadoop-hdfs and hadoop-mapreduce and thus want to use the same build.properties file for all three sub-projects.

1
2
3
4
5
6
7
8

$ pwd
/your/path/to/hadoop-common

# Create/edit the build.properties file
$ vi ../build.properties

# Create a symlink to it
$ ln -s ../build.properties build.properties

Now we are ready to compile Hadoop from source with ant. I used the command ant mvn-install as described on Git and Hadoop. The build itself should only take a few minutes. Be sure to run ant test as well (or only ant test-core if you’re lazy) but be aware that the tests take much longer than the build (two hours on my 3-year old MacBook Pro, for instance).

1
2
3
4
5
6
7
8
9

# Make sure we are using the branch-0.20-append sources
$ git checkout branch-0.20-append

# Run the build process
$ ant mvn-install

# Optional: run the full test suite or just the core test suite
$ ant test
$ ant test-core

The build test fails, now what?

Now comes the more delicate part: If you run the build tests via ant test, you will notice that the build test process always fails! One consistent test error is reported by TestFileAppend4 and logged to the file build/test/TEST-org.apache.hadoop.hdfs.TestFileAppend4.txt. Here is a short excerpt of the test’s output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

2011-04-06 09:40:28,666 INFO  ipc.Server (Server.java:run(970)) - IPC Server handler 5 on 47574, call append(/bbw.test, DFSClient_1066000827) from 127.0.0.1:45323: error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /bbw.test for DFSClient_1066000827 on client 127.0.0.1, because this file is already being created by DFSClient_-95621936 on 127.0.0.1
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1054)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1221)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:396)
        [...]
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955)
2011-04-06 09:40:28,667 INFO  hdfs.TestFileAppend4 (TestFileAppend4.java:recoverFile(161)) - Failed open for append, waiting on lease recovery

[...]
Testcase: testRecoverFinalizedBlock took 5.555 sec
    Caused an ERROR
No lease on /testRecoverFinalized File is not open for writing. Holder DFSClient_1816717192 does not have any open files.
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /testRecoverFinalized File is not open for writing. Holder DFSClient_1816717192 does not have any open files.
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1439)
        [...]
    at org.apache.hadoop.hdfs.TestFileAppend4$1.run(TestFileAppend4.java:636) 

Fortunately, this error does not mean that the build is not working. From what we know this is a problem of the unit tests in branch-0.20-append themselves (see also Michael Stack’s comment on HBASE-3285).

Occasionally, you might run into other build failures and/or build errors. On my machine, for instance, I have also seen the following tests fail:

• org.apache.hadoop.hdfs.server.namenode.TestEditLogRace (see error)
• org.apache.hadoop.hdfs.TestMultiThreadedSync (see error)

I do not know what might cause these occasional errors – maybe it is a problem of the machine I am running the tests on. Still working on this.

Frankly, what I wrote above may sound discomforting to you. At least it does to me. Still, the feedback I have received on the HBase mailing list indicates that the Hadoop 0.20-append build as done above is indeed correct.

Locate the build output (Hadoop JAR files)

By default, the build run via ant mvn-install places the generated Hadoop JAR files in $HOME/.m2/repository. You can find the actual JAR files with the following command.

1
2
3
4
5
6
7

$ find $HOME/.m2/repository -name "hadoop-*.jar"

.../repository/org/apache/hadoop/hadoop-examples/0.20-append-for-hbase/hadoop-examples-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-test/0.20-append-for-hbase/hadoop-test-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-tools/0.20-append-for-hbase/hadoop-tools-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-streaming/0.20-append-for-hbase/hadoop-streaming-0.20-append-for-hbase.jar
.../repository/org/apache/hadoop/hadoop-core/0.20-append-for-hbase/hadoop-core-0.20-append-for-hbase.jar

Install your Hadoop 0.20-append build in your Hadoop cluster

The only thing left to do now is to install the Hadoop 0.20-append build in your cluster. This step is easy: simply replace the Hadoop JAR files of your existing installation of Hadoop 0.20.2 release with the ones you just created above. You will also have to replace the Hadoop core JAR file in your HBase 0.90.2 installation ($HBASE_HOME/lib/hadoop-core-0.20-append-r1056497.jar) with the Hadoop core JAR file you created above (hadoop-core-0.20-apppend-for-hbase.jar if you followed the instructions above).

Rename the build JAR files if you run Hadoop 0.20.2

Hadoop 0.20.2 release names its JAR files in the form of hadoop-VERSION-PACKAGE.jar, e.g. hadoop-0.20.2-examples.jar. The build process above uses the different scheme hadoop-PACKAGE-VERSION.jar, e.g. hadoop-examples-0.20-append-for-hbase.jar. You might therefore want to rename the JAR files you created in the previous section so that they match the naming scheme of Hadoop 0.20.2 release (otherwise the bin/hadoop script will not be able to add the Hadoop core JAR file to its CLASSPATH, and also command examples such as hadoop jar hadoop-*-examples.jar pi 50 1000 in the Hadoop docs will not work as is).

1
2
3
4
5
6
7

# When you replace the Hadoop JAR files *in your Hadoop installation*,
# you might want to rename your Hadoop 0.20-append JAR files like so.
hadoop-examples-0.20-append-for-hbase.jar  --> hadoop-0.20-append-for-hbase-examples.jar
hadoop-test-0.20-append-for-hbase.jar      --> hadoop-0.20-append-for-hbase-test.jar
hadoop-tools-0.20-append-for-hbase.jar     --> hadoop-0.20-append-for-hbase-tools.jar
hadoop-streaming-0.20-append-for-hbase.jar --> hadoop-0.20-append-for-hbase-streaming.jar
hadoop-core-0.20-append-for-hbase.jar      --> hadoop-0.20-append-for-hbase-core.jar

In contrast, HBase uses the hadoop-PACKAGE-VERSION.jar scheme. So when you replace the Hadoop core JAR file shipped with HBase 0.90.2 in $HBASE_HOME/lib, you can here opt for leaving the name of the newly built Hadoop core JAR file as is.

Maintaining your own version of Hadoop 0.20-append

If you must integrate some additional patches into Hadoop 0.20.2 and/or Hadoop 0.20-append (normally in the form of backports of patches for Hadoop 0.21 or 22.0), you can create a local branch based on the Hadoop version you are interested in. Yes, this creates some effort on your behalf so you should be sure to weigh the pros and cons of doing so.

Imagine that, for instance, you use Hadoop 0.20-append based on branch-0.20-append because you also want to run the latest stable release of HBase on your Hadoop cluster. While doing your >benchmarking and stress testing of your cluster, you have unfortunately discovered a problem that you could track down to HDFS-611. Now a patch is actually available (you might have to do some tinkering to backport it properly) but it is not in the version of Hadoop you are running, i.e. it is not in the vanilla branch-0.20-append.

What you can do is to create a local git branch based on your Hadoop version (here: branch-0.20-append) where you can integrate and test any relevant patches you need. Please understand that I will only describe the basic approach here – I do not go into details on how you can make sure to stay current with any changes to the Hadoop version you are tracking after you followed the steps below. There are a lot of splendid git introductions such as the Git Community Book that can explain this much better and thoroughly than I am able to.

1
2
3
4
5

# Make sure we are in branch-0.20-append before running the next command
$ git checkout branch-0.20-append

# Create your own local branch based on the latest version (HEAD) of the official branch-0.20-append
$ git checkout -b branch-0.20-append-yourbranch

Verify that the two append branches are identical up to now.

1
2
3
4
5
6
7
8

# Verify that your local branch and the "official" (remote) branch are identical
$ git show-branch branch-0.20-append branch-0.20-append-yourbranch
! [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
 * [branch-0.20-append-yourbranch] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
--
+* [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.

# Yep, they are.

Apply the relevant patch to your branch. In the example below, I apply a backport of the patch for HDFS-611 for branch-0.20-append via the file HDFS-611.branch-0.20-append.v1.patch. Note that this backport is not available on the HDFS-611 page – I created the backport myself based on the HDFS-611 patch for Hadoop 0.20.2 release (HDFS-611.branch-20.v6.patch).

1
2
3
4
5
6
7
8
9
10
11

# Apply the patch to your branch
$ patch -p1 < HDFS-611.branch-0.20-append.v1.patch

# Add any modified or newly created files from the patch to git's index
$ git add src/hdfs/org/apache/hadoop/hdfs/protocol/FSConstants.java \
         src/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDataset.java \
         src/hdfs/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java \
         src/test/org/apache/hadoop/hdfs/TestDFSRemove.java

# Commit the changes from the index to the repository
$ git commit -m "HDFS-611: Backport of HDFS-611 patch for Hadoop 0.20.2 release"

Verify that your patched branch is one commit ahead of the original (remote) append branch.

1
2
3
4
5
6
7
8
9

# Compare the commit histories of your local append branch and the original (remote) branch
$ git show-branch branch-0.20-append branch-0.20-append-yourbranch
! [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
 * [branch-0.20-append-yourbranch] HDFS-611: Backport of HDFS-611 patch for Hadoop 0.20.2 release
--
 * [branch-0.20-append-yourbranch] HDFS-611: Backport of HDFS-611 patch for Hadoop 0.20.2 release
+* [branch-0.20-append] HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.

# Yep, it is exactly one commit ahead.

Voilà!

And by the way, if you want to see the commit differences between Hadoop 0.20.2 release, the official branch-0.20-append and your own, patched branch-0.20-append-yourbranch, run the following git command:

1

$ git show-branch release-0.20.2 branch-0.20-append branch-0.20-append-yourbranch

Conclusion

I hope this article helps you to build a Hadoop 0.20.x version for running HBase 0.90.2 on in a production environment! Your feedback and comments are as always appreciated.

Before we start

Let me first talk about a few things that you should be aware of while reading through this article.

What we (not) want to do

The purpose of this article is to give you a quick overview and walkthrough of some of Hadoop’s most popular benchmarking and testing tools. All the tools described in the following sections are part of the Apache Hadoop distribution, so they should already be available in your cluster and waiting to be (ab)used! I do not focus on the general concepts and best practices of benchmarking an Hadoop cluster in this article but rather want to get you up to speed with actually using these tools. My motivation comes from my own – sometimes frustrating – experience with using these tools, as they often lack clear documentation.

That said, at any time feel free to contribute back to the Hadoop project by helping to improve the status of the documentation. I know I’ll try to do my fair share.

Prerequisites

If you want to follow along the examples in this article, you obviously need access to a running cluster. In the case that you are still trying to install and configure your cluster, my following tutorials might help you to build one. The tutorials are tailored to Ubuntu Linux but the information does also apply to other Linux/Unix variants.

• Running Hadoop On Ubuntu Linux (Single-Node Cluster) How to set up a single-node Hadoop cluster using the Hadoop Distributed File System (HDFS) on Ubuntu Linux

• Running Hadoop On Ubuntu Linux (Multi-Node Cluster) How to set up a multi-node Hadoop cluster using the Hadoop Distributed File System (HDFS) on Ubuntu Linux

Version focus: Hadoop 0.20.2

I put the focus on the benchmark and testing tools shipped with Hadoop version 0.20.2. At this moment this is the latest production-ready release Hadoop (0.21 is not!). Until Hadoop 0.22 is out, a lot of Hadoop users (including me) are therefore sticking to the tested and true Hadoop 0.20.2 release.

NotReplicatedYetException and AlreadyBeingCreatedException errors

If your cluster is running the stock version of Hadoop 0.20.2 release, you might run into NotReplicatedYetException and/or AlreadyBeingCreatedException “errors” while test-driving your cluster (as you can see in the output below that they are actually logged as INFO):

1
2
3
4
5
6
7
8
9
10
11
12
13

2011-02-11 03:48:05,367 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310, call addBlock(/user/hduser/terasort-input/_temporary/_attempt_201102110201_0008_m_000020_0/part-00020, DFSClient_attempt_201102110201_0008_m_000020_0) from 192.168.0.2:45133: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/user/hduser/terasort-input/_temporary/_attempt_201102110201_0008_m_000020_0/part-00020
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/user/hduser/terasort-input/_temporary/_attempt_201102110201_0008_m_000020_0/part-00020
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1257)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

This often happens when you delete a large amount of HDFS data on the cluster – and this is something that you might want to do in between test runs so that your cluster does not fill up. This is a known problem in 0.20.2 and fixed in 0.21.0 (see HDFS-611). Fortunately, there is a patch available: HDFS-611.branch-20.v6.patch (link) is the file you need for Hadoop 0.20.2.

Describing how to patch Hadoop and rebuild it from source is unfortunately beyond the scope of this article. Sorry! If you actually happen to run into this problem, please have a look at the Hadoop documentation (e.g. the FAQ, How to Contribute, Working with Hadoop under Eclipse or Git and Hadoop are a very good start) and/or consult the Hadoop mailing lists. You can also check out my article Building an Hadoop 0.20.x version for HBase 0.90.2, which should jumpstart you with creating your own custom Hadoop build.

Another workaround is to simply wait some minutes after deleting a large amount of data before starting another test run. You might want to monitor the NameNode’s log file to check when the asynchronous background delete processes have finished their work.

Overview of Benchmarks and Testing Tools

The Hadoop distribution comes with a number of benchmarks, which are bundled in hadoop-*test*.jar and hadoop-*examples*.jar. The four benchmarks we will be looking at in more details are TestDFSIO, nnbench, mrbench (in hadoop-*test*.jar) and TeraGen / TeraSort / TeraValidate (in hadoop-*examples*.jar).

Here is the full list of available options in hadoop-*test*.jar:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# change to your Hadoop installation directory;
# if you followed my Hadoop tutorials, this is /usr/local/hadoop
$ cd /usr/local/hadoop

$ bin/hadoop jar hadoop-*test*.jar
An example program must be given as the first argument.
Valid program names are:
  DFSCIOTest: Distributed i/o benchmark of libhdfs.
  DistributedFSCheck: Distributed checkup of the file system consistency.
  MRReliabilityTest: A program that tests the reliability of the MR framework by injecting faults/failures
  TestDFSIO: Distributed i/o benchmark.
  dfsthroughput: measure hdfs throughput
  filebench: Benchmark SequenceFile(Input|Output)Format (block,record compressed and uncompressed), Text(Input|Output)Format (compressed and uncompressed)
  loadgen: Generic map/reduce load generator
  mapredtest: A map/reduce test check.
  mrbench: A map/reduce benchmark that can create many small jobs
  nnbench: A benchmark that stresses the namenode.
  testarrayfile: A test for flat files of binary key/value pairs.
  testbigmapoutput: A map/reduce program that works on a very big non-splittable file and does identity map/reduce
  testfilesystem: A test for FileSystem read/write.
  testipc: A test for ipc.
  testmapredsort: A map/reduce program that validates the map-reduce framework's sort.
  testrpc: A test for rpc.
  testsequencefile: A test for flat files of binary key value pairs.
  testsequencefileinputformat: A test for sequence file input format.
  testsetfile: A test for flat files of binary key/value pairs.
  testtextinputformat: A test for text input format.
  threadedmapbench: A map/reduce benchmark that compares the performance of maps with multiple spills over maps with 1 spill

And here is the full list of available options for hadoop-*examples*.jar:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ bin/hadoop jar hadoop-*examples*.jar
An example program must be given as the first argument.
Valid program names are:
  aggregatewordcount: An Aggregate based map/reduce program that counts the words in the input files.
  aggregatewordhist: An Aggregate based map/reduce program that computes the histogram of the words in the input files.
  dbcount: An example job that count the pageview counts from a database.
  grep: A map/reduce program that counts the matches of a regex in the input.
  join: A job that effects a join over sorted, equally partitioned datasets
  multifilewc: A job that counts words from several files.
  pentomino: A map/reduce tile laying program to find solutions to pentomino problems.
  pi: A map/reduce program that estimates Pi using monte-carlo method.
  randomtextwriter: A map/reduce program that writes 10GB of random textual data per node.
  randomwriter: A map/reduce program that writes 10GB of random data per node.
  secondarysort: An example defining a secondary sort to the reduce.
  sleep: A job that sleeps at each map and reduce task.
  sort: A map/reduce program that sorts the data written by the random writer.
  sudoku: A sudoku solver.
  teragen: Generate data for the terasort
  terasort: Run the terasort
  teravalidate: Checking results of terasort
  wordcount: A map/reduce program that counts the words in the input files.

The wordcount example, for instance, is a typical “Hello, World” test that you can run after you have finished installing a cluster.

And before we start, here’s a nifty trick for your tests: When running the benchmarks described in the following sections, you might want to use the Unix time command to measure the elapsed time. This saves you the hassle of navigating to the Hadoop JobTracker web interface to get the (almost) same information. Simply prefix every Hadoop command with time:

1
2
3
4
5

$ time hadoop jar hadoop-*examples*.jar ...
[...]
real    9m15.510s
user    0m7.075s
sys     0m0.584s

The relevant metric is the real value in the first row.

TestDFSIO

The TestDFSIO benchmark is a read and write test for HDFS. It is helpful for tasks such as stress testing HDFS, to discover performance bottlenecks in your network, to shake out the hardware, OS and Hadoop setup of your cluster machines (particularly the NameNode and the DataNodes) and to give you a first impression of how fast your cluster is in terms of I/O.

An official documentation does not seem to exist in Hadoop 0.20.2, so the only way to understand the test in greater detail is to inspect the source code found in $HADOOP_HOME/src/test/org/apache/hadoop/fs/TestDFSIO.java.

Preliminaries

1. The default output directory is /benchmarks/TestDFSIO

When a write test is run via -write, the TestDFSIO benchmark writes its files to /benchmarks/TestDFSIO on HDFS. Files from older write runs are overwritten. If you want to preserve the output files of previous runs, you have to copy/move these files manually to a new HDFS location. Benchmark results are saved in a local file called TestDFSIO_results.log in the current local directory (results are appended if the file already exists) and also printed to STDOUT. If you want to use a different filename, set the -resFile parameter appropriately (e.g. -resFile foo.txt).

2. Run write tests before read tests

The read test of TestDFSIO does not generate its own input files. For this reason, it is a convenient practice to first run a write test via -write and then follow-up with a read test via -read (while using the same parameters as during the previous -write run).

Run a write test (as input data for the subsequent read test)

The syntax for running a write test is as follows:

1
2

TestDFSIO.0.0.4
Usage: hadoop jar $HADOOP_HOME/hadoop-*test*.jar TestDFSIO -read | -write | -clean [-nrFiles N] [-fileSize MB] [-resFile resultFileName] [-bufferSize Bytes]

TestDFSIO is designed in such a way that it will use 1 map task per file, i.e. it is a 1:1 mapping from files to map tasks. Splits are defined so that each map gets only one filename, which it creates (-write) or reads (-read).

The command to run a write test that generates 10 output files of size 1GB for a total of 10GB is:

1

$ hadoop jar hadoop-*test*.jar TestDFSIO -write -nrFiles 10 -fileSize 1000

Run a read test

The command to run the corresponding read test using 10 input files of size 1GB is:

1

$ hadoop jar hadoop-*test*.jar TestDFSIO -read -nrFiles 10 -fileSize 1000

Clean up and remove test data

The command to remove previous test data is:

1

$ hadoop jar hadoop-*test*.jar TestDFSIO -clean

The cleaning run will delete the output directory /benchmarks/TestDFSIO on HDFS.

Interpreting TestDFSIO results

Let’s have a look at this exemplary result for writing and reading 1TB of data on a cluster of twenty nodes and try to deduce its meaning:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

----- TestDFSIO ----- : write
           Date & time: Fri Apr 08 2011
       Number of files: 1000
Total MBytes processed: 1000000
     Throughput mb/sec: 4.989
Average IO rate mb/sec: 5.185
 IO rate std deviation: 0.960
    Test exec time sec: 1113.53

----- TestDFSIO ----- : read
           Date & time: Fri Apr 08 2011
       Number of files: 1000
Total MBytes processed: 1000000
     Throughput mb/sec: 11.349
Average IO rate mb/sec: 22.341
 IO rate std deviation: 119.231
    Test exec time sec: 544.842

Here, the most notable metrics are Throughput mb/sec and Average IO rate mb/sec. Both of them are based on the file size written (or read) by the individual map tasks and the elapsed time to do so (see this discussion thread for more information).

Throughput mb/sec for a TestDFSIO job using N map tasks is defined as follows. The index 1 <= i <= N denotes the individual map tasks:

Average IO rate mb/sec is defined as:

Two derived metrics you might be interested in are estimates of the “concurrent” throughput and average IO rate (for the lack of a better term) your cluster is capable of. Imagine you let TestDFSIO create 1,000 files but your cluster has only 200 map slots. This means that it takes about five MapReduce waves (5 * 200 = 1,000) to write the full test data because the cluster can only run 200 map tasks at the same time. In this case, simply take the minimum of the number of files (here: 1,000) and the number of available map slots in your cluster (here: 200), and multiply the throughput and average IO rate by this minimum. In our example, the concurrent throughput would be estimated at 4.989 * 200 = 997.8 MB/s and the concurrent average IO rate at 5.185 * 200 = 1,037.0 MB/s.

Another thing to keep in mind when interpreting TestDFSIO results is that the HDFS replication factor plays an important role. If you compare two otherwise identical TestDFSIO write runs which use an HDFS replication factor of 2 and 3, respectively, you will see higher throughput and higher average IO numbers for the run with the lower replication factor. Unfortunately, you cannot simply overwrite your cluster’s default replication value on the command line via

1

$ hadoop jar hadoop-*test*.jar TestDFSIO -D dfs.replication=2 -write ...

because as I said above the TestDFSIO shipped with Hadoop 0.20.2 does not parse command line parameters properly. A quick workaround is to create or update the conf/hdfs-site.xml file on the machine you are running the TestDFSIO write test from. TestDFSIO will read this configuration file during startup and use the value specified for the dfs.replication property therein.

More information about TestDFSIO

The comments in HDFS-1338 include some general remarks on the concept and design of TestDFSIO. And of course you can have a look at the source code in TestDFSIO.java.

TeraSort benchmark suite

The TeraSort benchmark is probably the most well-known Hadoop benchmark. Back in 2008, Yahoo! set a record by sorting 1 TB of data in 209 seconds – on an Hadoop cluster of 910 nodes as Owen O’Malley of the Yahoo! Grid Computing Team reports. One year later in 2009, Yahoo! set another record by sorting a 1 PB (1’000 TB) of data in 16 hours on an even larger Hadoop cluster of 3800 nodes (it took the same cluster only 62 seconds to sort 1 TB of data, easily beating the previous year’s record!).

Basically, the goal of TeraSort is to sort 1TB of data (or any other amount of data you want) as fast as possible. It is a benchmark that combines testing the HDFS and MapReduce layers of an Hadoop cluster. As such it is not surprising that the TeraSort benchmark suite is often used in practice, which has the added benefit that it allows us – among other things – to compare the results of our own cluster with the clusters of other people. You can use the TeraSort benchmark, for instance, to iron out your Hadoop configuration after your cluster passed a convincing TestDFSIO benchmark first. Typical areas where TeraSort is helpful is to determine whether your map and reduce slot assignments are sound (as they depend on the variables such as the number of cores per TaskTracker node and the available RAM), whether other MapReduce-related parameters such as io.sort.mb and mapred.child.java.opts are set to proper values, or whether the FairScheduler configuration you came up with really behaves as expected.

A full TeraSort benchmark run consists of the following three steps:

1. Generating the input data via TeraGen.
2. Running the actual TeraSort on the input data.
3. Validating the sorted output data via TeraValidate.

You do not need to re-generate input data before every TeraSort run (step 2). So you can skip step 1 (TeraGen) for later TeraSort runs if you are satisfied with the generated data.

Figure 1 shows the basic data flow. We use the included HDFS directory names in the later examples.

The next sections describe each of the three steps in greater detail.

TeraGen: Generate the TeraSort input data (if needed)

TeraGen (source code) generates random data that can be conveniently used as input data for a subsequent TeraSort run.

The syntax for running TeraGen is as follows:

1

$ hadoop jar hadoop-*examples*.jar teragen <number of 100-byte rows> <output dir>

Using the HDFS output directory /user/hduser/terasort-input as an example, the command to run TeraGen in order to generate 1TB of input data (i.e. 1,000,000,000,000 bytes) is:

1

$ hadoop jar hadoop-*examples*.jar teragen 10000000000 /user/hduser/terasort-input

Please note that the first parameter supplied to TeraGen is 10 billion (10,000,000,000), i.e. not 1 trillion = 1 TB (1,000,000,000,000). The reason is that the first parameter specifies the number of rows of input data to generate, each of which having a size of 100 bytes.

Here is the actual TeraGen data format per row to clear things up:

1

<10 bytes key><10 bytes rowid><78 bytes filler>\r\n

where

1. The keys are random characters from the set ‘ ‘ .. ‘~’.
2. The rowid is the right justified row id as a int.
3. The filler consists of 7 runs of 10 characters from ‘A’ to ‘Z’.

If you have a very fast cluster, your map tasks might finish in a few seconds if you use a relatively small default HDFS block size such as 128MB (which is not a bad idea though in general). This means that the time to start/stop map tasks might be larger than the time to perform the actual task. In other words, the overhead for managing the TaskTrackers might exceed the job’s “payload”. An easy way to work around this is to increase the HDFS block size for files written by TeraGen. Keep in mind that the HDFS block size is a per-file setting, and the value specified by the dfs.block.size property in hdfs-default.xml (or conf/hdfs-site.xml if you use a custom configuration file) is just a default value. So if, for example, you want to use an HDFS block size of 512MB (i.e. 536870912 bytes) for the TeraSort benchmark suite, overwrite dfs.block.size when running TeraGen:

1

$ hadoop jar hadoop-*examples*.jar teragen -D dfs.block.size=536870912 ...

TeraSort: Run the actual TeraSort benchmark

TeraSort (source code) is implemented as a MapReduce sort job with a custom partitioner that uses a sorted list of n-1 sampled keys that define the key range for each reduce.

The syntax to run the TeraSort benchmark is as follows:

1

$ hadoop jar hadoop-*examples*.jar terasort <input dir> <output dir>

Using the input directory /user/hduser/terasort-input and the output directory /user/hduser/terasort-output as an example, the command to run the TeraSort benchmark is:

1

$ hadoop jar hadoop-*examples*.jar terasort /user/hduser/terasort-input /user/hduser/terasort-output

TeraValidate: Validate the sorted output data of TeraSort

TeraValidate (source code) ensures that the output data of TeraSort is globally sorted.

TeraValidate creates one map task per file in TeraSort’s output directory. A map task ensures that each key is less than or equal to the previous one. The map task also generates records with the first and last keys of the file, and the reduce task ensures that the first key of file i is greater than the last key of file i-1. The reduce tasks do not generate any output if everything is properly sorted. If they do detect any problems, they output the keys that are out of order.

The syntax to run the TeraValidate test is as follows:

1

$ hadoop jar hadoop-*examples*.jar teravalidate <terasort output dir (= input data)> <teravalidate output dir>

Using the output directory /user/hduser/terasort-output from the previous sections and the report (output) directory /user/hduser/terasort-validate as an example, the command to run the TeraValidate test is:

1

$ hadoop jar hadoop-*examples*.jar teravalidate /user/hduser/terasort-output /user/hduser/terasort-validate</pre>

Further tips and tricks

Hadoop provides a very convenient way to access statistics about a job from the command line:

1

$ hadoop job -history all <job output directory>

This command retrieves a job’s history files (two files that are by default stored in <job output directory>/_logs/history) and computes job statistics from them:

1
2
3
4

$ hadoop fs -ls /user/hduser/terasort-input/_logs/history
Found 2 items
-rw-r--r--   3 hadoop supergroup      17877 2011-02-11 11:58 /user/hduser/terasort-input/_logs/history/master_1297410440884_job_201102110201_0014_conf.xml
-rw-r--r--   3 hadoop supergroup      36758 2011-02-11 11:58 /user/hduser/terasort-input/_logs/history/master_1297410440884_job_201102110201_0014_hadoop_TeraGen

Here is an exemplary snippet of such job statistics for TeraGen from a small cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

$ hadoop job -history all /user/hduser/terasort-input

Hadoop job: job_201102110201_0014
=====================================
Job tracker host name: master
job tracker start time: Fri Feb 11 2011
User: hadoop
JobName: TeraGen
JobConf: hdfs://master:54310/app/hadoop/tmp/mapred/system/job_201102110201_0014/job.xml
Submitted At: 11-Feb-2011
Launched At: 11-Feb-2011 13:58:14 (0sec)
Finished At: 11-Feb-2011 15:00:56 (1hrs, 2mins, 41sec)
Status: SUCCESS
=====================================

Task Summary
============================
Kind    Total   Successful      Failed  Killed  StartTime       FinishTime

Setup   1       1               0       0       11-Feb-2011 13:58:15    11-Feb-2011 13:58:16 (1sec)
Map     24      24              0       0       11-Feb-2011 13:58:18    11-Feb-2011 15:00:47 (1hrs, 2mins, 28sec)
Reduce  0       0               0       0
Cleanup 1       1               0       0       11-Feb-2011 15:00:50    11-Feb-2011 15:00:53 (2sec)
============================

Analysis
=========

Time taken by best performing map task task_201102110201_0014_m_000006: 59mins, 5sec
Average time taken by map tasks: 1hrs, 1mins, 24sec
Worse performing map tasks:
TaskId          Timetaken
task_201102110201_0014_m_000004 1hrs, 2mins, 24sec
task_201102110201_0014_m_000020 1hrs, 2mins, 19sec
task_201102110201_0014_m_000013 1hrs, 2mins, 9sec
[...]

NameNode benchmark (nnbench)

NNBench (see src/test/org/apache/hadoop/hdfs/NNBench.java) is useful for load testing the NameNode hardware and configuration. It generates a lot of HDFS-related requests with normally very small “payloads” for the sole purpose of putting a high HDFS management stress on the NameNode. The benchmark can simulate requests for creating, reading, renaming and deleting files on HDFS.

I like to run this test simultaneously from several machines – e.g. from a set of DataNode boxes – in order to hit the NameNode from multiple locations at the same time.

The syntax of NNBench is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

NameNode Benchmark 0.4
Usage: nnbench <options>
Options:
        -operation <Available operations are create_write open_read rename delete. This option is mandatory>
         * NOTE: The open_read, rename and delete operations assume that the files they operate on, are already available. The create_write operation must be run before running the other operations.
        -maps <number of maps. default is 1. This is not mandatory>
        -reduces <number of reduces. default is 1. This is not mandatory>
        -startTime <time to start, given in seconds from the epoch. Make sure this is far enough into the future, so all maps (operations) will start at the same time>. default is launch time + 2 mins. This is not mandatory
        -blockSize <Block size in bytes. default is 1. This is not mandatory>
        -bytesToWrite <Bytes to write. default is 0. This is not mandatory>
        -bytesPerChecksum <Bytes per checksum for the files. default is 1. This is not mandatory>
        -numberOfFiles <number of files to create. default is 1. This is not mandatory>
        -replicationFactorPerFile <Replication factor for the files. default is 1. This is not mandatory>
        -baseDir <base DFS path. default is /becnhmarks/NNBench. This is not mandatory>
        -readFileAfterOpen <true or false. if true, it reads the file and reports the average time to read. This is valid with the open_read operation. default is false. This is not mandatory>
        -help: Display the help statement

The following command will run a NameNode benchmark that creates 1000 files using 12 maps and 6 reducers. It uses a custom output directory based on the machine’s short hostname. This is a simple trick to ensure that one box does not accidentally write into the same output directory of another box running NNBench at the same time.

1
2
3
4

$ hadoop jar hadoop-*test*.jar nnbench -operation create_write \
    -maps 12 -reduces 6 -blockSize 1 -bytesToWrite 0 -numberOfFiles 1000 \
    -replicationFactorPerFile 3 -readFileAfterOpen true \
    -baseDir /benchmarks/NNBench-`hostname -s`

Note that by default the benchmark waits 2 minutes before it actually starts!

MapReduce benchmark (mrbench)

MRBench (see src/test/org/apache/hadoop/mapred/MRBench.java) loops a small job a number of times. As such it is a very complimentary benchmark to the “large-scale” TeraSort benchmark suite because MRBench checks whether small job runs are responsive and running efficiently on your cluster. It puts its focus on the MapReduce layer as its impact on the HDFS layer is very limited.

This test should be run from a single box (see caveat below). The command syntax can be displayed via mrbench --help:

1
2
3
4
5
6
7
8
9

MRBenchmark.0.0.2
Usage: mrbench [-baseDir <base DFS path for output/input, default is /benchmarks/MRBench>]
          [-jar <local path to job jar file containing Mapper and Reducer implementations, default is current jar file>]
          [-numRuns <number of times to run the job, default is 1>]
          [-maps <number of maps for each run, default is 2>]
          [-reduces <number of reduces for each run, default is 1>]
          [-inputLines <number of input lines to generate, default is 1>]
          [-inputType <type of input to generate, one of ascending (default), descending, random>]
          [-verbose]

In Hadoop 0.20.2, the parameters default to:

1
2
3
4
5
6

-baseDir: /benchmarks/MRBench  [*** see my note above ***]
-numRuns: 1
-maps: 2
-reduces: 1
-inputLines: 1
-inputType: ascending

The command to run a loop of 50 small test jobs is:

1

$ hadoop jar hadoop-*test*.jar mrbench -numRuns 50

Exemplary output of the above command:

1
2

DataLines       Maps    Reduces AvgTime (milliseconds)
1               2       1       31414

This means that the average finish time of executed jobs was 31 seconds.

Summary

I hope you have found my quick overview of Hadoop’s benchmarking and testing tools useful! Feel free to provide your feedback, corrections and suggestions in the comments below.

Related Articles

From yours truly:

• Running Hadoop On Ubuntu Linux (Single-Node Cluster)
• Running Hadoop On Ubuntu Linux (Multi-Node Cluster)
• Writing An Hadoop MapReduce Program In Python

From others:

• HiBench - an Hadoop benchmark suite developed by Intel

I won’t go into the details of using quotas in Hadoop, but here it is in a nutshell. Hadoop differentiates between two kinds of quotas: name quotas and space quotas. The former limits the number of file and (sub)directory names whereas the latter limits the HDFS “disk” space of the directory (i.e. the number of bytes used by files under the tree rooted at this directory).

You can set HDFS name quotas with the command

1

$ hadoop dfsadmin -setQuota <max_number> <directory>

and you can set HDFS space quotas with the command

1

$ hadoop dfsadmin -setSpaceQuota <max_size> <directory>

To clear quotas, use -clrQuota and -clrSpaceQuota, respectively.

So much for the introduction. Recently, I stumbled upon a problem where Hadoop (version 0.20.2) reported a quota violation for a directory for which a space quota of 200 MB (209,715,200 bytes) was set but no name quota:

1
2

$ hadoop fs -count -q /user/jsmith
none    inf    209715200    209715200    5   1   0

The output columns for fs -count -q are: QUOTA, REMAINING_QUOTA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME.

The directory /user/jsmith was empty and not a single byte was used yet (according to the quota report above). However, when I tried to upload a very small file to the directory, the upload failed with a (space) quota violation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ hadoop fs -copyFromLocal small-file.txt /user/jsmith
11/03/25 13:09:16 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/jsmith is exceeded: quota=209715200 diskspace consumed=384.0m
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java               :39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorI               mpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:96)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:58)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:293               9)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:28               19)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2288)
11/03/25 13:09:16 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
11/03/25 13:09:16 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/jsmith/small-file.txt" - Aborting...
copyFromLocal: org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/jsmith is exceeded: quota=209715200 diskspace consumed=384.0m
11/03/25 13:09:16 ERROR hdfs.DFSClient: Exception closing file /user/jsmith/small-file.txt :
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException:
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /user/jsmith is exceeded: quota=209715200 diskspace consumed=384.0m

Clearly, the small file could not have exceeded the space quota of 200MB I thought. I also checked whether the Trash feature of Hadoop could be the culprit but that wasn’t the case.

Eventually, the mystery was solved: Hadoop checks space quotas during space allocation. This means that HDFS block size (here: 128MB) and the replication factor of the file (here: 3, i.e. the default value in the cluster set by the dfs.replication property) play an important role. In my case, this is what seems to have happened: When I tried to copy the local file to HDFS, Hadoop figured it would require a single block of 128MB size to store the small file. With replication factored in, the total space would be 3 * 128MB = 384 MB. And this would violate the space quota of 200MB.

1
2

  required_number_of_HDFS blocks * HDFS_block_size * replication_count
= 1 * 128MB * 3 = 384MB > 200MB.

I verified this by manually overwriting the default replication factor of 3 to 1 via

1

$ hadoop fs -D dfs.replication=1 -copyFromLocal small-file.txt /user/jsmith

which worked successfully.

To be honest, I was a bit puzzled at first because – remembering Tom White’s Hadoop book – a file in HDFS that is smaller than a single HDFS block does not occupy a full block’s worth of underlying storage (i.e. a kind of “sparse” use of storage).

Related Articles

• Understanding HDFS quotas and Hadoop fs and fsck tools

Such virtual environments are very helpful to make sure that each of your Python applications runs in a healthy environment. It allows your applications to use different - even conflicting - versions of Python modules, say, app A requires YourModule v1.0 whereas app B requires YourModule 3.1. In addition, virtual environments make sure that updating Python modules in one place/for one application does not break other applications.

The full documentation is available at http://pypi.python.org/pypi/virtualenv, from which parts of this text has been taken/modified.

Installation

These instructions might vary depending on your operating system. The next lines are the instructions for Ubuntu.

Install setuptools:

1

$ sudo apt-get install python-setuptools

Install virtualenv and pip (actually, pip should be installed automatically as a dependency of recent virtualenv versions):

1

$ sudo easy_install virtualenv pip

Creating a Python sandbox

Create a virtual environment

1

$ virtualenv yourenv

or

1

$ virtualenv --no-site-packages yourenv

If you build with the optional --no-site-packages switch, your virtual environment will not inherit any packages from your “global” site-packages directory. This can be used if you don’t have control over site-packages and don’t want to depend on the packages there, or you just want more isolation from the global system.

Activate the virtual environment

1

$ source yourenv/bin/activate

This will change your $PATH to point to the virtualenv bin/ directory, and update your prompt. This is all it does. If you use the complete path like /path/to/yourenv/bin/python myscript.py, you do not need to activate the environment first. You must use source because it changes the environment in-place.

Adding custom Python modules to the virtual environment

1

(yourenv)$ pip install YourFancyModule

If you have activated the virtual environment, running pip within your environment will install YourFancyModule only to this virtual environment. After all, this was the purpose of using virtualenv in the first place, right?

Deactivate the virtual environment

After activating an environment you can use the function deactivate to undo the changes.

1

(yourenv)$ deactivate

Here’s a quick example on how to use it from the Python interpreter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

>>> import spear
>>> activities = [
... (datetime.datetime(2010,7,1,9,0,0), "alice", "http://www.michael-noll.com/"),
... (datetime.datetime(2010,8,1,12,45,0), "bob", "http://www.michael-noll.com/"),
... ]
>>> spear_algorithm = spear.Spear(activities)
>>> expertise_results, quality_results = spear_algorithm.run()
</pre>
<!--more-->
Get the top user and his expertise score:

<pre>
>>> expertise_score, user = expertise_results[0]
>>> print "%s => %.4f" % (user, expertise_score)
alice => 0.5858

Get the top resource and its quality score:

1
2
3

>>> quality_score, resource = quality_results[0]
>>> print "%s => %.4f" % (resource, quality_score)
http://www.michael-noll.com/ => 1.0000

You can also use the library to simulate the HITS algorithm of Jon Kleinberg. Simply supply a credit score function C(x) = 1 to the SPEAR algorithm (see the documentation of the Spear.run() method).

Feel free to play around with it and send me feedback!

PS: The SPEAR Python library requires SciPy/NumPy. If you don’t have these installed already, here are some installation instructures to get you started.

1. Open the FLV file with VLC and stop it as soon as it starts playing.
2. Open the VLC Wizard by clicking on File > Streaming/Exporting Wizard…
3. Select Transcode/Save to file. Next.
4. Select your file from the Playlist. Next.
5. Check only the Transcode Audio checkmark (leave Video unchecked). Select 192 kb/s bitrate, and MP3 as the Audio codec. Next.
6. Select MPEG-1 as encapsulation method. Save the file with any name, but with the extension MPG. (Don’t use MP3 at this time). Press Finish.

Once the transcoding is finished, repeat steps 1 to 6, except for:

• Choose the MPG file you just created as your input file.
• Change the encapsulation method as RAW. Save the new file with the extension MP3.

Well, I’m happy to announce that the article is published now: How SPEAR Identifies Domain Experts within Delicious Check it out while it’s still hot ;-)

Thanks again to Vik Singh and Ariel Seidman from Yahoo! for this great opportunity and for their support!

The basic rule is Ctrl-Alt plus the special character as listed on my Logitech G15 keyboard (which has special characters listed in the bottom right of each key).

Abstract

Full Paper & Presentation

• M. G. Noll, C.-M. Au Yeung, N. Gibbins, C. Meinel, N. Shadbolt Telling Experts from Spammers: Expertise Ranking in Folksonomies Proceedings of 32nd ACM SIGIR Conference, Boston, USA, July 2009, pp. 612-619, ISBN 978-1-60558-483-6 (ACM Link, BibTeX)
• Presentation: Telling Experts from Spammers (Talk), our talk at SIGIR 2009

Related Links

• The Spear Algorithm - website dedicated to SPEAR
• List of my publications
• Article on Technology Review about this work: A Better Way to Rank Expertise Online, July 2009
• 32nd Annual ACM SIGIR Conference, Boston, USA, July 2009

Writing a Personal Link Recommendation Engine There is so much going on in the Internet today that it’s hard to keep track of it all. Whether you need to stay up to date with job-related information, the current state of research, or the latest developments in the Open Source world, finding relevant information quickly on the Internet is difficult. So-called “social” services such as Delicious.com or Digg.com try to support users by allowing them to collaboratively share their knowledge about interesting web sites. In this article, we will design and implement a simple, yet powerful link recommender whose analysis is based solely on Delicious.com social bookmarks information.