Friday, November 21, 2014

Setup Storm in a cluster

To setup storm in a cluster, suppose we have the following computers networked together:

Let's say we use computer and to run the zookeepers which maintain states and communications for storm cluster. We also use as the storm master node (i.e, the nimbus host) and and (i.e., the supervisor nodes) as the storm work nodes. Furthermore, let's suppose we work on as a client which submit storm topology to the storm cluster.

Firstly we need to setup zookeeper in the cluster consisting of and Following this link for instructions to setup the zookeeper in a cluster:

Next on computers,,, and which forms the storm cluster, let's setup the storm using instruction for storm setup in a single machine from this link:

Once the storm has been set up in the 4 computers, on each computer, navigate to the STORM_HOME/conf directory and edit the storm.yaml using the following commands:

> cd $STORM_HOME/conf
> gedit storm.yaml

In the storm.yaml, modify the settings as followings:

  - ""
  - ""
storm.zookeeper.port: 2181 ""
storm.local.dir: "/tmp/storm-data"
java.library.path: "/usr/local/lib"
storm.messaging.transport: baketype.storm.messaging.netty.Context
 - 6700
 - 6701
 - 6702
 - 6703

Save and close the storm.yaml in each of the storm cluster computers. Now on the client computer (i.e., create a .storm folder its user root folder and copy the updated STORM_HOME/conf/storm.yaml into the .storm folder created:
> cd $HOME
> mkdir .storm
> cp $STORM_HOME/conf/storm.yaml $HOME/.storm

Now on the master node computer, run the following command to start the master node:

> bin/storm nimbus

Now on the two supervisor node computer ( and, run the following command to start the work nodes:
> bin/storm supervisor

At this point, the storm has been setup and running. To view the status of the storm cluster, choose one of the computer (or the client computer) and run the following command to start the storm web front ui:

> bin/storm ui

Now navigate to the http://localhost:8080 on that computer's web browser, you should see the storm cluster status

Now we are ready to submit a storm topology, on the client computer, run the following command to submit the topology
> bin/storm jar [fullPathToTopoloyJarFile] [mainClassInJar] [arg0] [arg1] ...

Now navigate back to the http://localhost:8080, you should see the submitted topology running in the cluster.

No comments:

Post a Comment