Setup Mesos Cluster in CentOS VMs

This post summarizes my experiences in setting up mesos cluster for spark and hadoop cluster deployment.

Firstly, before start, set up a zookeeper cluster (link: http://czcodezone.blogspot.sg/2016/03/setup-zookeeper-cluster-in-centos-vms.html) to run on 3 nodes with hostnames zoo01, zoo02, zoo03  as well as set up 3 mesos master eligable nodes and 4 mesos slave nodes (Refers to this on how to setup CentOS VMs for cluster: http://czcodezone.blogspot.sg/2016/03/setup-centos-vm-in-virtualbox-for.html)

1. Disable Firewall and SELinux

1.1. Disable firewalld

Run the following command to disable the firewalld
```bash
`systemctl stop firewalld.service
`systemctl disable firewalld.service
`iptables -F

1.2. Disable SELinux,

Run the following command to open the /etc/selinux/config

```bash
`vi /etc/selinux/config

Edit the to file to include the following lines:

SELINUX=disabled
#SELINUXTYPE=targeted

Run the following command to shutdown selinux immediately:

```bash
`setenforce 0

2. Start the zookeepers

Start the zookeepers in the zoo01/2/3

2.1. Install Mesos

Run the following line to install development tools

```bash
`yum install -y java-1.8.0-openjdk-devel
`yum install -y maven

Add the following line to the /root/.bashrc file, add the following line:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk

Run the following commands to install the following libraries required to compile mesos:

```bash
`rpm -Uvh http://repos.mesosphere.io/el/7/noarch/RPMS/mesosphere-el-repo-7-2.noarch.rpm
# to install master and/or slave
`yum -y install mesos
# to install marathon
`yum -y install marathon
# to install chronos
`yum -y install chronos.x86_64

2.2. Configure Master with Zookeeper

Edit /etc/mesos/zk:
zk://zoo01:2181,zoo02:2181,zoo03:2181/mesos

3. Start Mesos

#
# Mesos Master
#
```bash
`systemctl enable mesos-master.service
`systemctl start mesos-master.service
`systemctl mask mesos-slave.service
`systemctl stop mesos-slave.service

#
# Marathon
#
```bash
`systemctl enable marathon.service
`systemctl start marathon.service

#
# Mesos Slave
#
```bash
`systemctl mask mesos-slave.service
`systemctl stop mesos-slave.service

#
# Chronos
#
```bash
`systemctl enable chronos.service
`systemctl start chronos.service

Personal Note: I noticed that if the mesos services are started immediately after the zookeepers are started,
it is possible that the mesos won't be able to detect the master from zookeeper. The way to solve this is to restart the mesos services on each node again.

4. Web Interface

Mesos-Master:
```bash
`curl http://mesos01:5050

Marathon:
```bash
`curl http://mesos01:8080

5. Command line Interface

Getting master from CLI
```bash
`mesos-resolve `cat /etc/mesos/zk`

Killing a framework:

```bash
`curl -XPOST http://mesos02:5050/api/v1/scheduler -d '{ "framework_id": { "value": "[task_id]" }, "type": "TEARDOWN"}' -H Content-Type:application/json

6. Submit a spark job via MESOS cluster

To run a spark application in MESOS cluster instead, setup and configure HDFS (link http://czcodezone.blogspot.sg/2016/03/setup-hdfs-cluster-in-centos-vms.html) and Spark (link http://czcodezone.blogspot.sg/2016/03/setup-spark-cluster-in-centos-vms.html).

Put the spark bin package in the hdfs (run the following command on the hadoop namenode centos01):

```bash
`wget http://www.apache.org/dyn/closer.lua/spark/spark-1.6.0/spark-1.6.0-bin-hadoop2.6.tgz
`hadoop/bin/hdfs dfs -mkdir /pkg
`hadoop/bin/hdfs dfs -put spark-1.6.0-bin-hadoop2.6.tgz /pkg/spark-1.6.0-bin-hadoop2.6.tgz

Run the following command to modify the spark-env.sh in spark/conf

```bash
`vi spark/conf/spark-env.sh

In the spark-env.sh, add the following lines:

export MESOS_NATIVE_LIBRARY=/usr/local/lib/libmesos.so
export SPARK_EXECUTOR_URI= hdfs://centos01:9000/pkg/spark-1.6.0-bin-hadoop2.6.tgz

Where centos01 is the hadoop namenode

To submit a spark job, run the following command:

```bash
`spark/bin/spark-submit --class com.tutorials.spark.WordCountDriver --master mesos://mesos01:5050 word-count.jar

Important: mesos01 must be the current leader master node, otherwise, the command such as "spark-shell --master mesos://mesos01:5050" will cause the spark-shell to hang on the line "No credential provided. attempting to register without authentication". The solution is to find out which node is the active leader master node by running the command "mesos-resolve `cat /etc/mesos/zk`" and the luanch the spark shell by specifying the active leader master as in the --master option instead.

Setup ZooKeeper cluster in CentOS VMs.

This post summarizes my experience on setting up a zookeeper cluster in CentOS VMs.

Before start, setup three VMs with the following hostname/ipaddress (Refer to this on how to set up and configure CentOS VMs using VirtualBox  http://czcodezone.blogspot.sg/2016/03/setup-centos-vm-in-virtualbox-for.html):

zoo01/192.168.56.91
zoo02/192.168.56.92
zoo03/192.168.56.93

1. Edit /etc/hosts

On each VM, edit /etc/hosts to add the following lines:

192.168.56.91 zoo01
192.168.56.92 zoo02
192.168.56.93 zoo03

2. Download and configure zookeeper

2.1 Download zookeeper

On each VM, download and unzip the zookeeper to the folder "/root/zookeeper".

2.2. Configure zookeeper

On each VM, run the following command to edit the /root/zookeeper/conf/zoo.cfg:

```bash
`cp zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
`vi zookeeper/conf/zoo.cfg

In the zookeeper/conf/zoo.cfg, add the following configuration:

dataDir=/root/zookeeper_data

server.1=zoo01:2222:2223
server.2=zoo02:2222:2223
server.3=zoo03:2222:2223

Run the following comand to create the directory for holding the data storage for zookeeper in each VM (the /root/zookeeper_data/myid must have an unique id on each VM):

```bash
`ssh root@zoo01
`mkdir /root/zookeeper_data
`touch /root/zookeeper_data/myid
`echo 1 >> /root/zookeeper_data/myid
`ssh root@zoo02
`mkdir /root/zookeeper_data
`touch /root/zookeeper_data/myid
`echo 2 >> /root/zookeeper_data/myid
`ssh root@zoo03
`mkdir /root/zookeeper_data
`touch /root/zookeeper_data/myid
`echo 3 >> /root/zookeeper_data/myid

3. Start zookeeper cluster

To start the zookeeper cluster, on each VM zoo01/2/3, run the following command:

```bash
`zookeeper/bin/zkServer.sh start zookeeper/conf/zoo.cfg

(Note that for testing, you may want to use the command "start-foreground" instead of "start")

4. Test zookeeper running

To check whether the zookeeper cluster is running, run the following command:

```bash
`zookeeper/bin/zkServer.sh status

To testing the zookeeper cluster, run the following command:

```bash
`zookeeper/bin/zkCli.sh -server 192.168.56.91:2181,192.168.56.92:2181,192.168.56.93:2181

Setup Kafka in a cluster

To setup Kafka in a cluster, first we must have the zookeeper cluster setup and running (follow this link: http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html), suppose that the zookeeper cluster consists of the zookeeper servers running at the following hostname:ports:

192.168.2.2:2181
192.168.2.4:2181

As I have only two computers, therefore i will use the same computers (but at different ports) to host the kafka cluster. For this case, the Kafka servers/brokers will be running at the following hostname:ports

192.168.2.2:9092
192.168.2.4:9092

To do this, follow this link (http://czcodezone.blogspot.sg/2014/11/setup-kafka-in-single-machine-running.html) to setup the kafka server. Now navigate to the kafka root folder oif each computer and modify the server.properties in "config" sub folder:

> cd $KAFKA_HOME
> gedit config/server.properties

In the server.properties file, search the line "zookeeper.connect" and change it to:

zookeeper.connect=192.168.2.2:2181,192.168.2.4:2181

Then search the line "broker.id" (unique id for each broker node) and change it to "broke.id=1" on computer 192.168.2.2 and to "broker.id=2" on computer 192.168.2.4

Next search the line "host.name" and change it to "host.name=192.168.2.2" on computer 192.168.2.2 and to "host.name=192.168.2.4" on computer 192.168.2.4

Make sure that the line "port=9092" is there and uncommented in the server.properties

Save and close the server.properties. Now start the kafka server on each computer:

> cd $KAFKA_HOME

> bin/kafka-server-start.sh config/server.properties

At this point, the kafka cluster is set up and running. We can test the cluster by creating a topic named "v-topic":

> bin/kakfa-topics.sh --create --zookeeper 192.168.2.4:2181 --partitions 2 --replication-factor 1 --topic v-topic

Now run the following commands to list the topics in the kafka brokers:

> bin/kafka-topics.sh --zookeeper 192.168.2.4:2181 --list

Now run the following commands to get a description how the topic "v-topic" is partitioned in each broker:

> bin/kafka-topics.sh --describe --zookeeper 192.168.2.4:2181 --topic v-topic

To test the producer and consumer interaction, let's start a consoler producer on the computer 192.168.2.4 by running the following command on that computer's terminal:

> cd $KAFKA_HOME
> bin/kafka-console-producer.sh --broker-list 192.168.2.2:9092,192.168.2.4:9092 --topic v-topic

Now open a terminal of the other computer 192.168.2.4 and start a console consumer:

> cd $KAFKA_HOME
> bin/kafka-console-consumer.sh --zookeeper 192.168.2.4:2181 --topic v-topic --from-beginning

Begin to type something in the console producer on 192.168.2.2 terminal and press ENTER, you will see the output displayed in the console consumer on 192.168.2.4 terminal.

Note:

It is also ok to set up multiple Kafka brokers on the same computer. For example, if we want to have two Kafka brokers running at two different ports on computer 192.168.2.2, say:

192.168.2.2:9092
192.168.2.2:9093

Now all that we need to do is to duplicate the server.properties after it is updated, and rename it server1.properties in the same "config" folder (note that name is not important, can be anything that make sense). Now in the server1.properties, modify to have the following settings:

broker.id=3
log.dirs=/var/kafka1-logs
port=9093

Save and close server1.properties (remember to create the folder /var/kafka1-logs with write permission), open two terminal in 192.168.2.2 and run the following command in the first terminal to start a kafka broker at port 9092:

> $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties

On the second terminal, run the following command to start a second kafka broker at port 9093:

> $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server1.properties

Now you will have two kafka brokers running on 192.168.2.2 on two different ports. To include the second broker for the console producer, change its start command to:

> $KAKFA_HOME/bin/kafka-console-producer.sh --broker-list 192.168.2.2:9092,192.168.2.2:9093,192.168.2.4:9092 --topic v-topic

Setup Kafka in a single machine running Ubuntu 14.04 LTS

Kafka is a messaging system that can acts as a buffer and feeder for messages processed by Storm spouts. It can also be used as a output buffer for Storm bolts. This post shows how to setup and test Kafka on a single machine running Ubuntu.

Firstly download the kafka 0.8.1.1 from the link below:

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1/kafka_2.8.0-0.8.1.1.tgz

Next "tar -xvzf" the kafka_2.8.0-0.8.1.1.tgz file and move it to a destination folder (say, /Documents/Works/Kafka folder under the user root directory):

> tar -xvzf kafka_2.8.0-0.8.1.1.tgz
> mkdir $HOME/Documents/Works/Kafka
> mv kafka_2.8.0-0.8.1.1 $HOME/Documents/Works/Kafka

Now go back to the user root folder and open the .bashrc file for editing:

> cd $HOME
> gedit .bashrc

In the .bashrc file, add the following line to the end:

export KAFKA_HOME=$HOME/Documents/Works/Kakfa/kafka_2.8.0-0.8.1.1

Save and close the .bashrc and run "source .bashrc" to update the environment variables. Now navigate to the kafka home folder and edit the server.properties in its sub-directory "config":

> cd $KAFKA_HOME/config
> gedit server.properties

In the server.properties file, search the line "zookeeper.connect" and change it to the following:

zookeeper.connect=192.168.2.2:2181,192.168.2.4:2181

search the line "log.dirs" and change it to the following:

log.dirs=/var/kafka-logs

Save and close the server.properties file (192.168.2.2 and 192.168.2.4 are the zookeeper nodes). Next we go and create the folder /var/kafka-logs (which will store the topics and partitions data for kafka) with write permissions:

> sudo mkdir /var/kafka-logs
> sudo chmod -R 777 /var/kafka-logs

Now set up and run the zookeeper cluster by following instructions in the link http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html. Once this is done, we are ready to start the kafka messaging system by running the following commands:

> cd $KAFKA_HOME
> bin/kafka-server-start.sh config/server.properties

To start testing kafka setup, Ctrl+Alt+T to open a new terminal and run the following command to create a topic "verification-topic" (a topic is a named entity in kafka which contain one or more partitions which are message queues that can run in parallel and serialize to individual folder in /var/kafka-log folder):

> cd $KAKFA_HOME
> bin/kafka-topics.sh --create --zookeeper 192.168.2.2:2181 --topic verification-topic --partitions 1 --replication-factor 1

The above command creates a topic named "verification-topic" which contains 1 partition (and with no replication)

Now we can check the list of topics in kafka by running the following command:

> bin/kafka-topics.sh --zookeeper 192.168.2.2:2181 --list

To test the producer and consumer interaction in kafka, fire up the console producer by running

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic verification-topic

9092 is the default port for a kafka broker node (which is localhost at the moment). Now the terminal enter interaction mode. Let's open another terminal and run the console consumer:

> bin/kafka-console-consumer.sh --zookeeper 192.168.2.2:2181 --topic verification-topic

Now enter some data in the console producer terminal and you should see the data immediately display in the console consumer terminal.

Create a passwordless SSH login to remote computers to automate launch of a zookeeper cluster

Passwordless SSH Login can be useful in situations, for example, when one likes to write a bash script that automatically setup or launch a zookeeper cluster. Suppose we want to ssh login to a computer at 192.168.2.4 without a password from a our client computer at 192.168.2.2, we can do so by creating a RSA pair of authentication key at 192.168.2.2 terminal:

> ssh-keygen -t rsa

In the terminal prompt, just press the ENTER key multiple times for each prompt. Once this is done, a public key rsa_id.pub is stored in the .ssh folder under the user root folder. Now ssh login to the 192.168.2.4 (password required at this point) to create the .ssh folder under the user root folder on that computer:

> ssh username@192.168.2.4 mkdir .ssh

The last step is to append the .ssh/rsa_id.pub file (which is the public key) from 192.168.2.2 to the .ssh/authorized_keys in 192.168.2.4 (password required at this point):

> cat .ssh/id_rsa.pub | ssh username@192.168.2.4 'cat >> .ssh/authorized_keys'

Now we can ssh login to 192.168.2.4 from the client 192.168.2.2 without password. For example, run the following command in the client 192.168.2.2 terminal:

> ssh username@192.168.2.4

Now we are ready to write a bash script to launch the zookeeper cluster which consists of the following networked computers:

192.168.2.1
192.168.2.3
192.168.2.4

First make sure that the zookeeper is setup in the cluster properly (follow instructions from this link: http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html and for storm at this link http://czcodezone.blogspot.sg/2014/11/setup-storm-in-cluster.html) and make sure the above steps of passwordless SSH login is repeated for 192.168.2.1 and 192.168.2.3 so that user can log into these two computers without password from the client 192.168.2.2. Now on the terminal of 192.168.2.2, create a start_zookeeper_cluster.sh script:

> touch start_zookeeper_cluster.sh
> gedit start_zookeeper_cluster.sh

In the start_zookeeper_cluster.sh that opens, enter the following lines:

ssh username@192.168.2.1 $ZK_HOME/bin/zkServer.sh start
ssh username@192.168.2.3 $ZK_HOME/bin/zkServer.sh start
ssh username@192.168.2.4 $ZK_HOME/bin/zkServer.sh start

Save and close the start_zookeeper_cluster.sh, and run the following command on 192.168.2.2 client terminal to make it executable:

> chmod +x ./start_zookeeper_cluster.sh

Now run the script to start the zookeeper cluster:

> ./start_zookeeper_cluster.sh

We can also write a stop_zookeeper_cluster.sh, start_storm_cluster.sh, stop_storm_cluster.sh in a similar way to automate the process.

Setup ZooKeeper in a cluster

To setup ZooKeeper in a cluster, suppose we have the following computers interconnected (via a switch, e.g.) with the following ip address:

192.168.2.2

192.168.2.4

Lets assign a unique ID to each computer (ID taken from 1 to 255), said, we assign 192.168.2.2 with id = 1, and assign 192.168.2.4 with id = 2. 

Firstly lets setup the Zookeeper on each machine. This can be done by following instructions in http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-on-single-machine.html 

Once the zookeeper is setup, log into the terminal of each computer, create a "zookeeper" folder under /var directory and create a myid file in it to contains the unique id assigned to the computer, by running the following commands:

> cd /var

> sudo mkdir zookeeper

> sudo chmod -R 777 zookeeper

> cd zookeeper

> sudo touch myid

> sudo gedit myid

In the myid opened, put the unique id and save the file. For example the content of myid file on 192.168.2.2 is

1

and the content of myid file on 192.168.2.4 is 

2

Now navigate to the ZK_HOME directory on each computer and update the zoo.cfg in the conf sub-directory, by running the following commnads:

> cd $ZK_HOME

> cd conf

> gedit zoo.cfg

In the zoo.cfg file, write the following as its content:

tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.2.2:2888:3888
server.2=192.168.2.4:2888:3888

Save and close the zoo.cfg file. Run start the zookeeper on each computer by running the following commands in their terminal:

> cd $ZK_HOME

> bin/zkServer.sh start

To check the status of the zookeeper cluster, type the following command in each computers' terminal:

> cd $ZK_HOME

> bin/zkServer.sh status

To stop a zookeeper on a computer, run the following command:

> cd $ZK_HOME

> bin/zkServer.sh stop

Common problems encountered will cause the zkServer.sh status command to display the following messages:

Error contacting service. It is probably not running.

When this message appeared, it may be because one of the computer is not connected to the network (e.g. the ethernet cable is loose). In this case, try to ping each computer to see if they are connected.

Another caution is to make sure the the /var/zookeeper folder on each computer has write permission (e.g. by running the "sudo chmod -R 777 /var/zookeeper" command in the terminal)

Setup ZooKeeper on a single machine running Ubuntu 14.04 LTS

This post shows how to set up ZooKeeper on a single machine running Ubuntu 14.04 LTS.

Firstly download ZooKeeper from the following link:

http://mirror.nus.edu.sg/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

Unzip the downloaded file and put the unzipped folder in a folder, say the following

$HOME/Documents/Works/ZooKeeper/zookeeper-3.4.6

Now go the the $HOME directory and run the following command to open the .bashrc file in terminal:

> gedit .bashrc

In the .bashrc file, add the following line to the end:

export ZK_HOME=$HOME/Documents/Works/ZooKeeper/zookeeper-3.4.6

Save and close the .bashrc file, run the following command to update the environment var:

> source .bashrc

Now run the following command to go to the $ZK_HOME/conf directory

> cd $ZK_HOME/conf

Run the following command to create the configuration zoo.cfg:

> touch zoo.cfg
> gedit zoo.cfg

In the zoo.cfg opened, add the following lines:

tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181

Save and close zoo.cfg, go back to the ZooKeeper home directory:

> cd $ZK_HOME

Run the following command to start the zookeeper on the machine:

> bin/zkServer.sh start

Run the following command to check the java process running:

> jps

Run the following command to check the status of zookeeper:

> bin/zkServer.sh status

Run the following command to stop the zookeeper:

> bin/zkServer.sh stop