Showing posts with label Ubuntu. Show all posts
Showing posts with label Ubuntu. Show all posts

Tuesday, May 2, 2017

Vagrantfile for latest magento 2 (version 2.16) and ubuntu (Ubuntu 14.04.5 LTS)

I have just upload Vagrantfile for latest magento (2.16) and latest Ubuntu (bento/ubuntu-14.04) to github so that interested readers can install the latest magento for development via Vagrant on their computer:

Magento & Vagrant
The github repo contains the Vagrantfile and Vagrantfile.config.yml adopted from the "Magento Developer Guide" book but updated based on the trial-and-errors with the latest magento, ubuntu and PHP
The objective is to address several issues with the original Vagrantfile provided by "Magento Developer Guide" so that it is compatible with the latest magento (current version 2.16) and Ubuntu (current version: Ubuntu 14.04.5 LTS)
Some known issues with the original Vagrantfile provided by "Magento Developer Guide" are:
  • The vm.box "ubuntu/vivid64" for vagrant is not longer provided
  • The default PHP version (7.1) provided by latest ubuntu (after running apt-get update on bento/ubuntu-14.04) is not compatible with magento 2.16 due to the issues with mcrypt no longer supported in PHP 7.1
  • php5.6-mbstring and php5.6-zip are needed for the magento 2.16 to work correctly

Version

The following list the version of the various softwares installed by the Vagrantfile on the ubuntu VM:
  • vm.box: bento/ubuntu-14.04
  • magento: magento 2 (version: 2.16)
  • PHP: 5.6
  • MySQL: 5.6
Currently the Vagrantfile is tested to be working on host computer which is Windows 10

Usage

  • Install VirtualBox
  • Install Vagrant
  • Obtain the magento public and private secret keys (Follow this link) and replace the "[magent_public_key]" and "[magento_private_key]" in Vagrantfile.config.yml with the obtained keys
  • Obtain a github oauth access key (Follow this link and replace the "[github_oauth]" in Vagrantfile.config.yml with the obtained github oauth token
  • Change the "email@change.me" in Vagrantfile.config.yml to your email of choice
  • Change the "[magento_host_path]" in Vagrantfile.config.yml to your local path on your host computer on which you want the magento files to be installed
  • Run the following command on your host computer:
git clone https://github.com/chen0040/vagrant-magento-2.16.git
cd vagrant-magento-2.16
vagrant up
  • Add the line "192.168.10.10 magento.box" to /etc/hosts on your host computer (If you host computer is Windows, then the hosts file is located at C:\Windows\System32\drivers\etc\hosts)
  • Open your browser and enter "http://magento.box" (To change this to your url of choice, replace it in the Vagrantfile.config.yml)
  • To login to the vagrant vm box, run the command "vagrant ssh". If you are using putty, ssh to 192.168.10.10 (username: vagrant, password: vagrant)

Issues and Solutions

If you encounter some of the issues when running Vagrantfile, you can refer to the following:
  • Issue: Vagrant was unable to mount VirtualBox shared folders. This is usually because the filesystem "vboxsf" is not available. This filesystem is made available via the VirtualBox Guest Additions and kernel module.
Solution: Run the following command:
vagrant plugin install vagrant-vbguest

Sunday, May 10, 2015

Build and install grok on Ubuntu 14.04

Thanks the this post, got grok built and installed on Ubuntu 14.04:

Link: http://stackoverflow.com/questions/15287765/error-compiling-grok

Steps:

> sudo apt-get update
> sudo apt-get install -y ctags flex gperf libevent-dev libpcre3-dev libtokyocabinet-dev
> sudo apt-get install build-essential
> sudo apt-get install wget
> wget https://semicomplete.googlecode.com/files/grok-1.20110630.1.tar.gz
> tar -zxvf grok-1.20110630.1.tar.gz
> cd grok-1.20110630.1
> vi Makefile

Look for the following two lines:

gcc $(LDFLAGS) $^ -o $@

Change to:

gcc $^ $(LDFLAGS) -o $@

Enter the following command to build and install grok
> make
> sudo make install


Tuesday, January 27, 2015

Install D-Link DWM-221 Broadband Modem on Ubuntu LTS 14.04 64 Bit

This post shows how to connect to Internet using a D-Link broadband modem dongle (Model DWM-221) on the Ubuntu LTS 14.04 64 Bit OS. Most of the posts i searched online on this topic does not seem to work (most of them on DWM-156 and older versions of Ubuntu), therefore i decides to write something about how to do it after successfully get it to work on my Ubuntu OS.

Firstly, one may need to install the ia32-libs, the following command lines can be followed to install ia32-libs on Ubuntu 14.04:

sudo apt-get install libc6:i386
sudo -i
cd /etc/apt/sources.list.d
echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" >ia32-libs-raring.list
apt-get update
apt-get install ia32-libs
rm /etc/apt/sources.list.d/ia32-libs-raring.list
apt-get update
exit
sudo apt-get install gcc-multilib
Next plug in your D-link broadband modem to your computer and run the following command lines to mount it

sudo -su
mkdir /media/cdrom
mount /dev/sr1 /media/cdrom
You may try "mount /dev/sr0 /media/cdrom" if "mount /dev/sr0 /media/cdrom" does not work.

Once the D-Link folder has been mounted, you will notice there is a zip file in the folder named "DTLW3_D600B_linux_2.0.0WW_140305.tar.gz". If you try to extract it to a folder on your computer, you may encounter "Cannot create hard links ..." error messages. To get around list, right-click the file and select "Open with Archive Manager", extracts all the files excepts the hard links "device.conf" (which has size 0). Now in the extracted folder, you should have something like the following files and folders:

install.sh
uninstall.sh
linux
  /usr
  /bin
  /etc
    /3g_modem_connection
    /ppp
    /cm.version
    /device.conf

To make it easier, i have share the contents in the extracted folder here:

https://dl.dropboxusercontent.com/u/113201788/Hardware/D-Link%20Modem/DWM-221.tar.gz

In the extracted folder, run the following command:

sudo bash install.sh

Once this is done, go to the networking icons on the upper right corner of the Ubuntu desktop and select "Create new mobile broadband", follow the simple steps (basically a wizard asking several simple questions such as your internet provider and data plan in the form of drop down lists) and you should be able to connect to the internet via your D-Link broadband modem (Can try "sudo 3g_connect.sh ttyUSB0" if still got problem).

Sunday, December 14, 2014

XMPP: IM using OpenFire and Smack

While searching for some useful tutorials on how to do IM using OpenFire and Smack, I found there is no a satisfactory one-stop basic tutorial to get one easily started on creating an IM application, therefore i decided to write something about it after some experiments. This post will show how to setup OpenFire for instant messaging application and send notification via XMPP using Smack.

Setup Droplet

As OpenFire needs public IP address in order to send and receive messages, to this end, a droplet is created at digitalocean. Follow the following link to create droplet at digitalocean:

https://www.digitalocean.com/community/tutorials/how-to-create-your-first-digitalocean-droplet-virtual-server

Suppose the droplet that we created has the following settings:

hostname: droplet.demo.com
IP: 1.1.1.1
port: 9090
user: root

Setup OpenFire on Droplet

The following describes how to set up openfire on droplet:

https://www.digitalocean.com/community/tutorials/how-to-install-openfire-xmpp-server-on-a-debian-or-ubuntu-vps

Below I describe my process of setting up openfire on a droplet which may be a bit different. Suppose you have set up your droplet at root@1.1.1.1. Run the following command to login to root@1.1.1.1:

> ssh root@1.1.1.1

Once login to root@1.1.1.1, run the following command to download the latest openfire:

> wget -O openfire_3.9.3_all.deb http://www.igniterealtime.org/downloadServlet?filename=openfire/openfire_3.9.3_all.deb

Once downloaded, run the following commands to update the environment:

> sudo apt-get update
> sudo apt-get install openjdk-7-jre

The above commands install the openjdk-7 on the droplet. Now run the following command to install openfire:

> dpkg --install openfire-3.9.3_all.deb

Open the web browser and navigate to http://1.1.1.1:9090/ to configure the openfire. After the installation the files can be found in "/usr/share/openfire"

After the openfire is installed, one can check whether it is running by the following command:

> sudo ps -ef | grep openfire

To stop openfire at root@1.1.1.1, run the following command after login:

> /etc/init.d/openfire stop

To start openfire at root@1.1.1.1, run the following command after login to root@1.1.1.1

> /etc/init.d/openfire start

Lets visits 1.1.1.1:9090 to add to users account (non-admin) to the Users/Groups:

  • demo
  • reader
Install Spark IM Client to receive IM
Download and install Spark IM Client from the following link:


After the installation, we will login the Spark IM client using the "reader" account.

Smack as OpenFire Client to send IM

Now we will develop the java code which allow the "demo" account to send IM to the "reader" account (which we have login Spark IM client in the previous step). To this end, the Smack library is used to connect to the openfire server running in the droplet (which is at 1.1.1.1:5222). Lets create a Maven project and add the following dependencies:

<dependency>
<groupId>jivesoftware</groupId>
<artifactId>smack</artifactId>
<version>3.1.0</version>
</dependency>
<dependency>
<groupId>jivesoftware</groupId>
<artifactId>smackx</artifactId>
<version>3.1.0</version>
</dependency>

Now in the main class java file App.java, add the following implementation:

package com.memeanalytics.openfire_client;

import java.util.Calendar;

import org.jivesoftware.smack.ConnectionConfiguration;
import org.jivesoftware.smack.RosterEntry;
import org.jivesoftware.smack.XMPPConnection;
import org.jivesoftware.smack.XMPPException;
import org.jivesoftware.smack.packet.Message;
import org.jivesoftware.smack.packet.Message.Type;

public class App 
{
    public static void main( String[] args )
    {
     sendMessage();
    }
    
    private static void sendMessage()
    {
     String hostname =  "1.1.1.1";
     int clientPort = 5222;
        ConnectionConfiguration config=new ConnectionConfiguration(hostname, clientPort);
        XMPPConnection conn=new XMPPConnection(config);
        
        String username="demo";
        String password="[demo_password]";
        
        String to_username = "reader";
        String to = to_username + "@droplet.demo.com";
        
        try {
         conn.connect();
         conn.login(username, password);
   
  listContacts(conn);
   
  sendMessage(conn, to, "This is some message! The time is "+(Calendar.getInstance().getTime()));
   
  if(!hasContact(conn, to))
  {
   addFriend(conn, to, "reader");
  }
   
   
   
  conn.disconnect();
   
   
 } catch (XMPPException e) {
  // TODO Auto-generated catch block
  e.printStackTrace();
 }
    }
    
    private static void sendMessage(XMPPConnection conn, String to, String message_body)
    {
     Message message = new Message(to, Type.normal);
 message.setBody(message_body);
 conn.sendPacket(message);
 System.out.println("Message sent!");
    }
    
    private static boolean hasContact(XMPPConnection conn, String to)
    {
     RosterEntry entry = conn.getRoster().getEntry(to);
     return entry != null;
    }
    
    private static void addFriend(XMPPConnection conn, String to, String to_nickname) throws XMPPException
    {
     String[] groups=new String[] { "Friends" };
  
 conn.getRoster().createEntry(to, to_nickname, groups);
 System.out.println("Invitation sent!");
    }
    
    private static void listContacts(XMPPConnection conn)
    {
     for(RosterEntry entry : conn.getRoster().getEntries())
     {
      String userId = entry.getUser();
      String username = entry.getName();
      System.out.println(username + " >> " + userId);
     }
    }
    
   
}

The above implementation should be quite well self-explainable. The code start with opening an XMPP connection to 1.1.1.1:5222 which runs the OpenFire XMPP server at DigitalOcean's Droplet, and end by closing the connection.

After connection, the java code login using the demo's account, next it invokes the listContacts() method which list down all the contacts in demo's contact list. and sends a message to "reader" account. Lastly it checks whether "reader" is in demo's contact list, if not, then it sends an invitation to "reader".


Thursday, December 11, 2014

Ubuntu: copy file from local computer to another computer.

To copy a file (e.g. /var/app.jar) from the local computer to the "/tmp" folder of a remote computer (said root@192.168.2.2), run the following command:

> scp /var/app.jar root@192.168.2.2:/tmp

Wednesday, November 26, 2014

Setup and run ElasticSearch on Ubuntu 14.04 LTS

Run the following command to download the ElasticSearch 1.4.0:

> wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.deb

Setup ElasticSearch

After download, run the following command to install the debian package:

> sudo dpkg -i elasticsearch-1.4.0.deb

After installation, the configuration files for ElasticSearch are stored in the following directory:

/etc/elasticsearch/

The bin folder for ElasticSearch can be found in the following directory:

/usr/share/elasticsearch/

Now navigate to the user root folder, and open the .bashrc for editing:

> sudo gedit $HOME/.bashrc

In the .bashrc file, add in the following lines to the end

export ES_HOME=/usr/share/elasticsearch
export ES_CONF_HOME=/etc/elasticsearch

Save and close the .bashrc file.

Check whether the elasticsearch is running as a service on Ubuntu:

> sudo service elasticsearch status

if not, start the elasticsearch service by running the following commands:

> sudo service elasticsearch start

Run the following command to test the ElasticSearch:

> curl localhost:9200

You can also open a web browser and enter "localhost:9200" to see ElasticSearch in action.

Create a document

From the terminal, execute a PUT call such as the following:

> curl -XPUT http://127.0.0.1:9200/myindex/mytype/1 -d '{
"start_time" : "2014-11-26T14:00:00",
"end_time" : "2014-11-26T17:30:00",
"subject" : "Elastic Search Tryout",
"agenda" : {
 "item 1" : "Create an index",
 "item 2" : "Delete an index",
 "item 3" : "Open an index",
 "item 4" : "Close an index" }
}'

The _index is specified as "myindex", _type is specified as "mytype", and _id is specified as "1". After the index has been created, it can be accessed by running a GET call

> curl http://127.0.0.1/myindex/mytype/1

We can execute another PUT call as below to generate one more document under myindex/mytype:

> curl -XPUT http://127.0.0.1:9200/myindex/mytype/2 -d '{
"start_time" : "2014-11-27T14:00:00",
"end_time" : "2014-11-27T17:30:00",
"subject" : "Elastic Search Tryout 2",
"agenda" : {
 "item 1" : "Create a document",
 "item 2" : "Delete a document",
 "item 3" : "Execute a search",
 "item 4" : "Delete by search" }
}'

After the index has been created, it can be accessed by running a GET call

> curl http://127.0.0.1/myindex/mytype/2

To create an index document with an auto-generated id, we can do the following:

We can execute another PUT call as below to generate one more document under myindex/mytype:

> curl -XPUT http://127.0.0.1:9200/myindex/mytype/11122 -d '{
"start_time" : "2014-11-28T14:00:00",
"end_time" : "2014-11-28T17:30:00",
"subject" : "Elastic Search Tryout 2",
"agenda" : {
 "item 1" : "Create a document with auto-generated  index",
 "item 2" : "Delete a document",
 "item 3" : "Execute a search",
 "item 4" : "Delete by search" }
}'

The _id should not be omitted, but can be generated using UUID from a programming language.

Create an index

Creating an index is the same as creating a document under an index. From the terminal, execute a PUT call such as the following:

> curl -XPUT http://127.0.0.1:9200/myindex/ -d '{
"settings"  {
 "item 1" : "true",
 "item 2" : "false",
 "item 3" : "1",
 "item 4" : "2" }
}'

Update a document by id

The following commands can be used to update the document with id=11122:

> curl -XPUT http://127.0.0.1:9200/myindex/mytype/11122 -d '{
"start_time" : "2014-11-21T14:00:00",
"end_time" : "2014-11-21T17:30:00",
"subject" : "Elastic Search Tryout 4",
"agenda" : {
 "item 1" : "Create a document with auto-generated  index",
 "item 2" : "Delete a document",
 "item 3" : "Execute a search",
 "item 4" : "Delete by search" }
}'

> curl -XPOST http://127.0.0.1:9200/myindex/mytype/11122 -d '{
"start_time" : "2014-11-24T14:00:00",
"end_time" : "2014-11-24T17:30:00",
"subject" : "Elastic Search Tryout 4",
"agenda" : {
 "item 1" : "Create a document with auto-generated  index",
 "item 2" : "Delete a document",
 "item 3" : "Execute a search",
 "item 4" : "Delete by search" }
}'

Delete a document

Deleting an document will remove the document from the ElasticSearch nodes. From the terminal, execute a DELETE call such as the following:

> curl -XDELETE http://127.0.0.1:9200/myindex/mytype/1

Delete an index

Deleting an index will remove the index from the ElasticSearch nodes. From the terminal, execute a DELETE call such as the following:

> curl -XDELETE http://127.0.0.1:9200/myindex/

Close an index

Closing an index is an alternative to the deleting the index as ElasticSearch put it in an offline mode. From the terminal, execute a POST call such as the following:

> curl -XPOST http://127.0.0.1:9200/myindex/_close

After this, if you run the "curl http://127.0.0.1:9200/myindex", you will get a "not found" message

Open an index

Opening an index will put the offline index online again. From the terminal, execute a POST call such as the following:

> curl -XPOST http://127.0.0.1:9200/myindex/_open

Execute an search

Suppose we want to retrieve a list of documents under an index having a particular word (e.g. "elastic") in their content, we can run a GET call from the terminal:

> curl -XGET http://127.0.0.1:9200/myindex/_search?q=elastic

Suppose we want to search in under a _type = "mytype":

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_search?q=elastic

The query can be written in more details by specifying search in the "subject" field of the document:

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_search -d '
{"query":{"match": { "subject" : "elastic" }}}'

To search all documents under myindex/mytype:

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_search -d '
{"query":{"match_all":{}}}'

Sort a search

Suppose we want the json results returned in certain sorting order (e.g. sort by start_time asc):

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_search -d '
{"query":{"match_all":{}}, "sort":[{"start_time": "asc" }]}'

Filtering


Suppose we want to search records filtered by a range of a values in one its parameter (said, age), and have a particular match query (said, "name" must be "James"), and we want the results to be returned sorted by a field (said, age). Furthermore, we also want the total number of records returned to be limited to 20:

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_search -d'
{
"from": 0, "size": 20,
"query" : {
  "filtered" :
    "query" : {
      "match" : { "name" : "James" }
     },
     "filter": {
       "range" : {
          "age" : { "lte" : 20, "gte" : 10 }
        }
     }
"sort" : { "age" : {"order" : "asc" }}
};

Count by Query

To count the number of documents in myindex/mytype matching a particular query, from the terminal, run the following command:

> curl -XGET http://127.0.0.1:9200/myindex/mytype/_count -d '
{"query":{"match": { "subject" : "elastic" }}}'

Delete by Query

To delete all documents in myindex/mytype matching a particular query,  from the terminal, run the following command:

> curl -XDELETE http://127.0.0.1:9200/myindex/mytype/_query -d '
{"query":{"match": { "subject" : "elastic" }}}'







Monday, November 24, 2014

Setup and run MySQL on Ubuntu 14.04 LTS

Suppose you want to install and run MySQL on Ubuntu, run the following commands to install the MySQL in the terminal:

> sudo apt-get install mysql-server
> sudo apt-get install mysql-client

During the installation of the mysql-server, the user will prompted to enter the password for the root user account. After the installation is completed, the user can run the following command to find out whether mysql-server is running on the computer:

> sudo netstat -tap | grep mysql

To start, stop or restart mysql, run one of the following commands:

> sudo /etc/init.d/mysql start
> sudo /etc/init.d/mysql stop
> sudo /etc/init.d/mysql restart

To log into mysql, run the following command

> sudo mysql -u [username] -p

Enter the password in the prompt and press ENTER. For example, to log into the root user account:

> mysql -u root -p

After logging into mysql, first select the database by running the following sql command (which select the database "mysql" which is the default):

> use mysql;

To list the tables in the "mysql" database, run the following sql command:

> show tables;

Now suppose we want to take a look at what is inside the "user" datatable. Run the following sql command:

> select * from user;

To create and select a new database, e.g., "mylog", run the following sql command:

> create database mylog;
> use mylog;

To create a datatable, e.g., "mylogtable", in the database "mylog", run the following sql commands:

> use mylog;
> create table mylogtable(
id INT NOT NULL AUTO_INCREMENT,
country VARCHAR(255) NULL);

To exit mysql, run the following sql command:

> quit;

Saturday, November 22, 2014

Setup Kafka in a single machine running Ubuntu 14.04 LTS

Kafka is a messaging system that can acts as a buffer and feeder for messages processed by Storm spouts. It can also be used as a output buffer for Storm bolts. This post shows how to setup and test Kafka on a single machine running Ubuntu.

Firstly download the kafka 0.8.1.1 from the link below:

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1.1/kafka_2.8.0-0.8.1.1.tgz

Next "tar -xvzf" the kafka_2.8.0-0.8.1.1.tgz file and move it to a destination folder (say, /Documents/Works/Kafka folder under the user root directory):

> tar -xvzf kafka_2.8.0-0.8.1.1.tgz
> mkdir $HOME/Documents/Works/Kafka
> mv kafka_2.8.0-0.8.1.1 $HOME/Documents/Works/Kafka

Now go back to the user root folder and open the .bashrc file for editing:

> cd $HOME
> gedit .bashrc

In the .bashrc file, add the following line to the end:

export KAFKA_HOME=$HOME/Documents/Works/Kakfa/kafka_2.8.0-0.8.1.1

Save and close the .bashrc and run "source .bashrc" to update the environment variables. Now navigate to the kafka home folder and edit the server.properties in its sub-directory "config":

> cd $KAFKA_HOME/config
> gedit server.properties

In the server.properties file, search the line "zookeeper.connect" and change it to the following:

zookeeper.connect=192.168.2.2:2181,192.168.2.4:2181

search the line "log.dirs" and change it to the following:

log.dirs=/var/kafka-logs

Save and close the server.properties file (192.168.2.2 and 192.168.2.4 are the zookeeper nodes). Next we go and create the folder /var/kafka-logs (which will store the topics and partitions data for kafka) with write permissions:

> sudo mkdir /var/kafka-logs
> sudo chmod -R 777 /var/kafka-logs

Now set up and run the zookeeper cluster by following instructions in the link http://czcodezone.blogspot.sg/2014/11/setup-zookeeper-in-cluster.html. Once this is done, we are ready to start the kafka messaging system by running the following commands:

> cd $KAFKA_HOME
> bin/kafka-server-start.sh config/server.properties

To start testing kafka setup, Ctrl+Alt+T to open a new terminal and run the following command to create a topic "verification-topic" (a topic is a named entity in kafka which contain one or more partitions which are message queues that can run in parallel and serialize to individual folder in /var/kafka-log folder):

> cd $KAKFA_HOME
> bin/kafka-topics.sh --create --zookeeper 192.168.2.2:2181 --topic verification-topic --partitions 1 --replication-factor 1

The above command creates a topic named "verification-topic" which contains 1 partition (and with no replication)

Now we can check the list of topics in kafka by running the following command:

> bin/kafka-topics.sh --zookeeper 192.168.2.2:2181 --list

To test the producer and consumer interaction in kafka, fire up the console producer by running

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic verification-topic

9092 is the default port for a kafka broker node (which is localhost at the moment). Now the terminal enter interaction mode. Let's open another terminal and run the console consumer:

> bin/kafka-console-consumer.sh --zookeeper 192.168.2.2:2181 --topic verification-topic

Now enter some data in the console producer terminal and you should see the data immediately display in the console consumer terminal.

Thursday, November 20, 2014

Setup Storm on a single machine running Ubuntu LTS 14.04

This post is about how to setup storm to run on a single machine with Ubuntu LTS 14.04. Firstly download storm-0.9.0.1 from the following link:

https://dl.dropboxusercontent.com/s/tqdpoif32gufapo/storm-0.9.0.1.tar.gz

Next "tar -xvzf" the downloaded gz file to the folder storm-0.9.0.1, let's suppose the full path is
$HOME/Documents/Works/Storm/storm-0.9.0.1

Now cd to the $HOME directory and open the .bashrc file to edit by running the following commands:
> cd $HOME
> gedit .bashrc

In the .bashrc file, add the following line to the end:
export STORM_HOME=$HOME/Documents/Works/Storm/storm-0.9.0.1

Save and close the .bashrc file, Run the following command to update the environment var:
> source .bashrc

Now navigate to the storm's home directory and modify the storm.yaml in its conf sub-directory:
> cd $STORM_HOME
> cd conf
> gedit storm.yaml

Replace the content of storm.yaml with the following lines:

storm.zookeeper.servers:
  - "127.0.0.1"
storm.zookeeper.port: 2181
nimbus.host: "127.0.0.1"
java.library.path: "/usr/local/lib"
storm.local.dir: "/tmp/storm-data"
storm.messaging.transport: backtype.storm.messaging.netty.Context
supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

Save and close the storm.yaml file. Below are some explanation of the above configurations in the storm.yarm file:

storm.zookeeper.servers: the ip addresses of the zookeepers used to maintain the states of the storm cluster
storm.zookeeper.port: the port used by the zookeepers to communicate with its clients
nimbus.host: the ip address of the nimbus node in storm cluster
supervisor.slots.ports: the ports used by each of the work processes (each of which is a jvm) in the supervisor nodes to listen and communicate.

Now run the following commands to start the nimbus node in the storm:
> cd $STORM_HOME
> bin/storm nimbus

Now Ctrl+Alt+T to open another terminal and start a supervisor node in the storm:
> cd $STORM_HOME
> bin/storm supervisor

To submit a jar containing storm topology, run the following command:
> cd $STORM_HOME
> bin/storm jar [jarName] [mainClassName] [mainClassArguments]

[mainClassName] is the fullname of the class containing main() method (which builds and submit the topology), [mainClassArguments] are the arguments passed to the main() method, [jarName] is the full path of the jar containing all the classes and dependencies

To see the java processes running:

> jps

To activate and deactivate a topology in storm:
> cd $STORM_HOME
> bin/storm activate [topologyName]

To kill a topology in storm
> cd $STORM_HOME
> bin/storm kill [topologyName]




Wednesday, November 19, 2014

Setup ZooKeeper on a single machine running Ubuntu 14.04 LTS

This post shows how to set up ZooKeeper on a single machine running Ubuntu 14.04 LTS.

Firstly download ZooKeeper from the following link:

http://mirror.nus.edu.sg/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

Unzip the downloaded file and put the unzipped folder in a folder, say the following
$HOME/Documents/Works/ZooKeeper/zookeeper-3.4.6

Now go the the $HOME directory and run the following command to open the .bashrc file in terminal:

> gedit .bashrc

In the .bashrc file, add the following line to the end:
export ZK_HOME=$HOME/Documents/Works/ZooKeeper/zookeeper-3.4.6

Save and close the .bashrc file, run the following command to update the environment var:

> source .bashrc

Now run the following command to go to the $ZK_HOME/conf directory

> cd $ZK_HOME/conf

Run the following command to create the configuration zoo.cfg:

> touch zoo.cfg
> gedit zoo.cfg

In the zoo.cfg opened, add the following lines:

tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181

Save and close zoo.cfg, go back to the ZooKeeper home directory:

> cd $ZK_HOME

Run the following command to start the zookeeper on the machine:

> bin/zkServer.sh start

Run the following command to check the java process running:

> jps

Run the following command to check the status of zookeeper:

> bin/zkServer.sh status

Run the following command to stop the zookeeper:

> bin/zkServer.sh stop



Remote access Ubuntu or MacOS using OpenSSH

To remotely access Ubuntu or MacOS using OpenSSH, type the following command:
$ssh username@ipaddress

where ipaddress is the ip address of the computer to be remotely access and the username is a authorized user on that computer. If you received the following error:

ssh: connect to host ipaddress port 22: Connection refused

The error is caused by the target computer does not have updated openssh server.

For Ubuntu, to rectify the problem, login to the target computer and launch the terminal. Then type the following commands:
$sudo apt-get install openssh-client openssh-server

For MacOS, go to Finder, and launch the "System Preferences...", click the "Sharing", check "File Sharing"and "Remote login". And in the "Remote Login", set "Allow access for " to "All Users"

Run LibSVM in Octave on Ubuntu 14.04 LTS

If you do not have octave installed, run the following command in the terminal:
$sudo apt-get upgrade
$sudo apt-get update
$sudo apt-get install octave
$sudo apt-get install liboctave-dev

 After this, download the LibSVM from the following link:

http://www.csie.ntu.edu.tw/~cjlin/libsvm/

unzip by running the tar xfz [libsvm tar file] command

create a working directory under your $HOME directory with a name, said, "OctaveProjects". now copy the unzipped libsvm-3.20 folder into the OctaveProjects folder.

Now type "octave" in the search box of Ubuntu to launch it, in the octave program, type the following command to goto the libsvm's matlab folder:
$cd $HOME/OctaveProjects/libsvm-3.20/matlab

type the following command to invoke the make.m in the libsvm's matlab folder:

$make

Once the matlab interface for libsvm has been built, go back to the OctaveProjects folder by typing the following command:
$cd $HOME/OctaveProjects

Now run the following command to add the libsvm path:
$addpath './libsvm3.20/matlab';

You can now run the libsvm, below is the usage from README in libsvm:

Usage
=====

octave> model = svmtrain(training_label_vector, training_instance_matrix [, 'libsvm_options']);

        -training_label_vector:
            An m by 1 vector of training labels (type must be double).
        -training_instance_matrix:
            An m by n matrix of m training instances with n features.
            It can be dense or sparse (type must be double).
        -libsvm_options:
            A string of training options in the same format as that of LIBSVM.

octave> [predicted_label, accuracy, decision_values/prob_estimates] = svmpredict(testing_label_vector, testing_instance_matrix, model [, 'libsvm_options']);
octave> [predicted_label] = svmpredict(testing_label_vector, testing_instance_matrix, model [, 'libsvm_options']);





Dual Installation of Ubuntu 14.04 LTS along side Windows 7

This is share some experiences in installing Ubuntu 14.04 LTS along side Windows 7 using wubi.exe. It turns out the process is not as straight forward. First I will describe a few processes which were tried but won't work. The scenarios describe involves installing Ubuntu 14.04 LTS 64 bit on Windows 7 64bit

Firstly download the Ubuntu ISO from

http://www.ubuntu.com/download

After download the ISO, on the Windows 7, mount the ISO image in the MagicISO (which can be downloaded and install from http://www.magiciso.com/). In the mounted disk, copy the wubi.exe out into an empty folder. Now unmount the ISO and copy the ISO image file into the same folder as the wubi.exe.

Now double click the wubi.exe and follow the instruction to install. After the installation is completed, restart the computer and selects the Ubuntu in the operating system manager. When entered in the Ubuntu, with the "*Ubuntu" option selected, press the key k to go into GRUB boot entry, now locate the following line:

RO- linux /boot/vmlinuz-3.13.0-24-generic root=UUID=AAC884AC1F144321 loop=/ubuntu/disks/root.disk ro quiet splash $vt_handoff

Change the RO and ro to RW and rw, as shown below:

RW- linux /boot/vmlinuz-3.13.0-24-generic root=UUID=AAC884AC1F144321 loop=/ubuntu/disks/root.disk rw quiet splash $vt_handoff

Now save and continue, the above step is to prevent the error: "Serious errors were found while checking the disk drive for /". Once goes into the Ubuntu, we want to make the change to the GRUB entry permanent so that the error won't appear next time. To do so, open the terminal and type the following commands:

$sudo gedit /etc/grub.d/10_linux

In the file, change ro to rw. Now save and closes the file. and type the following command line:

$sudo upgrade-grub


Below describes a few "traditional" ways which won't work for the dual installation of LTS 14.04 on Windows 7:

1. Mount the ISO in MagicISO and double-click wubi.exe to install: this is the way for the Ubuntu 12.02 and earlier, but if you do it for LTS 14.04, you will probably get the following error: "None Type" object has no attribute 'get_info'
2. Burn the ISO image into a DVD and tries to boot from DVD to install: this is another way which works for the previous ubuntu version but not for LTS 14.04, it will throws the error: "None Type" object has no attribute 'get_info'

Tuesday, November 18, 2014

Setup Maven on Ubuntu and MacOS

This post contains how to set up maven on Ubuntu and MacOS as well as finding their installation location. On Ubuntu, run the following command in the terminal:
$mvn 
The printout will indicate whether maven is installed or not. If not, run the following command to install:
$sudo apt-get install maven2
After maven is installed, its home directory is at /usr/share/maven2 (depending on your version)
To install maven on MacOS, run "$mvn --version" to check whether it is installed. If not, run the following command in the terminal to install:
$brew install maven
After the installation, to find the installation path of maven, type the following command:
$mvn --version

Install Oracle JDK 8 on Ubuntu 14.04 LTS

Due to license issue, Ubuntu 14.04 LTS does not come with Oracle JDK. you can check the current java installed by running:

$java -version

To install the latest Oracle JDK 1.8 and set as default JDK on Ubuntu. First download the JDK 1.8 from the Oracle download page:


or


Download the jdk-8u25-linux-x64.rpm (assuming that you have a 64bit Ubuntu installed). As Ubuntu is based on Debian, we need to convert the *.rpm to *.deb fileformat. To do so, install the alien using the following command line in Ubuntu terminal:

$sudo apt-get install alien

After the alien is installed, cd to the download folder containing the jdk-8u25-linux-x64.rpm. Run the following command line:

$alien -k jdk-8u25-linux-x64.rpm --scripts

After the above command line returns, the user should find the jdk-8u25-linux-x64.deb in the same folder. Now run the following command line to install JDK:

$sudo dpkg -i jdk-8u25-linux-x64.deb

After the JDK 1.8 is installed, open the .bashrc file in the $HOME directory:

$cd $HOME
$gedit .bashrc

In the .bashrc opened, add the following lines to the end:

export JAVA_HOME=/usr/java/jdk1.8.0_25/
export PATH=$JAVA_HOME/bin:$PATH

Save and close the .bashrc, now run the following command to update the environment var:

$source .bashrc

Done. we can check the current JDK on Ubuntu to confirm that the current version is indeed Oracle JDK 1.8 by running the following command:

$java -version

which should prints the following lines:

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)