Programming Breaks: October 2014

PersistentDictionary from ESENT is a useful class to be used in place of Dictionary object in C# where Dictionary is not big enough to hold large number of items. it can be downloaded from:

https://managedesent.codeplex.com/wikipage?title=PersistentDictionaryDocumentation&referringTitle=Home

Because it uses ESE (Extensible Storage Engine, available on Windows since 2000) database for storage. PersistentDictionary can store much more objects than Dictionary when storing large number items. However, PersistentDictionary has some restrictions as well as performance behavior which users should take precaution when using it. Below is some of these that i came across when using this class.

Key in PersistentDictionary must implement IComparable<T> interface
Key in PersistentDictionary must not be a struct, should be a class or valuetype
Value in PersisentDictionary must a system valuetype or a struct that implements a System.Serializable attribute. It is not straight forward to use any class as the value in PersistentDictionary (even if it implements Serializable or DataContract interface), if it is so desired that the value should be a class then refers to the DemoDictionary in the above link
The Clear() method in PersistentDictionary is very tricky to use, it almost throttled my program when it is called. My recommendation is that don't use Clear() method. Instead i used the "new PersistentDictionary<KeyType, ValueType>("[dictionary_name]")" to reassign the object. This solved the problem, at least in my case.
When creating a PersistentDictionary, do note that the constructor "new PersistentDictionary<KeyType, ValueType>("[dictionary_name]")" will invoke from the ESE database in the directory [dictionary_name], therefore if the PersistentDictionary is created previously, and new content is to be generated for the same PersistentDictionary when the constructor is called, then the user should call the PersistentDictionaryFile.Exists([dictionary_name]) to check whether cache file is already generated and call PersistentDictionaryFile.Delete([dictionary_name]) to delete the cache files previously generated for the same dictionary object (also remember to dispose the PersistentDictionaryObject first before calling Delete method as otherwise ESENT will have the database of the corresponding object open until the instance is finalized).

In this section, the details of sensing power, temperature sensing and monitoring are discussed. I will start by discussing the hardware that you need to buy and build a starter kit for power, temperature and humidity sensing and monitoring.

1. Build and Set up Base Station

1.1. Purchase Hardwares

First of all, we need to have a base station which receives transmission from remote sensors and stores / processes / redirects the sensor data. In this case, we will be using a Raspberry Pi Model B, which the user can purchase from the following link:

Raspberry Pi Model B Web-Connected Base Station

Apart from the above Raspberry Pi Model B base station, we will also purchase a set of accessories that go with it (e.g. for radio transmission, gateway, etc):

Micro USB Power Cable (for power supplier to the Raspberry Pi base station)

RFM12Pi - Raspberry Pi Base Station Receiver Board (for receiving radio transmission from remote sensors)

Pre-Loaded Gateway on microSD Card for Raspberry Pi ('Model B+' Compatible) (microSD card inserted into the base station with preloaded gateway)

Ethernet Cable (Connect the Raspberry Pi Model B base station to a PC or laptop)

MicroSD Card Reader (optional, in fact, with Ethernet cable, Raspberry Pi Model B, and a laptop with Ethernet adapter, the microSD card reader is not necessary as the preloaded gateway on the microSD card can be directly modified using these)

1.2. Assemble the base station

Now we have all the components for the base stations, I will now show how they can be assembled into the base station.

Firstly slot the RFM12Pi onto the Raspberry Pi Model B as shown in the figure below (Note that the RFM12Pi is slotted into the GPIO with the first pin of the Raspberry Pi aligned)

Next plug in the microSD Card as shown in the figure below:

Next connect the USB power supply as shown in the figure below (do not turn on the power yet)

Finally connect the Ethernet cable between the Raspberry Pi and the PC as shown in the figure below:

2. Access and Run Base Station microSD card using Ethernet Cable from PC

Now that we have assemble the base station (do not turn on the power of the base station yet). we need to configure the PC Ethernet adapter so that we can access the base station microSD card from PC.

2.1 Change Ethernet Adapter settings

To do this, open the "Network and Sharing Center" on the Windows platform, then click the "Change Adapter Settings" link on the left panel of the Network and Sharing Center. Now right click the "Local Area Network" which is the Ethernet adapter, and select "Properties" from the pop-up menu. In the properties dialog popup, select the "Networking" tab and select the "Internet Protocol Version 4 (TCP/IPv4)" from the list box, and click the "Properties" button below the list box. Enter the IP address as "192.168.2.1" and the mask as "255.255.255.0", as shown in the figure below

Note that if your PC or laptop also has wireless connection, you may want to allow the Raspberry Pi to access internet via your PC wireless connection, in that case, right click on the "Wireless Network Connection" in the network adapter settings page and select "Properties", and check "Allow other network users to connect through this computer's internet connection" with "Local Area Connection" select as the "Home network connection" in the combo box below it, as shown in the figure below, then click "OK" (Note that if this is the case, your IP address in the "Local Network Connection" Ethernet adapter settings will be changed to 192.168.137.1).

2.2. Configure a DHCP server for the Raspberry Pi base station

Once this is done, go and download the DHCP Server from http://www.dhcpserver.de/cms/ . Unzip it and double click dhcpwiz.exe to DHCP Wizard (i.e. dhcpwiz.exe in its root folder). On the wizard, click "Next" and then select "Local Area connection" from the Network Interface cards dialog (if you notice that "192.168.137.1" does not appear as the address for the "Local Area Connection", you can turn on the power of the Raspberry Pi temporarily for the Ethernet adapter to pick up this, then turn off the Raspberry Pi base station again until the end of this section), and click "Next again", as shown in the figure below:

leave the next screen "Supported Protocols" as default and click "Next", as shown in the figure below:

In the next screen "Configuring DHCP for Interface", set the IP-Pool from "192.168.137.100" to "192.168.137.110" (Note that earlier we change from "192.168.2.1" to "192.168.137.1" as we want Raspberry Pi base station to access internet via Ethernet cable, otherwise the range will be "192.168.2.100" to "192.168.2.110"). This is as shown in the figure below. Now click "Next" to proceed.

In the next screen "Writing the "INI" file", check the "Overwrite existing file" option and click "Write INI file" button. as shown in the figure below. Now click "Next" to proceed to the next screen.

In the next screen "DHCP configuration completed", click the "Install", "Configure" and "Start" buttons and click "Finish" button to complete the wizard, as shown in the figure below:

Now double click the dhcpsrv.exe to launch the server (i.e. dhcpsrv.exe in the root folder) in the "DHCP Server" screen as shown below, click "Continue as tray app" button.

At this point the DHCP server has been setup and properly running. Turn on the power of the Raspberry Pi base station, the server will then popup a notification indicating that a DHCP address "192.168.137.100" has been assigned to the Raspberry Pi base station, as shown in the figure below:

2.3. Use PuTTY client to access Raspberry Pi base station

Now download the free ssh and telnet client, PuTTY from the following link and unzip.

http://www.putty.org/

double click to launch the PuTTY client and enter the "192.168.137.100" DHCP address into the PuTTY configuration dialog as shown in the figure below:

Click "Open" button, the user will be presented with the PuTTY console as shown in the figure below:

Now enter "pi" and press ENTER. When the prompt "pi@192.168.137.100's password:" appear, enter "raspberry" as the password and press ENTER, as shown in the figure below:

3. Run Raspberry Pi base station

3.1. Configure the date for the Raspberry Pi base station

Before we proceed to make other changes to the base station's gateway. the first thing we want to do is to configure the date. To do this, first check the current time by entering the following command in PuTTY

$date

The current time will be print as shown in the figure below:

The time is obviously wrong. To correct this, run the following commands:

$rpi-rw

$sudo /etc/init.d/ntp stop

$sudo raspi-config

The commands "rpi-rw" unlock the file system for write access ("rpi-ro" will lock the file system). "ntp stop" command stops the ntp server"; "raspi-config" launch the "Raspberry Pi Software Configuration Tool" as shown in the figure below:

Now select "4 Internationalisation Options", then select "I2 Change Timezone", then

follows the screens to select the date and regional settings and select "<Finish>"

Once this is done, enter the following commands to update the current date and time

$sudo date -s '12 Oct 2014 22:19:00'

$sudo /etc/init.d/ntp start

Once this is done, the system is in correct time format. At this point the time should be replaced with your current time. Now that if you do not perform this step, later your input and feeds at your local emoncms web will have "inactive" as its "Last Updated" value since they would count from 1970 instead of the current time. This is a common error. (PS: based on my experience, it is not necessary to stop and restart NTP server, simply reset the date will do, also the raspi-config only need to do it once).

3.2. Start the local emoncms

Now we can start the local emoncms preloaded in the microSD card, by entering the following commands in the PuTTY client:

$sudo /usr/bin/localemoncms-enable

This will start the local emoncms if it is not yet started (Note that you can also run "sudo /usr/bin/localemoncms-disable" command to shutdown the local emoncms). The commands will start the local emoncms, the redis-server, the mysql, etc.

At this point if you open your web browser and navigate to the following link

http://192.168.173.100/emoncms

http://192.169.173.100

you will see the local emoncms running as shown in the figure below

You can login using the default username and password here:

username: raspberry

password: raspberry

Once logged in, you will see the following account page:

Notice the Write API key and the Read API key at the lower left corner of the account page, we would show how to use the write API key later to modify the emonhub.conf file so that remote sensors can post data onto the emoncms.

3.3 Learn a bit more using mysql

The mysql allows the user to explore a bit more of the Raspberry Pi base station's database, to this, enter the following commands in the PuTTY client:

$mysql -u root -p emoncms

This will prompt user to enter the password, the default password is "raspberry", key in and press ENTER.

Now in the mysql console, you can list down all the tables by entering the following command:

$show tables;

You will notice there is a users table, which stores the user accounts. Run the following command to see its content

$select * from users;

4. Setup Remote Sensors

Now we have the base station up and running, we would need to set up the remote sensors, there are two different remote sensors that we would use in this blog, the name and link to buy are listed below:

emonTH - Temperature & Humidity Node (remote sensors measuring temperature and humidity)

emontx-v3-electricity-monitoring-transmitter-unit (remote sensors measuring power and energy)

The emonTH is pretty standalone and only requires A5 battery, for the emonTx v3 to work, we need the following components:

100A max clip-on current sensor CT (for measuring the current and power)

AC-AC Power Supply (power supply for the emonTx v3)

The figure below shows how the emonTx v3 is assembled from the current sensor CT and power supply as shown below:

5. Receive and store Sensor Data from Remote Sensors

Now that the base station and the remote sensors are ready, we have even made sure they can communicate via RFM12 radio transmission, the final step is to allow remote sensor data to be posted to the local emoncms, this can be done by replacing the write api key in the emonhub.conf file. To do this, in the PuTTY client, enter the following command:

$sudo nano /boot/emonhub.conf

this will show the emonhub.conf in the nano text editor, search the line starting with "apikey = " and replace the "xxxxx...xxx" with your write API key. as shown in the green region in the figure below.

Once this is done, press "Ctrl + o" to save and "Ctrl + x" to exit. now run the following commands:

$sudo /usr/bin/localemoncms-disable

$sudo /usr/bin/localemoncms-enable

then navigate to http://192.168.173.100/emoncms and log in using username=raspberry and password=raspberry, you will see the inputs shown as follow:

Node 10 is the emonTx v3 power measure (from a single current sensor CT) and Node 19 is the emonTH which measure temperature and humidity. Note that edit of each row allow user to specify a name of that measurement. Also the user can edit each row so that a log to feed is added for each input measurement, as shown by the "log" icon in the Process List column, and with this set, user can see the feed in the "Feed" page as shown in the figure below:

6. Retrieve data by third-party software

If the user is using third-party software for processing and analyzing sensor data / feed from the remote sensors, they can use a set of web service in terms of csv, json, xml outputs, some examples are as below:

http://192.168.137.100/emoncms/feed/list.json?userid=0 - returns a list of public feeds made public by the given user.

http://192.168.137.100/emoncms/feed/value.json?id=0 - returns the present value of a given feed

http://192.168.137.100/emoncms/feed/get.json?id=0&field= - returns the present value of a given feed

http://192.168.137.100/emoncms/feed/data.json?id=0&start=&end=&dp= - returns feed data

http://192.168.137.100/emoncms/feed/histogram.json?id=0&start=&end= - returns histogram data

More can be found on the page

http://192.168.137.100/emoncms/feed/api

Below is the screenshots showing the feed in an application that i developed:

7. Other Useful Links and Pointers

http://www.instructables.com/id/Use-ssh-to-talk-with-your-Raspberry-Pi/
http://www.instructables.com/id/Direct-Network-Connection-between-Windows-PC-and-R/
http://openenergymonitor.org/emon/
http://www.raspberrypi.org/forums/viewtopic.php?f=63&t=59893
http://openenergymonitor.org/emon/node/2444
https://github.com/emoncms/emoncms/blob/bufferedwrite/docs/install.md

Programming Breaks

Sunday, October 19, 2014

Parse JSON string using JSON.Net

Tips on using PersistentDictionary from ESENT

Sunday, October 12, 2014

Power, Temperature and Humidity Sensing and Monitoring using Raspberry Pi and Emoncms