Temperature sensor with Raspberry Pi 3 and AWS

This tutorial describes how to set-up the communication between Raspberry Pi 3 and AWS cloud. As a concrete example, we will store temperature data tin the cloud and visualize it. This is accomplished in several steps:

  1. Connect Raspberry Pi 3 to AWS
  2. Read temperature from I2C sensor
  3. Store data time-series in DynamoDB
  4. Visualize the historical temperature data
  5. Receive alerts on low and high temperature

Connecting RPi 3 to AWS

As a starting point, we have the Raspberry Pi 3 board with Raspbian Linux  and fully configured WiFi connection. Let’s connect it to the Amazon IoT!

This process can be split up into two parts – set-up of security credentials and SDK installation. During the security credentials set-up, it does not matter which SDK you prefer. Amazon has pretty detailed description of how to connect RPi for Javascript SDK, but we can for the large part follow the same steps. Follow this AWS manual to

  • Create and Attach a Thing (Device)
  • Generate certificates and policy
  • Download certificates

Once all the certificates (i.e. device certificate, private key, and root CA certificate) are downloaded from AWS, let’s upload them into $ ~/certs folder on the device.

Now, we will install Python SDK rather than NodeJS. Simply following the instructions here and executing $pip3 install AWSIoTPythonSDK

Let’s also clone the Git repository so we can use its sample code to learn the SDK basics:

$ git clone https://github.com/aws/aws-iot-device-sdk-python.git
$ cd aws-iot-device-sdk-python/samples
$ ls 
basicPubSub
basicShadow
ThingShadowEcho

We’ll take advantage of the basicPubSub.py script. It should be called together with the certificates (so that AWS IoT can identify the device):

# Certificate based mutual authentication
$ python3 basicPubSub.py -e <endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>

This sample code will subscibe to sdk/test/Python topic and publish to the same topic in the infinite loop. It is possible to use the external MQTT client (or the on built-into AWS IoT ) to see these messages and validate that the connection with AWS IoT is successful.

We are going to use the connection established in this way to publish ambient temperature to the appropriate topic. But first, we need to read the sensor data.

Reading Temperature Sensor…

I am using an affordable and easy to set-up TMP102 sensor from SparkFun, connected to the board using Pi Wedge. The connections are straightforward, as there are only four wires: SDA, SCL, +3.3V and GND.

Raspberry Pi with TMP 102
Raspberry Pi with TMP102 – I2C connections

Now let’s make sure that the I2C is enabled by typing  $ sudo raspi-config and going to the advanced options menu. If I2C is currently off,  enabling it requires reboot of RPi board. We also need to install i2c-tools by executing  $ sudo apt-get install -y i2c-tools  to be able to communicate through I2C. Let’s see what’s connected by executing i2cdetect command:

$ i2cdetect -y 1
   
  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- 48 -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

We have the TMP102 sensor at its standard address 0x48. The full temperature reading is stored in the first 12 bits with address 0. The first 8 bits provide  to get the rounded temperature reading:

$ i2cget -y 1 0x48 0 b
0x19

The returned result 0x19 = 25 C is the room temperature as measured by the sensor. The full 12 but reading will be used later to get more accurate data. For now,  we are assured that the sensor is connected and works. All we need is a python script that will read the temperature and send it to AWS.

…with Python

Let’s use pigpio library to communicate with I2C – seems like it comes pre-installed with Python3. Let’s also install the pyth0n-smbus library: $ sudo apt-get install python-smbus.

First, start a pigpio daemon by executing  $ sudo pigpiod.  After that, the script should acquire and print the temperature. We can start it by executing $ python tmp102.py The source code  of tmp102.py is below:

#!bin/python3
#reading TMP 102 sensor

import pigpio
import time

def tmp102_reading(byte0, word0):
 #calculation of the temperature based on 
 #the first word and the first byte
 
 # !!! not tested for negative temperatures !!!!
 #last 4 bits of the word0
 l4b = (word0 & 0b1111000000000000)>>12
 temperature = ((byte0<<4) | l4b) * 0.0625
 return temperature
 
 

#i2c bus of the Raspberry Pi 3
i2c_bus = 1
#TMP 102 address on the i2c bus
addr = 0x48


dev_pi = pigpio.pi()
dev_tmp = dev_pi.i2c_open(i2c_bus, addr, 0)
register_n = 0
try:
 while True:
 t_byte = dev_pi.i2c_read_byte_data(dev_tmp, 0)
 t_word = dev_pi.i2c_read_word_data(dev_tmp, 0)
 t = tmp102_reading(t_byte, t_word)
 print(' Temperature: {} C'.format(t))
 time.sleep(1)
except KeyboardInterrupt:
 pass
print('Exiting the loop');
r = dev_pi.i2c_close(dev_tmp)

This script will output the sensor reading every second until Ctrl+C is pressed:

Temperature: 25.59375 C
Temperature: 25.59375 C
Temperature: 25.625 C
Temperature: 25.625 C
Temperature: 25.625 C
Temperature: 25.625 C
Temperature: 25.625 C

Sending data to AWS

Instead of printing the temperature in the console, we need to send it to AWS IoT and repeat the whole thing after a set interval – every minute, for example. The Raspberry Pi has already been connected to AWS during the first step. We will re-use the publish/subscribe code example (basucPubSub.py) from the AWS SDK example to publish to a new topic for our new sensor.

First, let’s create variables for the topic and delay between temperature updates. Plus, we can give our sensor a serial number – in case we build more than one and need to distinguish between them:

delay_s = 60
sensor_sn = '00000001'
topic = 'myrpi/'+sensor_sn

Let’s now read the temperature in the loop and send it to AWS (until Ctrl+C):

try:
  while True:
    loopCount += 1
    t_byte = dev_pi.i2c_read_byte_data(dev_tmp, 0)
    t_word = dev_pi.i2c_read_word_data(dev_tmp, 0)
    t = tmp102_reading(t_byte, t_word)
    timestamp = datetime.datetime.now()
    print(' Temperature: {} C   Loop # {:d}'.format(t,loopCount))
    print(' Time: {} \n'.format(timestamp))
    msg = '"Device": "{:s}", "Temperature": "{}", "Loop": "{}"'.format(sensor_sn, t,loopCount)
    msg = '{'+msg+'}'
    myAWSIoTMQTTClient.publish(topic, msg, 1)
    print('Sleeping...')
    time.sleep(delay_s)
except KeyboardInterrupt:
  pass
  
print('Exiting the loop');
r = dev_pi.i2c_close(dev_tmp)
myAWSIoTMQTTClient.disconnect()
print('Disconnected from AWS')

Remember – this(get the entire file here)  is only a (slight) modification of the basicPubSub.py file provided in AWS Python SDK. Path to certificated is still necessary for it to run properly. The shell command to start publishing to AWS is therefore

$ python3 publis_temp.py -e <Your:AWS:endpoint> -r <rootCAFilePath> -c <certFilePath> -k <privateKeyFilePath>

The script publishes the following payload to the myrpi/00000001 topic:

{
 "Device": "00000001", 
  "Temperature": "27.0",
  "Loop": "2670"
}

 

The structure of this payload is very similar to the data sent by  the AWS  IoT button. Once the data is published we can time-stamp and log it the DynamoDB table. This was described in detail in the post about IoT button.

Visualizing AWS IoT Button data (part II)

The IoT button is connected to the AWS cloud and every click is recorded in the DynamoDB. It’s now time to query the database and present the data, preferably in a visual form. Here, we will do everything in the browser, building our app using JavaScript AWS SDK. Thus, the first step is to add the script to the HTML page:

<script src="https://sdk.amazonaws.com/js/aws-sdk-2.5.4.min.js">
</script>

Configure the SDK

Create and configure user credentials

We would like to send requests to the DynamoDB, but the SDK has to be configured before we can do that. The two required pieces of information are the region and credentials. There are several ways to provide credentials securely:

  1. Using Amazon Cognito to authenticate users
  2. Using web identity federation to authenticate users
  3. Hard-coded in your application

We are going to pick the easiest one and hard-code the credentials into the application. This is also the least desirable method from the security standpoint, used here for the sake of convenience and speed. To do so, open the Identity and Access Management (IAM) console, click on Users menu and Create User. Immediately after the user is created, you can copy (or download) the security credentials and hard-code them into the application:

var user_credentials = {
        accessKeyId: 'AKIAIV6CXXXXXXXXXXXX', 
        secretAccessKey: 'i8+vtmKkXXXXXXXXXXXXX'
};

Allow read-only access to DynamoDB

This newly created user still does not have any service permissions and thus can’t perform any actions. We need to attach a specific policy to allow DynamoDB access. Click on the Users name in the IAM, select the user, open Permissions tab and click Attach policy. Type AmazonDynamoDBReadOnlyAccess, select it and click Attach policy. We are now officially allowed to query the DynamoDB. Permissions are set as read only to mitigate the security risk due to hard-coding the credentials.

Let’s finish configuring the JavaScript SDK by executing a command to update user credentials:

AWS.config.region = 'us-east-1';
AWS.config.update(user_credentials);

List DynamoDB tables

To test the newly created user credentials and verify the read-only access to Dynamo, lest first list all Dynamo tables. This is done by calling the listTables() method of the DynamoDB object. Let’s also put all tables into the drop-down menu for easy selection.

var dynamodb = new AWS.DynamoDB();
dynamodb.listTables(function(err, data) {
 var html_str;
 if (err == null){
   html_str = ' DynamoDB tables: <select>';
   $.each(data.TableNames, function(index, value){
     html_str += '<option value="' + value + '">' + index+1 + '. '+ value + '</option>';
   });
   html_str += '</select>';
  }
  else{
   console.log(err);
   html_str = err;
  }
  $('#table-list').html(html_str); 
 });

If the credentials are invalid, we will see an error message. If the credentials were crated properly, we can select the table iot-buttons from the menu (that’s the name we decide on in the first part).

Display table info

Let’s now obtain table info by calling describeTable() method and passing the table name as parameter. For example, let’s get the total number of records in a table:

var params = {
   TableName: $(this).val()
};
dynamodb.describeTable(params, function(err, data) {
   if (err) {
      //an error occurred 
       $('#table-list').append('<p>' + JSON.stringify(err) + '</p>');
    }
    else {
       //successful response
       var table = data.Table;
       $('#table-list').append('<p> '+table.TableName + ": # of items= "+table.ItemCount + '</p>');
    }
 });

Query the table

Finally, let’s query the table to get the data for a particular sensor. Device serial number serves as the primary partition key. Let’s obtain all data from an IoT button with a given S/N. First, we need to create a DocumentClient object:

var docClient = new AWS.DynamoDB.DocumentClient();

We need to pass query parameters to the query() method of this object. This includes the  deviceS/N:

var device_sn = 'G030JF05XXXXXXXX';
var params = {
    TableName : table_name,
    KeyConditionExpression: "#device = :dev_sn",
    ExpressionAttributeNames:{
       "#device": "device_id"
    },
    ExpressionAttributeValues: {
       ":dev_sn":device_sn
    }
 };

Finally, we can get the data from the table:

docClient.query(params, function(err, data) {
   if (err) {
      //output error message if query failed
      $('#query').append('<p> Unable to query. Error: </br>'+JSON.stringify(err, null, 2));
   } 
   else {
      $('#query').append(JSON.stringify(data.Items));
      //data.Items now contains the array with returned item
    } 
 });

After the successful query, returned data.Items  contains all returned items. The variable structured as an array of JSON object, each corresponding to a row in the DynamoDB table:

data.Items = [Object, Object,... Object]
Object = {"data":
             {
                "serialNumber":"G030JF05XXXXXXXX",
                "clickType":"SINGLE",
                "batteryVoltage":"1704mV"
             },
             "timestamp":"1472127527955",
             "device_id":"G030JF058432MLDB"
          }

The only thing left is to represent the data in a convenient way.

Process and visualize the data

There are many possible ways to process and visualize the data. For example, one can parse it and represent as HTML table. The voltage and click type can be represented as time-series using one of many available charting libraries.

I wrote two classes for data processing. The first one process data as a HTML table and then adds more functionality to it using dynatable jQeuery plug-in. The second one plots clicks and voltage vs time using plot.ly JavaScript.

To take advantage of these, add the following scripts:

<script src="http://cdnjs.cloudflare.com/ajax/libs/numeral.js/1.4.5/numeral.min.js"></script>
<script src="http://momentjs.com/downloads/moment.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/Dynatable/0.3.1/jquery.dynatable.min.js" type="text/javascript"></script>
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>

<script src="plotly_scatter.js" type="text/javascript"></script>
<script src="tableprocessor.js" type="text/javascript"></script>

Scripts table_processor.js and plotly_scatter.js  require moment.js and numeral.js to propely process the time-stamp data.

Pass the acquired data into the new FH.dataprocess object together with a jQuery object for the output HTML div, and the script takes care of tables and plotting:

new FH.dataprocess(query_data, $('#processed-data'));

To see the details of the data processing scripts, download the repository on GitHub. If your AWS IoT and DynamoDB endpoints are set-up correctly, this sample code will produce IoT button voltage and click time-series plots like the one below.

Visualizing AWS IoT Button data

I recently wrote two introductory posts about configuring the Amazon IoT button and using AWS SNS to receive SMS and email notifications on click.
To log the clicks and battery voltage over time, the payload sent with every click can be time-stamped and stored in a database. A script can be used then to query the database, retrieves the data and plot the time series of this data (using one of many d3js libraries, for example, plot.ly). (Will write about the script later in this post)
Some posts about the Dash button speculated that the battery would last for over 1000 clicks. So far, after >150 clicks the battery voltage shows no signs of decreasing.


At a starting point, we have a fully configured Amazon IoT button with uploaded certificates. We tested that it works and can send SMS notifications on click. We know that the button publishes the following message (payload) in the ‘iotbuttons’ topic of the IoT message broker:

{
  "serialNumber": "G030JF058432MLDB",
  "batteryVoltage": "1739mV",
  "clickType": "LONG"
}

We need to time-stamp this data and store it in a database. Let’s use Amazon’s NoSQL DynamoDB database to store and query the time-series data.


Create new DynamoDB table to store data

Go to the DynamoDB dashboard and click Create table button. The table requires a name and a primary key. Let’s name the table iot-buttons (we might need to log more than one button for a potential application). While the name choice is rather arbitrary, the partition key is important – elements with different partition keys should have uniform access frequency (more about partition keys). In our case, the serial number is a good partition key. Additionally, we can use a timestamp as a sort key to search within a partition. Both the serial number and the timestamp are strings. After specifying the keys, we can create a table using default remaining settings.

iot-buttons
device_id (String)
timestamp (String)

Create IoT rule

With the table created and keys specified, we now can create a rule in the AWS IoT console to write the data into DynamoDB on click. Clicking on the Create resource and choosing the Create a rule  opens a dialog window for the new rule. Name and Description fields are again arbitrary. The SQL query should capture all messages sent by (multiple) buttons to our MQTT broker without additional conditions. This is achieved by:

SELECT * FROM 'iotbutton/+'

We need to select the DynamoDB Action from the drop-down menu to insert  messages into a database. This selection extends the dialog by several fields. First, choose a table name  from the drop-down list (there is also an option to create new table, but we have taken care of this already). We decided to use the serial number as a hash key. It is transmitted in the “serialNumber” field of the payload, so we will specify ${serialNumber} as a hash key value. To specify time-stamp as a range key value,  type ${timestamp()} in the appropriate field. Finally, the payload field can be just data, meaning that the entire payload will be stored in the database.

The last step is to specify the role. Luckily, we can click on Create new role and simply specify role name. IoT interface will take care of the rest and create a new IAM role which allows writing data into a particular database.

Click Add action to add this DynamoDB action to the new rule (a rule can have several actions) and Create rule to complete the process. Pressing AWS IoT button now should create a new record in the DynamoDB table with the following content:

dataMap{3}
         batteryVoltage String: 1760mV
         clickType String: LONG
         serialNumber String: G030JF05XXXXXXXX
device_id String: G030JF05XXXXXXXX
timestamp String:1474212345678

Every click is now recorded into the database. Let’s now look at the ways to retrieve the data and visualize voltage and clicks as a function of  time.


To be continued…

Mapping the NSF funding data

NSF Funding levels:  funds obligated and estimated max per year (1996 - 2015)The National Science Foundation funds ~ 1/4 of all federally supported basic research, most of it in physical science and math. As a PhD student in physics,  I am supported in part by the NSF grants awarded to my PI and Brandeis University MRSEC. How does current level of  basic research funding compares to historical data? All the funding data by the government agencies like NSF is publicly available, but not easy to digest and analyze. I created a simple web-page that allows one to search NSF funding by PI,  institution or state. It also makes visualizing funding data on the state or city level very easy with interactive maps.

I looked up all NSF funding from 1996 through 2016 and sorted it by city and state. Looks like the amount of NSF funding peaked in 2009 and has been decreasing recently. Also there are two visible spikes in funding levels, in 2002 and 2010. I mapped the funding levels by city using plot.ly interactive maps. Here is one such map for 2015. You can find the rest of them on my plot.ly page.

2015 NSF and NASA Funding ($ Millions) - US Cities
2015 NSF and NASA Funding ($ Millions) – US Cities