Visualizing AWS IoT Button data (part II)

The IoT button is connected to the AWS cloud and every click is recorded in the DynamoDB. It’s now time to query the database and present the data, preferably in a visual form. Here, we will do everything in the browser, building our app using JavaScript AWS SDK. Thus, the first step is to add the script to the HTML page:

<script src="https://sdk.amazonaws.com/js/aws-sdk-2.5.4.min.js">
</script>

Configure the SDK

Create and configure user credentials

We would like to send requests to the DynamoDB, but the SDK has to be configured before we can do that. The two required pieces of information are the region and credentials. There are several ways to provide credentials securely:

  1. Using Amazon Cognito to authenticate users
  2. Using web identity federation to authenticate users
  3. Hard-coded in your application

We are going to pick the easiest one and hard-code the credentials into the application. This is also the least desirable method from the security standpoint, used here for the sake of convenience and speed. To do so, open the Identity and Access Management (IAM) console, click on Users menu and Create User. Immediately after the user is created, you can copy (or download) the security credentials and hard-code them into the application:

var user_credentials = {
        accessKeyId: 'AKIAIV6CXXXXXXXXXXXX', 
        secretAccessKey: 'i8+vtmKkXXXXXXXXXXXXX'
};

Allow read-only access to DynamoDB

This newly created user still does not have any service permissions and thus can’t perform any actions. We need to attach a specific policy to allow DynamoDB access. Click on the Users name in the IAM, select the user, open Permissions tab and click Attach policy. Type AmazonDynamoDBReadOnlyAccess, select it and click Attach policy. We are now officially allowed to query the DynamoDB. Permissions are set as read only to mitigate the security risk due to hard-coding the credentials.

Let’s finish configuring the JavaScript SDK by executing a command to update user credentials:

AWS.config.region = 'us-east-1';
AWS.config.update(user_credentials);

List DynamoDB tables

To test the newly created user credentials and verify the read-only access to Dynamo, lest first list all Dynamo tables. This is done by calling the listTables() method of the DynamoDB object. Let’s also put all tables into the drop-down menu for easy selection.

var dynamodb = new AWS.DynamoDB();
dynamodb.listTables(function(err, data) {
 var html_str;
 if (err == null){
   html_str = ' DynamoDB tables: <select>';
   $.each(data.TableNames, function(index, value){
     html_str += '<option value="' + value + '">' + index+1 + '. '+ value + '</option>';
   });
   html_str += '</select>';
  }
  else{
   console.log(err);
   html_str = err;
  }
  $('#table-list').html(html_str); 
 });

If the credentials are invalid, we will see an error message. If the credentials were crated properly, we can select the table iot-buttons from the menu (that’s the name we decide on in the first part).

Display table info

Let’s now obtain table info by calling describeTable() method and passing the table name as parameter. For example, let’s get the total number of records in a table:

var params = {
   TableName: $(this).val()
};
dynamodb.describeTable(params, function(err, data) {
   if (err) {
      //an error occurred 
       $('#table-list').append('<p>' + JSON.stringify(err) + '</p>');
    }
    else {
       //successful response
       var table = data.Table;
       $('#table-list').append('<p> '+table.TableName + ": # of items= "+table.ItemCount + '</p>');
    }
 });

Query the table

Finally, let’s query the table to get the data for a particular sensor. Device serial number serves as the primary partition key. Let’s obtain all data from an IoT button with a given S/N. First, we need to create a DocumentClient object:

var docClient = new AWS.DynamoDB.DocumentClient();

We need to pass query parameters to the query() method of this object. This includes the  deviceS/N:

var device_sn = 'G030JF05XXXXXXXX';
var params = {
    TableName : table_name,
    KeyConditionExpression: "#device = :dev_sn",
    ExpressionAttributeNames:{
       "#device": "device_id"
    },
    ExpressionAttributeValues: {
       ":dev_sn":device_sn
    }
 };

Finally, we can get the data from the table:

docClient.query(params, function(err, data) {
   if (err) {
      //output error message if query failed
      $('#query').append('<p> Unable to query. Error: </br>'+JSON.stringify(err, null, 2));
   } 
   else {
      $('#query').append(JSON.stringify(data.Items));
      //data.Items now contains the array with returned item
    } 
 });

After the successful query, returned data.Items  contains all returned items. The variable structured as an array of JSON object, each corresponding to a row in the DynamoDB table:

data.Items = [Object, Object,... Object]
Object = {"data":
             {
                "serialNumber":"G030JF05XXXXXXXX",
                "clickType":"SINGLE",
                "batteryVoltage":"1704mV"
             },
             "timestamp":"1472127527955",
             "device_id":"G030JF058432MLDB"
          }

The only thing left is to represent the data in a convenient way.

Process and visualize the data

There are many possible ways to process and visualize the data. For example, one can parse it and represent as HTML table. The voltage and click type can be represented as time-series using one of many available charting libraries.

I wrote two classes for data processing. The first one process data as a HTML table and then adds more functionality to it using dynatable jQeuery plug-in. The second one plots clicks and voltage vs time using plot.ly JavaScript.

To take advantage of these, add the following scripts:

<script src="http://cdnjs.cloudflare.com/ajax/libs/numeral.js/1.4.5/numeral.min.js"></script>
<script src="http://momentjs.com/downloads/moment.js" type="text/javascript"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/Dynatable/0.3.1/jquery.dynatable.min.js" type="text/javascript"></script>
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>

<script src="plotly_scatter.js" type="text/javascript"></script>
<script src="tableprocessor.js" type="text/javascript"></script>

Scripts table_processor.js and plotly_scatter.js  require moment.js and numeral.js to propely process the time-stamp data.

Pass the acquired data into the new FH.dataprocess object together with a jQuery object for the output HTML div, and the script takes care of tables and plotting:

new FH.dataprocess(query_data, $('#processed-data'));

To see the details of the data processing scripts, download the repository on GitHub. If your AWS IoT and DynamoDB endpoints are set-up correctly, this sample code will produce IoT button voltage and click time-series plots like the one below.

Visualizing AWS IoT Button data

I recently wrote two introductory posts about configuring the Amazon IoT button and using AWS SNS to receive SMS and email notifications on click.
To log the clicks and battery voltage over time, the payload sent with every click can be time-stamped and stored in a database. A script can be used then to query the database, retrieves the data and plot the time series of this data (using one of many d3js libraries, for example, plot.ly). (Will write about the script later in this post)
Some posts about the Dash button speculated that the battery would last for over 1000 clicks. So far, after >150 clicks the battery voltage shows no signs of decreasing.


At a starting point, we have a fully configured Amazon IoT button with uploaded certificates. We tested that it works and can send SMS notifications on click. We know that the button publishes the following message (payload) in the ‘iotbuttons’ topic of the IoT message broker:

{
  "serialNumber": "G030JF058432MLDB",
  "batteryVoltage": "1739mV",
  "clickType": "LONG"
}

We need to time-stamp this data and store it in a database. Let’s use Amazon’s NoSQL DynamoDB database to store and query the time-series data.


Create new DynamoDB table to store data

Go to the DynamoDB dashboard and click Create table button. The table requires a name and a primary key. Let’s name the table iot-buttons (we might need to log more than one button for a potential application). While the name choice is rather arbitrary, the partition key is important – elements with different partition keys should have uniform access frequency (more about partition keys). In our case, the serial number is a good partition key. Additionally, we can use a timestamp as a sort key to search within a partition. Both the serial number and the timestamp are strings. After specifying the keys, we can create a table using default remaining settings.

iot-buttons
device_id (String)
timestamp (String)

Create IoT rule

With the table created and keys specified, we now can create a rule in the AWS IoT console to write the data into DynamoDB on click. Clicking on the Create resource and choosing the Create a rule  opens a dialog window for the new rule. Name and Description fields are again arbitrary. The SQL query should capture all messages sent by (multiple) buttons to our MQTT broker without additional conditions. This is achieved by:

SELECT * FROM 'iotbutton/+'

We need to select the DynamoDB Action from the drop-down menu to insert  messages into a database. This selection extends the dialog by several fields. First, choose a table name  from the drop-down list (there is also an option to create new table, but we have taken care of this already). We decided to use the serial number as a hash key. It is transmitted in the “serialNumber” field of the payload, so we will specify ${serialNumber} as a hash key value. To specify time-stamp as a range key value,  type ${timestamp()} in the appropriate field. Finally, the payload field can be just data, meaning that the entire payload will be stored in the database.

The last step is to specify the role. Luckily, we can click on Create new role and simply specify role name. IoT interface will take care of the rest and create a new IAM role which allows writing data into a particular database.

Click Add action to add this DynamoDB action to the new rule (a rule can have several actions) and Create rule to complete the process. Pressing AWS IoT button now should create a new record in the DynamoDB table with the following content:

dataMap{3}
         batteryVoltage String: 1760mV
         clickType String: LONG
         serialNumber String: G030JF05XXXXXXXX
device_id String: G030JF05XXXXXXXX
timestamp String:1474212345678

Every click is now recorded into the database. Let’s now look at the ways to retrieve the data and visualize voltage and clicks as a function of  time.


To be continued…

Connecting MQTT.fx to AWS IoT

AWS IoT MQTT client
Built-in IoT MQTT client

AWS IoT functionality is based to a large extent on a publish/subscribe message broker that supports MQTT protocol. The concept is simple: a device can publish messages (for example, sensor readings) to a topic or subscribe to receive messages. AWS Developer Guide has an excellent and detailed explanation of the way message broker operates. AWS IoT main page has a built-in MQTT client that can be used in a web-browser with ease, but the session is lost if the browser window is closed.

It is possible to use one of the freely available MQTT clients to connect to the IoT endpoint.  This can be useful in development and debugging of the IoT applications. Here is how to set-up MQTT.fx  to work with AWS IoT.


1 Download and install MQTT.fx

MQTT.fx logoThe latest version of MQTT.fx client can be downloaded here. At the moment a large variety of installation packages for Windows, MacOS, Linux is  available for version 1.1.0

 


2 Create a connection profile

The nice thing about MQTT.fx is that it allows you to create multiple connection profiles. If you work with several different MQTT brokers, loading saved profiles increases convenience. Let’s first create a new profile for the AWS connection. Either go to Extras > Edit Connection Profiles in the menu or click on the gear icon to open Connection Profiles window.

There should be two default profiles, “local mosquitto” and “M2M Eclipse”. Click on the plus sign (+) all the way on the bottom to create a new profile and give it a name. The Broker Address is your AWS IoT endpoint, usually URL in the form: (iot-address).iot.(region).amazonaws.com  Here, region can be, for example,  us-east-1; and the iot-address is your personal AWS identifier. AWS IoT uses Broker Port 8883. Unique Client ID can be generated on the spot with the Generate button.

Create new connection profile
Creating new connection profile

Finally, we need is to provide the SSL/TLS certificates to establish secure connection to the message broker. Click on the SSL/TLS tab, check the Enable SSL/TLS  box, and select TLS v. 1.2 as the appropriate protocolNow, we are going to provide Self-signed certificates (generated by the AWS).  Upon selecting this option, several input fields appear, asking us to provide three files: CA, Client Certificate and Client Key.

MQTT.fx SSL options
MQTT.fx SSL options

The root CA file, which serves as proof that we are really communicating with AWS IoT server, can be downloaded here. (More about authentication can be found in the Developer Guide.) The other two files, Client Certificate and Client Key, have to be downloaded when the new certificate is created as the IoT resource.

 


3 Create new IoT certificate

Creation of certificates was covered in the previous post about the IoT button. When the new certificate is created, the IoT web interface gives you a chance to save keys and certificate files, which are PEM-encoded. Saving the public key is not necessary in this case, we really need only the private key and the certificate.

Create a new certificate
Create a new certificate
Save certificate files
Save certificate files
MQTT.fx SSL options
MQTT.fx SSL options with certificates.

Going back to the profile set-up window of the MQTT.fx, specify the newly downloaded certificate and private key in the SSL/TSL tab and click OK. The connection parameters are now set up. However, the policy certificates alone does not give a permission to the MQTT client to connect and subscribe/publish to topics in AWS IoT. If you try pressing connect now, the MQTT.fx will display “Connection lost” message. Permissions are granted through specific policies. Therefore, we need to create a policy and attach it to the certificate.


4 Create a policy for the certificate

We need to create a policy that allows the MQTT client to connect to the AWS server, receive data, subscribe and publish to topics. This is accomplished in the web-interface by creating a new policy with the following Actions:  iot:Connect, iot:Receive, iot:Publish, iot:Subscribe. We are going to use * as a Resource, which allows selected actions on any resource. Finally, we need to explicitly Allow specify that the listed actions. Once all fields in the Create a policy dialog are filled, click Add statement.

Create AWS IoT policy for MQTT
Create AWS IoT policy for MQTT

The newly crated policy needs to be attached to the certificate. Click on the certificate and choose Attach a policy in drop-down list of available actions. Enter the Policy Name and click attach. Finally, MQTT.fx set-up is complete.


5 Conncet to AWS IoT and subscribe to a topic

Click on the Connect button in the MQTT.fx now, and the connection indicator will light up green, along with the lock sign indicating secure TSL connection.

MQTT.fx connection status
MQTT.fx connection status

You now can subscribe and publish to any topic. For example, to see messages sent by the Amazon IoT button connected previously, subscribe to the “iotbutton/+”  topic. The + is a wildcard that matches one level in topic hierarchy, so we are subscribing to all iotbutton messages, even though they might be sent by multiple buttons. Pressing the IoT Dash button now results in the new message in MQTT.fx, displaying the message and previous subscription history.

MQTT.fx subscribed to the iotbutton topic
MQTT.fx subscribed to the iotbutton topic

Getting started with Amazon IoT button (Part II)

Amazon IoT button
AWS IoT button

In the first part of this Getting started with IoT button post, we configured the WiFi connection and set up certificates for secure communication with AWS cloud. The data sent with every button press looks like that:

{"serialNumber": "G030JF05XXXXXXXX","batteryVoltage": "1592mV","clickType": "SINGLE"}

Besides sending, the device has to do operations with this data like publishing it on a MQTT topic. Since all actions in AWS are tightly controlled for security purposes, the device needs a permission to perform actions. Policy is a form of such permission.


Create a policy

Creating a policy starts with clicking Create a resource and choosing Create a policy tab.

Create AWS IoT policy
Create AWS IoT policy

In order to publish to a particular topic, select iot:Publish action (it is easy to find it in the drop-down list after typing the first few letters). The content of the resource field depend on the selected action.  Since we selected Publish, we need to provide topic identifier as a resource (read more about resources and topics here). Resources follow somewhat complex naming convention:

arn:aws:iot:your-region:your-aws-account:topic/iotbutton/your-button-serial-number

This is similar to the REST API endpoint, since it contains your AWS account number (endpoint subdomain) and region (for example, us-east-1). We already used the serial number as well when setting up WiFi access. With all previously collected info, it is easy to create the correct ARN:

arn:aws:iot:us-east-1:xxxxxxxxxxxx:topic/iotbutton/G030JF05XXXXXXXX

Check the Allow box, click on Add statement  (single policy can contain multiple statements) and Create.  The policy is now visible in the list of resources.


Attach the policy and the thing to the device certificate

In order to work properly, the newly created policy has to be attached to the device certificate. Also, the certificate should be associated with the thing we created in the first part. To perform this association, select the certificate in the resource list and click on Actions menu. Attach both the policy and the thing to the certificate. The certificate is finally complete!

Attach policy & thing to the certificate
Attaching the policy & thing to the certificate
IoT policy properties
IoT policy properties after attachments

 

 

 

 

 

 

 

If you press a button on the device at this point, it will send the data and publish it to the iotbutton topic. LED indicator should turn solid green for several seconds, indication successful publish outcome. As a test, try to detach the policy – the LED will flash red. Also try detaching the thing. Surprisingly, the thing really does not make a difference – even without it the message is published.

You can subscribe to a topic and receive messages by using the MQTT client built into AWS IoT or with the standalone application like MQTT.fx (more MQTT clients here). Subscribe to the iotbutton/G030JF05XXXXXXXX topic to view messages for a specific device or iotbutton/+ if you have multiple buttons.


Create and configure a rule

AWS IoT can perform actions when a messages is published through the use of rules. Creating a rule is similar to creating any other resource. In the Create rule dialog, fill in the name and description fields first. We need to create an SQL query that will be used to monitor published messages. This is done by specifying the Attribute field and the Topic filter. Since we are interested in the IoT button topic, type in the familiar iotbutton/G030JF05XXXXXXXX. Attribute can be a specific field of the payload, such as clickType or batteryVoltage, but we can also use  for all fields. Condition is not required and can be left blank, unless we want to trigger the rule only when battery voltage falls below a certain threshold, for example.

 

Create a rule dialog
Create a rule dialog

Finally, we need to select an action from the expansive list of available actions. We are currently interested in the sending a text message or an email as the click notification, so select the SNS service.

Select action from AWS services
Select an action from the list of AWS services

We have not selected any targets for the notification. Let’s follow Create a new resource link to create a target for the AWS Push Notification Service (SNS). Targets for the rules are called topics in the SNS parlance.


Create SNS topic

Hit Create new topic button on the newly opened SNS dashboard and input name and display name of your topic.

Create SNS topic
Create SNS topic

Create the topic and click on its ARN in the list of topics. This should display topic details. As you see, the topic currently has no subscriptions – i.e. addresses or phone numbers to send notifications to. You can add a number or an email address by creating new subscription.

Creating new SNS subscription
Creating new SNS subscription

This process is straightforward, as Topic ARN field gets auto-populated, all you need to do is to choose the desired protocol (SMS, email or AWS Lambda function to name a few), enter the endpoint (cell number, email address, etc) and create the subscription. With the subscription and topic in place, lets go back to the IoT dashboard and continue with the rule creation.

Create a role
Create a role

SNS target is now the topic name. Message format field is not required and the only remaining thing to do is to specify the Role name. Click Create a new role  and specify role name – this is all. AWS will automatically add a new role into the Identity and Access Management (IAM) system and grant it a permission to push IoT notifications. With the role in place, Add action and Create  the rule. Notice that a rule can have multiple assigned actions.

Rules are independent of a particular device or certificate. They do not have to be attached to anything. AWS IoT rules engine will continuously monitor the messages and push notifications if the message matches SQL query specified in the rule.


Press the IoT button and receive text message

With all steps now complete, pressing a button sends a text message withing a few seconds. There are three distinct click types: single, double and long. In addition, the button sends voltage, which can be monitored over time. Counting clicks is another interesting project for learning AWS basics.

Final SMS notification
Final SMS notification from the IoT button

Continue reading  Amazon Web Services IoT and DynamoDB tutorials:

Storing and visualizing IoT button data

Connecting MQTT.fx to AWS IoT

Getting started with Amazon IoT button (Part I)

Dash button CLIF
Press to order protein bars…
Amazon IoT button
or to learn AWS IoT

Dash Buttons help you buy stuff on Amazon by simply pressing a button. You can place an order on anything from laundry detergent to soft drinks by pressing one of these. They are essentially free after the first placed order. Amazon recently released the IoT button,  which orders absolutely nothing  and will cost you $20. Why would anyone buy that? Well, because it is easy and fun introduction to the Amazon IoT platform.

It serves as a perfect introduction to the workflow of the AWS IoT – part of the Amazon Web Services that helps connect devices to the cloud. The Internet of Things has two essential aspects to it. The “things” part are physical devices that interact with the user (a button) or the environment (a temperature sensor). The “Internet” part is the software responsible for processing the data and acting on it (placing an order or sending a text message alert). Getting started with the IoT thus requires working both with hardware and software. Designing and building connected devices might be challenging (although extremely interesting as well). The IoT  Button takes care of the hardware part of the equation, and gives you an opportunity to learn the “Internet” side of IoT.

Here is my take on the IoT button workflow – from unpacking to receiving SMS notifications.


Unpacking and AWS set-up

Amazon IoT button
Amazon IoT button packaging

The button comes in a neatly packaged and fully charged. Press it right out-of-the-box, and you will see red flashing light, which means the button has not been set up yet. It has no record of WiFi network SSID and password. There is also no destination to send the data.

First, create the Amazon Web Services Account here. After that you should see AWS console with the multitude of provided services. All we need now is to click AWS IoT icon to go to the AWS IoT homepage.

AWS Console
AWS Console

Create a thing

AWS IoT contains the list of all resources you created, a button to create a them and a link connect AWS IoT button. I found myself confused after following the link and decided to go back. I recommend clicking on the Create a resource.

AWS IoT homepage
AWS IoT homepage – click on Create a Resource

Choose Create a thing from the list of possible options and give it a name (for example, iot-button). It will have no type and no additional attributes so far. Clicking Create will add a thing, which is a virtual representation of some arbitrary physical device. Next, we need to make sure this newly created “thing” in the cloud is associated with the IoT button we have.

IoT Create a thing dialog
IoT Create a thing dialog

Create a certificate and download private key

The communication between the “virtual” thing and the physical button happens via secure MQTT protocol. This security scheme is using a certificate in combination with the public and private keys. All of them can be created in one simple step. Click on Create a resource again and choose Create a certificate. Certificates have to be activated, you can check the Activate box to do it now (or do it as a separate step later). 1-Click certificate button will create the certificate with keys and add them to the list of resources.

Create security certificate
Create and activate the security certificate

Right after creating the certificate you will have the only chance to save public and private key files. It is extremely important to do it. Private key is essential for the IoT button – this is how the button will identify itself and secure the connection to the cloud. Save all three files in a separate folder.

Save key files
Save the certificate and key files

Record the AWS endpoint

Apart from the certificates, the IoT button needs a web-address associated with the AWS cloud. You can get this information by clicking on the icon of the newly created ‘iot-button’ thing. The properties tab contains the REST API endpoint. It typically looks like this: https://xxxxxxxxxxx.iot.us-east-1.amazonaws.com/things/iot-button/shadow  Here, xxxxxxxxxxx is unique identifier of your IoT endpoint, us-east-1 stands for AWS service region. The rest – /things/iot-button/shadow specifies a particular devices. Record the xxxxxxxxxxx and the region (ex. us-east-1) – this information will be necessary for the IoT button set-up.

Device properties with AWS endpoint
Device properties with AWS endpoint

Upload certificates and WiFi set-up

We are finally ready to work with the IoT button itself. In order to enter the set-up mode, press and hold the button for 15 seconds, until the LED turns blinking blue. The device switches into Access Point mode and creates a WiFi network with with SSID Button ConfigureMe – xxx. The password is the last eight digits of the serial number located on the back of the button. Connect to the network and type http://192.168.0.1/index.html in your browser. This opens a simple web-form where we enter the previously collected information and attach key and certificate files.

IoT button set-up
IoT button set-up

Click Configure and the button will attempt to connect to the provided network, obtain IP and send the info to the IoT endpoint while flashing white LED light.

The certificate identifies the device to the IoT cloud back-end. However, it has not been given permission to actually publish any data. At this point, the LED will be flashing red, indicating unsuccessful publish process. Even though the WiFi connection works, we need to create appropriate rules


In the next post – Getting started with Amazon IoT button (Part II) – I will describe how to set up rules to process this information and send SMS and email click notifications.

Also, check out this post about storing and visualizing IoT button data.

Mapping the NSF funding data

NSF Funding levels:  funds obligated and estimated max per year (1996 - 2015)The National Science Foundation funds ~ 1/4 of all federally supported basic research, most of it in physical science and math. As a PhD student in physics,  I am supported in part by the NSF grants awarded to my PI and Brandeis University MRSEC. How does current level of  basic research funding compares to historical data? All the funding data by the government agencies like NSF is publicly available, but not easy to digest and analyze. I created a simple web-page that allows one to search NSF funding by PI,  institution or state. It also makes visualizing funding data on the state or city level very easy with interactive maps.

I looked up all NSF funding from 1996 through 2016 and sorted it by city and state. Looks like the amount of NSF funding peaked in 2009 and has been decreasing recently. Also there are two visible spikes in funding levels, in 2002 and 2010. I mapped the funding levels by city using plot.ly interactive maps. Here is one such map for 2015. You can find the rest of them on my plot.ly page.

2015 NSF and NASA Funding ($ Millions) - US Cities
2015 NSF and NASA Funding ($ Millions) – US Cities