IoT Device and Application Use Case

Use Cases
,
1

In this scenario, we are creating a device based on the ESP8266. The device stores some of its data on the cloud. The end-user’s application is developed with Flutter and needs to access the device’s data and also read and modify the state of the device.

Before switching to Upwind Cloud, the device was using Firebase and the Realtime Database to store the data. A few major issues were resolved by switching to the new cloud hosting:

  • Firebase requires an SSL connection to its servers. Implementing SSL connections on an ESP8266 device that is limited to 80K of RAM creates a number of issues and frequent crashes. The alternative is to use HTTP and encrypt sensitive data on the server and the device.
  • Some of the processing that was done on the device and the mobile app could be moved to the web services component, improving, performance, reliability and compatibility between the device and the mobile application.
  • The number of devices and how much these devices could access the database could not be determined in advance, making it impossible to predict the cost of using Firebase and its associated database.
  • Minimal number of changes were required on both the device firmware and mobile application in order to follow very strict time constraints.

Commented extracts from the device’s firmware code (C++):


//
// Including a slightly modified version of FirebaseArduino
// The modified version replaces WifiClientSecure by WifiClient to overcome memory limitation issues
//
#include <FirebaseArduino.h>

//
// The Arduino crypto library is used to encrypt sensitive data before sending it to the server
//
#include <Crypto.h>
#include "FirebaseInterface.h"

//
// Initialize the Firbase library to connect to our new URL:Port address
// The _deviceid parameter uniquely identifies each device on the server
//
void FirebaseInterface::Setup(const char* url,
							const int port, 
							const char *_deviceid, 
							const char *_password)
{
	if (initialized)
		return;
	initialized = true;

	// initialize the Firebase library
	firebase = new FirebaseArduino();
	firebase->begin(url, port, AUTH_KEY);

	// store the device Id for later use
	strncpy(deviceId, _deviceid, sizeof(deviceId) - 1);

	// start monitoring changes to the On/Off status of the device in the database
	// Firebase uses Server Sent Events (SSEs) to communicate back to the device
	{
		char buf[64];
		snprintf_P(buf, sizeof(buf), PSTR("/%s/OnOff/on"), deviceId);
		firebase->stream(buf);
	}

	// set the device's password in the database after encrpyting it
	SetPassword(_password);
}

//
// Call the web services application on the server to set the OnOff state of the device
//
void FirebaseInterface::SetRelay(bool onOff)
{
	char buf[64];

	if (!initialized) return;

	snprintf_P(buf, sizeof(buf), PSTR("/%s/OnOff/on"), deviceId);
	firebase->setBool(buf, onOff);
	if (firebase->failed())
	{
		log_message(true, PSTR("SetRelay failed\n"));
	}
}

//
// Check if an event is available from web services and set the status of the device accordingly
//
bool FirebaseInterface::PollEvent(bool &newValue)
{
	if (!initialized)
	{
		return false;
	}

	if (firebase->available())
	{
		FirebaseObject event = firebase->readEvent();
		if (event.getString("type") == "put")
		{
			newValue = event.getBool("data");
			return true;
		}
	}

	return false;
}

//
// Send an encrypted version of the device's password
// The mobile application can only change the status of the device if the passwords match
//
void FirebaseInterface::SetPassword(const char* _password)
{
	char buf[64];

	if (!initialized) return;
	
	// encrypt the password using the device ID and our secret keyword
	experimental::crypto::SHA256 hasher;
	byte hash[experimental::crypto::SHA256::NATURAL_LENGTH];
	String input = String(_password) + String(deviceId) + SECRET;
	
	// set the password in the database
	snprintf_P(buf, sizeof(buf), PSTR("/%s/Password"), deviceId);
	firebase->setString(buf, hasher.hash(input).c_str(), true);
	if (firebase->failed())
	{
		log_message(true, PSTR("SetPassword failed\n"));
	}
}

Commented extracts from the mobile application’s source-code (Dart):

import 'dart:async';
import 'dart:io';
import 'dart:typed_data';
import 'package:flutter/material.dart';

/* Remove all firebase related imports
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_auth/firebase_auth.dart';
import 'package:firebase_database/firebase_database.dart';
*/
// replace previous imports with Upwind Cloud API
import 'package:upwindcloud/upwindcloud.dart';

//
// Set the device's On/Off state by querying the OnOff/on URL
// No code changes needed when switching from Firebase to Upwind Cloud
//
Future<void> setDeviceState(DeviceIdentifier identifier) async {
	await getFirebaseRTDB();
	if (rtdb != null) {
	  if (await checkDevicePassword(identifier.chipID, identifier.password)) {
		await rtdb!
			.ref("${identifier.chipID}/OnOff/on")
			.set(identifier.state == "on" ? true : false);
	  }
	}
}

//
// Get the device's On/Off state by querying the OnOff/on URL
// No code changes needed when switching from Firebase to Upwind Cloud
// returns true if the device state is modified, false otherwise
//
Future<bool> getDeviceState(DeviceIdentifier identifier) async {
	bool modified = false;
	await getFirebaseRTDB();
	if (rtdb != null) {
	  final event =
		  await rtdb!.ref("${identifier.chipID}/OnOff/on").get();
	  if (event.exists) {
		String deviceState = ((event.value as bool) ? "on" : "off");
		modified = (deviceState != identifier.state);
		identifier.state = deviceState;
	  }
	}
	return modified;
}

//
// Initialize the database parameters on the first call
// On further calls, simply return the database instance
//
static Future<void> getFirebaseRTDB() async {
	if (rtdb == null) {
	  rtdb = FirebaseDatabase.instanceFor(
		  databaseURL: UpwindCloudDatabaseURL);
	}
}

Commented extracts from the Web Services application’s source-code (C#):

using Microsoft.AspNetCore.Http.HttpResults;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using Npgsql;
using System.Diagnostics;
using System.Net;
using System.Net.WebSockets;
using System.Runtime.CompilerServices;
using System.Security.Cryptography.X509Certificates;
using System.Text.Json;
using WmsWebServices;

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.UseKestrel(serverOptions =>
{
    // listening to non-SSL port is for compatibility with existing devices
    serverOptions.Listen(IPAddress.Any, 8080);
	// listening to SSL port
	serverOptions.Listen(IPAddress.Any, 8089,
        listenOptions =>
        {
            listenOptions.UseHttps(new X509Certificate2(@"cert/certificate.pfx", "****"));
        }
    );

});

// in production, the connection string can be provided as an environment variable, in development it is provided by the WMS connection string
string? connectionString = Environment.GetEnvironmentVariable("WMS_CONNECTION_STRING");
if (connectionString == null)
{
    connectionString = builder.Configuration.GetConnectionString("WMS");
}

// use PostgreSQL as database provider
builder.Services.AddDbContext<DevicesDb>(opt => 
        opt.UseNpgsql(connectionString)
        );

// Create another DataSource to receive notifications from the Database
builder.Services.AddSingleton<NpgsqlDataSource>((sp) =>
{
    var dataSourceBuilder = new NpgsqlDataSourceBuilder(connectionString);
    return dataSourceBuilder.Build();
});

// Database Notification Handler
builder.Services.AddSingleton<IPostgresNotificationHandler>(DevicesDb.notificationHandler);

// Add the Background Service processing the Notifications
builder.Services.Configure<PostgresNotificationServiceOptions>(o => o.ChannelName = "datachange");
builder.Services.AddHostedService<PostgresNotificationService>();

var app = builder.Build();

app.MapGet("/{id}/OnOff/on.json", GetDeviceState);
app.MapPut("/{id}/OnOff/on.json", SetDeviceState);

app.MapGet("/{**catchAll}", CatchAllUnhandled);

app.Run();

/// <summary>
/// Logs any unhandled request path for diagnostics.
/// </summary>
/// <param name="request">The HTTP request.</param>
/// <param name="catchAll">The unmatched route path.</param>
/// <returns>A BadRequest result.</returns>
static IResult CatchAllUnhandled(HttpRequest request, string catchAll)
{
    Trace.WriteLine($"*** Received unhandled {request.Method} {request.Path}");
    return TypedResults.BadRequest();
}

/// <summary>
/// Sets the on/off state of a device.
/// </summary>
/// <param name="request">The HTTP request.</param>
/// <param name="id">The device ID.</param>
/// <param name="db">The database context.</param>
/// <returns>Ok if successful, otherwise NotFound.</returns>
static async Task<IResult> SetDeviceState(HttpRequest request,
                    [FromRoute] string id,
                    DevicesDb db)
{
	// read the body as string
	// Note: the body is expected to be a simple string "true" or "false", without JSON formatting, to be compatible with existing devices.
	request.EnableBuffering();
	request.Body.Position = 0;
	var str = await new StreamReader(request.Body).ReadToEndAsync().Trim('\"');
	
    Device[] devices = await db.Devices.Where(t => t.DeviceId == id)
        .Include(device => device.Settings)
        .ToArrayAsync();

    try
    {
        if (devices.Length > 0)
        {
            Device device = devices[0];
            device.Value = new OnOff();
            device.Value.on = (str.ToLower() == "true");
            await db.SaveChangesAsync();
            return TypedResults.Ok();
        }
    }
    catch (Exception ex)
    {       
    }

    return TypedResults.NotFound();
}

/// <summary>
/// Retrieves the on/off state of a device. Supports both SSE streaming and single value responses.
/// SSE streaming is compatible with Firebase client libraries (no chunking).
/// </summary>
/// <param name="id">The device ID.</param>
/// <param name="Accept">The Accept header value.</param>
/// <param name="response">The HTTP response.</param>
/// <param name="db">The database context.</param>
/// <returns>NotFound if device doesn't exist, otherwise streams or returns the device state.</returns>
static async Task<IResult> GetDeviceState([FromRoute] string id,
                    [FromHeader] string? Accept,
                    HttpResponse response,
                    DevicesDb db)
{
    if (Accept is not null && Accept == "text/event-stream")
    {
        //
        // case of SSE streaming
        //
        PostgresNotificationListener listener = new PostgresNotificationListener();
        DevicesDb.notificationHandler.AddListener(listener);

        response.ContentType = "text/event-stream";

        // This call has the effect of sending the response without chunking.
        // This is to be compatible with the Firebase client libraries
        response.Headers.TransferEncoding = "";

        while (true)
        {
            Device[] devices = await db.Devices.AsNoTracking().Where(t => t.DeviceId == id)
                .ToArrayAsync();
            Device? device = null;

            if (devices.Length == 0)
            {
                return TypedResults.NotFound();
            }
            else
            {
                device = devices[0];
                SSEResponse resp = new SSEResponse { path = "/", data = (device.Value?.on) };
                // return the initial state
                var json = JsonSerializer.Serialize(resp);
                await response.WriteAsync($"event: put\ndata: {json}\n\n");
                await response.Body.FlushAsync();
            }

            // monitor state changes
            listener.ewh.WaitOne();
        }
    }
    else
    {
        // No streaming, simply return the value as true or false
        Device[] devices = await db.Devices.AsNoTracking().Where(t => t.DeviceId == id)
            .ToArrayAsync();
        Device? device = null;

        if (devices.Length == 0)
        {
            return TypedResults.NotFound();
        }
        else
        {
            device = devices[0];
            return TypedResults.Ok(device.Value is null ? false : device.Value.on);
        }
    }
}

Reading and writing the device state from/to the database is straightforward. The ESP device and the web application send REST requests to the server application which does all the accesses to the database. A GET request for example is routed to GetDeviceState which checks the device ID and password, reads the OnOff attribute of the device and returns it to the caller.

A bit more work is needed for the device to automatically respond to changes to the state in the database. E.g. a user turns on the device from the mobile application and needs the device to respond rather quickly to the On command. The device could continously poll the server application for the OnOff state, but a more efficient solution is to use HTTP streaming and Server-Sent Events to respond more quickly to status change. This requires the device to keep a connection open to the server which is done through the call to firebase->stream. In the case of streaming, the HTTP request contains the header Accept: “text/event-stream”. When this header is received, the method GetDeviceState starts a loop where it waits for events from the database and notifies the connected device about changes to the OnOff state through the call to response.WriteAsync.

Note that monitoring for changes to a specific table and field of the Postgres database from C# requires changes to the database and the server application. This will be discussed in details in a separate post.