Sunday, 15 July 2018

Hybrid ESP8266+UNO Energy Measurement

To complement my home automation system I needed to add a multi-channel power measurement system to my DB board. I figured four channels is a good place to start: total incoming (or one day outgoing) power, house power, geyser (hot water heater) power and cottage power.

I tried the Espurna analog energy meter and it's great, the only issue is it's one channel. While it's not too hard to add some analog current transformer multiplexing, I decided it would be easier to use an Arduino Nano for measurement and an ESP8266 (ESP-01) for transmitting the measurements via MQTT. This was version 1.0. Unfortunately, with the installation in a nice IP65 box mounted on the wall next to the DB board, it wasn't so easy to update the firmware with gradual improvements...

I discovered this hybrid board at Banggood recently and it's great! It has an Arduino Uno form factor with the usual Atmega328p microcontroller which is good for stability and robustness. It then couples a ESP8266 via the serial ports and also breaks out the ESP GPIOs just in case. Perfect! I have so many plans for these boards :) I am concerned about the stability and robustness of ESP8266s for long-term installations such as home automation systems and so I think that devices using this are the way forward given that the Atmega328 is well proven.


The on-board USB / ESP / Atmega serials can be configured with DIP switches:



I used clip-on current transformers (30 A ones for the various circuits and a 100 A one for the incomer) and a normal linear transformer for the voltage measurement. Much to my surprise, the mains voltage at my house fluctuates between around 208 V and 230 V! It's important to measure this as this is a very large error on the "ideal" 220 V.

I built four of the standard Arduino Emonlib (Open Energy Monitor) circuits for each of the channels as well as a voltage channel. My 50% DC bias voltage was common - no need to build four of these too! Unfortunalty, I don't have a circuit diagram of the final design. I used 1% resistors everywhere and I also used 3.3 V zeners for spike protection as used in the standard designs. A photo of the (almost) final product is below. I simply soldered all the bits to the plugs and wired the plugs to arduino pins.


The wiring isn't neat yet but it's a work in progress!

I found that while it is indeed possible to power the board from the 6 V 500 mA linear transformer that is used for voltage measurement, the start up time is very slow and the power on reset didn't always trigger. Instead, I wired up a 5 V switch mode power supply.

The firmware is separated into two parts: the ESP and the Atmega. The Atmega simply runs a fleshed out Emonlib example (for four channels) and formats the measurements into JSON and then dumps that long string to the serial port every five seconds. For the ESP I had initially written custom firmware which simply packaged the received JSON and sent it out via MQTT but for stability and flexibility I now use Sonoff-Tasmota which supports transmitting arbitrary text from the serial port. I suppose at a later stage I could also parse the text in Tasmota and automate some things without a central server...

On my server I run Node-Red which parses the received MQTT JSON message and splits it into separate MQTT messages with more meaningful topics. The server also initially calculated the total energy per channel in kWh but I am not sure about the accuracy due to missed messages, etc. and so I now do that calculation on the Atmega which is about as good as it gets! I'll feed back the comparison soon...

The code for the Atmega is below (it's a bit rushed...). 

#define V_CALIBRATION 991.6   //1017.35//1065.79
#define I_CALIBRATION 100
#define I_OFFSET -0.06
#define half_wavelengths 100
#define mtimeout 2000

#include "XEmonLib.h"             // Include Emon Library

EnergyMonitor ct1, ct2, ct3, ct4;
float kWh1, kWh2, kWh3, kWh4;           //energy since last restart

int led = LOW;
String json;

long deltaT = 0;
long timeNow = 0;
long lastTX = 0;
long txPeriod = 5000;

void setup()
{
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);

  Serial.begin(115200);

  pinMode(A1, INPUT);
  pinMode(A2, INPUT);
  pinMode(A3, INPUT);
  pinMode(A4, INPUT);
  pinMode(A5, INPUT);

  ct1.current(2, I_CALIBRATION, I_OFFSET);      // Current: input pin, calibration.
  ct2.current(3, I_CALIBRATION, I_OFFSET);
  ct3.current(4, I_CALIBRATION, I_OFFSET);
  ct4.current(5, I_CALIBRATION, I_OFFSET);

  ct1.voltage(1, V_CALIBRATION, 0.79);           // Voltage: input pin, calibration, phase_shift
  ct2.voltage(1, V_CALIBRATION, 0.79);           // Tweak phase_shift until power factor ~ 1 on
  ct3.voltage(1, V_CALIBRATION, 0.79);           // a resistive load such as a hot water geyser.
  ct4.voltage(1, V_CALIBRATION, 0.79);

  //Do some fake readings to force everything to settle otherwise first reading is very wrong
  analogReference(INTERNAL); //internal 1.1V ref
  float temp;
  for (int i = 0; i < 100; ++i) {
    temp += analogRead(A1);
  }
  json = String(temp);

  delay(1000); //wait for power to settle

  digitalWrite(LED_BUILTIN, LOW);

  timeNow = millis();
}

void loop()
{
  led = !led;
  digitalWrite(LED_BUILTIN, led);

  ct1.calcVI(half_wavelengths, mtimeout);
  ct2.calcVI(half_wavelengths, mtimeout);
  ct3.calcVI(half_wavelengths, mtimeout);
  ct4.calcVI(half_wavelengths, mtimeout);

  deltaT = millis() - timeNow;
  timeNow = millis();

  // Calculate energy from last measurements:
  float dT = (float)deltaT / (60.0 * 60.0 * 1000.0 * 1000.0);
  kWh1 = kWh1 + ct1.realPower * dT;
  kWh2 = kWh2 + ct2.realPower * dT;
  kWh3 = kWh3 + ct3.realPower * dT;
  kWh4 = kWh4 + ct4.realPower * dT;

  // Transmit to ESP:

  if (millis() >= (lastTX + txPeriod)) {
    //ESP buffer is 128 bytes so must keep it less than that
    json = "{\"V\":" + String(ct1.Vrms, 0)
           + ",\"I\":[" + String(ct1.Irms) + "," + String(ct2.Irms) + "," + String(ct3.Irms) + "," + String(ct4.Irms)
           + "],\"R\":[" + String(ct1.realPower, 0) + "," + String(ct2.realPower, 0) + "," + String(ct3.realPower, 0) + "," + String(ct4.realPower, 0)
           + "],\"A\":[" + String(ct1.apparentPower) + "," + String(ct2.apparentPower) + "," + String(ct3.apparentPower) + "," + String(ct4.apparentPower)
           //  + "],\"F\":[" + String(ct1.powerFactor, 1) + "," + String(ct2.powerFactor, 1) + "," + String(ct3.powerFactor, 1) + "," + String(ct4.powerFactor, 1)
           + "],\"E\":[" + String(kWh1, 2) + "," + String(kWh2, 2) + "," + String(kWh3, 2) + "," + String(kWh4, 2) + "]}" ;

    Serial.print(json);

    lastTX = millis();
  }
}

Once it's all fired up this is how it looks from the ESP's perspective:


I hope you find it useful!

Sunday, 8 July 2018

OpenWRT Mosquitto Gateway

After setting up my Armbian based Mosquitto gateway and having it run for a few months I wasn't happy with the stability. The Orange Pi Zero with read-only OverlayFS, etc. just didn't seem to work 100% and sometimes on reboot mosquitto didn't come up, etc. I tried DietPI, which is really great, but it doesn't support a read-only mode out of the box.

The solution (I hope) is OpenWRT! I couldn't find a version for my OrangePI Zero, although I suspect that any AllWinner H3 based version would work. I did have a Cubieboard2 with a nice 3D printed enclosure lying around though. I also have a laptop drive addition to the enclosure which I can use for CCTV and storage intensive tasks.

The benefit of OpenWRT (or LEDE) is that it's designed to be read only and super robust. It's no good for a router to crash after sudden power outages... I downloaded the squashfs version (I don't have time to waste on ext4 stability). This post lists the details to get it installed and running.

Installing and Customization

I opted to use the latest stable version, 17.0.1.4, for the sunxi target: https://downloads.lede-project.org/releases download the squashfs version: direct link. I suggest using the latest stable version that's available when you read this.

I had no luck using Etcher but I suspect the issue was related to the default network settings and not Ethcher (see later). Instead I dd'd the image after extracting it:

sudo dd if=./lede.img of=/dev/sdb

Make sure that you replace the input file with the image you downloaded and the output file with the SD card (check with sudo fdisk -l)

If you eject the SD now and put it in the Cubieboard it will boot but unfortuately the default IP is status and is 192.168.1.1 with login root and no password. If you have another device on your network with this IP you have three options:

1. Connect the board to a computer directly. Set the computer static IP to 192.168.1.2. Login and make the board some other static IP or use DHCP.
2. Use a 3.3V serial to USB and edit the network config at /etc/config/network and set a proper IP then reboot.
3. Mount the SD card and edit the file with your computer (do this just after dd'ing the card).

Now you should be able to access the web UI. Set up a password, etc. Over SSH or from the web UI I ran the following to get mosquitto up and running:

opkg update
opkg install nano
opkg install wget
opkg install mosquitto
opkg install mosquitto-client

You have to run the update command after a reboot as the files seem to get cleared to save flash space (clever).

In addition, I plan to use the board to save files from my CCTV cameras. Samba is pretty heavy weight and FTP is lightweight. Performance is no issue on the Cubieboard and the camera's support both so I'm installing both. I'll use samba to access the files from Windoze, etc. and FTP to write from the cameras.

opkg install vsftpd
opkg install luci-app-samba

It seems that samba comes with a nice web config with luci! Sweet! FTP needs a little bit of manual config though... I'll be using the SATA port for the hard drive but you can use USB too. See the official guide for USB.

opkg install kmod-fs-ext4
opkg install block-mount
opkg install e2fsprogs 
opkg install luci-app-hd-idle
opkg install kmod-ata-sunxi
opkg install hdparm

Before continuing, first reboot the system to ensure everything is saved and working.

Check this tutorial for sharing via samba and this for FTP. I set up guest access to my CCTV share which is a directory on the shared drive. I made this directory readable and writable by anyone suing chmod 777 ./cctv otherwise I would have to set up users.

Add this the the bottom of /etc/vsftpd.conf so that files are created with 777 permissions:

local_root=/mnt/data/cctv (or whatever your mount is)
local_umask=0000

A couple of other useful features:

1. Enable a heartbeat "flashing" LED so that you know at a glance that the system is up and running.
2. Set up a fsck on the mounts just to keep the clean (in the mount points settings).
3. Use the following mount options: rw,async,relatime,lazytime

Lets hope the mosquitto gateway is rock solid now!



Wednesday, 21 March 2018

Orange Pi MySensors Gateway

After several stability issues with my home automation controller I have decided to separate things across different devices. My latest (and soon to be superseded) setup is an Amlogic S905x Android TV Box running Armbian with Home Assistant, samba, Node-Red, mosquitto and Pi-Hole. I've had the device crash a few times due to random kernel Out Of Memory (OOM) issues, which required a power cycle to fix, eventually resulting in a corrupt SD card... :( The solution: a dedicated Orange Pi Zero (with overlayfs) for core functions and the TV box for Home Assistant and Node-Red.

I have been using a Arduino Nano based MySensors gateway which also presented some stability issues. After power up the /dev/ttyUSB0 device didn't always come up automatically. Soldering the TEST pin on the FT232 seemed to do the trick, but it's an annoying thing when half the light switches around my house stop working and then I lose Wife Acceptance Factor. This post briefly explains the steps required to install it on an Orange Pi Zero.

I tried to use the mainline kernel, and even after enabling SPI, etc. it just didn't want to talk. With the legacy version I simply installed the overlayroot package to make everything read-only when I was done setting up for maximum robustness. I installed the watchdog package too.
The rest of the tutorial is the same. For convenience this is my configure line and the commands:

git clone https://github.com/mysensors/MySensors.git cd MySensors

./configure --spi-spidev-device=/dev/spidev1.0 --my-transport=nrf24 --my-rf24-ce-pin=2 --my-rf24-cs-pin=13 --my-rf24-irq-pin=10 --my-gateway=mqtt --my-controller-ip-address=127.0.0.1 --my-mqtt-user=*** --my-mqtt-password=*** --my-mqtt-publish-topic-prefix=mysensors-out --my-mqtt-subscribe-topic-prefix=mysensors-in --my-mqtt-client-id=mygateway1 --soc=H3

make
sudo make install
sudo mysgw 
sudo systemctl start mysgw
sudo systemctl enable mysgw

Now that it's all installed and "working" software-wise, set up the hardware according to the tutorial. You can see a photo of mine below. I have opted to solder everything for long-term reliability.


I also installed Node-Red, which works perfectly on overlayfs (just not deploying new flows that persist across reboots). I use the Node-Red on this device to simply convert the MySensors messages which use their serial protocol into something more applicable for my light switches, etc. It's easy enough to update every now and then when I install new things. Here's the gist of it: https://gist.github.com/uberflyx/b456fc90ca1435b17c47ae9e59c9f4a9


Hybrid ESP8266+UNO Energy Measurement

To complement my home automation system I needed to add a multi-channel power measurement system to my DB board. I figured four channels is ...