Sunday, 15 July 2018

Hybrid ESP8266+UNO Energy Measurement

To complement my home automation system I needed to add a multi-channel power measurement system to my DB board. I figured four channels is a good place to start: total incoming (or one day outgoing) power, house power, geyser (hot water heater) power and cottage power.

I tried the Espurna analog energy meter and it's great, the only issue is it's one channel. While it's not too hard to add some analog current transformer multiplexing, I decided it would be easier to use an Arduino Nano for measurement and an ESP8266 (ESP-01) for transmitting the measurements via MQTT. This was version 1.0. Unfortunately, with the installation in a nice IP65 box mounted on the wall next to the DB board, it wasn't so easy to update the firmware with gradual improvements...

I discovered this hybrid board at Banggood recently and it's great! It has an Arduino Uno form factor with the usual Atmega328p microcontroller which is good for stability and robustness. It then couples a ESP8266 via the serial ports and also breaks out the ESP GPIOs just in case. Perfect! I have so many plans for these boards :) I am concerned about the stability and robustness of ESP8266s for long-term installations such as home automation systems and so I think that devices using this are the way forward given that the Atmega328 is well proven.

The on-board USB / ESP / Atmega serials can be configured with DIP switches:

I used clip-on current transformers (30 A ones for the various circuits and a 100 A one for the incomer) and a normal linear transformer for the voltage measurement. Much to my surprise, the mains voltage at my house fluctuates between around 208 V and 230 V! It's important to measure this as this is a very large error on the "ideal" 220 V.

I built four of the standard Arduino Emonlib (Open Energy Monitor) circuits for each of the channels as well as a voltage channel. My 50% DC bias voltage was common - no need to build four of these too! Unfortunalty, I don't have a circuit diagram of the final design. I used 1% resistors everywhere and I also used 3.3 V zeners for spike protection as used in the standard designs. A photo of the (almost) final product is below. I simply soldered all the bits to the plugs and wired the plugs to arduino pins.

The wiring isn't neat yet but it's a work in progress!

I found that while it is indeed possible to power the board from the 6 V 500 mA linear transformer that is used for voltage measurement, the start up time is very slow and the power on reset didn't always trigger. Instead, I wired up a 5 V switch mode power supply.

The firmware is separated into two parts: the ESP and the Atmega. The Atmega simply runs a fleshed out Emonlib example (for four channels) and formats the measurements into JSON and then dumps that long string to the serial port every five seconds. For the ESP I had initially written custom firmware which simply packaged the received JSON and sent it out via MQTT but for stability and flexibility I now use Sonoff-Tasmota which supports transmitting arbitrary text from the serial port. I suppose at a later stage I could also parse the text in Tasmota and automate some things without a central server...

On my server I run Node-Red which parses the received MQTT JSON message and splits it into separate MQTT messages with more meaningful topics. The server also initially calculated the total energy per channel in kWh but I am not sure about the accuracy due to missed messages, etc. and so I now do that calculation on the Atmega which is about as good as it gets! I'll feed back the comparison soon...

The code for the Atmega is below (it's a bit rushed...). 

#define V_CALIBRATION 991.6   //1017.35//1065.79
#define I_CALIBRATION 100
#define I_OFFSET -0.06
#define half_wavelengths 100
#define mtimeout 2000

#include "XEmonLib.h"             // Include Emon Library

EnergyMonitor ct1, ct2, ct3, ct4;
float kWh1, kWh2, kWh3, kWh4;           //energy since last restart

int led = LOW;
String json;

long deltaT = 0;
long timeNow = 0;
long lastTX = 0;
long txPeriod = 5000;

void setup()
  digitalWrite(LED_BUILTIN, HIGH);


  pinMode(A1, INPUT);
  pinMode(A2, INPUT);
  pinMode(A3, INPUT);
  pinMode(A4, INPUT);
  pinMode(A5, INPUT);

  ct1.current(2, I_CALIBRATION, I_OFFSET);      // Current: input pin, calibration.
  ct2.current(3, I_CALIBRATION, I_OFFSET);
  ct3.current(4, I_CALIBRATION, I_OFFSET);
  ct4.current(5, I_CALIBRATION, I_OFFSET);

  ct1.voltage(1, V_CALIBRATION, 0.79);           // Voltage: input pin, calibration, phase_shift
  ct2.voltage(1, V_CALIBRATION, 0.79);           // Tweak phase_shift until power factor ~ 1 on
  ct3.voltage(1, V_CALIBRATION, 0.79);           // a resistive load such as a hot water geyser.
  ct4.voltage(1, V_CALIBRATION, 0.79);

  //Do some fake readings to force everything to settle otherwise first reading is very wrong
  analogReference(INTERNAL); //internal 1.1V ref
  float temp;
  for (int i = 0; i < 100; ++i) {
    temp += analogRead(A1);
  json = String(temp);

  delay(1000); //wait for power to settle

  digitalWrite(LED_BUILTIN, LOW);

  timeNow = millis();

void loop()
  led = !led;
  digitalWrite(LED_BUILTIN, led);

  ct1.calcVI(half_wavelengths, mtimeout);
  ct2.calcVI(half_wavelengths, mtimeout);
  ct3.calcVI(half_wavelengths, mtimeout);
  ct4.calcVI(half_wavelengths, mtimeout);

  deltaT = millis() - timeNow;
  timeNow = millis();

  // Calculate energy from last measurements:
  float dT = (float)deltaT / (60.0 * 60.0 * 1000.0 * 1000.0);
  kWh1 = kWh1 + ct1.realPower * dT;
  kWh2 = kWh2 + ct2.realPower * dT;
  kWh3 = kWh3 + ct3.realPower * dT;
  kWh4 = kWh4 + ct4.realPower * dT;

  // Transmit to ESP:

  if (millis() >= (lastTX + txPeriod)) {
    //ESP buffer is 128 bytes so must keep it less than that
    json = "{\"V\":" + String(ct1.Vrms, 0)
           + ",\"I\":[" + String(ct1.Irms) + "," + String(ct2.Irms) + "," + String(ct3.Irms) + "," + String(ct4.Irms)
           + "],\"R\":[" + String(ct1.realPower, 0) + "," + String(ct2.realPower, 0) + "," + String(ct3.realPower, 0) + "," + String(ct4.realPower, 0)
           + "],\"A\":[" + String(ct1.apparentPower) + "," + String(ct2.apparentPower) + "," + String(ct3.apparentPower) + "," + String(ct4.apparentPower)
           //  + "],\"F\":[" + String(ct1.powerFactor, 1) + "," + String(ct2.powerFactor, 1) + "," + String(ct3.powerFactor, 1) + "," + String(ct4.powerFactor, 1)
           + "],\"E\":[" + String(kWh1, 2) + "," + String(kWh2, 2) + "," + String(kWh3, 2) + "," + String(kWh4, 2) + "]}" ;


    lastTX = millis();

Once it's all fired up this is how it looks from the ESP's perspective:

I hope you find it useful!

Sunday, 8 July 2018

OpenWRT Mosquitto Gateway

After setting up my Armbian based Mosquitto gateway and having it run for a few months I wasn't happy with the stability. The Orange Pi Zero with read-only OverlayFS, etc. just didn't seem to work 100% and sometimes on reboot mosquitto didn't come up, etc. I tried DietPI, which is really great, but it doesn't support a read-only mode out of the box.

The solution (I hope) is OpenWRT! I couldn't find a version for my OrangePI Zero, although I suspect that any AllWinner H3 based version would work. I did have a Cubieboard2 with a nice 3D printed enclosure lying around though. I also have a laptop drive addition to the enclosure which I can use for CCTV and storage intensive tasks.

The benefit of OpenWRT (or LEDE) is that it's designed to be read only and super robust. It's no good for a router to crash after sudden power outages... I downloaded the squashfs version (I don't have time to waste on ext4 stability). This post lists the details to get it installed and running.

Installing and Customization

I opted to use the latest stable version,, for the sunxi target: download the squashfs version: direct link. I suggest using the latest stable version that's available when you read this.

I had no luck using Etcher but I suspect the issue was related to the default network settings and not Ethcher (see later). Instead I dd'd the image after extracting it:

sudo dd if=./lede.img of=/dev/sdb

Make sure that you replace the input file with the image you downloaded and the output file with the SD card (check with sudo fdisk -l)

If you eject the SD now and put it in the Cubieboard it will boot but unfortuately the default IP is status and is with login root and no password. If you have another device on your network with this IP you have three options:

1. Connect the board to a computer directly. Set the computer static IP to Login and make the board some other static IP or use DHCP.
2. Use a 3.3V serial to USB and edit the network config at /etc/config/network and set a proper IP then reboot.
3. Mount the SD card and edit the file with your computer (do this just after dd'ing the card).

Now you should be able to access the web UI. Set up a password, etc. Over SSH or from the web UI I ran the following to get mosquitto up and running:

opkg update
opkg install nano
opkg install wget
opkg install mosquitto
opkg install mosquitto-client

You have to run the update command after a reboot as the files seem to get cleared to save flash space (clever).

In addition, I plan to use the board to save files from my CCTV cameras. Samba is pretty heavy weight and FTP is lightweight. Performance is no issue on the Cubieboard and the camera's support both so I'm installing both. I'll use samba to access the files from Windoze, etc. and FTP to write from the cameras.

opkg install vsftpd
opkg install luci-app-samba

It seems that samba comes with a nice web config with luci! Sweet! FTP needs a little bit of manual config though... I'll be using the SATA port for the hard drive but you can use USB too. See the official guide for USB.

opkg install kmod-fs-ext4
opkg install block-mount
opkg install e2fsprogs 
opkg install luci-app-hd-idle
opkg install kmod-ata-sunxi
opkg install hdparm

Before continuing, first reboot the system to ensure everything is saved and working.

Check this tutorial for sharing via samba and this for FTP. I set up guest access to my CCTV share which is a directory on the shared drive. I made this directory readable and writable by anyone suing chmod 777 ./cctv otherwise I would have to set up users.

Add this the the bottom of /etc/vsftpd.conf so that files are created with 777 permissions:

local_root=/mnt/data/cctv (or whatever your mount is)

A couple of other useful features:

1. Enable a heartbeat "flashing" LED so that you know at a glance that the system is up and running.
2. Set up a fsck on the mounts just to keep the clean (in the mount points settings).
3. Use the following mount options: rw,async,relatime,lazytime

Lets hope the mosquitto gateway is rock solid now!

Wednesday, 21 March 2018

Orange Pi MySensors Gateway

After several stability issues with my home automation controller I have decided to separate things across different devices. My latest (and soon to be superseded) setup is an Amlogic S905x Android TV Box running Armbian with Home Assistant, samba, Node-Red, mosquitto and Pi-Hole. I've had the device crash a few times due to random kernel Out Of Memory (OOM) issues, which required a power cycle to fix, eventually resulting in a corrupt SD card... :( The solution: a dedicated Orange Pi Zero (with overlayfs) for core functions and the TV box for Home Assistant and Node-Red.

I have been using a Arduino Nano based MySensors gateway which also presented some stability issues. After power up the /dev/ttyUSB0 device didn't always come up automatically. Soldering the TEST pin on the FT232 seemed to do the trick, but it's an annoying thing when half the light switches around my house stop working and then I lose Wife Acceptance Factor. This post briefly explains the steps required to install it on an Orange Pi Zero.

I tried to use the mainline kernel, and even after enabling SPI, etc. it just didn't want to talk. With the legacy version I simply installed the overlayroot package to make everything read-only when I was done setting up for maximum robustness. I installed the watchdog package too.
The rest of the tutorial is the same. For convenience this is my configure line and the commands:

git clone cd MySensors

./configure --spi-spidev-device=/dev/spidev1.0 --my-transport=nrf24 --my-rf24-ce-pin=2 --my-rf24-cs-pin=13 --my-rf24-irq-pin=10 --my-gateway=mqtt --my-controller-ip-address= --my-mqtt-user=*** --my-mqtt-password=*** --my-mqtt-publish-topic-prefix=mysensors-out --my-mqtt-subscribe-topic-prefix=mysensors-in --my-mqtt-client-id=mygateway1 --soc=H3

sudo make install
sudo mysgw 
sudo systemctl start mysgw
sudo systemctl enable mysgw

Now that it's all installed and "working" software-wise, set up the hardware according to the tutorial. You can see a photo of mine below. I have opted to solder everything for long-term reliability.

I also installed Node-Red, which works perfectly on overlayfs (just not deploying new flows that persist across reboots). I use the Node-Red on this device to simply convert the MySensors messages which use their serial protocol into something more applicable for my light switches, etc. It's easy enough to update every now and then when I install new things. Here's the gist of it:

Saturday, 23 December 2017

Home Assistant + Node-RED with BTRFS on Arch

After using Home Assistant for a few months and having set it up to use external MySQL databases, etc. I have decided to re-install in a robust manner. I found that the ext4 that I was using kept going corrupt after power outages. I stupidly turned off the journal to "save" the sdcard but this was the last straw and many of my config files went missing... Time to start afresh!

This post pretty much contains a list of commands. I won't go into detail on each of them but they should be fairly self explanatory. I probably missed a bunch of stuff too but this is the gist of it!

First off, I am using Arch because I like the amount of control it gives me... I am also going to go with btrfs instead of ext4 this time around. My hardware is a Cubieboard2 with an external SATA 1TB drive which I use to store CCTV footage, the MySQL database, etc. I want to minimise writes to the sdcard and so that is the thing to keep in mind.

Following the Arch installation tutorial here:

Using an Ubuntu 16.04 VM to get the sdcard prepped, install btrfs-tools first.

Partition and create btrfs instead of ext4 and create a boot partition too. Make a 100MB boot partition and use the rest for root. Follow the guide for the start location.

sudo mkfs.ext2 /dev/sdc1
sudo mkfs.btrfs /dev/sdc2

I would like to use the btrfs compression for a performance boost (allegedly), space savings and perhaps a boost in reliability because of less space usage...

cd /tmp

mkdir boot
mkdir root

sudo mount -o compress=lzo /dev/sdc2 root
sudo mount /dev/sdc1 boot

Download the arch tar.gz file and decompress as per their guide. I got a few warnings about flags due to the btrfs so I used tar instead of bsdtar. I'll ignore them for now and hope for the best! It may take a while to do the extraction - depending on the speed of your SD Card.

sudo tar -xzvf ArchDownloadFile.tar.gz -C root

Create a file on the boot partition called boot.cmd and put this inside it:

if test -n ${distro_bootpart}; then setenv bootpart ${distro_bootpart}; else setenv bootpart 1; fi
part uuid ${devtype} ${devnum}:${bootpart} uuid

setenv bootargs console=${console} root=/dev/mmcblk0p2 rootfstype=btrfs rootflags=autodefrag,compress=lzo rw rootwait

if load ${devtype} ${devnum}:${bootpart} ${kernel_addr_r} /zImage; then
  if load ${devtype} ${devnum}:${bootpart} ${fdt_addr_r} /dtbs/${fdtfile}; then
    if load ${devtype} ${devnum}:${bootpart} ${ramdisk_addr_r} /initramfs-linux.img; then
      bootz ${kernel_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r};
      bootz ${kernel_addr_r} - ${fdt_addr_r};

This is essentially what is in the default boot.scr, with mods for btrfs and the boot partition. We need to compile this file. Get the u-boot-tools and the run with appropriate modifications to your paths if you aren't following mine:

mkimage -C none -A arm -T script -d boot/boot.cmd boot/boot.scr

Unmount everything and pop the sdcard into the Cubieboard. It should boot up to a login prompt. If not, start from scratch and try again :) the serial console helps a lot! It's probably worthwhile to edit the boot.scr to output to HDMI rather that serial.

When you are up and running, before installing uboot or other update stuff (as per the Arch tutorials), update your fstab to include a line for the boot partition:

/dev/mmcblk0p2 / btrfs defaults,relatime 0 0
/dev/mmcblk0p1 /boot ext2 defaults,noatime 0 0

tmpfs   /tmp tmpfs  rw,nodev,nosuid,size=512M  0  0

It's also important to install btrfs-tools and a couple of other things with pacman:

pacman -Syu btrfs-prog ntp wget nano

General Stuff

Do the usual things at:

Also, install yaourt for AUR stuff:

su (the password is root by default)
useradd ...
pacman -S sudo

Follow the instructions and then check if your new user has sudo by typing su newuser and trying out a sudo command. If all goes well, ssh in via the new user and sudo userdel alarm.

Make sure your boot partition is mounted properly and then:

sudo pacman -Syu

Home Assistant and Node-Red Installation

yaourt -S home-assistant --noconfirm
yaourt -S nodejs-node-red --noconfirm

The default home assistant AUR install puts the config is /var/lib/hass and runs home assistant with user hass. Let's keep it this way but I want read write access to the config as my normal user.

sudo usermod -a -G hass username
sudo systemctl start home-assistant
sudo systemctl enable home-assistant

The first run of home-assistant (hass going forward) takes a while as it downloads and sets up all of the components.

Node-Red doesn't seem to come with a predefined service, user, etc. so let's create one...

sudo useradd -m -s /sbin/nologin nodered
sudo nano /lib/systemd/system/nodered.service

Paste this inside:

# systemd service file to start Node-RED
Description=Node-RED graphical event wiring tool.
# Run as root user in order to have access to gpio pins
ExecStart=/usr/bin/env node-red-pi $NODE_OPTIONS $NODE_RED_OPTIONS

sudo systemctl start nodered
sudo systemctl enable nodered

The Node-Red config is in /home/nodered/.node-red

I then modified the permissions on the nodered home directory so that I could edit as my normal user:

sudo usermod -a -G nodered username
sudo chmod 770 /home/nodered

There's lots more to do such as installing pi-hole, samba, etc!

Friday, 22 December 2017

Sonoff External Button Input

It seems like an obvious thing that's missing from the excellent Sonoff modules out there: an input for an external button (or two or four!). A common thing to do is to solder some wires across the existing switch on the PCB, but this isn't the best idea. In this post I will quickly show what I did, which is a more robust solution for a permanent installation.

The built in button(s) on the Sonoff's are hooked up directly to one of the GPIO inputs on the ESP8266. This is fine for a button on a PCB as no ESD or surge or wrong connections are likely. On the other hand, hooking a pin to a long wire which will be handled during installation, is susceptible to interference coupling, etc. is not the best idea. The ESD protection diodes on the chip may be able to handle some punishment but over the years it's going to be a real hassle to replace the Sonoff...

In the photo and rough circuit diagram below you can see I have simply soldered everything (with some heatshrink for insulation) to a pin header which plugs onto the Sonoff serial header. A 2.2k pullup resistor is there to provide a strong pull up. I have found that the built in pullups aren't strong enough, causing the inputs to trigger sporadically and sometimes even "randomly" - possibly when lightning strikes... A series 470 ohm resistor prevents too much current going into or coming out of the pin in case of mis-connection, surge or even static to some extent.

I plan on designing some small input protection PCBs to use with all of my Sonoffs which will have proper ESD and surge protection by using clamp diodes, etc. These will be totally bullet-proof and could probably even handle a mains wire shorting to them!

Sunday, 3 December 2017

Home Assistant Hue Emulation

My Home Assistant setup uses nginx to proxy the web interface to HTTPS. I bought a Google Assistant for black friday and I want it to be able to control my lights... I haven't tested it yet but I figure I may as well start doing some setup!

The Home Assistant docs say that the emulated_hue component needs to run on port 80. This won't work by default because it doesn't have root permissions. I also didn't want to grant python the ability using cap_net_bind. My solution in the end was to create a NAT firewall rule on my server (see the end of this post). This takes anything that comes in via TCP on port 80 and redirects it to port 8300 (which is the port that Home Assistant was told to use for the emulated hue).

The issue with this is that nothing else that uses port 80 will work anymore - in particular nginx pages such as pi-hole. I figured I could just run pi-hole through HTTPS!

sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8300

and for localhost redirects too (optional):

sudo iptables -t nat -I OUTPUT -p tcp -d --dport 80 -j REDIRECT --to-ports 8300

To remove these rules:

iptables -t nat --line-numbers -n -L

HDMI to HDMI+Audio Teardown

I recently purchased an HDMI to HDMI+Optical and stereo analog output box from Banggood. I was curious at to what was inside as well as the build quality. It's pretty good! This post has photos as well as links to some of the components.

First off, why do I need one of these? My current TV and media PC setup is a little too spread out for my liking. The TV is mounted above the fireplace and the media PC, PS4, Chromecast, etc. are scattered around, but not above the fireplace obviously! I don't want a bunch of separate HDMI cables running up the wall (not to mention power cables) and then optical or coax S/PDIF cables running back down to the HiFi...

The solution is to use an HDMI selector ($10 or so!) and then afterwards place the audio extractor shown above. That way I get surround sound for every device (and I don't need to use the TV passthrough which doesn't really work anyway since it has to be on!), and I only need one remote control to select what I want to watch / listen to. This is really convenient for my home automation system too, since I can use one IR blaster to turn on everything, and simply send the commands to the selector and everything works... The selector actually has an external IR receiver so perhaps I'll make it work without IR :) stay tuned!

This is what I found inside when I cracked it open:

It's a good quality looking PCB without any dodgy soldering and nice looking solder mask / silk screen. What's important it the components though. In terms of power supply, the input is 5V and there is a 3.3V and 1.8V linear regulator immediately visible. Linear regulators have good noise performance compared to switch mode so this bodes well for potential buzzing on the analog audio output.

Interestingly the heart of the device is a large looking IC with markings rubbed off. This is probably to avoid royalties and other issues. I'm not complaining...

The Analog to Digital Converter (ADC) is the small 8 pin chip close to the RCA outputs. It's a CS4344 from Cirrus Logic. It's a pretty good ADC: 24 bit, 192 kHz stereo (obviously). The THD+N is pretty average at -90 dB which is about 0.003%. This is still a pretty good THD given that most affordable HiFi power amplifiers can only do 0.01% (and sometimes as bad as 0.1%). You'll probably find that your typical power amplifier at home is as bad as 10% THD when you crank it up - so nothing to complain about here!

Overall, having used this device for a few weeks now I'm happy with the performance! The only issue is that sometimes when I change HDMI source using the selector, this audio extractor doesn't immediately figure out that the source has changed and I need to power-cycle to get the audio back. It's pretty rare so no big deal! One other issue is that my BluRay player seems to know something is up, and it won't work through this device due to the HDMI encryption.

Hybrid ESP8266+UNO Energy Measurement

To complement my home automation system I needed to add a multi-channel power measurement system to my DB board. I figured four channels is ...