TL;DR: Skip to "Setting up mTLS with Cloudflare and Android" for the high level instructions.

There are many different ways to set up Home Assistant so that it is accessible when you're away from home, which have various pros and cons, but I had some extra requirements beyond just securing it with HTTPS/TLS which I haven't found any articles about (in bold):

1. Does not expose my home IP address
2. Does not publicly expose my Home Assistant instance (i.e. no publicly-accessible login form)
3. Secured using TLS/SSL
4. Accessible from any device
5. Works with the Home Assistant Android native app
6. Does not require running a network client "tool" on my phone (e.g. VPN client, Cloudflare WARP client)
7. Ideally doesn't require a paid subscription to something
8. Can support Google Assistant integration in future

This blog post, as with most of my posts, serves as a combination of a reminder to my future self on how I got it working, and hopefully also a useful resource for others. As such, this post will be rough-around-the-edges in places.

Things this blog post doesn't cover:

• Setting up Cloudflare with your domain
• Setting up Cloudflare Tunnel
• Configuring Home Assistant to accept reverse-proxied traffic through Cloudflare Tunnel (see Home Assistant docs)

Ways of exposing Home Assistant

Some different ways of setting up Home Assistant so that it's accessible publicly:

• Paying for Home Assistant Cloud
  • Pro: easiest, supports the maintainers
  • Con: subscription service, leaves your HA instance exposed to the internet
  • Satisfies requirements 1-6, 8 (I think)
• Forwarding a port on your home router
  • Pro: easy to set up
  • Con: exposing your home IP, and has potential risks from security holes in Home Assistant
  • Satisfies requirements 3-8
• Using a VPN (incl. Tailscale) to connect back into your home network
  • Pro: secure, no attack surface for your home server
  • Con: you have to install the VPN client wherever you want to use it, and connect/disconnect when needed
  • Satisfies requirements 1-5, 7
• Hiding your Home Assistant instance behind Cloudflare, using a Cloudflare Tunnel to connect outbound from your home network to Cloudflare.
  • Pro: secure, no direct route of attack to your home server
  • Con: has potential risks from security holes in Home Assistant
  • Satisfies requirements 1, 3-8

Requirement 2 (does not publicly expose my Home Assistant instance to the internet) can be mitigated when using Cloudflare by using Cloudflare Access to require visitors to log in (e.g. via Google, Okta, one-time PIN via email) before they even get to your Home Assistant login form. This is great from a security standpoint, but it does cause issues with the Android app.

Cloudflare Access and the Android app

If you set up Cloudflare Access so that it prompts the user to log in via either an OTP sent via email, or via another identity provider before exposing your Home Assistant login form, this breaks compatibility with the Android app. This is because when you access your Home Assistant domain, e.g. ha.mydomain.co.uk, you will be redirected to your identity provider domain before you get to see your Home Assistant login form (e.g. myname.cloudflareaccess.com). This redirect takes you out of the Home Assistant app and into your browser, and so once you've logged into your identity provider, the Android app never receives the auth token generated as part of the final login step, which it needs to be able to communicate with Home Assistant. As such, we need to find another way of authenticating the Android app which does not require redirecting to another domain.

There have been issues raised on GitHub about this exact issue, but there is no solution for the redirect problem at present.

Just get to the point

What does seem to work is having two subdomains, which both back off to the same Home Assistant instance, one which is for use in a browser, and one which is for use from the Android app.

• ha.mydomain.co.uk - for use from a browser
  • Uses Cloudflare Access to enforce login via an identity provider
  • Accessible from anywhere, as long as you can log into your identity provider
• ha-android.mydomain.co.uk - for use from the Android app
  • Uses mTLS (mutual TLS) to require the Android app to present a client TLS certificate to Cloudflare
  • Requires manual setup on each Android device, but only once

You will need two separate Cloudflare Tunnels, one for each domain, both of which should back off to the same origin URL.

I've seen a number of other guides/tutorials online describing how to set up Cloudflare Access, but I haven't seen much explaining how to set up mTLS with Cloudflare, so that's what I'll do next.

Setting up mTLS with Cloudflare and Android

Tl;dr this is how I did it:

1. Remove Cloudflare Access rules for your ha-android domain
   
   • Cloudflare Access rules will prompt the user to log in, which we don't want
2. Generate a client certificate in Cloudflare
   
   1. Use the "SSL/TLS" pane of Cloudflare to generate and download a client certificate and secret (or generate it yourself offline). Store the certificate as cf.pem and the private key as cf.key
   2. Convert these two files into a pfx file, which is importable in Android, using the following command (source): openssl pkcs12 -export -out cf.pfx -inkey cf.key -in cf.pem

   3. Securely transfer this file to your Android phone
3. Download and install Cloudflare CA cert on Android device
   
   1. See guide on Cloudflare's site. This seems to be required to install the client cert.
4. Install client certificate into Android cert store
   
   1. Go to Settings > Security > Credential Storage, tap Install from storage, then locate the file you transferred from step 2.
   2. Give the certificate a sensible name so that future-you remembers what it is
5. Create an mTLS rule in Cloudflare WAF to block non-certificate-holders
   
   1. From the "SSL/TLS" pane in Cloudflare, click "Client certificates", then click the "Create mTLS rule" button. If this isn't visible in Cloudflare, it's because they've moved it to the API Shield section which they are warning about right now ('Nov 2022)
6. Configure Home Assistant app to use cert
   
   1. Change the "External URL" in your app settings to use your ha-android domain, then restart the app. You should be prompted to select a client certificate on next start.

That's it. You now have a domain for browser use, and a domain which works in the Android app, without having to start any other VPN apps first.

References

Some posts I found useful while setting this up:

• https://empty.coffee/home-assistant-cloudflare-zero-trust-setup/ - looking back, this blog post is a much more nicely written version of a significant portion of what I wrote!

Problem: some podcasts that I like to listen to vary considerably in volume throughout, which means I have to keep adjusting the volume in loud environments, such as when cooking. I wanted to avoid having having to do this when listening to podcasts on my laptop.

Requirements:

• No change to music player required (I want to keep using Spotify)
• No work required ahead of time to prepare each podcast episode
• Simple to enable/disable

General solution technique: Employ dynamic range compression to reduce the volume of the loud parts of the podcast, so that they're closer to the quiet parts, meaning no manual volume adjustment is necessary. This seemed easier to do than trying to control my laptop volume in sync with the podcast.

Specific solution: run all of my laptop audio output through a software dynamic range compression filter before it goes to the speakers. This means all application audio output is covered which makes it easy to use with Spotify.

This is how I did it on macOS Monterey.

Side note: This will make music sound awful and remove any punchiness from it (which is already slowly happening across the music industry anyway - see Loudness war), but for spoken word, it does what I wanted it to do. The end result will make podcasts sound more like radio programmes in terms of having consistent volume.

1. Install Blackhole, a free audio loopback driver. This presents an extra input and output device to the system. I used to use Soundflower for this but it hasn't been maintained for a while and has some issues.
2. Install Apple AU Lab (link at bottom of page), a free audio mixer allowing you to route audio through various effects processing units, a number of which are included with macOS.
3. On recent versions of macOS, you need to grant microphone permissions to AU Lab so it can receive audio from BlackHole. Most applications prompt you, but AU Lab doesn't for some reason. Run this command to grant the permission (source):
   sqlite3 ~/Library/Application\ Support/com.apple.TCC/TCC.db "INSERT INTO access VALUES('kTCCServiceMicrophone','com.apple.audio.aulab',0,2,2,1,NULL,NULL,NULL,'UNUSED',NULL,0, CAST(strftime('%s','now') AS INTEGER)) ;"

4. Set your system output audio device to BlackHole. This captures audio from all applications (excepting those which are configured to output to a specific other device) - including Spotify, in my case. Note: you will no longer hear any sound until the next step, as it's no longer being routed to your speakers.
5. Create a new document in AU Lab, using the "Stereo In/Stereo Out" template. Specify BlackHole as the input source, and your chosen speaker as your output device (for me, a Bluetooth speaker). You should now be able to hear application audio again, as the audio from the BlackHole input is now being routed to your output speaker.
6. In the "Audio 1" column, under the "Effects" section, choose "AUDynamicsProcessor" from the drop-down menu to add a dynamic compressor.
7. In the popup window for the AUDynamicsProcessor, set the parameters like so:
   
   • Compression threshold (centre dot): -50 dB
   • Headroom (right-hand side dot): only just above the compression threshold dot. ~9 dB
8. Expand the "Details" section and set the following parameters:
   
   • Attack time: 0.002
   • Master Gain: ~11 dB (to amplify the output signal, as the compression makes it significantly quieter)
9. Done

I run all of my home server infrastructure in Docker containers as I like the isolation and distribution model that they provide.

The default sqlite3 database used by Home Assistant does not scale well to storing a large number of events and state changes, which makes using the History or Logbook panes of the Home Assistant interface slow to use. MySQL performs significantly better at this, at the expense of having to run a separate Docker container to handle the database.

Migration process

The overall process for migrating from the default sqlite3 database to a MySQL looks like the following:

1. Start a containerised instance of MySQL 8.0
2. Stop Home Assistant
3. Configure HA to point at the containerised MySQL instance
4. Start HA, allow it to create its database schema, then stop it again
5. Truncate all tables in the MySQL database
6. Dump sqlite3 database into an SQL file
7. Transform the SQL file from sqlite3-flavour to MySQL-flavour
8. Import the SQL file into MySQL
9. Start HA

Largely following this guide: https://gist.github.com/seidler2547/93012edf3c7a2414ec1d9a8ebbc9c1a6

Below are my "notable points" relating to steps in the above process.

1. Start a containerised instance of MySQL 8.0

There are a choice of Docker images available - mysql:8.0 and mysql/mysql-server:8.0. The difference is apparently that the image with the longer name is an Oracle fork of the "official" mysql image, which uses Red Hat as the base image. The "official" image uses Debian as the base.

In practice, I found that when I tried to run the Oracle-maintained image, it didn't handle being run with a different uid/gid well (didn't have permission to write temporary files). The "official" mysql image didn't have this problem.

4. Start HA to allow it to create database schema

Home Assistant wouldn't connect to MySQL on startup, and it threw an error into the logs containing the following line:

Plugin caching_sha2_password could not be loaded: Error loading shared library lib/mariadb/plugin/caching_sha2_password.so: No such file or directory  

This led me to a post on the Home Assistant forum, which pointed me in the right direction. MySQL 8.0 changed the default authentication method to one which is more secure, and as part of the MySQL protocol client-server handshake, the server tells the client which auth method to use. The version of SQLAlchemy ORM used in Home Assistant doesn't yet support the new caching_sha2_password plugin, so the easiest solution is to modify the MySQL server settings to default to using the previous mysql_native_password method, which Home Assistant does support.

This can be done by modifying the MySQL container start command to include the --default-authentication-plugin=mysql_native_password flag.

7+8. Transform SQL file from sqlite3 format and import into MySQL

As I didn't have any sqlite tooling installed locally on my server, I used a Docker image containing sqlite to dump the database contents as SQL.

Importing this into MySQL involved transforming the SQL format first, and with a couple of small modifications to the gigantic shell command included in the guide linked above, I was able to do this in one go:

# not shown: cd into home assistant directory
docker run --rm -it -v $(pwd):/db keinos/sqlite3 sqlite3 /db/home-assistant_v2.db .dump | \  
    sed -re 's/^PRAGMA .+OFF/SET FOREIGN_KEY_CHECKS=0;SET UNIQUE_CHECKS=0/' \
        -e 's/^CREATE INDEX .+//' \
        -e 's/^BEGIN TRANSACTION;\r$/SET autocommit=0;BEGIN;/' \
        -e '/CREATE TABLE/,/\);/ d' \
        -e 's/^INSERT INTO "([^"]+)"/INSERT INTO \1/' \
        -e 's/\\n/\n/g' | \
    perl -pe 'binmode STDOUT, ":utf8";s/\\u([0-9A-Fa-f]{4})/pack"U*",hex($1)/ge' | \
    docker exec -i <MYSQLCONTAINERID> mysql --default-character-set=utf8 -u hass -p<YOURMYSQLPASSWORD> homeassistant

This command took about 30 minutes to run, but completed the full migration from a SQLite file to a MySQL database.

tl;dr: Using Rufus inside a Windows XP VM works

I happened to have one lying around, and passing the USB stick device through to the XP VM worked perfectly.

Methods that don't work:

• sudo dd if=win10.iso of=/dev/rdisk3s1 bs=1m
• unetbootin (the USB stick boots, but Windows 10 isn't an option in the list)

Where are the intro docs?

Some docs are located here. There are a number of interesting/useful repos in the same GitHub organisation.

Connecting to Wi-Fi

Initially the ReSpeaker broadcasts its own wireless network. After connecting to this, visit http://192.168.100.1/home.html to access the main ReSpeaker menu, which contains an option to connect the ReSpeaker to a wireless network of your choice.

There's also an installation of JUCI running at http://192.168.100.1/#!/overview, which is a web interface to most of OpenWRT's configuration (since ReSpeaker's Linux core is based on OpenWRT).

Side note about networking

The ReSpeaker can act in station and access point mode simultaneously; it uses a bridge with an IP of 192.168.100.1 to bridge the station interface (WAN) of apcli0 (also listed as ra0) with eth0.1 (LAN):

root@ReSpeaker:~# brctl show  
bridge name   bridge id           STP enabled   interfaces  
br-lan        7fff.9c65f91dc3cf   no            eth0.1  
                                                ra0

If you set up a route on your PC manually, you can access 192.168.100.1 when connected to your normal wireless network. The firewall is configured to allow WAN traffic into the LAN but not vice-versa.

The config files for this are in the standard OpenWRT location of /etc/config/{firewall,network,wireless}. Most of these settings are also editable through JUCI.

After connecting the ReSpeaker to your wireless network, you can disable the ReSpeaker's own access point by going to JUCI, then Wifi => Interfaces => ReSpeakerXXXXXX @, and turning off "Enabled".

Accessing the ReSpeaker console

If you mess up your networking somehow and can't access its web UI, you can access a shell through an emulated serial console to fix it when connected over USB. Example command to connect:

screen /dev/tty.usbmodem1411 57600

Running code samples

I wanted to play with the LEDs, but out of the box the music player Mopidy has exclusive claim to their usage (presumably to support the intro "Hi, ReSpeaker" demo) - attempting to use them within Python will give you a "resource busy" error. Stop Mopidy running with:

/etc/init.d/mopidy stop

Now you can use the LEDs as described in the Programming Guide.

It may have just been because I was powering ReSpeaker from my laptop, but it doesn't seem to be able to supply enough power to the LEDs to run them at full power - the board browned out at around 80% of full power (setting them to colour 0xCCCCCC made it a bit unstable).

Misc useful info I've gleaned

The onboard Atmega32u4 is running this code by default.

Intuitively, installing the JRE appears to be the obvious choice, as you're not bothered about javac or other developer tools. However, the JRE DMG package doesn't register itself as a full JVM, merely a "Internet Plug-In". It'll allow you to run GUI applications that use Java, but it will still result in a prompt to download a JVM if you run java from a terminal. This is because the JRE is installed into /Library/Internet Plug-Ins/JavaAppletPlugin.plugin, but /usr/bin/java only looks for JVMs in /Library/Java/JavaVirtualMachines.

Instead, download the Oracle JDK - this installs itself into /Library/Java/JavaVirtualMachines and therefore can be used from both the command-line and from GUI apps.

Controlling existing lamps

A range of 433Mhz plug-in remote controlled mains switches are available from various brands, which are able to either switch on/off a device with a standard UK plug, or dim it. Initially I used a set of Maplin switches, but later replaced them with Nexa units as they seemed more reliable (and the relays are quieter when switching). These can be controlled with a handheld remote, but the signal that it emits can be emulated using an Arduino and a 433Mhz radio transmitter.

Other people have done the hard work in reverse engineering the radio protocols, and others have written Arduino libraries to transmit the commands, but I've combined control of both Maplin and Nexa sockets into one Arduino sketch. switcherduino is my Arduino firmware that can be used as a USB serial device to dim or turn on/off these devices.

Orchestration software

I use a simple Node.js-based webapp to act as an interface to the USB switcherduino, which maintains state about devices, allows associating devices into groups, executes pre-computed scenes, runs sequences on certain events (e.g. connecting to wifi), and provides an API to enable voice control (integrating with Tasker and AutoVoice for Android).

This is currently not open-source as the software is minimal and quite rough-around-the-edges, but there seems to be a range of other projects for this sort of thing such as Domoticz, Home Assistant, OpenHAB etc. I haven't tried any of them, but I've heard good things about Domoticz. Here's a list of similar projects.

Products available

Most of my sockets are these Nexa EYCR-250 dimmable sockets, which are part of the "System Nexa" range. Mine were bought at Clas Ohlson for the bargain price of £3.99 each.

Other sockets that I have or have used are the Nexa EYCR-2300 non-dimmable sockets which are better suited for non-dimmable equipment (such as a Raspberry Pi), and this set (N79KA) from Maplin. The Maplin set has a less robust radio protocol and seems to have more trouble with interference and has a shorter range (in my experience).

Nexa-compatible products/brands

Some other brands are apparently compatible with Nexa's protocol. I'll list them here as I find out about them.

Apparently the Nexa switches are manufactured by Arctech, and so share the same protocol as a number of other brands, documented here at the PiLight wiki.

Maplin-compatible products

This GitHub repo's wiki has a good description of hardware using the same protocol.

My setup tries to modify as little of the out-of-the-box Debian 8 configuration as possible. I enable public bridging so that all of my VMs are accessible from my wider LAN, but a private bridge could be added later.

Let's go

Make yourself comfortable

sudo apt-get install tmux vim  

Install KVM and tools

Install KVM, CLI tools (including virsh) and tunctl utility (required for setting up public bridging):

sudo apt-get install qemu-kvm libvirt-bin uml-utilities  
sudo adduser <user> kvm  
sudo adduser <user> libvirt  

This should now work (no VMs listed):

virsh --connect qemu:///system list --all  

Set up bridge networking

Add a bridge adapter. This will wrap the existing eth0 interface, which I'm using DHCP on (as I use address reservation on my home router to gain a "static" address). Debian 8 uses NetworkManager to configure networking, so in /etc/network/interfaces no adapters (besides loopback) will be listed.

Add the following to /etc/networking/interfaces:

# Bridge for KVM
auto br0  
iface br0 inet dhcp  
    bridge_ports eth0
    bridge_stp off
    bridge_maxwait 0
    bridge_fd 0

NetworkManager will stop managing any devices listed in /etc/networking/interfaces.

Reboot to ensure network changes have taken effect (restarting networking using the init script wasn't enough):

sudo reboot  

You should now be able to see the network bridge containing eth0 (bridge id redacted):

$ sudo brctl show
bridge name    bridge id       STP enabled interfaces  
br0        8000.xxxxxxxxxxxx   no      eth0  

And ip addr should show that the bridge adapter br0 has received an IP address via DHCP, and that eth0 no longer does:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000  
    link/ether [redacted MAC] brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000  
    link/ether [redacted MAC] brd ff:ff:ff:ff:ff:ff
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  
    link/ether [redacted MAC] brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.XX/24 brd 192.168.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 [redacted MAC]/64 scope global mngtmpaddr dynamic
       valid_lft 6917sec preferred_lft 3316sec
    inet6 [redacted MAC]/64 scope global mngtmpaddr dynamic
       valid_lft forever preferred_lft forever
    inet6 [redacted MAC]/64 scope link
       valid_lft forever preferred_lft forever

I haven't tested this setup, but I believe it should be complete. Next step is to install Docker, on which I will install a container for the Kimchi KVM management tool.

These instructions (largely based on this page) assume you've already got Homebrew installed, with Homebrew git.

$ brew cask install xquartz

$ brew install python3

$ pip3 install --upgrade pip setuptools

$ brew install python

$ pip install --upgrade pip setuptools

$ brew linkapps python3

$ brew linkapps python

$ brew install sdl sdl_image sdl_ttf portmidi libogg libvorbis

$ brew install sdl_mixer --with-libvorbis

$ brew tap homebrew/headonly

$ brew install smpeg

$ brew install mercurial

$ pip3 install hg+http://bitbucket.org/pygame/pygame

$ pip3 install pygame

$ pip3 install pyopengl

Now you've got a working PyGame setup (don't forget to run your script using python3).

I was having difficulty getting Android File Transfer to detect that my Nexus 5 (running KitKat 4.4.4) was connected via USB, when I discovered that MTP mode has to be manually enabled on the phone in order for Android File Transfer to detect it. The setting is hidden in Settings -> Storage, and can be found by tapping the menu button and tapping "USB computer connection". After enabling MTP mode using the checkbox, Android File Transfer detected the phone a couple of seconds later and file transfer then worked as expected.

Thanks to the original blog post I found explaining the setting and where to find it.

RewriteCond %{HTTP_HOST} !^www\.alexsilcock\.net$ [NC]
RewriteRule . http://www.alexsilcock.net%{REQUEST_URI} [R=301,L]

This works with requests to the base site URL, subdirectories of the site, and is SEO-friendly with its 301 Moved Permanently HTTP response code.

tl;dr: snippet at the end of this post

If my application references its CSS and JS using relative paths, e.g. <script src="assets/main.css">, the following ProxyPass rule in the Apache config appears to work correctly:

ProxyPass /someapp http://localhost:23456

This works fine if you request example.com/someapp/, because the browser will translate a script source of assets/main.css into an HTTP request for http://example.com/someapp/assets/main.css.

However, visiting http://example.com/someapp (note the lack of trailing-slash) will mean the CSS request goes to http://example.com/assets/main.css, which is outside the directory of the proxied app. To ensure that the application is only visited with the trailing slash present, requests can be redirected to the application using the following mod_rewrite rule (assuming you've already turned RewriteEngine on):

RewriteRule ^/someapp$ someapp/ [L,R=301]

These rules combine to form the following snippet that I use when mounting a node.js web application under a subdirectory on my website:

RewriteRule ^/someapp$ someapp/ [L,R=301]
ProxyPass /someapp http://localhost:23456
ProxyPassReverse /someapp http://localhost:23456

At long last I can finally tick off "set up some sort of blog-type thing" on my mental to-do list! I expect most of my posts here to be about techy things, often for my own reference, but undoubtedly I'll make the occasional post about something less electronic.

Listening to: Knife Party - EDM Trend Machine