Over the Air update (Ota) for ESP8266

Thanks to the esp8266 project on github there is a very convenient way how an ESP can be updated over the air. There are three different ways available.

1. The first one is via the arduino IDE itself where the esp opens a port and is available for firmware upload just like with a serial connection. Very convenient if you are in the same network.
2. The second one is via http upload. So the esp provides a web server to upload the bin file. In this case there is no need to be in the same network but it is still a push and for each installed esp individual necessary.
3. The third and most convenient way for a bigger installation base or in case the devices are behind a firewall (as they always should be) and no remote access is possible. In this case the device can download the firmware itself via http(s) download from a web server somewhere in the internet.

For a complete dev-ops pipeline from pushing to a repository to flashing a device the third scenario it the easiest one. So we need a place to store the binary files. For convenience I use amazon s3 to host my binary files as travis easily supports s3 upload. But it can be every internet platform where files can be stored and downloaded via http(s). The necessary code on arduino side looks like this:

#define BUILD_VERSION        REPLACE_WITH_CURRENT_VERSION
#define ULR_FIRMWARE_BIN     "http://s3.amazonaws.com/feelflight/firmware/blanked.bin"
#define URL_FIRMWARE_VERSION "http://s3.amazonaws.com/feelflight/firmware/blanked.version"

void checkForNewFirmware(void){

    HTTPClient http;
    http.begin(URL_FIRMWARE_VERSION);
    int httpCode = http.GET();

    if(httpCode == HTTP_CODE_OK) {
        String payload = http.getString();
        int newVersion = payload.toInt();

        if (BUILD_VERSION < newVersion){
            Serial.println("I need to update");
            t_httpUpdate_return ret = ESPhttpUpdate.update(ULR_FIRMWARE_BIN);

            if (ret == HTTP_UPDATE_FAILED){
                Serial.printf("HTTP_UPDATE_FAILD Error (%d): %s\n", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
            }
        }
    }
}

This arduino function can be called from time to time (at startup or on constant running systems every now and then) to check for a new firmware version and in case there is a new version available automatic flash it and restart.

• Line 1 is a #define with a placeholder for the current version of the installed firmware. This placeholder is replaced in the build pipeline at travis with an increasing number. So the compiled code has something like 23 or 42 instead of REPLACE_WITH_CURRENT_VERSION.
• Line 2 is the URL for a latest version of a firmware.
• Line 3 is the URL to a file with only one line with the latest build number in it.
• Line 7-9 loads the version file from s3.
• Line 12-13 converts the file into a number which can be compared with the define from line 1.
• Line 17 is the firmware update itself. A detailed description of the ESPhttpUpdate class can be found here.

There are two ways to check if there is a new version available and only flash if there is something new. The one we use here is to have an own mechanism for it. I do it because on s3 I can only host static files and therefore I place the latest build number in a static file next to the firmware itself. The other way is build in into ESPhttpUpdate. The update function can be called with a build number which will be compared on the server and the return code will reflect if there is a new version or not. In this case we would need a script on the server to check for it.

Get an increasing build version number

With a little bash script we could load the last build number from s3 and then increase it in order to have the current number for our build.

#!/usr/bin/env bash

let oldversion=`curl http://s3.amazonaws.com/feelflight/firmware/blanked.version`
let newversion=oldversion+1

echo "============="
echo "New Version:"
echo $newversion
echo "============="

sed -i "s/REPLACE_WITH_CURRENT_VERSION/$newversion/g" src/main.cpp

echo $newversion > upload/blanked.version

This script loads the version file (line 3), increases the number (line 4) and patches our source code file (line 11) with this number instead of REPLACE_WITH_CURRENT_VERSION. After running this script the current source code contains the latest number and also the upload folder for s3 has a new file with the newest number in order to inform the polling ESPs.

Travis config file

Travis-ci is incredible easy to use and very reliable for continuous integration. In combination with platformio it is very easy to compile arduino code for several types of hardware. Simply configure the hardware in the platformio.ini file:

[env:huzzah]
platform  = espressif8266
board     = huzzah
framework = arduino

In this case we use the esp8266 feather board aka Huzzah. Just set the framework to your kind of esp.

Travis itself is configured by the .travis file in the root directory of your repository on github:

language: python
python:
- '2.7'
sudo: false
cache:
  directories:
  - "~/.platformio"

install:
- pip install -U platformio

script:
- mkdir upload
- ./updateBuildVersion.sh
- sed -i "s/WLANSSID/$WLANSSID/g"     src/main.cpp
- sed -i "s/WLANPASSWD/$WLANPASSWD/g" src/main.cpp
- platformio run
- cp .pioenvs/huzzah/firmware.bin upload/blanked.bin

deploy:
  provider: s3
  bucket: 'feelflight'
  secret_access_key:
    secure: amzqBC+rs+S860Z6ABQNAseKYL+7UgNnJGhF7jGkc6Aq/e8JmPqRSYHrEESM4S1jkOXYR5WouX04ytZqoXnrt0E625LT0+rLUGjyZ1QpGlrI5dwhOP4TagT+A90DtRI77TGf4qgnAkX+wAOufKehMKNms8jL6M64vwR5mIg3veiewZFRBtpvlkqCS55+rdWmYbFuT+UYNdq5UItJkfY1HNunafDvS1qfCTwBzoa5Yro/pyGA5cSdKDEZrJ+WfgCm03PZHVMKARm07lOcAZTstp5qQCbG8S4jE0OA8Q+AQ/mcwzB+JRHrJZdoQmpNjAsREnRDvv/Zz88V4JluPVrgk1B3mWw7tAPGnxT+N/Kwj+f455AMjsEcJ3z3YdGeJtftqYtr9kbcECWt7puPILpRhSKkAMGEPAQhOQAdLqQvfL1qZQbunexDShKkpMbpmVvyTYQXXmixoc26dB7MJpbw4UHNui3zpb5fWDHuJ3EIEvHvuoMDT2Dk2GTpStBqACrbo74Orsfah6DvEuJXXbmBIChfDufalNA5CNkjhIfBSDQpu5HE6UEylPDYcwgXwvhIl9zSXljYcH6LBP18axwheCmyeolVse3a3h9GF3tSfcJrlMshZ0oZ0WuwvLflE4ZzDWMT1XX8kgHrvaYklagwKbgltMYkq7R04kD++h32J8s=
  access_key_id:
    secure: jZiEn7PTFRrwFu8ZmDEkUGjYuKSWmP+kI6biVKaRwhcqA+WeYjKOH3r5NgR5V+xoYcZA47Qm41pl6gi71aMEU2Xil4+HUdmLM6pXLU87Q0NG974cLesccaDy2/rkADmLP/jaqN66Pavd4l2tRxGOP+p1QQRXQOccFEW6j95PdPOyzppPZc6h8yzqmxerIgDSDFQuF4pRWjEtJSPrEyw79p834wvVVahlXRJ6jrqy5X4CiqabYmaR3QuT0W9tBHHtfMfgPJBCooTxqT0uqDnOSN0wU6TmQ8ZHg9y7d4ChOWLbpHwHhOk3UrDrTllbTSr7zRjqzwW69yivZX2e0XR89X8PFcLg8jIcZxgKyIKGo+BpCnaLlVQ1dxmIrDfcComino+3ZWC4lZDLgaw/uTfcAapn1sPBNhnxed7kr7u/RkZIfdXWZn4GSO1aDAJWXMF6lC2lq1JN7FlfpyGJuvsN6FQcIrq0W8jghZ0+8AAgwOzzNPG5bY34+8R3Qtp1d4hwqkan4peF0vVfeVRldtkissmup+bQRyU7xyYUPqL7EdtjJXBXwP3ChTv/FGu2eQhjgweLHsyrkcFBeqpKwjHnG0jUSV3QQPq9hpO8mk3eSjSbM91cY9S5t2BKtIR0ALCyyAn+B40P6OiJ+4v4d0ZdyXGjL3aRyOSg4jIOl5Awa10=
  skip_cleanup: true
  acl: public_read
  local_dir: upload/
  upload-dir: firmware

• Line 1: Platformio is based on python so the build environment (although the code is c++) is python for maintaining platformio.
• Line 3: Right now platformio is only available for python 2.7 so this line gets the latest stable version of python 2.7.
• Line 5-7: Gets the latest cache files from the last build in order to save compile time and reduce the costs for travis. As this service is for free for open source projects it is always nice to save some money for the cool guys.
• Line 10: Installs the latest version of platformio itself.
• Line 13: Creates the upload directory which we will upload to s3 later on.
• Line 14: Calls the build number increase and patch script.
• Line 15-16: Patches the wireless lan config in case it is not handled inside the arduino code itself.
• Line 17: Calls platformio to download all libraries and compile the arduino code itself.
• Line 18: Platformio generates a lot of files for the linker and several other files. We only need the bin file later on, so we copy it here to the upload folder.
• Line 20: Travis has a build in functionality to upload files after compilation. This is the part where we upload the files to s3.
• Line 22: Defines the s3 bucket to upload the files.
• Line 23-26: Provides the encrypted s3 credentials. See travis documentation on how to create these lines.
• Line 29: Defines the local folder to be uploaded. Otherwise travis will upload everything from the current run.
• Line 30: Defines the s3 folder in the bucket where the files will be stored.

With this files in place travis monitors your github repository and creates / uploads new firmware versions each time you push changes to your repository. The arduino code checks for new versions and patches itselfs as soon as there is a new version available. A complete project can be found here in my github repository.