OTA Upgrade Improvements #5
Conversation
The current 3.5 platform version doesn't seem to correctly pick-up the board_build.filesystem = littlefs option when using the uploadfs target, and builds a SPIFFS image instead of the desired LittleFS image... Upgrading the platform to 4.4, which has now got LittleFS built in, rather than as an external library fixes the uploadfs target.
Something in the newer platform code doesn't like the arguments we're passing here, and fails to initialize the bus at all, whereas it works perfectly without the arguments. The only argument that is potentially effectual here is the clock, but that's already reset in the SCD30 specific code anyway, which appears to be the main motivation for running at half the default speed, so should be fine?
In the upgraded platform, failing to set the ledPin to OUTPUT prior to calling ledcSetup results in nothing happening at all!
To avoid needing to hardcode certificates into the source, upgrade to >0.1.6 of esp32FOTA which supports reading both cert (and an optional signing key from the filesystem). But the default package only supports SPIFFS, and our FS is LittleFS, so use the thorrak branch that's lingering in chrisjoyce911/esp32FOTA#79 to enable that.
This is the currently active LetsEncrypt root CA[0], which seems like a reasonable default that many folks may be using on their webservers and therefore may "just work". In the case that your server is secured with another CA, simply replace this file. Note: this CA cert will change over time and will need updating before it expires to prevent OTAs breaking (TODO!), but this is no worse than the previous hard-coding in ota_cert.h really. [0] sourced from https://letsencrypt.org/certificates/, specifically: https://letsencrypt.org/certs/isrgrootx1.pem
Support supplying the ota URL via the configuration structure to avoid having to hardcode private hostnames, etc into the source tree, or alternatively having to maintain local diffs. Also include support for viewing/modifying the otaUrl via the get/setConfig MQTT messages.
Add a new MQTT message and associated handlers for triggering an OTA with a specified firmware file, rather than relying on the "is there a new version" logic from parsing firmware.json. This is useful for debug/testing over OTA when you don't want to be frequently updating firmware.json.
To support easily flashing different board configs (aka models) without having to edit the source to change defines every time, set up platformio environments for the different board configs (leds and neopixel) so you can simply use -e MODEL to do the right thing with no editing required. To support understanding what has been flashed, include a version number, build timestamp and git revision hash in the built image, and display that in both serial debug and MQTT messages. This works via a platformio extra_scripts pre hook and associated python script. This hook also sets the generated firmware filenames to include the model and version for convenience/clarity. The versioning strategy is semver (for compatibility with esp32FOTA logic) with the release version set in platformio.ini and used when the :release environments are built. The default environments for each model, increment the patch level of the release version and add a -BUILD_TIMESTAMP pre-release suffix to the version they are built with, such that they can be forceOTA'd to a device and a regular ota check will not cause them to be downgraded, but a later release will cause them to be upgraded.
This makes the screen flash every few seconds as the message briefly appears which can be distracting.
These helper scripts ease the common actions involved in developing via OTA and maintaining an OTA repository for the different models of board/firmware that need to be flashed. push-ota will build the specified environment and if it is a release, copy it to the webserver and call the gen-firmwarejson.py script on that remote host to update firmware.json accordingly. Optionally, if you also supply a device ID it will then use trigger-ota to tell the device to check for a new version. When operating on a non-release environment, the new firmware will be copied to the webserver, but firmware.json is not updated. If you suply a device ID, trigger-ota will then send a forced OTA request for that specific newly built firmware to the device.
It makes everything messy...
To avoid needing to hardcode certificates into the source, upgrade to >0.1.6 of esp32FOTA which supports reading both cert (and an optional signing key from the filesystem). But the default package only supports SPIFFS, and our FS is LittleFS, so use the thorrak branch that's lingering in chrisjoyce911/esp32FOTA#79 to enable that.
This is the currently active LetsEncrypt root CA[0], which seems like a reasonable default that many folks may be using on their webservers and therefore may "just work". In the case that your server is secured with another CA, simply replace this file. Note: this CA cert will change over time and will need updating before it expires to prevent OTAs breaking (TODO!), but this is no worse than the previous hard-coding in ota_cert.h really. [0] sourced from https://letsencrypt.org/certificates/, specifically: https://letsencrypt.org/certs/isrgrootx1.pem
Support supplying the ota URL via the configuration structure to avoid having to hardcode private hostnames, etc into the source tree, or alternatively having to maintain local diffs. Also include support for viewing/modifying the otaUrl via the get/setConfig MQTT messages.
Add a new MQTT message and associated handlers for triggering an OTA with a specified firmware file, rather than relying on the "is there a new version" logic from parsing firmware.json. This is useful for debug/testing over OTA when you don't want to be frequently updating firmware.json.
|
Question: Is there much value in adding the OTA URL to the configuration? I'd have thought that it's rather static and sufficient to be specified at build time, especially with the additional option to forceOTA with providing a URL which should cover the need to overwrite any given URL. |
|
It would be much nicer if we could avoid linking to this branch https://github.com/thorrak/esp32FOTA/tree/multi_filesystem and instead try to get this PR merged chrisjoyce911/esp32FOTA#79 |
|
I don't like build parameters that can't be committed to the repo was why I made it a config option initially, but you're right that now that there's also the forceota method, it's hard to see a justification for this. Thinking about my use-cases going forward, I can't see myself wanting devices to poll for OTAs, I'd much rather co-ordinate and trigger that centrally. So yeah, I'd be happy to remove that code if you like. |
|
Cool. No need to do anything, I've almost got it working in a local branch, that saves you resolving the merge conflicts. |
|
See 2072097 |
- Removes config support for otaURL (only hardcoded) - But can always be specified via the forceOTA MQTT message
|
I had to merge this anyway to pick up your changes so no drama. There's just two little diffs left:
Thanks. |
|
actually, I see you added root_ca.pem to .gitignore already, which confirms my (2) question. I'll delete this, and you can take card of the header prototype direclty if you think important. I'm going to blow away my forked repo and restart now that I've got a better idea how to work with github's workflow so it's not so messy in future... |
|
I'm currently not using Let's Encrypt for my OTA server, but I take your point it would be nice to have a default cert bundle that just works. |
|
Ah, that's a neat set of certs, that would be fine for me, but I don't think it's a big deal either way, I can easily push a cert in my provisioning script, but if you have other users wanting a more plug and play experience, having some default certs loaded would be good. |
3 OTA improvements in this pull request:
(sorry, this pull request is a bit messy because it's from a branch off the LittleFS upgrade branch in my repo, but I don't know git/giithub well enough to figure out how to get that reflected here, so you see all the changes, not just the ones on top of the LittleFS branch...)