implement swupdate - replacing opkg based updating

[mpvader]

the problem which we want to fix with swupdate

After being flashed with an image during manufacturing, the systems out in the field are auto-updated using a package system (opkg). Except for offline units, they are updated manually with the recovery image; manual update instructions. And also when the package based updating has caused issues, we tell people to update / restart from scratch by doing an update from an sdcard, using the recovery image.

This package based updating has certain advantages, but also disadvantages:

• Very sensitive to errors made in the versioning numbers
• Not all systems that report themselves as a certain system version l, for example Ccgx v1.40, are the same: some have updated themselves in many little steps, other have updated themselves at once, not always resulting in the same end-result etcetera.
• Sometimes systems end up being half updated: certain packages are updated, others are not because of power cycle, loss of internet connection or otherwise.
• Testing all of this is cumbersome.
• Not easy, or at least risky, to do major updates.

the solution

Go to image updating. Using the swupdate solution from Stefano Babic / Denx.

requirements

• Many installations have a bad internet connection: resume transfer is a hard requirement.
• Some installations are on an expensive internet connection: transferring only deltas is a nice to have. A bonus feature: not in the first release! Maybe later. By reducing the data cost of checking for an update daily, which is high with the opkg mechanism, we can already gain much.
• Checking for updates should cost less data then current solution. Easy to accomplish.
• Use https for the transfer
• Also a user can trigger a manual check for updates from the gui (and from command line). When available, install the update.
• There should be feedback to the user that download of a new version has started
• Log updating progress, on the data partition. Like we do now to /data/log/opkg/
• Keep the four feeds: develop, testing, candidate, release
• Keep the existing package based update / install mechanism intact. Not to be used for normal updates, but to offer extra optional package for users. Like we currently do with qt5, git, gdb, etc etc.

targets

1. machine bpp3: only active machine which is used to build code for devices in the field. All code and ongoing development for devices in the field is built using the danny config. And using the make ccgx command, with MACHINE=bpp3. To get started quickly, it is ok to work under Jethro instead of Danny. We can then later see if we first backport it to Danny and use that, or do the big jump at once. This would be a big jump because it is not so sure if building the current kernel version (3.7) with Jethro works at all or is stable, etc. etc.
2. machine beaglebone: A Venus image is currently being developed for the beaglebone black, on jethro. Note that this still in progress: focus first on bpp3. Start on the bbb only after checking with me on beaglebone status.

Once this image update mechanism is in place, our hands are free and we'll then start testing bpp3 built under Jethro in order to asap drop Danny and continue on Jethro.

We can then also drop the current bpp3-rootfs image, and use the venus-image instead.

And same time or later move from MACHINE=bpp3 to MACHINE=ccgx.

one time manual user intervention to switch

Chosen solution is: there will be one last opkg update. There will be an explanation that user need to use the sdcard (once) for an update. After which the ccgx can download its own updates again.

[mpvader]

/data details, as on a unit today, working with opkg:

16.0K   ./db/vrm-sqlite.db              <-- not needed
 6.0M   ./db/vrmlogger-backlog.sqlite3  <-- nice to have    - 48h backlog buffer
14.0M   ./log/*                         <-- not needed
 8.0K   ./ccgx/*                        <-- needed          - user settings
 8.0K   ./conf/*                        <-- needed          - serial no, production date, etc.
24.0K   ./keys/*                        <-- nice to have    - ssh keys
 8.0K   ./vrmfilescache                 <-- not needed
14.5M   .

Shown sizes are representative for all units in the field, except for vrmlogger-backlog.sqlite3. It is a buffer of max 48 hours of data. Grows up to 48 hours if there is no internet for 48 hours. Amount of data stored depends on number of connected chargers, inverters etc. I'll double check with the developer (Wiebe) for more details about vrmlogger-backlog.sqlite3 file size.

Histogram of free space (MBs) on /data partition:

Bin        Freq      Perc
   Less      11      0.3%
140-145       0      0.0%
145-150       2      0.0%
150-155       2      0.0%
155-160       2      0.0%
160-165      68      1.6%
165-170    2544     61.5%
170-175    1081     26.1%
175-180     398      9.6%
180-185       0      0.0%
   More      29      0.7%
          ---------------
           4137    100.0%

I scrolled through the list, and all the abnormals (x < 160 and x > 180) are CCGX-es from developers and other machines such as bbb, rpi.

[mpvader]

Status

All future updates of this issue are maintained on the project page on wiki, here.

[mpvader]

one final thanks to @mansr @jepefe and @jhofstee, and lets close this issue now. swupdate is still rock solid.