ESP32 OTA Example with a DHT11 data sent - fail to finish download

[csvke]

I am planning to write code using Arduino framework to use a ESP32 to send DHT11 sensor data to Thingsboard while checking if there's a new version of OTA.

My thingsboard rule chain has been set and have tested with the 0009-esp8266_esp32_process_OTA_MQTT sample.

I suspect it's either the way I code the dht sensor sending to thingsboard that causes the problem.

Here's the main.cpp

https://gist.github.com/csvke/b649ae904155312580a9d9b9c6ddaf58

#include <Arduino.h>
#include <Update.h>
#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <Arduino_MQTT_Client.h>
#include <ThingsBoard.h>
#include <Espressif_Updater.h>
#include <DHT.h>

constexpr char CURRENT_FIRMWARE_TITLE[] PROGMEM = "TEST";
constexpr char CURRENT_FIRMWARE_VERSION[] PROGMEM = "1.0.3";

// Maximum amount of retries we attempt to download each firmware chunck over MQTT
constexpr uint8_t FIRMWARE_FAILURE_RETRIES PROGMEM = 12U;
constexpr uint16_t FIRMWARE_PACKET_SIZE PROGMEM = 4096U;

constexpr char WIFI_SSID[] PROGMEM = "ssid";
constexpr char WIFI_PASSWORD[] PROGMEM = "password";

constexpr char TOKEN[] PROGMEM = "xa0xh7tej8jk05br4f8i";

constexpr char THINGSBOARD_SERVER[] PROGMEM = "xxx.xxx.xxx.xxx";

constexpr uint16_t THINGSBOARD_PORT PROGMEM = 1883U;

// Maximum size packets will ever be sent or received by the underlying MQTT client,
// if the size is to small messages might not be sent or received messages will be discarded
constexpr uint16_t MAX_MESSAGE_SIZE PROGMEM = 512U;

constexpr uint32_t SERIAL_DEBUG_BAUD PROGMEM = 115200U;

constexpr char FW_STATE_UPDATED[] PROGMEM = "UPDATED";

// Initialize underlying client, used to establish a connection
WiFiClient espClient;
// Initalize the Mqtt client instance
Arduino_MQTT_Client mqttClient(espClient);
// Initialize ThingsBoard instance with the maximum needed buffer size
ThingsBoard tb(mqttClient, MAX_MESSAGE_SIZE);
// Initalize the Updater client instance used to flash binary to flash memory
Espressif_Updater updater;

// Statuses for updating
bool currentFWSent = false;
bool updateRequestSent = false;

// Sensor definitions
#define DHTPIN 13
#define DHTTYPE DHT11

DHT dht(DHTPIN, DHTTYPE);

/// @brief Initalizes WiFi connection,
// will endlessly delay until a connection has been successfully established
void InitWiFi() {
  Serial.println(F("Connecting to AP ..."));
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    // Delay 500ms until a connection has been successfully established
    delay(500);
    Serial.print(F("."));
  }
  Serial.println(F("Connected to AP"));
}

/// @brief Reconnects the WiFi uses InitWiFi if the connection has been removed
/// @return Returns true as soon as a connection has been established again
bool reconnect() {
  // Check to ensure we aren't connected yet
  const wl_status_t status = WiFi.status();
  if (status == WL_CONNECTED) {
    return true;
  }

  // If we aren't establish a new connection to the given WiFi network
  InitWiFi();
  return true;
}

/// @brief Updated callback that will be called as soon as the firmware update finishes
/// @param success Either true (update successful) or false (update failed)
void updatedCallback(const bool& success) {
  if (success) {
    Serial.println(F("Done, Reboot now"));
    esp_restart();
    return;
  }
}

/// @brief Progress callback that will be called every time we downloaded a new chunk successfully
/// @param currentChunk 
/// @param totalChuncks 
void progressCallback(const size_t& currentChunk, const size_t& totalChuncks) {
  Serial.printf("Progress %.2f%%\n", static_cast<float>(currentChunk * 100U) / totalChuncks);
}

void setup() {
  Serial.begin(SERIAL_DEBUG_BAUD);
  dht.begin();
  InitWiFi();
  Serial.print("MAC Address: ");
  Serial.println(WiFi.macAddress());

  // Connect to ThingsBoard
  if (!tb.connect(THINGSBOARD_SERVER, TOKEN)) {
    Serial.println("Failed to connect to ThingsBoard");
  }
}

void loop() {
  delay(1000);

  if (!reconnect()) {
    return;
  }
  if (!tb.connected()) {
    // Reconnect to the ThingsBoard server,
    // if a connection was disrupted or has not yet been established
    Serial.printf("Current firmware version is: (%s)\n", CURRENT_FIRMWARE_VERSION); // q: how to print the CURRENT_FIRMWARE_VERSION
    Serial.printf("Connecting to: (%s) with token (%s)\n", THINGSBOARD_SERVER, TOKEN);
    if (!tb.connect(THINGSBOARD_SERVER, TOKEN, THINGSBOARD_PORT)) {
      Serial.println(F("Failed to connect"));
      return;
    }
  }

  if (!currentFWSent) {
    // Firmware state send at the start of the firmware, to inform the cloud about the current firmware and that it was installed correctly,
    // especially important when using OTA update, because the OTA update sends the last firmware state as UPDATING, meaning the device is restarting
    // if the device restarted correctly and has the new given firmware title and version it should then send thoose to the cloud with the state UPDATED,
    // to inform any end user that the device has successfully restarted and does actually contain the version it was flashed too
    currentFWSent = tb.Firmware_Send_Info(CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION) && tb.Firmware_Send_State(FW_STATE_UPDATED);
  }

  if (!updateRequestSent) {
    Serial.println(F("Checking for updates..."));
    Serial.println(F("Firwmare Update..."));
    const OTA_Update_Callback callback(&progressCallback, &updatedCallback, CURRENT_FIRMWARE_TITLE, CURRENT_FIRMWARE_VERSION, &updater, FIRMWARE_FAILURE_RETRIES, FIRMWARE_PACKET_SIZE);
    // See https://thingsboard.io/docs/user-guide/ota-updates/
    // to understand how to create a new OTA pacakge and assign it to a device so it can download it.
    updateRequestSent = tb.Start_Firmware_Update(callback);
  }

  // Read DHT11 sensor data
  float humidity = dht.readHumidity();
  float temperature = dht.readTemperature();

  // Check if any reads failed and exit early (to try again).
  if (isnan(humidity) || isnan(temperature)) {
    Serial.println("Failed to read from DHT sensor!");
    return;
  }

// Print data to Serial Monitor
Serial.print("Humidity: ");
Serial.print(humidity);
Serial.print(" %\t");
Serial.print("Temperature: ");
Serial.print(temperature);
Serial.println(" *C");

// Print timestamp
Serial.print("Timestamp: ");
Serial.println(millis());

// Get MAC address
String macAddress = WiFi.macAddress();
Serial.print("MAC Address: ");
Serial.println(macAddress);

// Prepare data for ThingsBoard
tb.sendTelemetryData("temperature", temperature);
tb.sendTelemetryData("humidity", humidity);
tb.sendTelemetryData("macAddress", macAddress.c_str());

tb.loop();
}

Here's the ESP32 serial monitor output
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost

[MathewHDYT]

First of all I'm pretty sure this issue occurs with the last chunk that we attempt to download, because the size is smaller than the chunk size of 4096. Meaning it is the last chunk which contains the remaining data.

It is weird tough that the calculation seems to be off, because if the received firmware size shared attribute is correct, then it should be 189 * 4096 and 1 * 2816 bytes big. However in your case it seems like the received firmware size was 189 * 4096 and 1 * 2832.

How do you set the firmware size shared attribute. Are you simply using the Firmware Dashboard and assigning a new firmware or are you chaging the shared attributes by hand?

[csvke]

First of all I'm pretty sure this issue occurs with the last chunk that we attempt to download, because the size is smaller than the chunk size of 4096. Meaning it is the last chunk which contains the remaining data.

It is weird tough that the calculation seems to be off, because if the received firmware size shared attribute is correct, then it should be 189 * 4096 and 1 * 2816 bytes big. However in your case it seems like the received firmware size was 189 * 4096 and 1 * 2832.

How do you set the firmware size shared attribute. Are you simply using the Firmware Dashboard and assigning a new firmware or are you chaging the shared attributes by hand?

I'm new to thingsboard.

I simply used the Firmware Dashboard to assign a new firmware.

[MathewHDYT]

Okay weird, if you assign the firmware from the dashboard it should work. The only thing I can currently imagine is that it is sending while receiving the data.

For now at the top of your main.cpp file can you add #define THINGSBOARD_ENABLE_DEBUG 1.

Additionally can you switch the PubSubClient version you are currently using to my fork. Currently both sending and receiving write into the same buffer my fork has seperated that behaviour into 2 buffers instead, which means you should be able to send while receiving data. Perhaps that was the issue.

[csvke]

Okay weird, if you assign the firmware from the dashboard it should work. The only thing I can currently imagine is that it is sending while receiving the data.

For now at the top of your main.cpp file can you add #define THINGSBOARD_ENABLE_DEBUG 1.

Additionally can you switch the PubSubClient version you are currently using to my fork. Currently both sending and receiving write into the same buffer my fork has seperated that behaviour into 2 buffers instead, which means you should be able to send while receiving data. Perhaps that was the issue.

I have enabled #define THINGSBOARD_ENABLE_DEBUG 1 SO HANDY

And also replaced the PubSubClient to your fork.

Still the same. Let me chop out all the sendTelemetryData and see what I will have.

[MathewHDYT]

You should see more log messages if you retry the udpate with the #define THINGSBOARD_ENABLE_DEBUG 1, mode enabled. Can you send thoose additional log messages, because currently I'm not really sure what the issue could be. I've tried the update mechanism a lot especially on ESP32 and have never seen such behaviour before.

[csvke]

More weird is that despite I deleted v1.0.3 of the firmware and explicitly set the assigned firmware in dashboard for the device to use v1.0.2, the device has still detected a new firmware of v1.0.3. Is there some sort of cache behind the scene for firmware versions binaries in thingsboard?

As I have been getting myself familiar with thingsboard I might have created a version of 1.0.3, deleted it, and re-uploaded a different v1.0.3 with a different binary size.

[csvke]

have

[TB] Received data from server over topic (v2/fw/response/0/chunk/188)
[TB] Receive chunk (188), with size (4096) bytes
Progress 98.95%
[TB] Received data from server over topic (v2/fw/response/0/chunk/189)
[TB] Receive chunk (189), with size (4096) bytes
Progress 99.48%
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Received chunk size (2832), not the same as expected chunk size (2816)
[TB] Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Failed to receive requested chunk (190) in (5000000) us. Internet connection might have been lost","fw_state":"FAILED"})

[MathewHDYT]

More weird is that despite I deleted v1.0.3 of the firmware and explicitly set the assigned firmware in dashboard for the device to use v1.0.2, the device has still detected a new firmware of v1.0.3. Is there some sort of cache behind the scene for firmware versions binaries in thingsboard?

As I have been getting myself familiar with thingsboard I might have created a version of 1.0.3, deleted it, and re-uploaded a different v1.0.3 with a different binary size.

There is not really a cache per se, however the fw_size, fw_title, fw_version, ... of the assigned firmware binary are saved into shared attributes. Those shared attributes are then what the device receives and checks. If of course the firmware size of v1.0.2 is received but it attempts to update to v1.0.3 it will fail.

---

The first thing could you assign v1.0.2 and check if it is really assigned with the shared attributes of your device.

[csvke]

I checked the device's shared attributes and there's no entry. However, there are current_fw_title, current_fe_version, fw_error and fw_state sent from the device to thingsboard.

Is there a detailed up-to-date (thingsboard v3.7) version of an OTA tutorial I can follow?

[MathewHDYT]

Okay if the shared attributes do not look like this, that would be very weird. Because the update shouldn't even be able to start if these keys do not exist on the device.

Can you make sure that in the Shared Attributes Tab of the Device you want to update, if those keys exist and if yes what there values are.

[csvke]

No, it doesn't have any Shared attributes (admittedly there was a device that had shared attributes but after a few attempts of deleting devices and re-doing the Rule chains etc., nothing shows up in shared attributes and latest telemetry anymore, only server attributes)

I have trimmed it down to only running the example 0009 as-is with #define THINGSBOARD_ENABLE_DEBUG 1 on, this is what serial monitor comes out:

Connecting to AP ...
....Connected to AP
Connecting to: (xxx.xxx.xxx.local) with token (p2qknhhdpc7jeg85f7ad)
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11","current_fw_version":"1.0.0"})
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATED"})
Firwmare Update...
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11","current_fw_version":"1.0.0"})
[TB] Sending data to server over topic (v1/devices/me/attributes/request/1) with data ({"sharedKeys":"fw_checksum,fw_checksum_algorithm,fw_size,fw_title,fw_version,"})
[TB] Received data from server over topic (v1/devices/me/attributes/response/1)
[TB] Calling subscribed callback for request with response id (1)
[TB] No new firmware assigned on the given device
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"No new firmware assigned on the given device","fw_state":"FAILED"})

[MathewHDYT]

Yeah that error atleast makes sense you have no shared attributes assigned that could inform the device that it has a new firmware version to update to so it doesn't.

What exactly have you changed on the rule chain. Because the OTA update should run perfectly without any changes to it, perhaps those are somehow causing problems.

Because normally if you assign the Firmware over the Firmware Dashboard it simply sets the Shared Attribute values as shown above, if that is not the case then something goes wrong in the rule chain probably.

[csvke]

Please confirm the steps for the OTA process:

1. Setup all the device and get the access token
2. Setup device profile
3. Add 'Check is ESP32 firmware synced' rule chains, add 'Check is ESP32 firmware synced' rule chains to Root Rule Chain with the links Attributes Updated and Success
4. Edit Device Profile's Default rule chain to Check is ESP32 firmware synced
5. Load the firmware with access token via serial UART to ESP32
6. Let ESP32 connect to Thingsboard
7. (Wait until ESP32 sends shared attributes to Thingsboard?)
8. (Do I need to load v1.0.0 firmware with the exact checksum to Thingsboard?)
9. Upload firmware 1.0.1 to Thingsboard
10. Change device profile to assigned firmware v1.0.1
11. Wait for OTA firmware update??

[MathewHDYT]

I'm highly confused by at least half of theese steps, the OTA update works perfectly with the default rule chain, the checking if the firmware is synced is done by this library you don't need the rule chain to do that. (Step 3, 4)

Furthermore I don't know why the ESP32 should send shared attributes that isn't even possible (without workarounds), the device can only ever subscribe or request shared attributes but not create them. You don't need step 8.

Step 9 to 11 are correct, I'm pretty sure these rule chains you added are somehow borking the expected update process of this library. Can you retry simply with the default Rule chain. So just skip step 3, 4, 7 and 8

---

I might have an idea, perhaps you are using a really old Firmware Update dashboard which we don't support. Download the firmware dashboard from this documentation page and use that one if you aren't already doing that.

[csvke]

May I ask where should I put the tb.sendTelemetryData(); inside the void loop ();?

[csvke]

After lots of tweaking, I can only make the OTA process partially work.

What's working is that the firmware is actually pushed to the device, but only if I specifically choose the version of firmware from the Firmware dashboard, assign the firmware in device or device profile will not work.
Also after the update is completed, the device reboot and is with the desired updated version of firmware via OTA (I added a Serial.printf("Current firmware version is: (%s)\n", CURRENT_FIRMWARE_VERSION); for debugging), however, the device somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED. So the dashboard is displaying the device stuck in Initiated status.

I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.

Progress 99.48%
Current firmware version is: (1.0.1)
[TB] Received data from server over topic (v2/fw/response/0/chunk/190)
[TB] Receive chunk (190), with size (4032) bytes
Progress 100.00%
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"DOWNLOADED"})
[TB] Checksum is the same as expected
[TB] Update success
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATING"})
Done, Reboot now
ets Jul 29 2019 12:21:46

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
Connecting to AP ...
......Connected to AP
MAC Address: E8:31:CD:32:05:E0
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11-SENSOR","current_fw_version":"1.0.2"})
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_state":"UPDATED"})
Checking for updates...
Firwmare Update...
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"current_fw_title":"ESP32-DHT11-SENSOR","current_fw_version":"1.0.2"})
[TB] Sending data to server over topic (v1/devices/me/attributes/request/1) with data ({"sharedKeys":"fw_checksum,fw_checksum_algorithm,fw_size,fw_title,fw_version,"})
Current firmware version is: (1.0.2)
[TB] Received data from server over topic (v1/devices/me/attributes/response/1)
[TB] Calling subscribed callback for request with response id (1)
[TB] Firmware version (1.0.2) already up to date
[TB] Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"})

[MathewHDYT]

I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.

No problem I hope it hasn't been too much of a hassle.

---

For the OTA process it is not partially working, OTA is working exactly as expected now. Because if you assign the firmware over the Device Profile or directly to the Device it will not work, because it does not change the Shared Attribute values and therefore does not inform the device correctly that it can update to a new version.

If you assign it over the Dashboard however it does change thoose Shared Attributes so it does not cause any issues.

---

Somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED

Will be fixed it the newest version, but currently if you call Start_Firmware_Update it checks the received shared attributes, if we even need to update and if we don't it simply sends the failed status with the given error message. With my newest fork this will be resolved as well and send the UPDATED state automatically.

[csvke]

I am so sorry that it's taken so long to implement this feature. But I'm really struggling between reading the documentation, putting the settings into my thingsboard instance and guided (or mis-guided) by a very limited video in Youtube together with a 5-year old official video of the OTA feature.

No problem I hope it hasn't been too much of a hassle.

---

For the OTA process it is not partially working, OTA is working exactly as expected now. Because if you assign the firmware over the Device Profile or directly to the Device it will not work, because it does not change the Shared Attribute values and therefore does not inform the device correctly that it can update to a new version.

If you assign it over the Dashboard however it does change thoose Shared Attributes so it does not cause any issues.

---

Somehow send Sending data to server over topic (v1/devices/me/telemetry) with data ({"fw_error":"Firmware version (1.0.2) already up to date","fw_state":"FAILED"}) after the status UPDATED

Will be fixed it the newest version, but currently if you call Start_Firmware_Update it checks the received shared attributes, if we even need to update and if we don't it simply sends the failed status with the given error message. With my newest fork this will be resolved as well and send the UPDATED state automatically.

I finally got it working for what I want with 'examples/0011-esp8266_esp32_subscribe_OTA_MQTT/'

My expected behaviour is to have a device loaded with v1.0.0 version of firmware, i upload the firmware v1.0.1 in ThingsBoard OTA, change the device profile assigned firmware, the device detect version increment, the device download and flash the v1.0.1 firmware, the device sent updated state when completed, ThingsBoard OTA dashboard shows updated.

On the side note, what's the approach if I have a bunch of ESP32 grouped in the same device profile that I want to deploy an update to all of them? Because the sample code has the access token hard coded in, do I have to create partitions in the ESP32 and store the credentials in a partition that will not be updated (so I can write firmware code without the access token in it)?

Many many thanks for the help along the way. I feel the whole OTA topic deserves a lot more TLC from an overview write up in the documentation to an updated guide either in video or writing. I can only understand half of what it does after investing 20 hours in it :( but worth the effort

[MathewHDYT]

Yeah sorry to hear I hope the OTA process atleast makes more sense now.

On the side note, what's the approach if I have a bunch of ESP32 grouped in the same device profile that I want to deploy an update to all of them? Because the sample code has the access token hard coded in, do I have to create partitions in the ESP32 and store the credentials in a partition that will not be updated (so I can write firmware code without the access token in it)?

The easiest way which I would recommend is to use NVS, it is exactly meant for this use case, being storing small key value pairs in a partition that is safe from updating.

The official documentation from ESP32 on NVS might help.

For small stimple steps what you need to do is. Write the access token into NVS once and then you can read it again at the start of the application and then use that token to start connecting with ThingsBoard.

I feel the whole OTA topic deserves a lot more TLC from an overview write up in the documentation to an updated guide either in video or writing

If you want to understand how it works under the hood on the ESP32 with the partitions and what it writes this documentation from ESP32 might be useful. And for the general process on the Cloud and how it get's notified this page does exaplain it pretty well and is up to date.

I think the main reason for confusion was simply that you used this guide which is pretty old and is not compatible with this library.

[csvke]

@MathewHDYT once again thank you for your help, I have written up a firmware that uses NVS based on the examples/0011-esp8266_esp32_subscribe_OTA_MQTT.
However, I wonder: if assigning a firmware to a particular device profile cannot trigger a batch firmware update (that's what I thought Device profile > Assigned Firmware is for) and I have to individually choose what device to assign to what firmware in the Firmware Dashboard, what is the best practice to batch update firmware?

Once I iron this issue out, I thought it'd be useful if I release the source code of that, together with another example of self-provisioning, in case someone wants to jump-start and build a close-to-production workflow for a ESP32 sensor data logger with self-provisioning to Thingsboard.

I have also written up a build script and a upload script for the PlatformIO IDE and the code base is under Arduino framework.

[MathewHDYT]

What is the best practice to batch update firmware?

There is no premade widget available that fufills that functionality and I think there are 2 major ways to solve it both including custom JavaScript code on the ThingsBoard cloud instance.

Either you detect the change of the assigned firmware in teh device profile on the rule chain and then write the necessary shared attribute values to every device of that type from the assigned firmware.

Or alternatively you make a completly custom widget that allows to select multiple devices or a device profile in a dropdown and select an OTA package and when a submit button is clicked all selected device will have the necessary shared attribute values written,

---

I thought it'd be useful if I release the source code of that, together with another example of self-provisioning, in case someone wants to jump-start and build a close-to-production workflow for a ESP32 sensor data logger with self-provisioning to Thingsboard.

Sounds good to me the example provided in the library are only meant to provide a jump start, so more complete exampled providing combined functionality would be surely interesting as well.