Settings overwritten after NoFUSS OTA update

[mcspr]

I too faced similar issues and device lost configuration after update via Nofuss.
I enabled debug in eeprom rotate and saw this -

Magic value for sector #1019 is 71
Calculated CRC: 0x8ADE
Stored CRC    : 0x8ADE
Magic value for sector #1018 is 12
Calculated CRC: 0x3A78
Stored CRC    : 0x40B9
Sector #1018 has not passed the CRC check
Magic value for sector #1017 is 0
Calculated CRC: 0x2DCD
Stored CRC    : 0x765F
Sector #1017 has not passed the CRC check
Magic value for sector #1016 is 161
Calculated CRC: 0xA924
Stored CRC    : 0xFFA0
Sector #1016 has not passed the CRC check
Current sector is #1019
Current magic value is #71

Clearly blocks 1016,1017 and 1018 are overwritten. While 1019 is intact so EEPROM Rotate picked up configuration from 1019, but since the latest data was not present in the sector 1019 . So the device lost some configuration keys.

But once again question is, why are these sectors are being overwritten during OTA. For information - I did a custom build with no SPIFSS enabled(In both previous and new binary).

I also disabled calling of custom_crash_callback as well, via an addition flag
#define SAVE_CRASH_INFO 0.

I think it is not an issue exactly with Espurna, but something with nofuss or esp8266httpUpdater itself(Which is used by Nofuss).
However its note worthy for all espura users as well, that if we build it with latest core, and deliver OTAs to devices, few sectors will be overwritten. Depending on the code size it may overflow all 4 sectors also.
Anyway, i'll create another issue on arduino esp8266 core as well, but the same could also be true for ArduinoOta library which is used for web based OTA.

Sectors from 378000 to 3f9000,3f9000 to 3fa000 , 3fa000 to 3fb000, 3fb000 to 3fc000 were used by eeprom rotate. And were clean. In flash dump also as expected. Then i took the dump of these same sectors right after OTA and saw these sectors were filled with something.

Originally posted by @arihantdaga in #1595 (comment)

[mcspr]

Which Core version was it? Release, git? If git, were there any modifications to libraries / espurna / ldscripts?
edit: Assuming it is Arduino IDE, what board config was used to build the firmware?

About the flash dump, these are not really random. Check out the last bytes of the .bin you are using with `xxd'. I do see dhcps strings right at the end.

[arihantdaga]

@mcspr Thank you for creating this issue. I am sorry for not doing it myself. I was trying to pin point the issue.
I am using latest git - slightly ahead of version 2.5.2.
However i found out the solution for the problem. In my case it happened because i was building firmware with setting- "Flash Size 4Mb, No SPIFFS". Because i was under the assumption from the memory block diagram of ESP that After SPIFFS comes EEPROM. But if we use eeprom rotate, i think we are entering the territory of SPIFFS. And hence even if we are not using file system, we have to keep minimum SPIFFS.
After that i built the sketch again this time keeping 4MB, 1M SPIFFS. And it worked fine.
Thank a lot for posting here and helping me figure out this.
I am not sure, the built binaries in the #1595 uses which SPIFFS configuration.

[mcspr]

I think I have not read updater code clearly, for some reason I thought it measures from the sketch end address. If it positions itself at the spiffs start and steps back based on OTA size, it will surely overwrite the eeprom sectors that are not defined at build. In platformio builds we offset it specifically for this reason (see #1559 files for arduino)

You can check the updater code though! DEBUG_UPDATER (UPDATER in debug menu) will print a bunch of stuff about the process.
https://github.com/esp8266/Arduino/blob/master/tools/sdk/ld/eagle.flash.4m1m.ld
https://github.com/esp8266/Arduino/blob/master/tools/sdk/ld/eagle.flash.4m.ld
have the exact same end address. Updater itself uses starting point to avoid writing to the filesystem part.
https://github.com/esp8266/Arduino/blob/16312949c976629db948b1daa635948064cfd037/cores/esp8266/Updater.cpp#L114
Meaning, you do overwrite hidden eeprom part with the last 12KB

[mcspr]

EEPROM_rotate handles such case by disabling rotation. However, there needs to be eeprom commit to record the last known config to the last sector. having newest sector somewhere in the danger zone results in a data loss, since we never copy it to the last one.

• Can you try adding eepromCommit() here? This is to handle common cases of async OTA process:
  

  espurna/code/espurna/eeprom.ino

  Line 25 in dc26a69

  EEPROMr.rotate(value);

• NoFuss specifically needs a couple modifications though.
  This line should be removed:
  https://bitbucket.org/xoseperez/nofuss/src/9f13d9e8f268aa5a1dc2bd2b7f7e0b3270b72689/src/NoFUSSClient.cpp#lines-201
  And deferredReset(100, CUSTOM_RESET_OTA); should be added here:
  

  espurna/code/espurna/nofuss.ino

  Line 148 in dc26a69

  nice_delay(100);

[arihantdaga]

Instead of Updating nofuss, what do you say if we can update in espurna itself @mcspr this pull request will work ?

[mcspr]

It will do the same thing, yes.

And I just noticed that CUSTOM_RESET_NOFUSS is missing from NoFuss reset. defferedReset would've set it via customResetReason(...), but it can also happen right there. Info banner would show "Reboot after successful NoFUSS update"

[mcspr]

Closing via #1809