Fake EEPROM module just added

[gfwilliams]

Posted at 2015-11-13 by @gfwilliams

Hi,

I just uploaded this: http://www.espruino.com/FlashEEPROM

Hopefully it'll give you nice easy access to nonvolatile memory, without having to faff around with pages.

/* No arguments mean use the default, which will work on the Original Espruino or Espruino Pico
 You can supply extra arguments to choose which flash page you'll use, and even whether you
 use external flash memory - but be careful when doing this. You can accidentally overwrite
 Espruino itself! */
var f = new (require("FlashEEPROM"))(); 

f.write(0, "Hello");
f.write(1, "World");
//.. you can write to any address between 0 and 255, with any data up to 256 chars long

f.read(0)
// returns new Uint8Array([72, 101, 108, 108, 111])

E.toString(f.read(1))
// returns "World"

f.readAll();
// returns [
//  new Uint8Array([72, 101, 108, 108, 111]),
//  new Uint8Array([87, 111, 114, 108, 100])
// ]

f.write(1, "Espruino");
// has now overwritten 'World'

E.toString(f.read(1))
// returns "Espruino"

/* You can tidy up the journal if you want, removing any values that were
overwritten by subsequent writes. It can take around a second though */
f.cleanup();

[espruino-discuss]

Posted at 2015-11-13 by the1laz

Nice! Features like this really make Espruino great, really really easy to use.

[espruino-discuss]

Posted at 2015-11-13 by DrAzzy

What do you mean by:

and even whether you use external flash memory

[espruino-discuss]

Posted at 2015-11-13 by tve

Q: what is flash.length set to? Basically are you putting the eeprom area by default just after the flashed code/data, such that it gets lots if a new version of espruino changes the length in pages? Wouldn't it be better to add a param to the board config to specify where the eeprom area should start? At least going forward that would be safer.

Have you thought of any CRC to validate that the eeprom contents is still valid and hasn't been corrupted?

This module only works on a single flash page, correct? Might be nice to state that.

It looks like the addr parameter of FlashEEPROM.prototype.readAll is not used?

What are the memory requirements of this module in order to rewrite the data? If the flash area contains, say, 1KB of data the memory requirements will be much more than that, right? Have you thought of using two pages in order to incrementally copy from one to the other? This would also avoid data loss in case the flash page write gets interrupted.

[gfwilliams]

Posted at 2015-11-16 by @gfwilliams

What do you mean by: and even whether you use external flash memory

You should be able to specify an object for flash as the last argument. If we have flash modules that implement read/write/getPage/erasePage (I think there was one?) then you can just use them.

are you putting the eeprom area by default just after the flashed code/data

STM32F4 has only a few pages, so there isn't any choice in the matter. It uses the page starting at 384kB - the one that the chip isn't supposed to have (but that exists anyway) - all the rest are used. I thought it said in the docs.

... and by using a page that the STM32 shouldn't have, it doesn't get overwritten by the bootloader :)

specify where the eeprom area should start?

That's the optional first argument of the module? I don't think it needs to go in the firmware...

CRC

Hmm... maybe - I wonder what the chances of it being wrong are? The flash memory seems very reliable - so much so I have now turned off verify when flashing as it just wastes time :)

Single page / two pages

It'd be nice to use a second page, but on STM32F4 there aren't enough pages, and on the F1 the pages are 1024 or 2048k, so storing the contents for that in RAM isn't a big deal (although yes, if it got interrupted while writing a page you'd lose stuff)

It depends what you've written to the EEPROM what memory gets used. You could obviously fill a 128kB page with data and it wouldn't rewrite, but if you just wrote values at maybe 100 addresses and changed them enough to fill up memory, rewriting would not be an issue.

I'll get rid of the readAll argument... That was a copy/paste bug :)

[espruino-discuss]

Posted at 2015-11-16 by tve

specify where the eeprom area should start?
That's the optional first argument of the module? I don't think it needs to go in the firmware...

Mhh, you seem to be taking a very stm32-centric view. On the esp8266 the flash_start and flash_length values don't make a ton of sense because there are multiple areas with settings and such. Flash_start+Flash_length=Flash_end, basically, so you can't put anything beyond that. So the way this is coded, a user has to go and look up the memory map and decide into which gap to put the 4KB eeprom page.
If you would allow process.memory() to have an "eeprom" field, or "scratch_flash" field that could be populated by the device dependent code then it could be coded to just work. As it is, it only "just works" on the pico and for the esp8266 the user has to (a) discover that there's a problem and (b) dig out the answer, and (c) modify the code. That doesn't make for portable code (consider what happens when someone publishes a module that uses the flasheeprom).

[gfwilliams]

Posted at 2015-11-16 by @gfwilliams

Well, as you say, something defining a spare area could be good? I'm happy to add something like that - and maybe it could be pulled out of BOARD.py. At least if it isn't defined, the module could throw an error rather than soldiering on regardless.

[gfwilliams]

Posted at 2015-11-16 by @gfwilliams

Just to add, it looks like there's a big rant on Gitter that this doesn't 'just work' on ESP8266.

All you have to do is var f = new (require("FlashEEPROM"))(whatever_address_works_on_esp8266);

if you want to use this at the moment.

[espruino-discuss]

Posted at 2015-11-16 by tve

Big rant on gitter? where? there were some questions and responses, but no rant that I see.

[gfwilliams]

Posted at 2015-11-16 by @gfwilliams

Ahh, sorry - I literally just scanned over it this morning while answering mail and forum posts and saw:

However, he has chosen to NOT expose the physical address within the address space where the flash is read and written

But as you spotted that's not right. So no, not much of a rant :)

[espruino-discuss]

Posted at 2015-11-16 by JumJum

I had to translate rant, interesting. Anyway lets go back to the facts:

• Flash-object works on ESP8266
• FlashEEPROM works on ESP8266
• with the actual firmware from EspruinoBuilds the free area starts at 0x6c000 and ends at 0x7c000, @tve please correct me if I'm wrong
• another option to load some data into flash is to create a binary file with a tool of your choice, and upload this with ESP8266Flasher.exe

[espruino-discuss]

Posted at 2015-11-17 by tve

with the actual firmware from EspruinoBuilds the free area starts at 0x6c000 and ends at 0x7c000, @tve please correct me if I'm wrong

That is probably correct if it works for you ;-). The start really depends on where the current firmware binary ends. However, with the soon-to-happen change to a single firmware image for all flash chip sizes this is likely to change and may be different between flash chip sizes.

[gfwilliams]

Posted at 2015-11-17 by @gfwilliams

I'd like to keep FlashEEPROM in a state where it still works on a normal 1v81 release, but I think reporting free flash areas in process.memory could be really neat - and it could be modified to check and use those if they existed.

Perhaps something like:

unused : [
 { addr : ..., length: ... },
 { addr : ..., length: ... },
]

I guess for now, only one might get reported, but in a lot of cases I imagine there may be several small areas free that could potentially be used. I don't know if we'd only report whole free pages, or maybe we could report if there were parts of pages that were free as well.

Any thoughts?

[espruino-discuss]

Posted at 2015-11-17 by JumJum

@gordon, AFAIK, erasing can only be done on whole page(usually 4K). Therefore I would recommend to publish empty pages only. Otherwise it would happen too often, that data needed somewhere else, will be deleted.
For example, someone could store some data in a page, used partly by save(). Erase this page and you could come in trouble.
@tve, I see the problem with single firmware for all ESP8266-boards. Would Gordons list of unused flash pages help ? Would it be possible to create something like unused_flash, similar to saved_code in ESP8266_BOARD.py ?

[gfwilliams]

Posted at 2015-11-17 by @gfwilliams

Would Gordon's list of unused flash pages help ?

To be fair, this was @tve's idea originally ;)

The list would be created by the process.memory function, so potentially some runtime checks could be done to see what part was being used?

[espruino-discuss]

Posted at 2015-11-21 by tve

I think a list of free page regions would be awesome. Would it make things too complicated to be able to reserve a chunk from that? That info would have to be persisted in the save() area, perhaps?
WRT esp8266: see https://github.com/tve/EspruinoDocs/blob/master/boards/EspruinoESP8266.md#flash-map-and-access if you want a preview for what I'm working towards.

[gfwilliams]

Posted at 2015-11-21 by @gfwilliams

That info would have to be persisted in the save() area

I'm not sure I understand this? You mean that when you update firmware you don't want to overwrite it?

Really good documentation on the memory layout though - I need one of those for Espruino boards :)

I think if we added int jshGetFreeFlashAreas(uint32_t **addressAndLength, int maxAreas) to jshardware.c then the process.memory implementation could just use that, and report each area back as addr and length?

I guess probably returning a set of ranges (from + length) would be best, as otherwise you're going to get swamped with data items if you ask how many pages are in the 4MB parts :)

... other option is to just have JsVar *jshGetFreeFlashAreas() and let each platform return its own array of data (I guess then some platform-specific hints could be added).

[espruino-discuss]

Posted at 2015-11-21 by @allObjects

...[[from,length,pageSize],...], or [[from, pageSize,numberOfPages],...]... if page size varies.

[espruino-discuss]

Posted at 2015-11-21 by tve

Adding the page size sounds like a good idea.

[gfwilliams]

Posted at 2015-11-23 by @gfwilliams

But you can just do flash.getPage(addr) on it and get the page size or work out the number of pages already? I'm not sure it's worth duplicating that work for every platform?

[espruino-discuss]

Posted at 2015-11-23 by tve

Ah, good point.

[gfwilliams]

Posted at 2015-11-25 by @gfwilliams

DO NOT USE THIS ON ESPRUINO PICO FIRMWARE 1v82 - it'll cause your Pico to overwrite part of its firmware, and you'll have to enter the bootloader and re-update it.

I'll fix this in 1v83 which I'll release shortly.

[gfwilliams]

Posted at 2015-11-27 by @gfwilliams

And 1v83 is released now, so it's fine to use again.