Extending the partition

[ksstms]

Hi!

We are currently trying to figure out a way to change our FS-less data storage to LittleFS.
The idea is:

• move everything to the end of the flash (keeping track of the bad sectors with our own solution)
• create LittleFS at the beginning of the flash.
• copy the existing data to LFS files
• expand the LittleFS partition to the whole chip

It's sort of easy to do, except the last step. The problem is that the number of blocks are written in the superblock at format, and I haven't found a way to expand the partition without reformatting. I have checked the code, and I couldn't find if the block number in the superblock is used at all.
So I guess we could:

• not care about it. Just format as if we would be using the whole chip, and the block allocator will (hopefully) allocate the beginning of the flash anyway (Haven't checked if this works though...)
• The same, but the flash driver gives an error if the FS tries to access the end of the flash.
• Format with only half flash size, and hack the superblock afterwards. I guess you can just change the block number and fix up the CRC. Might not be this easy.

All of these are pretty hacky, so I'm curious if anyone has other ideas on how to do it.

Thank you in advance.

[geky]

Huh, that's an interesting problem. I don't think there's an easy solution at the moment.

One option, if you're willing to modify littlefs, is to extend the lfs_fs_traverse function. It's purpose is to report all of the blocks that are in use by the filesystem. If, in addition, you make it report the blocks that contain the data on disk, littlefs should not allocate those blocks.

I had to do something similar for the v1->v2 migration, that may make a good starting point:
https://github.com/ARMmbed/littlefs/blob/master/lfs.c#L3487-L3498

[ksstms]

Thanks, I looked into it, but I'd rather leave the LFS code untouched.

There could be a little class between LFS and the flash driver during this firmware upgrade, which returns error if the FS tries to access the protected area. The question is, will this area be usable after the migration, or does LittleFS store that those blocks are bad?

There's another thing that came to my mind. Correct me if I'm wrong, but I think that the LFS code might not care about the block count written in the superblock. It uses lfs->cfg->block_count instead.
So it might be possible to create two configs with two different block_count values. Format with the big one, mount with the small one, perform the migration, change back to the big one.

[geky]

There could be a little class between LFS and the flash driver during this firmware upgrade

Oh! That is a better way to solve this. Currently littlefs does not track bad blocks, it will circle around and try to use those blocks again.

block allocator will (hopefully) allocate the beginning of the flash anyway

I should also mention that this isn't true. littlefs allocates from a random offset into flash that is generated every mount. Because of how its seeded it may always start at 0 after format, but I'm not sure you can rely on that.

There's another thing that came to my mind. Correct me if I'm wrong, but I think that the LFS code might not care about the block count written in the superblock. It uses lfs->cfg->block_count instead.
So it might be possible to create two configs with two different block_count values. Format with the big one, mount with the small one, perform the migration, change back to the big one.

Oh, that's true. But it sounds like a bug. At the very least it should print a warning if these don't match.

My gut says littlefs should pick up and use the superblock's value if it's <= cfg.block_count, but this would have a performance and code size impact...

My 2 cents is that having the block device report the reserved blocks as "bad" (LFS_ERR_CORRUPT) is the best way to solve this. Maybe we should add an additional error code to tell the filesystem the block is just unavailable (LFS_ERR_PERM?). That would make this an intentional feature and keep it from breaking if littlefs tracks bad blocks in the future.

[ksstms]

littlefs allocates from a random offset into flash that is generated every mount.

Oh, I totally missed that. Thanks!

Maybe we should add an additional error code to tell the filesystem the block is just unavailable (LFS_ERR_PERM?). That would make this an intentional feature and keep it from breaking if littlefs tracks bad blocks in the future.

Good idea. I thought about returning LFS_ERR_IO. I guess this isn't widespread problem, so maybe adding another error code isn't really neccesary.

[geky]

Good idea. I thought about returning LFS_ERR_IO. I guess this isn't widespread problem, so maybe adding another error code isn't really neccesary.

Well, one problem is that littlefs is cautious about error codes it doesn't recognize. Besides LFS_ERR_CORRUPT, any other error codes will cause littlefs to halt. Currently littlefs only takes corrective action on LFS_ERR_CORRUPT.

I think some sort of "block in use" error code would be quite useful (maybe actually LFS_ERR_BUSY?). It would at least let me clean up the migration code a bit.