metric for heap fragmentation #5090

d-a-v · 2018-08-28T16:58:44Z

~~Available only in debug mode,~~ to help debugging.

ESP.getHeapUnfragness() calculates "unfragmentation-ness" of the heap.
Whatever free heap value, it gives 1000 when the whole free memory is contiguous and decreases with fragmentation.
For example it gives 707 (which is 1000/sqrt(2)) for the same free size when it is in fact two equal-size continuous blocks. 999 is given with the same free size with a huge free block and a small one.

Here is a sketch log when allocating many blocks, then removing half of them (every other), then removing again half of them, with free heap and fragmentation metric:

allocating 24576 bytes with 24 blocks of 1024 bytes each
unfragness: 999/1000 (freeheap: 53304) <- start
unfragness: 999/1000 (freeheap: 28232) <- allocation
unfragness: 698/1000 (freeheap: 40712) <- freeing every other blocks
unfragness: 622/1000 (freeheap: 46952) <- freeing every other remaining blocks
unfragness: 999/1000 (freeheap: 53192) <- freeing everything
unfragness: 999/1000 (freeheap: 53304) <- end

allocating 21312 bytes with 1332 blocks of 16 bytes each
unfragness: 999/1000 (freeheap: 53304) <- start
unfragness: 998/1000 (freeheap:  5336) <- allocation
unfragness: 202/1000 (freeheap: 26648) <- freeing every other blocks
unfragness: 150/1000 (freeheap: 37304) <- freeing every other remaining blocks

(plenty of available space, but heap is in very bad shape: 150/1000)

unfragness: 999/1000 (freeheap: 47960) <- freeing everything
unfragness: 999/1000 (freeheap: 53304) <- end

devyte

Overall, I think this is a must-have that should always be available to the user. It adds to a "health" metric of the esp, along with free heap, and the stack metrics.
I'd like it more without global state, though. Maybe a new function specific to the calculation or something?

devyte · 2018-08-29T04:02:56Z

cores/esp8266/Esp-unfragness.cpp

+#include "coredecls.h"
+#include "Esp.h"
+
+uint16_t EspClass::getHeapUnfragness(uint32_t* freeHeap)


While I personally love the name, and the term "unfragmentation-ness", maybe it would be better for users to rename to something like contigousness (or an abbreviation like contig), or calculate the complement and call it fragmentation, which is a well-known term.

devyte · 2018-08-29T05:07:39Z

cores/esp8266/Esp-unfragness.cpp

+
+uint16_t EspClass::getHeapUnfragness(uint32_t* freeHeap)
+{
+    ummFreeSize2 = 1; // enable processing


This looks like global state. Shouldn't these lines be protected against interrupts?

Is there a way to do this without global state?

Can this code be explained with comments or a url or doc reference?

devyte · 2018-08-29T05:15:33Z

cores/esp8266/Esp-unfragness.cpp

+ Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+ */
+
+#ifdef DEBUG_ESP_PORT


This is a good metric to have available as a tool, just like free heap. I suggest not making it debug-only.

devyte · 2018-08-29T05:20:58Z

cores/esp8266/umm_malloc/umm_malloc.c

@@ -1025,6 +1029,11 @@ void ICACHE_FLASH_ATTR *umm_info( void *ptr, int force ) {
      ++ummHeapInfo.freeEntries;
      ummHeapInfo.freeBlocks += curBlocks;

+#ifdef DEBUG_ESP_PORT
+      if (ummFreeSize2)
+        ummFreeSize2 += (uint64_t)curBlocks * (uint64_t)sizeof(umm_block) * (uint64_t)curBlocks * (uint64_t)sizeof(umm_block);


Is it necessary for this to be a uint64_t? I think for the esp it's not necessary to use 64bit arithmetic, but as a general extension to umm it may be good. I can't tell what should be done

d-a-v · 2018-08-29T08:55:37Z

or calculate the complement and call it fragmentation, which is a well-known term

I have no idea. I reckon getUnfragness is quite a twisted name. I like getHeapContig() and its complement getHeapFrag() too.

ummFreeSize2 looks like global state. Shouldn't these lines be protected against interrupts?
Is there a way to do this without global state?

It is global but only used during the calculation. It's the way umm works: it uses a global structure (ummHeapInfo) and fills it from umm_info() / getFreeHeap(). Unfragness is an optional calculation within umm_info() and uses the same structure. To do this without global, umm_info() would have to change. With no side effect nor memory change this global can be moved to the (global) struct.

Can this code be explained with comments or a url or doc reference?

It's the L2 / Euclidian norm of free block sizes. We have getFreeHeap()=sum(hole-size), and
unfragness = 1000 * (sqrt(sum(hole-size²))) / sum(hole-size)
I will add that.

This is a good metric to have available as a tool, just like free heap. I suggest not making it debug-only.

As it is and if it is always enabled, it takes ~160 bytes in flash when not called, and 1KB when called (I guess sqrt is the reason, I'll check for integer version for this).

Is it necessary for this to be a uint64_t?

sqrt(2^32)=65536 so 32 bits is valid if we have always less then 64K free ram (which is the case today). I will change that.

d-a-v · 2018-08-29T10:02:33Z

Now always enabled. Adds 16 bytes in flash, and another 128 bytes when used.
bonus: cheap and accurate integer square root added for everybody isqrt32()
(tested across the full 32 bits range, thanks to Tristan Muntsinger @ http://www.codecodex.com)

earlephilhower

I'm still traveling so haven't looked in depth yet, but this seems like a great idea!

devyte · 2018-08-31T21:08:01Z

If
unfragness = 1000 * (sqrt(sum(hole-size²))) / sum(hole-size)
then
fragness = 1000 - 1000 * (sqrt(sum(hole-size²))) / sum(hole-size)
so an unfragness of 1000 is the same as a fragmentation of 0, and they both mean one contiguous block.

d-a-v · 2018-09-04T20:38:24Z

22:35:59.233 -> 
22:35:59.233 -> Filling memory with blocks of 8000 bytes each
22:35:59.233 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.233 -> frag:   1% (freeheap:  5360 - maxblock= 5352) <- allocation
22:35:59.233 -> frag:  50% (freeheap: 29384 - maxblock= 8008) <- freeing every other blocks
22:35:59.233 -> frag:  30% (freeheap: 45400 - maxblock=24024) <- freeing every other remaining blocks
22:35:59.263 -> frag:   1% (freeheap: 53408 - maxblock=53400) <- freeing everything
22:35:59.263 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.263 -> 
22:35:59.263 -> Filling memory with blocks of 4000 bytes each
22:35:59.263 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.263 -> frag:   1% (freeheap:  1256 - maxblock= 1248) <- allocation
22:35:59.297 -> frag:  63% (freeheap: 29312 - maxblock= 5256) <- freeing every other blocks
22:35:59.297 -> frag:  49% (freeheap: 41336 - maxblock=12024) <- freeing every other remaining blocks
22:35:59.297 -> frag:   1% (freeheap: 53360 - maxblock=53352) <- freeing everything
22:35:59.297 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.297 -> 
22:35:59.297 -> Filling memory with blocks of 2000 bytes each
22:35:59.297 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.329 -> frag:   1% (freeheap:  1040 - maxblock= 1032) <- allocation
22:35:59.329 -> frag:  74% (freeheap: 27144 - maxblock= 2008) <- freeing every other blocks
22:35:59.329 -> frag:  63% (freeheap: 41200 - maxblock= 6024) <- freeing every other remaining blocks
22:35:59.329 -> frag:   1% (freeheap: 53248 - maxblock=53240) <- freeing everything
22:35:59.330 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.330 -> 
22:35:59.330 -> Filling memory with blocks of 1000 bytes each
22:35:59.363 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.363 -> frag:   2% (freeheap:   624 - maxblock=  616) <- allocation
22:35:59.363 -> frag:  81% (freeheap: 26832 - maxblock= 1008) <- freeing every other blocks
22:35:59.363 -> frag:  73% (freeheap: 39936 - maxblock= 3024) <- freeing every other remaining blocks
22:35:59.363 -> frag:   1% (freeheap: 53040 - maxblock=53032) <- freeing everything
22:35:59.363 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.396 -> 
22:35:59.396 -> Filling memory with blocks of 500 bytes each
22:35:59.396 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.396 -> frag:   5% (freeheap:   192 - maxblock=  184) <- allocation
22:35:59.396 -> frag:  87% (freeheap: 26400 - maxblock=  504) <- freeing every other blocks
22:35:59.396 -> frag:  81% (freeheap: 39504 - maxblock= 1512) <- freeing every other remaining blocks
22:35:59.396 -> frag:   1% (freeheap: 52608 - maxblock=52600) <- freeing everything
22:35:59.429 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.429 -> 
22:35:59.429 -> Filling memory with blocks of 200 bytes each
22:35:59.429 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.429 -> frag:   5% (freeheap:   192 - maxblock=  184) <- allocation
22:35:59.429 -> frag:  92% (freeheap: 25776 - maxblock=  208) <- freeing every other blocks
22:35:59.429 -> frag:  88% (freeheap: 38672 - maxblock=  624) <- freeing every other remaining blocks
22:35:59.462 -> frag:   1% (freeheap: 51360 - maxblock=51352) <- freeing everything
22:35:59.462 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.462 -> 
22:35:59.463 -> Filling memory with blocks of 100 bytes each
22:35:59.463 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.463 -> frag:  17% (freeheap:    48 - maxblock=   40) <- allocation
22:35:59.463 -> frag:  94% (freeheap: 24696 - maxblock=  104) <- freeing every other blocks
22:35:59.532 -> frag:  91% (freeheap: 37072 - maxblock=  312) <- freeing every other remaining blocks
22:35:59.532 -> frag:   1% (freeheap: 49344 - maxblock=49336) <- freeing everything
22:35:59.532 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.532 -> 
22:35:59.532 -> Filling memory with blocks of 50 bytes each
22:35:59.532 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.532 -> frag: 100% (freeheap:     8 - maxblock=    8) <- allocation
22:35:59.532 -> frag:  96% (freeheap: 22800 - maxblock=   56) <- freeing every other blocks
22:35:59.532 -> frag:  93% (freeheap: 34168 - maxblock=  168) <- freeing every other remaining blocks
22:35:59.532 -> frag:   1% (freeheap: 45536 - maxblock=45528) <- freeing everything
22:35:59.532 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end
22:35:59.532 -> 
22:35:59.532 -> Filling memory with blocks of 15 bytes each
22:35:59.532 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- start
22:35:59.561 -> frag:  34% (freeheap:    24 - maxblock=   16) <- allocation
22:35:59.561 -> frag:  97% (freeheap: 15504 - maxblock=   40) <- freeing every other blocks
22:35:59.561 -> frag:  95% (freeheap: 23232 - maxblock=   72) <- freeing every other remaining blocks
22:35:59.561 -> frag:   1% (freeheap: 30960 - maxblock=30952) <- freeing everything
22:35:59.561 -> frag:   1% (freeheap: 53464 - maxblock=53456) <- end

devyte · 2018-09-04T23:15:00Z

cores/esp8266/Esp.cpp

@@ -171,6 +171,11 @@ uint32_t EspClass::getFreeHeap(void)
    return system_get_free_heap_size();
 }

+uint16_t EspClass::getMaxFreeBlockSize(void)
+{
+    return system_get_free_heap_size();


What's the difference between this and the return via maxBlock pointer in getHeapFragmentation?

This is a mistake :) (the example uses getHeapFragmentation parameters)

devyte · 2018-09-04T23:27:40Z

cores/esp8266/Esp-frag.cpp

+#include "coredecls.h"
+#include "Esp.h"
+
+uint8_t EspClass::getHeapFragmentation (uint32_t* freeHeap, uint16_t* maxBlock)


it bothers me that we have a getFreeHeap() that takes no args and return only the free heap, a getMaxBlockSize() that takes no args and returns only the max block size, and this that does take 2 optional args for free heap and maxblock and returns the fragmentation. It just seems inconsistent to me. I think I'd prefer to have 3 standalone functions one for each, plus a 4th function getHeapStats() or whatever that takes 3 args for getting everything at once (performance) and returns nothing (void).
Thoughts?

For the three functions (Free, MaxBlock, Fragmentation), the same umm_info() function is called which goes through every heap chunks. I figured that those who would call Fragmentation may optionally give pointers to fill those results for free (as this one may not be quite often used).
OK with getHeapStats().

devyte · 2018-09-04T23:35:17Z

Awesome test sequence. I was thinking it'd be great adding that as a device test, but then we'd have to update the test every time we implement something that frees more heap, because the numbers could change. Not sure if that's desirable. Maybe just add it as an example, and include the expected output at the time of implementation?

d-a-v · 2018-09-05T00:13:36Z

I will add the example.

devyte

Still some remaining questions at this point.

devyte · 2018-09-07T17:30:15Z

cores/esp8266/Esp-frag.cpp

+#include "coredecls.h"
+#include "Esp.h"
+
+void EspClass::getHeapStats(uint32_t* free, uint16_t* max, uint8_t* frag)


arg "free" shadows free(), arg max can shadow max() (many apps #define it). would be better to change the names.

devyte · 2018-09-07T17:44:01Z

cores/esp8266/umm_malloc/umm_malloc.c

@@ -1761,4 +1762,13 @@ size_t ICACHE_FLASH_ATTR umm_free_heap_size( void ) {
  return (size_t)ummHeapInfo.freeBlocks * sizeof(umm_block);
 }

+uint16_t ICACHE_FLASH_ATTR umm_max_block_size( void ) {


the umm code doesn't seem to rely on these uint8_t/uint16_t/uint32_t types. Maybe it would be better to stick to the types used in the rest of the code, i.e.: go low level adventurer and declare them as unsigned char/unsigned short/unsigned int

devyte · 2018-09-07T17:47:01Z

cores/esp8266/umm_malloc/umm_malloc.c

@@ -1024,6 +1024,7 @@ void ICACHE_FLASH_ATTR *umm_info( void *ptr, int force ) {
    if( UMM_NBLOCK(blockNo) & UMM_FREELIST_MASK ) {
      ++ummHeapInfo.freeEntries;
      ummHeapInfo.freeBlocks += curBlocks;
+      ummHeapInfo.ummFreeSize2 += (uint32_t)curBlocks * (uint32_t)sizeof(umm_block) * (uint32_t)curBlocks * (uint32_t)sizeof(umm_block);


Maybe don't rely on these types and use basic types like in the rest of umm

devyte · 2018-09-07T21:42:24Z

cores/esp8266/Esp-frag.cpp

+    if (max)
+        *max = ummHeapInfo.maxFreeContiguousBlocks * block_size;
+    if (frag)
+        *frag = 100 - (sqrt32(ummHeapInfo.ummFreeSize2) * 100) / fh;


Would something like this be more precise, or same output? I ask because paranoid about numerical stability.
100 - (sqrt32( ummHeapInfo.ummFreeSize2 * 10000) / (fh*fh) );

I'm not sure it'd be more numerically stable, but it would takes more flash (16 bytes), more ram (4 more bytes) and more times (64 instead of 32 bits ints).
About range, the current version works with at most 64K free heap.

devyte · 2018-09-07T21:48:31Z

cores/esp8266/Esp.cpp

@@ -171,6 +172,11 @@ uint32_t EspClass::getFreeHeap(void)
    return system_get_free_heap_size();


Are we sure that this sdk api is the same as:

uint32_t fh = ummHeapInfo.freeBlocks * block_size; ... *free = fh;

used in getHeapStats()? After a quick look I can't tell how the integration works between that sdk api and umm.

Indeed this is a fw call that calls fw api'sxPortGetFreeHeapSize in which we call umm (heap.c).

TD-er · 2018-09-08T13:46:00Z

This is a great addition.
Just one question, does it take a lot of resources to compute the fragmentation?

And just curious, what would be a good default allocation size for Strings to minimize fragmentation?
For example blocks of multiple of 32 Bytes for those strings that may be longer than a few bytes and have a short expected life span?

…ize())

Replaces esp8266#1829

fixes esp8266#4951

d-a-v · 2018-09-09T21:53:43Z

@TD-er I did not measure time.
getFreeHeap now also internally computes the sum of (int) squares,
and calling getHeapFragmentation additionally computes a (int) square-root (15 iterations of simple 32bits operations including only one multiplication each).

And just curious, what would be a good default allocation size for Strings to minimize fragmentation?
For example blocks of multiple of 32 Bytes for those strings that may be longer than a few bytes and have a short expected life span?

Only testing would answer this tricky question because it depends on everything (how are these strings used).
I believe we can use it to compare an algorithm against another, or like you say, adjust the granularity of some structure often calling realloc. I made it to measure the mess in memory according to policies of some libraries.
Another use is for the user to trigger a reboot when one or both of HeapFragmentation and MaxFreeBlockSize go wrong.

ayushsharma82 · 2019-02-26T09:03:17Z

V2.5.0 gives me this error now:

'class EspClass' has no member named 'getHeapFragmentation'

             stats["heapFragmentation"] = ESP.getHeapFragmentation();

Did you guys just removed the getHeapFragmentation() completly?

devyte requested changes Aug 29, 2018

View reviewed changes

d-a-v mentioned this pull request Aug 29, 2018

metric for heap fragmentation rhempel/umm_malloc#14

Closed

earlephilhower reviewed Aug 29, 2018

View reviewed changes

d-a-v force-pushed the fragness branch from 64d7335 to 0a338b6 Compare September 4, 2018 20:34

devyte reviewed Sep 4, 2018

View reviewed changes

devyte reviewed Sep 7, 2018

View reviewed changes

d-a-v and others added 12 commits September 9, 2018 23:17

+Esp.getHeapUnfragness()

d91ca71

only in debug mode

83b44d7

default value

26d64a4

always enable, 64->32, light 32 integer square root, comments

8f22381

fix when debugging is disabled

9bce009

give credits

08b2304

cosmetics

f55f0ba

fragmentation metric updates (doc, better api, added getMaxFreeBlockS…

0d2f7a0

…ize())

api reworked, +example

84c92b5

read all lines from slow servers (esp8266#5113)

e93e9f6

Replaces esp8266#1829

ESP8266HTTPClient: allow getString() more than once (esp8266#5091)

fc96312

fixes esp8266#4951

fixe types, fix names

ab5a94d

d-a-v force-pushed the fragness branch from e186b0e to ab5a94d Compare September 9, 2018 21:19

d-a-v and others added 3 commits September 9, 2018 23:57

coding style fix

19a3488

use astyle for example

469729a

Merge branch 'master' into fragness

40694d1

devyte approved these changes Sep 10, 2018

View reviewed changes

devyte merged commit ce28a76 into esp8266:master Sep 10, 2018

d-a-v deleted the fragness branch September 13, 2019 23:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metric for heap fragmentation #5090

metric for heap fragmentation #5090

d-a-v commented Aug 28, 2018 •

edited

Loading

devyte left a comment

devyte Aug 29, 2018

devyte Aug 29, 2018

devyte Aug 29, 2018

devyte Aug 29, 2018

devyte Aug 29, 2018

devyte Aug 29, 2018

d-a-v commented Aug 29, 2018

d-a-v commented Aug 29, 2018

earlephilhower left a comment

devyte commented Aug 31, 2018

d-a-v commented Sep 4, 2018

devyte Sep 4, 2018

d-a-v Sep 5, 2018

devyte Sep 4, 2018

d-a-v Sep 5, 2018

devyte commented Sep 4, 2018

d-a-v commented Sep 5, 2018

devyte left a comment

devyte Sep 7, 2018

devyte Sep 7, 2018

devyte Sep 7, 2018

devyte Sep 7, 2018

d-a-v Sep 9, 2018

devyte Sep 7, 2018

d-a-v Sep 9, 2018

TD-er commented Sep 8, 2018

d-a-v commented Sep 9, 2018

ayushsharma82 commented Feb 26, 2019

		@@ -171,6 +172,11 @@ uint32_t EspClass::getFreeHeap(void)
		return system_get_free_heap_size();

metric for heap fragmentation #5090

metric for heap fragmentation #5090

Conversation

d-a-v commented Aug 28, 2018 • edited Loading

devyte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

d-a-v commented Aug 29, 2018

d-a-v commented Aug 29, 2018

earlephilhower left a comment

Choose a reason for hiding this comment

devyte commented Aug 31, 2018

d-a-v commented Sep 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devyte commented Sep 4, 2018

d-a-v commented Sep 5, 2018

devyte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TD-er commented Sep 8, 2018

d-a-v commented Sep 9, 2018

ayushsharma82 commented Feb 26, 2019

d-a-v commented Aug 28, 2018 •

edited

Loading