Possible improvment to the benchmark

[X-Ryl669]

While reading the code, I'm spotting 3 potentials issues:

1. Your code should be placed in IRAM (using the IRAM_ATTR macro) so it's not evicted from cache when the PSRAM is loading (and the time code doesn't measure slowness of SPI Flash reading).
2. I'm not sure you're testing what you expect to test. The IRAM->IRAM test isn't doing what you think, since you've allocated the buffer with MALLOC_CAP_DMA and not MALLOC_CAP_32BIT. It's only with the latter that you may get an IRAM buffer (see here). The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here). Please notice that the memory bus of ESP32-S3 is 240MHz (32 bits, so a maximum theorical bandwidth is 960MB/s, a copy should halve that bandwidth so at best, you should get 480MB/s). I don't understand how you could get more than that from the results you've shown. The ESP32 doesn't do any DDR like behavior on the SRAM, only the PSRAM supports it.
   If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.
3. PSRAM tests are likely wrong. You need to evince the cache before starting the test, else you'll measure the time it takes to move from the cache to the cache. The default cache size in the ESP32S3 is 32KB (see here, p400 and later). In ESP-IDF, they usually configure it to 64KB. So the 100KB test already include a large part of it from the cache. You can use esp_cache_msync to evict it or simply read 64KB from somewhere here first to prune the cache, and then run the test. The size must also be larger than 128KB so when it finishes, the result is actually stored in the PSRAM and not in the cache. Else, what you'll see is like this:

Source PSRAM buffer
 32KB   |  32KB |  28KB  
[   A   |   B   |   C   ]

Dest PSRAM buffer  
 32KB   |  32KB |  28KB  
[   D   |   E   |   F   ]

Cache
[   x   |   x   ]

After warmup, you'll have this in the cache: (state diagram by time) 

|
|   S1    S2  S3    S4
+---*-----*----*----*--------------------> time

S1: Start of benchmark, cache contains [x|x]
S2: Read from PSRAM to cache and write to cache, cache contains [A|D] (actually measuring the PSRAM=>SRAM bandwidth for reading A + SRAM =>SRAM copying)
S3: End of reading A, starts reading B: doesn't fit. cache must evict its data to PSRAM, (actually measuring SRAM =>PSRAM bandwidth for transfering D to PSRAM, then PSRAM=>SRAM for reading B)
S4: End of reading C. Benchmarking stop, yet the data F isn't in PSRAM yet, since you haven't msync'ed yet.

The only part that would be relevant here for benchmark would be around state S3, since it's only there that the PSRAM is read and written. It happens after you've exhausted the cache and before it can leave lingering data in the cache, so after 32KB of source data and 32KB before its end. So in a 100KB test, you only experience this in the 28KB part between both ends. A 192KB test would have measured it in the inner 128KB area and left the 2 ends of 32KB unrelated.

Please notice that you can boost the PSRAM speed. On the ESP32-S3-WROOM-1U-N8R8, the PSRAM is mounted as a Octal SPI, and it's built by ESP-IDF by default as MSPI DDR mode running at 80MHz (see here). So the best bandwidth you could expect is 8bit * 80MHz * 2 = 160MB/s (a lot less, since there's the SPI and address selection overhead). Again, a copy operation will halve that (even more since you need to select the source and destination address for each transaction). So getting more than 50-60MB/s would be very excellent.

Yet, you can configure the MSPI to run at 120MHz (under the CONFIG_IDF_EXPERIMENTAL_FEATURES), with many caveats, but in that case you could expect a theorical bandwidth of 240MB/s or maybe ~100MB/s for a copy.

Anyway, thank you for your repository, it's excellent and I'm sure it can be even better. I've had hard time figuring out actual values for the DSP and SIMD stuff.

[project-x51]

While reading the code, I'm spotting 3 potentials issues:

Thanks @X-Ryl669

I modified the code to include the IRAM_ATTR prefix to any of the time sensitive code. This made a small improvement with the largest improvement (2.257%) being to the 'PIE 128-bit (32 byte loop) IRAM->IRAM' copy function. This now goes at 1,789.81 MB/s so not a bad way (its pretty much free) to go that little bit faster. After all every bit counts!

I'm still getting my head around the other two improvements you have suggested but wanted to post this so you knew I appreciated your comment.

I've created a Google Sheet to track improvements. You can find it here. Please take a look if you have the time so I know its available to anyone interested.

[project-x51]

Please notice that the memory bus of ESP32-S3 is 240MHz (32 bits, so a maximum theorical bandwidth is 960MB/s, a copy should halve that bandwidth so at best, you should get 480MB/s). I don't understand how you could get more than that from the results you've shown. The ESP32 doesn't do any DDR like behavior on the SRAM, only the PSRAM supports it.
If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.

@X-Ryl669 That would be the case if the ESP32 only had a 32bit internal memory bus. In fact its a 128bit data bus!! The only part of the processor that utilizes it is the PIE. Its capable of doing SIMD instructions such as 4 x 32 bit operations per clock cycle. Or as we use it a 1 x 128bit (16 bytes) load or store operation per clock cycle. This is how we manage to get the 1830.2 MB/s IRAM->IRAM result using the PIE. This is just shy of 4 x your theoretical max for moving 32 bits at 240 MHz.

[project-x51]

2. I'm not sure you're testing what you expect to test. The IRAM->IRAM test isn't doing what you think, since you've allocated the buffer with MALLOC_CAP_DMA and not MALLOC_CAP_32BIT. It's only with the latter that you may get an IRAM buffer (see here). The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here).

@X-Ryl669 I took a look at the documentation you refer to on the MALLOC_CAP_32BIT. If you look just above it you will see that the DMA version excludes PSRAM but makes no mention of where else it might be. I don't see any mention of DRAM in the ESP32-S3 datasheet so I'm not sure how it could allocate DRAM, which only leaves IRAM. Anyway I tried changing the memory allocation to 32BIT and it made no difference. The results were identical for all the tests as best I can tell.

[project-x51]

3. PSRAM tests are likely wrong. You need to evince the cache before starting the test, else you'll measure the time it takes to move from the cache to the cache. The default cache size in the ESP32S3 is 32KB (see here, p400 and later). In ESP-IDF, they usually configure it to 64KB. So the 100KB test already include a large part of it from the cache. You can use esp_cache_msync to evict it or simply read 64KB from somewhere here first to prune the cache, and then run the test. The size must also be larger than 128KB so when it finishes, the result is actually stored in the PSRAM and not in the cache. Else, what you'll see is like this:

@X-Ryl669 Very interesting and not something I was aware of. I found more information by looking up the esp_cache_msync function here. I'm not sure how to use this function yet as it wants me to specify memory starting addresses and size (no idea why) so I'll have a think on this one. I've also asked Microcontroller as he may have some more insight.

I did discover that the cache is only used when the CPU accesses PSRAM. It is bypassed for the DMA so some of the results may be skewed but the DMA ones should be correct. Defiantly a trap for someone not aware of this. I could just imagine trying to debug that one.

[project-x51]

Yet, you can configure the MSPI to run at 120MHz (under the CONFIG_IDF_EXPERIMENTAL_FEATURES), with many caveats, but in that case you could expect a theorical bandwidth of 240MB/s or maybe ~100MB/s for a copy.

@X-Ryl669 I tried the 120 MHz PSRAM on my board. However it failed to get past the RAM detection code. It found it and posted information on the RAM and then hung. :(

I guess this is why its considered experimental. Note the warning says its not stable if the temperature moves by more than 20 degrees so I guess this makes it pretty useless except for some pretty niche use cases (if it worked).

Thanks for the suggestions. I'll let you know if I figure out how to properly test the PSRAM performance taking into account the cache load/flush challenges.

[X-Ryl669]

Thanks for your complete answer. As for the PSRAM performance measurement, I would do this way (pseudo code):

char * inBuf = malloc_psram(192kB)
char * outBuf = malloc_psram(192kB)
// Pre fill the buffer
memset(inBuf, whatever)
// Start benchmarking 
clock false_start = now
copyBuffer(inBuf, outBuf, 32KB)
clock start = now
copyBuffer(inBuf+32KB, outBuf+32KB, 128KB)
clock end = now
copyBuffer(inBuffer+160kB, outBuf+160kB, 32kB)
clock false_end=now

// Mem benchmark speed:
totalWithCache = 192kB/(false_end - false_start)
totalWithoutCache = 128kB/(end - start)
totalPrePostLoad = 64kB/((false_end - end) + (start - false_start))

I've made the assumption that the cache is smart enough to split a 64kB R/W access in two independent 32kB access (one for read / one for write). Maybe I'm wrong, since there's no documentation about what algorithm is used on the caching process. Maybe the granularity is a lot smaller, so it'll be almost impossible to measure the effect of the caching, or it's larger (a single 64kB block?) in that case, the current test is already measuring the correct value.

One hint however: if you memory map a flash page with MMU (which is, what I think the cache is using for PSRAM), the granularity is 64kB for ESP32S3, IIRC (see here and here for the actual page size)

[BitsForPeople]

1. Your code should be placed in IRAM (using the `IRAM_ATTR` [macro](https://docs.espressif.com/projects/esp-idf/en/stable/esp32s2/api-guides/memory-types.html#how-to-place-code-in-iram)) so it's not evicted from cache when the PSRAM is loading (and the time code doesn't measure slowness of SPI Flash reading).

The cache is split into code and data cache, so data cannot push out code, and vice versa. But yes, for maximum accuracy pre-loading the timed code into RAM would be best. Or running the same code more than once and taking the minimum run time.

2. I'm not sure you're testing what you expect to test. ... The other flag won't give you IRAM (they might give you DRAM, so you're measuring the SRAM speed here).

The -S3 actually has no DRAM (data RAM) or IRAM (instruction RAM); all its RAM is byte-addressable and can be executed from. (Except of course the RTC RAM, but we're not interested in that for now :))
So "IRAM" in this benchmark just means "internal RAM" a.k.a. SRAM, as opposed to the external PSRAM.
Adding MALLOC_CAP_INTERNAL to the allocations would make it extra clear that we want internal RAM.

   If your results are confirmed, then this means that the DSP/PIE can access the SRAM in larger format than 32 bits (and the SRAM supports it), which would be excellent.

We confirmed that the results match what the TRM states, i.e. single-cycle 128-bit transfers from and to SRAM :)
(Actually, it's not 'fully' single-cycle: While a 128-bit load takes the CPU 1 clock cycle to execute, the data loaded is only available in the respective CPU register 1 clock cycle later, so worst case is 2 CPU clock cycles to load 128 bits; but by exploiting the CPU's pipeline you can achieve a throughput of 128 bits per clock cycle. That's why loading 2x16 bytes, then storing 2x16 bytes is faster than 2x loading+storing 16 bytes)

[BitsForPeople]

I've made the assumption that the cache is smart enough to split a 64kB R/W access in two independent 32kB access (one for read / one for write). Maybe I'm wrong, since there's no documentation about what algorithm is used on the caching process. Maybe the granularity is a lot smaller, ...

It is a lot smaller. The granularity is one cache line, which can be configured to be 32 or 64 bytes.

[BitsForPeople]

To avoid the effects of the cache when measuring non-DMA transfers from/to PSRAM, we would have to

1. esp_cache_msync( sourceBuffer, sourceSize, ESP_CACHE_MSYNC_FLAG_DIR_C2M ); // Make sure the source data we want to copy has actually made it to PSRAM
2. esp_cache_msync( sourceBuffer, sourceSize, ESP_CACHE_MSYNC_FLAG_DIR_M2C ); // Make the cache forget the cached source data
3. esp_cache_msync( destBuffer, destSize, ESP_CACHE_MSYNC_FLAG_DIR_M2C ); // Make the cache forget any data in the destination it may still want to write to PSRAM
4. start timing
5. copy data from/to PSRAM
6. esp_cache_msync( destBuffer, destSize, ESP_CACHE_MSYNC_FLAG_DIR_C2M ); // Make sure the data we copied to dest has actually made it to PSRAM
7. stop timing

Here, 1. and 2. only apply when we read from PSRAM; 6. only applies when we write to PSRAM. 3. applies if there may be some dirty data in the cache for destBuffer; it's redundant if we're sure that's not the case, e.g. because 6. was previously executed.

Edit:
It seems that 1. and 2. can be combined into a single esp_cache_msync( sourceBuffer, sourceSize, ESP_CACHE_MSYNC_FLAG_DIR_C2M | ESP_CACHE_MSYNC_FLAG_INVALIDATE ).

To be extra accurate, we should actually evict all PSRAM data from the cache first. Otherwise there may be some dirty PSRAM data from a previous run in the cache, which will likely get pushed out of the cache during our test, forcing the cache to write it back to PSRAM while we're timing the test.

[X-Ryl669]

It is a lot smaller. The granularity is one cache line, which can be configured to be 32 or 64 bytes.

I think we are not speaking about the same thing here. The cache line is 64 bytes on ESP32-S3. But this doesn't matter for the test. What matter is the cache block size (in SRAM) which is 32K or 64K since when you're benchmarking the memory copy, you're are actually measuring the time it takes to copy from SRAM to SRAM if the data you're copying to/from can fit in this cache block size. You need to overcome the block size to actually measure the PSRAM speed.

When using the CPU (not DMA), you can't skip the caching mechanism, so you must measure the time it takes to fill / empty the cache too, which the current code didn't do (the filling of the cache was here obviously, but the emptying wasn't, since the test stopped without knowing when the cache lines were transmitted). This is relevant to a real world usage. I don't know if the caching algorithm was dumb (like, every time a cache line is touched it'll be marked as dirty and will be transferred later on, and in that case, the benchmark is almost correct, except for the last cache line) or a bit more smarter (like, until all the cache block is used, no cache line is transferred to PSRAM and then only when new data are loaded in the cache that'll trigger transferring the stale/old data to PSRAM, and in that case, the benchmark is irrelevant, since it was fitting in the cache block size anyway).

From the post of Sprite_TM here it seems it's the latter, so it's definitively an issue in the benchmark.

However, since the DMA bypass the cache, the actual PSRAM speed is captured here (althrough it also capture a bit of the FreeRTOS's async scheduling overhead so for such a small buffer, it matters too).

I think to get a better idea of the memcpy speed from & to PSRAM, a larger block is required in all cases (cache or no cache) so that the caching overhead (which is useless in this usage pattern since data is never used twice) or async scheduling overhead is minimized.

[X-Ryl669]

To avoid the effects of the cache when measuring non-DMA transfers from/to PSRAM, we would have to [...]

Yes that's correct.

[BitsForPeople]

With proper(?) cache flushing the results look different w.r.t. IRAM<->PSRAM speed:

I (931) Memory Copy: 8-bit for loop copy IRAM->PSRAM took 796811 CPU cycles = 29.41 MB/s
I (981) Memory Copy: 16-bit for loop copy IRAM->PSRAM took 460818 CPU cycles = 50.86 MB/s
I (1031) Memory Copy: 32-bit for loop copy IRAM->PSRAM took 403215 CPU cycles = 58.13 MB/s
I (1081) Memory Copy: 64-bit for loop copy IRAM->PSRAM took 412809 CPU cycles = 56.78 MB/s
I (1131) Memory Copy: memcpy IRAM->PSRAM took 412836 CPU cycles = 56.77 MB/s
I (1181) Memory Copy: async_memcpy IRAM->PSRAM took 461617 CPU cycles = 50.77 MB/s
I (1231) Memory Copy: PIE 128-bit (16 byte loop) IRAM->PSRAM took 403207 CPU cycles = 58.13 MB/s
I (1281) Memory Copy: PIE 128-bit (32 byte loop) IRAM->PSRAM took 403209 CPU cycles = 58.13 MB/s
I (1331) Memory Copy: DSP AES3 IRAM->PSRAM took 405053 CPU cycles = 57.86 MB/s

I (1381) Memory Copy: Swapping source and destination buffers
I (1431) Memory Copy: 8-bit for loop copy PSRAM->IRAM took 864096 CPU cycles = 27.12 MB/s
I (1481) Memory Copy: 16-bit for loop copy PSRAM->IRAM took 736145 CPU cycles = 31.84 MB/s
I (1531) Memory Copy: 32-bit for loop copy PSRAM->IRAM took 717695 CPU cycles = 32.66 MB/s
I (1581) Memory Copy: 64-bit for loop copy PSRAM->IRAM took 717694 CPU cycles = 32.66 MB/s
I (1631) Memory Copy: memcpy PSRAM->IRAM took 720810 CPU cycles = 32.52 MB/s
I (1681) Memory Copy: async_memcpy PSRAM->IRAM took 444946 CPU cycles = 52.67 MB/s
I (1731) Memory Copy: PIE 128-bit (16 byte loop) PSRAM->IRAM took 720756 CPU cycles = 32.52 MB/s
I (1781) Memory Copy: PIE 128-bit (32 byte loop) PSRAM->IRAM took 720758 CPU cycles = 32.52 MB/s
I (1831) Memory Copy: DSP AES3 PSRAM->IRAM took 722728 CPU cycles = 32.43 MB/s

[X-Ryl669]

I suspect the "cache flushing" function to simply trash the cache lines by filling them with random memory addresses. I need to do my homework here and read them.

I don't get why you are flushing the memory for async_memcpy since DMA doesn't use the cache anyway.

The value are consistent with the architecture here, PSRAM is using 8 bits/cycle at a 80MHz clock in DDR fashion, so at best it can do 160MB/s (thus at best 80MB/s for a memory copy to/from PSRAM without cache). Since you need to select the address for transferring from the chip (10 cycles per transfer, provided it's a burst one with 64 bytes, so 32 cycles in DDR, that's a 1/3 overhead here), it's a lot less than that, so a maximum theoretical value of ~60MB/s is achievable (80MB/s * 32 cycles/42 cycles). Reaching a value of ~50MB/s doesn't look too far from the reality.

It's good to see that the effect of caching actually improve performance (38MB/s instead of 32MB/s), since this proves that the address selection overhead in the SPI memory is important. It's not as important as expected (I would have expected a 1/3 increase here) but it's nonetheless useful.

[BitsForPeople]

I'm only wondering why the DMA PSRAM->IRAM transfer would be so much faster than the CPU-driven one.
Seeing that, I added cache prepping around the DMA too, but as expected it made no difference.

I assume the DMA, knowing the size and sequential nature of the transfer beforehand, uses longer bursts than the cache, reducing the relative overhead.