[Android] Performance deteriorates over time

[Leopard20]

I'm not sure why, but recently I've noticed that performance gets worse over time (usually after 10-20 mins). I'm pretty sure it's not caused by thermal throttling, because if I save the state, exit the game and immediately continue from the saved state everything goes back to normal. So far I've only tested vulkan. I'll let you know if I observe the same behavior on OGL.
Device: Mi 5s Plus
OS: Android 8.0

[unknownbrackets]

Does this happen even with sustained performance mode enabled? Any more information on backends?

-[Unknown]

[RinMaru]

Performance has degraded allot on windows aswell no backend can reach fullspeed anymore with the settings I use. though I don't update PPSSPP very much anymore so it could have regressed like 3 stables back

[unknownbrackets]

I think that's not related, since this is about performance degrading after starting the emulator, not about newer versions.

It's been a while since I saw bad performance on a Windows device, but that probably belongs in a new issue. It'd help to try some builds in between and note what graphics card and which games are affected.

I haven't noticed any real decline in speed myself, but I also never use "disable slow effects" or disable "block transfer".

-[Unknown]

[Leopard20]

@unknownbrackets
Upon further testing, I found that the slowdown is more noticeable on Vulkan, but OGL is also affected to some lesser extent.

Also, another proof I have that this is not a device slowdown problem is that if I simply create a savestate (there's no need to even exit the game, as I said before) the speed goes back to normal (but simply pausing and resuming the emulator won't do).
What do you do when a savestate is created that could possibly make this happen? Maybe you could do that regularly (e.g every minute or something) to avoid slowdown? I think it is some sort of "garbage data accumulation" thing that gets cleaned up when a savestate is created?

[unknownbrackets]

We do a lot of things on save state - it's an expensive operation.

You can try turning on the rewind feature, which will probably make you drop frames every time it creates a save state, but it'll be regular. Does that make it behave more consistently?

Since it's an expensive operation, is it maybe that it's just triggering your CPU governor to (temporarily) step up the CPU speed it gives PPSSPP, then after we get back to emulating efficiently, it steps it back down to reduce thermal throttling? What happens with sustained performance mode on - is that consistent?

-[Unknown]

[Leopard20]

We do a lot of things on save state - it's an expensive operation.

I don't mean do it all (of course it would be silly)! I meant if there are any cache/old data flushing operations that you might do during the savestate creation process and it could also be performed regularly, it might be helpful.

You can try turning on the rewind feature, which will probably make you drop frames every time it creates a save state, but it'll be regular. Does that make it behave more consistently?

How does the rewind function work exactly? I enabled it in settings and set the rewind frequency to 1800 frames (making it every 30 seconds for a 60FPS game, correct?) but the rewind option in pause menu is grayed out. Am I supposed to manually trigger it?

Since it's an expensive operation, is it maybe that it's just triggering your CPU governor to (temporarily) step up the CPU speed it gives PPSSPP, then after we get back to emulating efficiently, it steps it back down to reduce thermal throttling? What happens with sustained performance mode on - is that consistent?

I don't think so. Firstly, I've tampered with my thermal config to increase the throttling temperature. Secondly, I've also tweaked the governer to regulate the frequency jumps and achieve a more linear and proportional load to frequency ratio. Thirdly, I monitored my CPU frequency and while it did go higher when I created the savestate, it returned to normal afterwards. This lasts for a relatively long time (more than 5 minutes) so it's definitely not a thermal throttling or similar issue. Also, my ROM doesn't support sustained performance mode so it's out of question.

BTW, if the speedup was actually because of a spike in CPU frequency, then doing something as simple as taking a screenshot (which, as I noticed during the test, also caused a noticeable increase in CPU frequency) would've done the trick but it does not.

Another interesting thing I noticed was that the game kept draining memory (possible memory leakage) but after a savestate a noticeable amount of RAM was freed up.

[hrydgard]

The only thing I could think of would be if our Vulkan slab memory allocator gets slower and slower due to fragmentation. If that was the cause though, OGL shouldn't be affected at all..

[unknownbrackets]

I meant if there are any cache/old data flushing operations that you might do during the savestate creation process

Well, we don't really clear caches on save state. We used to, but it made rewind slower, and we fixed it. If we're still clearing any caches on save state, it's a bug - but I'm not aware of any. Save states do take screenshots, as well, but sounds like that can't be related. This should make sense - it'd be silly to pointlessly clear caches for a feature a user might set to run every second.

Note that loading a save state certainly does clear caches. All these areas are protected by a if (p.mode == p.MODE_READ) { condition to make sure they apply only to loading a save state.

The only thing I could think of would be if our Vulkan slab memory allocator gets slower and slower due to fragmentation. If that was the cause though, OGL shouldn't be affected at all..

That's protected by if (p.mode == p.MODE_READ && !PSP_CoreParameter().frozen) {, so it doesn't happen when creating a save state anyway.

How does the rewind function work exactly?

Sorry, I forgot - it's entirely disabled on mobile. That's why it's doing nothing for you. Maybe it could be re-enabled, although it probably will cause noticeable periodic stutter on any mobile device...

but after a savestate a noticeable amount of RAM was freed up.

And this doesn't happen if you just go to the pause menu and then resume? Does it happen to the same degree if you put your phone's display to sleep and then wake it back up (which should flush ALL the graphics caches)?

Just to explain what happens when you save state:

1. First, PPSSPP goes over all the data related to emulation state, and determines how much data there is.
2. Next, a buffer is allocated and the above step is repeated, but now saving that data into the buffer.
3. PSP RAM is copied into the buffer, which is typically the most expensive (time wise) part of the process, since the 32 MB of PSP RAM are the largest part of the save state.
4. When saving RAM, save states and replacements are temporarily removed, but they are put back exactly as they were after the RAM is saved.
5. For saving to file (not rewind), the data is next compressed with Snappy. This is probably the second most expensive part of the process, but usually results in a much smaller file.
6. After that's written to storage, again when saving to a file, a screenshot is triggered which requires a stall for graphics (time expensive but not CPU expensive.)

Step 1/2 basically go through all of PPSSPP's code, so it's hard to make a list of everything. This search currently has 120 results which is probably a good measure of all the places it checks:
https://github.com/hrydgard/ppsspp/search?utf8=%E2%9C%93&q=dostate&type=

So this runs a bunch of save state specific code and reads a bunch of RAM that probably is not frequently read otherwise. This probably trashes CPU level data and instruction caches.

-[Unknown]

[Leopard20]

@unknownbrackets Both locking and unlocking the device and task switching cause a huge amount of memory drop, and doing this multiple times causes the OS to force stop the app (I think you created an issue about it before), also it doesn't fix the slowdown problem. Pausing and resuming the emulator has no effect at all (at least none that's noticeable) neither on RAM nor the game speed.
So there has to be something you do during the process of savestate creation alone. Based on your description, the last step seems to be the only one related (at least to my understanding). It could also be a driver issue.

If you have a hunch about what might be the cause, you can create a temporary apk (you don't have to commit the changes to Github, to avoid problems) and I will test it for you.

[unknownbrackets]

Well, you can try tying your volume down key or something to screenshot now with #12009.

-[Unknown]

[Leopard20]

Sadly it didn't have any effect.
Doss this screenshot function do everything you mentioned in step 6, including the "stall for graphics" part? I personally don't sense any stall.

[Leopard20]

@unknownbrackets A stupid question: Would it be possible to add a fake "optimization" function that does everything similar to the process of savestate creation minus saving to disk and compression steps? It would be worth a test if you think it might actually help. Maybe this function could then be iterated every minute or something.

[LunaMoo]

Rewind does that and can be set to activate once every 1800 frames max which is a minute for 30fps games, but the stutter it causes is noticeable in sound even if fps doesn't move on a powerful pc and it's very annoying to play with.

[Leopard20]

@LunaMoo Yeah but 1. Rewind doesn't work on Android. 2. It actually consumes memory. What I mean by above was that the used memory should be discarded as well. But it still sounds stupid, so...

[LunaMoo]

If you can build ppsspp, you could just build yourself a special build with rewind enabled and limited to for example 1 step to reduce mem usage or even keeping 0 steps just doing all the work, the point is there's no need for anything new doing that, also that's a really terrible workaround for a potential OS power management issue which this probably is.

People tend to say it's not thermal throttling whenever an app on android slows down hoping the problem can be felt on the app side, the thing is mobile operating systems, especially with modern android doesn't thermal throttle, they throttle based on some algorithms trying to keep battery life the longest - throttling much sooner than temperatures could be felt. It's possibly an increase in mem usage or something silly as that works around the throttling algorithm in your system, maybe even decides app needs more power based on memory it requires, you could try loading ISO to RAM, but I think that also might be disabled for android by default.

[Leopard20]

Good point. I need to investigate this further.
At least implementing #12011 would be great for now.
By the way, why is there no quick save/quick load (without screenshots, etc) in PPSSPP?

[unknownbrackets]

When I say stall, I'm talking from the perspective of the total time a save state actually takes - it's all relatively quick, so you probably wouldn't notice anything obvious, at worst a single dropped frame.

-[Unknown]

[unknownbrackets]

Since I wanted to enable it anyway, I created #13866 to enable rewind on mobile.

That said, I think this issue is a thermal throttling issue and probably not resolvable in any real way - even if this works around the heuristic for now, the device maker may just change the heuristic later.

-[Unknown]

[ghost]

I don't experience this issue on my Redmi Note 9 Mediatek Helio G85 12nm 🤔

[hrydgard]

We haven't gotten a lot of additional reports on this, so I'm just gonna close it. Some kind of thermal or similar throttling does seem likely.

[unknownbrackets]

I am hoping that some of my blind attempts at fixing sustained performance (while less blindly trying to fix the immersive resize thing) may have helped. I always felt like they were likely symptoms of the same root cause.

(I realize this issue specifically wasn't necessarily about sustained performance but if that's working better, it might be why we're getting less reports... also of course phones are getting more powerful.)

-[Unknown]

[Leopard20]

I am hoping that some of my blind attempts at fixing sustained performance (while less blindly trying to fix the immersive resize thing) may have helped. I always felt like they were likely symptoms of the same root cause.

(I realize this issue specifically wasn't necessarily about sustained performance but if that's working better, it might be why we're getting less reports... also of course phones are getting more powerful.)

-[Unknown]

Yeah it has gotten better. Not fully fixed.
I still don't believe it's related to thermal throttling (as I mentioned before, saving the game fixes the issue), but without any leads, this is a total dead end and not worth keeping open anymore.