Performance issue

[KHeket]

Is your feature request related to a problem? Please describe.

Is it only me, or performance became worse? I am not sure, that it the cause of problem, but I noticed most performance problems after bird wings glide ability #70479
I dunno, it the source of problem, but I noticed, that after some time of playing (first day of summer) - at the open spaces game became very slow, what is dangerous, when you are driving a car. And the car is not a cause, because I noticed game speed slow down even when I am traveling by foot, no matter, make I it manually, or using fast travel option
If its night, or you are in the house - everything is ok
May be performance problem caused by latest 3d FOV changes, I dunno, but I am creating this issue to understand, is it only my problem, or other have it too?

Solution you would like.

Good game speed

Describe alternatives you have considered.

No response

Additional context

No response

[KHeket]

And also I noticed, that, when I am creating minimized save achive and load it - I have good game performance. But previously, about ~ 6 months ago, I didn't noticed such problems even at the long run games

[worm-girl]

Nothing was added by that PR which would have any effect on game performance. There's only one bit that it's running outside of the action of gliding and related activities, and that's:

    // If you're inside a wall or something and haven't been telefragged, let's get you out.
    if( m.impassable( u.pos() ) && !m.has_flag( ter_furn_flag::TFLAG_CLIMBABLE, u.pos() ) ) {
        u.stagger();
    }

in do_turn. I don't think a single if check run only on the player would have any impact. The rest of the code isn't active unless you're poking at a ledge or trying to fly.

I also haven't noticed a performance hit, except for the stuff since charges were removed, and then only around lots of items.

[Rokharn]

I also feel like the game speed has slowed down. Loading has become slower even for fresh new saves. Actions such as crafting near a large number of NPCs of my own faction (starting from 4 and more) have seriously slowed down (and even slower, when craft just one item, so npcs cannot help and just messages appears if npcs can help batch craft). A large number of items, however, did not change the speed of the game much. Traveling above underground labs, however, was also a challenge before because of the game slowness, so not sure if it's become slower or not.

What's the reason - I have no idea. So many things had been changed in the game for this time.

[IdleSol]

I compared versions 0.G and fd88c9f (2024-02-01-1733). It seems to me that it's about the cities. Something that is in them. Maybe fields, maybe some additional checks. I don't understand much about this.

My initial assumption was that the slowdown was due to the large number of monsters. But now I have my doubts.

Check as follows:

1. Default world

2. Default character

3. Teleportation to an empty field (no creatures)

4. create a debug monster (i=10) and kill 11x11 of them (as in the image).
   

5. Create a large number of regular zombies around your "fortress". i created 6 groups of 121 zombies (i=5). That's a total of 726 zombies.
   

6. Wait a minute. (I didn't notice much difference between different versions the game. By feel, not exactly)

7. Try walking around inside the "fortress".

8. Mark a spot on the map and teleport to another location. So that the zombies are outside the bubble of reality.

9. Change the time. +10 years.

10. Go back to the "fortress". The zombies should turn into advanced versions.
    

11. Walk around the fortress. You may notice that the time per step increases compared to step 7.

12. Create a blindfold and put it on.

13. Repeat step 11. Less time is spent. Is the reason in the log?

14. Kill all the monsters

15. Walk around for a while. No lag.

16. Go to the town (big enough). Somewhere in the center.

17. Kill all the monsters. Keep moving by killing monsters through the debug menu.

18. Compare responsiveness with step 14.

19. Do a quick save, compare to step 18.

20. Exit the game to the main menu and load the save.

21. Move in the opposite direction (where everyone is already dead). Compare the responsiveness with point 19.

22. Walk around in a circle for a while.

Comments.
Step 16. Personally, for me, almost immediately the lags start. It comes to the point that the character continues to move. That is, I let go of the movement button, and the character still makes a few moves.
Step 17. Death of monsters, has no effect on lags.
Step 19. Quick save has no effect on lag.
Step 20. If you go to the menu and load the game. The speed of the game is significantly increased, but for a short time.
Steps 21 and 22. Even if you move in the opposite direction where all monsters are killed. You can notice a gradual slowing down of the game

On version 0.G I didn't notice this. But requires confirmation. Since 0.G is linux. And the experimental version is windows, running through portproton on linux.

So I'm assuming it's something else.

Additionally.
The experimental version slows down when loading at step: verifying / items

P.S. I apologize for the quality of the text. Deepl translate is used.

[worm-girl]

I really think the issue is items. When windows break, for example, they leave 50-100 glass shards on the ground. If you're in a town, that's thousands of items. Things like flour, sugar, and sand also leave thousands of items. When charges were removed, these powder items went from being a single object to thousands of objects. As I understand it, any time you have that many objects just lying on the ground inside the reality bubble, the game has to check how they're doing every single turn. 0.G still had charges, so there were fewer individual items lying around.

A big source of item spam is sandbags, which break open and turn from a single piece of furniture to thousands of bits of sand. This is a known issue and some work has been done to address it, but it sounds like there's still a ways to go.

If you're in your base, I understand that you can reduce lag by putting your powder-type items into containers. A bag of flour lying on the floor only causes the game to check the bag, it doesn't care about the items inside. Obviously that isn't a great solution for a town you're exploring.

I'm not sure what would be causing slowdown in fields. It's not something I've noticed, personally. I do wonder if some of the lightmap/line of sight changes have introduced slowdown. The map cache is pretty expensive in terms of processor power.

edit: Thank you @IdleSol, that is very helpful data.

[Zireael07]

In other words, when charges were removed I was afraid of exactly that sort of performance issues. I was told I was worrying over nothing...

[IdleSol]

I wish someone would double-check my tests. Still I'm not sure if playing through the emulator, doesn't create additional problems.

As I understand it, any time you have that many objects just lying on the ground inside the reality bubble, the game has to check how they're doing every single turn.

That sounds like a lot of useless work. Reducing productivity. Is that really the case?

Assumption 1. Large number of items. Sand, glass shards, flour.

1. Default world and character
2. Teleportation to a clear field
3. Generation of 100k shards + 100k flour + 100k sand

It felt like no performance issues at all.

Added a fortress and 1k zombies (see last post). Everything is ok.
Killed zombies: no lags.
Increased the number of shards to ~1kk. No lags.

Clarification. No lags. There is a slight slowdown when moving. It's not too noticeable. But you can feel it if you go outside the bubble radius. And it feels like an additional slowdown if you move on objects (shards). Suspecting the log, which displays the objects on which the character is stepping.

Additional information.
Who will check, it makes no sense to generate more than 100k shards at a time.

Tile capacity 4096 shards, radius of spawning 2 tiles. Total: 25 cells of 4096 shards. All other shards either disappear or are not generated.

Sand still has charges. And less volume per 1 charge than a shard. You can create 200k at a time. By the way, 200k is the maximum number of items you can create at a time.

[IdleSol]

Assumption 2. The problem is in the city itself.

In the first post, I got lags by moving to the city.

1. Default world and character
2. Teleportation to the center of the city
3. All monsters are killed through the debug menu.
4. Cleaning one long street (about 8 residential neighborhoods)
5. Several moves from beginning to end

No lags. One time there was a slowdown. For about a dozen moves.

The difference between this situation and 1 post in time: the first year of the cataclysm and the 11th year of the cataclysm.

6. Teleportation outside the city
7. Setting the time for the 11th year of the cataclysm. (11 because it's easy to just add 1).
8. Return to the same street and repeat steps 3-5.

The zombies are still dead. Come to life as the player advances. Zombies are still the initial version. On neighboring streets, advanced version zombies. Starting to lag.

No underground labs in the first post. This time there was an underground lab. 2 blocks away. I assume that by itself has no effect.

Conclusions
I assume the lag problem is due to monsters and the reality bubble. The difference between point 3 and 8 is monsters outside the reality bubble or on its border. Depends on the radius in which all monsters die via the debug menu.

I saw a PR about increasing the radius of the bubble. Has it been accepted?

The PR that added zombie hordes and their movement. By the way, why doesn't it depend on the horde option in the menu? Bug or a feature? I have Wandering hordes = false, but the hordes are there and they wander.

What other options are there? I have a few ideas, but don't know how to test them yet.

[KHeket]

Nothing was added by that PR which would have any effect on game performance. There's only one bit that it's running outside of the action of gliding and related activities, and that's:

Okay, may be it's not the glide ability issue, but I noticed, that when I am crossing the bridges by car/legs - I have performance slowdown
There was few prs #67997 #65738 and, I think, #67434 more fps eaters in that way and causing my problem
I don't know, how to check it, but as I can understand the logic, the game every tick checking all surroundings for angles to make the "ledges provide sight coverage" from #67434
And when I am crossing bridges - there is lots of ledges for game to check. I dunno, do the game check for ledges in the open fields, but, as I can check the logic, the game should check all you 3d FOV field for any ledges, so it can be the case of my performance death in traveling
I also remember, how this changes broke all FOV system and made game ultra laggy, so may be there is the one of the performance killing case
This also can affect @IdleSol case, because there is lots of Ledges in the cities, created by roofs, houses, basements - and game should check more ledges

[IdleSol]

I don't understand anything.

I've been testing the effects of portal storm, flour and bird droppings spoilage. I suspected the storm and the handling of expired items. Not confirmed.

But, in the process of checking, I got the following result. Same world. Two locations at a distance of about 10-15 cells of the global map.

Both locations - field, in both locations "fortress" of debug monsters + 700k shards. No lags. I change the time +10 years. In the first location lags, in the second location no lags. Under the ground there are no voids. Above ground, there are no animals. Trees and their tops, destroyed.

What is the difference between them, no idea. Flowers? Puddles? Burdock? I doubt it, but I'll try to remove it all.

Are there any checks that start after a certain event or time? The ones that don't work on the first day?

P.S. What about the game's multithreading? Is there any? During the lags I paid attention to the CPU load. Load 100% but only one thread. Sometimes split into several, but the total is not higher than 100% of one thread.

[Procyonae]

P.S. What about the game's multithreading? Is there any? During the lags I paid attention to the CPU load. Load 100% but only one thread. Sometimes split into several, but the total is not higher than 100% of one thread.

The game doesn't use and is unlikely to ever use multithreading

[IdleSol]

I think I found the cause of the lags in the cities. The reason is the items.

1. Default world and character
2. Set x0.01 on item generation
3. Add 10 years
4. Teleport to a large city
5. Move through the streets killing zombies through the debug menu

In my case, no lags. I think it's the food. Even though I wrote earlier that it wasn't her.

When experimenting in the field, I got different variations. Bird droppings had no effect, but it also spoils. Flour together with shards sometimes gave lags, sometimes not. That's why I thought it wasn't her.

But if you think logically, it is for food, there should be the most frequent checks. And it should also need operations to destroy it if it is spoiled.

[Rokharn]

And it should also need operations to destroy it if it is spoiled.

In my experience, food from 'out of reality bubble for the long time' become rotten mostly after player arrive to to new location and at the moment when player try to get firstly looking good food, so it's become rotten just into the inventory or after 1rst item eaten. Not by just walking around mostly. You can try to visit any out of reality town after years time skip and still find more or less fresh food of this kind.

[Kamayana]

Testing with #71472's save file.

Before the 10 year time jump

Looking at the call tree for the first stress test, before jumping ahead 10 years, one of the biggest slowdowns looks to be coming from item::can_revive.

item::can_revive actually gets called twice, as it's part of redraw_invalidated which gets called twice during do_turn:

There are two calls to redraw_invalidated here to handle the sporatic updates you get when auto-travelling. Unless I'm missing something, though, this implementation calls it twice unnecessarily during almost all normal turns, as g->handle_action calls it every time unless the player is auto-travelling. I feel like there must be a better way to handle this.

After the 10 year time jump

After the 10 year time jump, the main slowdown is from item::process_temperature_rot. That function uses so little processing before the jump it doesn't even show up on the call graph, but it's a massive issue here:

It appears that consumables that are in a stack are not rotting away like they're supposed to - if I put a single flour out, it rots away after the jump, but any stack amount stays intact. The game thinks they should be rotting though, so it's causing a slowdown because every turn the game is going through 11 years of temperature changes for every individual flour item. Normally the game is supposed to update an item's last_temp_check to the current time, but since all of the flour should be rotting away it leaves the loop early and never updates last_temp_check. Then, since the flour isn't rotting for some reason, it starts all over again from the beginning.

Edit: I was wrong here, see below.

[Kamayana]

Correction: the stacked flour items ARE rotting away, but it seems like the game is only able to rot away exactly 171 of them every turn. Every stack of flour started with 4096 items. After skipping 10 years and waiting one turn, they're all still 4096 except for the top-left one, which is now 3925 (171 less):

After another wait, 3754 (171 less again):

And they just continue to rot as you pass time, though it starts to jump around to other stacks:

[IdleSol]

Try replacing the flour with bird droppings. It should also rot, but I haven't had any lags.

Different mechanism?

Update.

And something that confuses me during the tests. Move to a remote location and create only flour, in the same amount (~100k). But no shards or monsters. No lags.

[Kamayana]

Are you still jumping forward 10 years? That's what's actually causing the major lag: the game trying to update the status of old food.

[IdleSol]

Yeah, it's still 10 years away. But you can try for two seasons (autumn). When moving, the lag is about 0.5 seconds.

[andrei8l]

Correction: the stacked flour items ARE rotting away, but it seems like the game is only able to rot away exactly 171 of them every turn. Every stack of flour started with 4096 items. After skipping 10 years and waiting one turn, they're all still 4096 except for the top-left one, which is now 3925 (171 less):

Item processing is staggered so only number / processing_speed gets processed every turn (per submap). There are 102400 flour items in that submap and the processing speed of food is 10 minutes. Hence, 171 items are processed every turn.

As I understand it, any time you have that many objects just lying on the ground inside the reality bubble, the game has to check how they're doing every single turn. 0.G still had charges, so there were fewer individual items lying around.

If you're in your base, I understand that you can reduce lag by putting your powder-type items into containers. A bag of flour lying on the floor only causes the game to check the bag, it doesn't care about the items inside. Obviously that isn't a great solution for a town you're exploring.

This is all false. There's a lot of baseless speculation in this thread. The situation with rot processing after 10 years is interesting and should be addressed if it's buggy, but you need to reproduce the performance issue with smaller time jumps that would actually occur in game (let's say max 1 year, but more realistically just a few weeks).

And btw, 102400 active items in one submap is completely bonkers. I know people like to hoard, but seriously we can't do magic with that many items.

[IdleSol]

1. Load the save
2. teleport to a free location (crossroads at the junkyard)
3. Create 102400 flour
4. +10 years
5. Compare lags.

In my case, there are no lags.

6. Walk around for a few moves. Pay attention to the amount of flour.

Flour does not spoil, in a few moves (~10) 102400 - 102375 = 25 flour disappeared. Further moves do not result in spoilage. I assume that the spoilage is a random amount. At repetition I got 26 flour.

8. Make a quick save.
9. +10 years
10. Repeat step 6.

No lags, flour does not spoil

11. Save and exit to the menu
12. Repeat step 6.

There are lags and flour spoilage.

[Kamayana]

Yeah sorry, got sorta lost in the 10 year sauce. Rot is still a significant issue after a shorter time jump of 20 days:

Also, fairly large time jumps aren't actually that uncommon because every time the player loads up a map that contains food for the first time the game is going to generate that food then spend CPU time determining that it should've rotted away. I did another test where I jumped time forward a year and went to a never-before-visited city. While not nearly as bad as the 102400 flour field, item rot still took a top spot in processing time:

Right now the game steps through every individual item's temperature history to calculate rot, but it would make a lot more sense, and be faster, to get the temperature history of a tile (or submap, I'm unsure at what scale weather operates at) and apply that to every item inside it at once.

Also, can we really stick to just processing 171 items at a time after charge removal? Food stores of 10,000+ items is not out of the question anymore, and only doing 171 at a time means the player could have enough time to reach their food store and watch it turn to rot in their hands.

[IdleSol]

@andrei8l

This is all false. There's a lot of baseless speculation in this thread. The situation with rot processing after 10 years is interesting and should be addressed if it's buggy, but you need to reproduce the performance issue with smaller time jumps that would actually occur in game (let's say max 1 year, but more realistically just a few weeks).

If you use the saves from the bug report. I have lag after two seasons. Small, but noticeable.

Going to a fresh autumn city. Does not cause noticeable lags. But the responsiveness is a little worse.

@Kamayana

the player could have enough time to reach their food store and watch it turn to rot in their hands.

Sounds like new mechanics for some new monster.

[SurFlurer]

Also, can we really stick to just processing 171 items at a time after charge removal? Food stores of 10,000+ items is not out of the question anymore, and only doing 171 at a time means the player could have enough time to reach their food store and watch it turn to rot in their hands.

The limit isn't 171. The active item cache is designed to process 1/600 of active items on the map every turn. So regardless of the number of food items, it's always possible to see them rot away in hands.

[Cenness]

to get the temperature history of a tile (or submap, I'm unsure at what scale weather operates at) and apply that to every item inside it at once.

There are containers with thermal insulation property, wouldn't that bypass them?

[IdleSol]

Maybe I'm just imagining things. Moving on the bridge is a little slower than moving on the road.

[KHeket]

Maybe I'm just imagining things. Moving on the bridge is a little slower than moving on the road.

No, you don't. And I think, the problem is in this pr's #71419 (comment)
Moving through the bridge creates lots of edges and game makes lots of field of view calculations

[IdleSol]

I don't see how this can be verified.

Manually build a long track at +1 level? And record the ground and track speeds?

[KHeket]

I don't see how this can be verified.

Manually build a long track at +1 level? And record the ground and track speeds?

I just revealed map with debug menu, found long bridge, spawn car and firstly riding it at the ground and then riding it at the bridge. Its possible to record it via obs with stopwatch before and after entering bridge, to show performance slowdown, but I dont have energy to do it, may be some visual studio masters like Kamaya or ZhilkinSerg with ability to work with solution explorer can find more obvious visible proofs

[KHeket]

I still noticed for now free performance dead situations

1. After driving near the slime pit I have big performance slowdown, and it continued after I traveled for few (5+) tiles away. And only quit and load game again solved the problem
2. After driving through some bridges - the same situation happed, and quit-load is solding the problem.
3. Traveling in the fields became slower, than before
   And I cant reproduce it because after quit-load or loading trimmed save files the problem is disappear
   And There was few prs
   Fix visible tiles revealing invisible tiles below #67997
   3D vision of lower levels with distance fog #65738
   Ledges provide sight coverage #67434
   which is one of the fps dead

There also was added Tree tops #70669 which as I can understand the logic - will make ledges fps dead situation worse by creation lots of "tree roofs"

[IdleSol]

I tried walking and driving near the slime and bridges. I didn't have any lags. Or they were imperceptible.

cdda-windows-tiles-x64-2024-03-01-0629.

[KHeket]

I tried walking and driving near the slime and bridges. I didn't have any lags. Or they were imperceptible.

Can you show screenshot of your safe mode manager? Because I think, this can be the reason of my laggys due to big number of creature in my manager

My safe mode manager

[IdleSol]

I was watching with a blank manager.

UPD.
I'm assuming it's a memory problem. Maybe the cache. I'm basing this on the fact that rebooting reduces the problem. But I'm a layman.

Am I understanding correctly that you encounter them as you play the game? It's not a specially created world for testing (in my case)?

Can you estimate how much real time passes, from the start of the game (or rebooting) and before the lags occur?

And try changing: settings - options - debug - vertical range of 3D field of vision: 4 -> 0

I didn't notice any difference, but I didn't get any lags (noticeable) either.

[KHeket]

Am I understanding correctly that you encounter them as you play the game?

About 1-2 real hours of playing or more

Turning off 3D field of vision solve the problem, but problem is, that I didnt have so big fps loss with 3D field of vision turned on about 1 year ago, but now, after recent updates - I do, and I cant understand, which one of new features destroying fps

[IdleSol]

I tried leaving the character near the pit for an hour. There's a slight slowdown. But I wouldn't say it's very noticeable.
Switching from linux to windows, slows it down more. Need to clarify: I wasn't playing, I was doing other things.

At the 5 minute interval (after loading), I didn't notice anything.

UPD. I tried using your safemode list. At the 5 minute interval (after loading), I didn't notice anything.

[IdleSol]

On the topic of performance. How does the sidebar load?

1. The game looks at panel_options.json
2. Looks at the value of "current_layout_id"
3. Finds the appropriate section
4. Loads the widgets according to the order specified in that section?

What happens if the language is other than English?

Here is a sample file panel_options.json in English locale:

[
  { "current_layout_id": "legacy_labels_sidebar", 
  "layouts": [ 
    { "layout_id": "custom_sidebar", "panels": [ 
      { "name": "Hit Points", "toggle": true },
      { "name": "Hit Points (num)", "toggle": false },
      { "name": "Stats", "toggle": true },
...

And this is him in the Russian locale:

[
  { "current_layout_id": "legacy_labels_sidebar", 
  "layouts": [ 
    { "layout_id": "custom_sidebar", "panels": [ 
      { "name": "Очки здоровья", "toggle": true }, 
      { "name": "Очки здоровья (цифр)", "toggle": false }, 
      { "name": "Хар-ки", "toggle": true }, 
...

These samples have been slightly corrected for better viewing. In the original version, it's all in one giant line.

Am I correct in understanding that there will be additional steps to translate names? How correct is that?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Please do not bump or comment on this issue unless you are actively working on it. Stale issues, and stale issues that are closed are still considered.