dart:analysis_server takes 19GB memory

[fzyzcjy]

Well, I have reported here #41793 (comment), but I am not sure whether it is more reasonable to create a new issue, since the memory size is really really a lot IMHO. The original issue complains when it takes <2GB, while here it is nearly 20GB!

[incendial]

@fzyzcjy you have dart_code_metrics plugin enabled, right? The plugin API is the problem.

[incendial]

If you want to check what is the difference in memory with the standalone, let me know.

[fzyzcjy]

@incendial Hi thanks for the reply, but indeed no. I do not enable the dart_code_metrics plugin when using the analysis server; indeed I run dart_code_metrics separately to check lint problems. See the screenshot below (no plugins).

[incendial]

Ah, my bad then, my assumption was wrong. 93 contexts though... 🥲

[fzyzcjy]

93 contexts though... 🥲

Well... I know it looks like a nontrivial number. But is that too many? I have not heard of documents saying that I should use packages as few as possible...
(I have checked my code, it seems not very easy to merge those packages...)

The total number of Dart lines is at the quantity of 100kloc (e.g. 200kloc, I forget the exact numbers). There are a few packages that are huge, and many are small.

[keertip]

/cc @scheglov @srawlins

[scheglov]

@bwilkerson @jacob314 as an anecdotal evidence that users with many analysis contexts exist :-D

[jensjoha]

From the screenshot you're running an old version of dart. The new version should have a "Collect Report" option at the top of the "Analysis Server Diagnostics" page which could help us diagnose this problem. (It also contains some memory optimizations so updating might even help the problem in itself).

[fzyzcjy]

@jensjoha Thanks for the information! Yes I am using Flutter 3.7 (and dare not update to 3.10.0 since many people are saying it is buggy (though exciting)... so may have to wait for something like 3.10.3). I will update this post with the collected report once I upgrade.

[srawlins]

Well... I know it looks like a nontrivial number. But is that too many? I have not heard of documents saying that I should use packages as few as possible...

We've been meaning to publish something along these lines, but we don't have the data to support a number. But 93 is a lot. Our analytics are showing that the vast majority of IDE sessions are opened to less than 5 contexts. And there is (currently) a linear correlation between # of contexts and memory usage.

[fzyzcjy]

@srawlins Thanks for the information! Then may I know how is this possible to only have 5 contexts? My usage is as follows (happy to provide more information when needed):

1. Totally at the quantity of 100kloc lines (e.g. 200kloc, I forget the exact numbers)
2. Have many packages which each one does one thing, just like this -> https://github.com/flutter/packages , instead of one huge giant package
3. Seems inevitable to split into multi packages for one (group of) things. Take my open source lib https://github.com/fzyzcjy/flutter_convenient_test/tree/master/packages as an example. To allow execution from pure-dart (e.g. command line), I have to have *_dart packages. To have some code to in testing, I added the *_dev packages.

[srawlins]

Then may I know how is this possible to only have 5 contexts?

A separate context is created for every pubspec.yaml file (technically a .dart_tool/package_config.json file) and for every analysis_options.yaml file (except a directory with a .dart_tool/package_config.json file and an analysis_options.yaml file only results in one context, not two). It would be interesting to know how many pubspec.yaml are found in the root where you run your IDE. And how many analysis_options.yaml files.

One way to reduce the number of contexts is to open the IDE window to a more narrowly focused directory, somewhere inside of the directory containing 93 contexts.

I expect most users have only one context in their DAS instance because their IDE is open to just one Flutter app.

Take my open source lib https://github.com/fzyzcjy/flutter_convenient_test/tree/master/packages as an example. To allow execution from pure-dart (e.g. command line), I have to have *_dart packages. To have some code to in testing, I added the *_dev packages.

Can you explain a little better why so many packages are needed to support one feature?

• convenient_test
• convenient_test/example
• convenient_test_common
• convenient_test_common_dart
• convenient_test_dev
• convenient_test_manager
• convenient_test_manager_dart

All of the READMEs point to the first one, but I don't think the first one explains what any of the others are. Is there a page describing these different packages? In particular, I'm curious to know why a *_dart package is needed, and why a *_dev package is needed.

[fzyzcjy]

@srawlins Hi thanks for the reply!

It would be interesting to know how many pubspec.yaml are found in the root where you run your IDE. And how many analysis_options.yaml files.

I checked the name of my 93 contexts listed in Analysis Server webpage. Seems that each name corresponds to a real Dart/Flutter package that I wrote. So it seems to be 93 pubspec.yaml and 93 analysis_options.yaml.

One way to reduce the number of contexts is to open the IDE window to a more narrowly focused directory, somewhere inside of the directory containing 93 contexts.

Well, I need all those packages and very frequently navigate here and there among them... So I cannot just throw away some of them inside the IDE :/

Can you explain a little better why so many packages are needed to support one feature?

Sure!

• convenient_test: Code here are used in a user's app. For example, in your app's main.dart, insert ConvenientTestWrapperWidget.
• convenient_test_dev: Used in a user's app's test directory. For example, in your app's home_page_test.dart, you may call EnhancedLocalFileComparator(...). This is a separate package instead of a folder in convenient_test, because it depends on things like test, integration_test, etc.
• convenient_test_manager: This is a desktop GUI app (instead of something that users need to import in their pubspec), so must be a separate package. This is built based on convenient_test_manager_dart.
• convenient_test_manager_dart: Command line utility for users. Cannot be merged into convenient_test_manager, because convenient_test_manager depends on Flutter, and thus cannot be run on command line.
• convenient_test_common: Common code shared between things above, with Flutter things. This cannot be merged into convenient_test_common_dart, because convenient_test_common_dart does not depend on flutter and cannot access any flutter features.
• convenient_test_common_dart: Common code shared between things above, pure-dart (no Flutter).
• convenient_test/example: You know, the standard package structure suggested by Flutter is to add such an example.

[srawlins]

Thanks for the explanation. I don't have any further advice on reducing the number of contexts in play.

[fzyzcjy]

@srawlins Thank you for the reply. Then maybe this is a legitimate scenario to have this many packages.

[incendial]

@srawlins how many people you need to upvote the issue to address the problem with monorepos and multiple contexts? I've spoken with too many people already that really suffer from the analyzer perf (without plugins) on a scale. I believe it's a small percentage of the overall Dart devs, but the thing is with time this number will only grow.

[srawlins]

A great question! Now that we have analytics, we don't need upvotes as a signal as much any more. Upvotes are still good, and analytics can't answer all the questions.

This is largely a question of ROI. Some of the ideas we've thrown around are very high effort and complexity (bug-prone) and maintenance cost, in order to serve users with many analysis contexts. So we weigh that ROI against other work in the queue.

One item I'd like to accomplish, which is not too complicated, is to not create new analysis contexts for each analysis_options.yaml file, unless that file specifies language experiments. However, in @fzyzcjy 's case, that would literally help 0. 93 pubspec files yield 93 contexts. So that idea might not have good ROI even though it's not too complex.

We have other ideas, and the priority is informed largely by analytics.

[fzyzcjy]

@srawlins I see. But I am a bit confused and curious: I am just an (individual) developer so my codebase is quite small. There must be (a lot of) teams and groups who are using Flutter to develop apps. On the other hand, you mentioned most people have few contexts. So I guess it is either because the large teams only have small number of packages (which I do not know how... since I have explained my scenario above and it seems it is hard to be reduced), or because most people use Flutter for small apps?

[incendial]

@srawlins so we basically come to a situation when people with actual performance problems can't get any attention to their problems because on a scale their problems are insignificant to you.

I'm asking about numbers, because I can easily find 200+ people with the same problem as @fzyzcjy has without any effort.

I understand that large codebases are locked into using the analyzer the way it works now (with no alternatives and no way to migrate off Dart / Flutter in a reasonable time), but are various hacks you suggest (like opening a subfolder) really the best option they can have in a long run?

Can you also give more details how exactly this analytics is collected? Do people who open hello-worlds count?

@fzyzcjy I believe it's the latter.

[srawlins]

I am just an (individual) developer so my codebase is quite small. There must be (a lot of) teams and groups who are using Flutter to develop apps.

CC @bwilkerson and @jacob314 for any insights and I don't know what user stories we can derive from just the analytics (it's pretty limited data), but as a hunch, I would guess that many users literally open their VS Code window to a single Flutter app, with no internal packages 🤷 .

Someone posited on a different issue that ~99% of Dart developers do not write "packages" which are published to pub; they are writing Flutter apps. It would be really interesting to get analytics that help us understand that.

(Most of my VS Code windows are a single analysis context but I won't claim to be a typical flutter developer by any stretch. I usually have 3-5 VS Code windows open to different packages I'm working on within a "few days" timespan, like linter, dartdoc, analyzer, markdown, mockito. I'm very happy with 3-5 VS Code windows 🤷 .)

@srawlins so we basically come to a situation when people with actual performance problems can't get any attention to their problems because on a scale their problems are insignificant to you.

I apologize if it feels that way. I haven't closed this issue as "Won't Fix." It's just a matter of binning into priorities. So I cannot promise a timeline for a better many-packages-in-workspace Developer Experience.

Can you also give more details how exactly this analytics is collected? Do people who open hello-worlds count?

CC @bwilkerson I imagine we publish somewhere what data we collect, right? But yes, I think even hello worlds count. I'm not sure how often the data is collected for each user, which might distinguish hello world users from large app / multi-context users.

[incendial]

they are writing Flutter apps.

Even a flutter app can be separated into packages. For example, if you have several apps with the same reusable core blocks (have discussed at least two cases like this).

Most of my VS Code windows are a single analysis context but I won't claim to be a typical flutter developer by any stretch.

Yep, me neither, so I can't judge from my exp, but talking to people that get no help upsets me a lot. DCM itself has 13 contexts which is fine for now (rarely get more than 1.5 GB).

linter, dartdoc, analyzer, markdown, mockito. I'm very happy with 3-5 VS Code windows 🤷

They all are not in a single monorepo (and not very related to each other in terms of code changes, at least not all of them), so you basically have no options 🙂.

I apologize if it feels that way. I haven't closed this issue as "Won't Fix." It's just a matter of binning into priorities.

How to say that... considering the overall planning approach, low priority thing can become a won't fix if there is always something more important. And as I know, the number of people with many contexts is low relative to the total number in the analytics data you have.

[nhannah]

I work on a project with a much greater number of LoC (but less contexts at 49) and we used to have a similar issue to this, in the end our fix was to remove almost all of the barrel files. We had a lot of circular references that both slowed down analysis greatly and caused memory pressure. This also put a stop to our ambition to further split our app into more packages in our monorepo as this would further push our use of barrel files. If you have a package A that 3 other packages use and you make a change to A any file referencing that package will re-analyze, as will any file referencing those files, and so on down the line. Our analysis time had become a crawl, unusable at times, this change (removing most barrel files) made it something we don't any longer think about. In addition we saw our memory usage drop from over 20GB that would continually leak higher as we switched branches and worked to a more stable ~2GB that barely grows.

[bwilkerson]

I imagine we publish somewhere what data we collect, right?

I'm not aware of anywhere that this information is published. That got me curious, so I tried to find out what Flutter documents, and the only thing I could find was information about crash reporting and how to disable analytics: https://docs.flutter.dev/reference/crash-reporting. I might have missed something, but I don't think Flutter provides that data either.

That said, I can confirm that we're collecting analytics for all projects no matter how large or small, so yes, if a disproportionate number of users (that have enabled analytics) are opening small hello-world sized packages that might impact the numbers.

[incendial]

Thanks, could you also please share how does it work for forks, for example? Or for projects copied from a template like it usually happens for job interviews? No differentiation?

[olexale]

@incendial, we use a single barrel file per package to expose classes that are hidden under the src folder, and we heavily rely on code generation, so we use part of directives a lot.

[fzyzcjy]

@jensjoha From the screenshot you're running an old version of dart. The new version should have a "Collect Report" option at the top of the "Analysis Server Diagnostics" page which could help us diagnose this problem. (It also contains some memory optimizations so updating might even help the problem in itself).

Hi, I upgraded to Dart 3.0 and here is the report. (It is 16-18GB at the time I create the report, but sometimes it again becomes 19GB. I also removed and added some packages after the first post of the issue, so the number of packages will not exactly match what I said before)

dart_analyzer_diagnostics_report.json.zip

[spkersten]

We also have many package for our project (an app), about ~120. Running all tests and codegen takes ~1h for the whole code base. So to be able to qualify pull requests in an acceptable time and to be able to run tests/codegen while developing, we split the code into packages for individual features, so we can limit test/codegen to only packages that are affected by the work you're doing. Over time, the number has grown with the number of features (and their interactions) and to avoid circular dependencies.

[jensjoha]

Thanks @fzyzcjy!
That's an interesting report.

It has 40,611,996 _Entry objects directly taking ~1.2GB of ram.
Looking at a local running instance of the analyzer I also have (much much fewer) _Entry classes --- which has a Map in it. Considering the report also has 40,991,788 _Map objects directly taking ~2.4GB of ram it seems likely that this in itself uses ~3.6GB.

(looking)

So I seem to be able to reproduce this in Windows (though Mac seems to be the same, Linux isn't) --- with a session with now 40,434,365 _Entrys... The "Retained size" according to Observatory is 7.39GB (though, to be fair the memory usage now is also 17.4GB --- so maybe there are more leaks and/or the observatory number is off). Anyway, I should probably fix that.

[fzyzcjy]

@jensjoha Thanks for the information!

A bit more context (for the zip I provided): It goes to the huge memory after using for a while (instead of being there in the first minute), so I also expect it can be some kind of leaks.

[polina-c]

I recently enabled auto-snapshotting for analyzer, in case of memory overuse. Here is instruction how to configure it: go/analyser-memory-troubleshooting-with-auto-snapshotting

[polina-c]

Public document with instructions how to setup auto-snapshotting for analyzer in case of huge memory usage: https://docs.google.com/document/d/1mxCKhvyHUtGVhIH5Apx9y7V7itqCxAQPse6ghbjaDMM

[polina-c]

@fzyzcjy @spkersten

Does the issue still reproduces for you?
May I ask you to configure auto-snapshotting in your IDE? Contact me directly at https://twitter.com/polinalinac if you have issues.

[fzyzcjy]

@polina-c Yes and no. It is ~10GB today (though it is still quite huge, it is much better than ~19GB). I am happy to configure auto snapshotting, but before that I need to confirm - I am on stable channel, is it OK, or do I need master Flutter?

[jensjoha]

FYI I would expect the issue to be fixed* on Flutter master --- the likely fix (5aee0f6) (and 5e08794 though that "leak" is only temporary) has landed a good while back. The fix seems to have been chery-picked in stable and landed (7475c2c) which I think means it will be included in the next stable release (though I don't know when that will be).

* At least I'd expect it to get way down from 19/10 GB --- we previously saw that @fzyzcjy had lots of _Entry objects.

[fzyzcjy]

That looks great! I will wait for next stable and report the updates here.

[polina-c]

@polina-c Yes and no. It is ~10GB today (though it is still quite huge, it is much better than ~19GB). I am happy to configure auto snapshotting, but before that I need to confirm - I am on stable channel, is it OK, or do I need master Flutter?

Auto-snapshotting is available starting Dart 3.1.0-200.0.dev. Current Flutter stable has 3.0.5. So, master has it, but stable does not yet.

[gabrielgarciagava]

@fzyzcjy The fix was released on flutter version 3.10.6. You can give it a try to see how is the memory usage now.

[fzyzcjy]

@gabrielgarciagava I will do that soon (when focusing on frontend again)!

[srawlins]

@fzyzcjy I'm going to close this issue as a few folks on the Dart team have landed many memory improvements in the Analyzer code bases over the last year, and even over the last six months (see below). I won't promise that your memory problems are solved, but I think at this point, it would be most effective to sort of rebase the discussion.

Anyone else experiencing similar "giant memory usage" issues with the Dart Analyzer, and you'd like to document a bit your situation, also please follow these instructions:

First, be sure to bump to the latest Dart, 3.2.0, as it has some performance improvements over 3.1.0. Then, if you experience what you feel is an inordinate amount of memory being used (we don't have any SLOs yet in this space, but anything north of 5GB is probably noteworthy):

1. We have a public document with instructions how to setup auto-snapshotting for analyzer in case of huge memory usage: https://docs.google.com/document/d/1mxCKhvyHUtGVhIH5Apx9y7V7itqCxAQPse6ghbjaDMM
2. You can file an issue in this repo and send us some other basic info:
   • How many .dart files are under the analysis root for the IDE?
   • In the Diagnostics pages, you may want to look at Memory, Plugins, how many Analysis Contexts exist, etc.
   • How long has the process been running? If you restart your IDE or and/or DAS, does it use less memory? What about hours later? What about days later?

Recent memory-reducing changes:

Dart ~ 2.19.0, Jan 2023

• 9306752 - large memory reduction
• b785f7d - small memory reduction
• 91c88ab - small memory reduction
• 4193bb3 - small memory reduction

Dart ~3.0.0, May 2023

• 3378c55 - small memory reduction

Dart ~3.1.0, August 2023

• bda10d7 - large memory reduction
• 5e08794 - memory leak fix
• 5aee0f6 - memory leak fix (also cherry-picked into Dart 3.0.6)
• 3dd9389 - memory leak fix (also cherry-picked into Dart 3.0.6)

Dart ~3.2.0, November 2023

• aba5663 - large memory reduction
• ef2422a - large memory reduction
• f28d4a5 - small memory reduction
• 2bbf993 - large memory reduction
• 807d1cf - large memory reduction
• 5f23e5b - large memory reduction

(There were also speed-up improvements made in this time frame, and improvements to our analytics, logging, benchmarking, and platform coverage, but I'm not documenting those here on this memory-focused issue.)

[fzyzcjy]

Thank you and the Dart team for improving this! I will try Dart 3.2 later and report if anything goes wrong.

[RichardGreat]

I faced the same problem in flutter 3.7.12 but upgrading flutter version was not a option because I have a huge code base with 3-4 big modules which I have to migrate all my modules to dart null safety, I found the solution which worked on me
the problem was in example project of an each module, which created I cyclic dependency source analyzing :
module -> example -> dependency -> module -> example ... etc.
and I reviewed my module examples and remove module dependency from pubspec.yaml file and now it helped and now it takes at most 1GB in ram

[polina-c]

For those who still experience this, you can enable auto memory snapshotting for analyzer.

Do not worry, it will not fill all you hard drive with snapshots, it will stop at the configured limit.

Here is the steps how to configure your IDE to turn on analyzer autosnapshotting: https://docs.google.com/document/d/1mxCKhvyHUtGVhIH5Apx9y7V7itqCxAQPse6ghbjaDMM/edit

Then, with DevTools starting flutter 3.22.0-1.0.pre.10 or fresher (you do not have to migrate your project, just switch to flutter master to get DevTools) you can import and diff the collected snapshots.
Or, if the snapshots do not contain sensitive information, just share them here so that analyzer team can check them.