SublimeText uses too many inotify instances in Linux

[3v1n0]

I'm quite heavily using SublimeText... And it's not uncommon that I've 7-8 windows of it open, where at least two or three projects have may opened files too.

However, it seems that SublimeText uses too many inotify instances even in this case. In fact, I'm running also other programs that are doing file monitoring (such as cloud sync ones), but they just use one or two inotify instances. Sublime Text seems to abuse of them.

You can check which process uses inotify, with this command:

for foo in /proc/*/fd/*; do readlink -f $foo; done |grep inotify |cut -d/ -f3 |xargs -I '{}' -- ps --no-headers -o '%p %U %a' -p '{}' |uniq -c |sort -n

and this is result for me (this is running in Ubuntu, and I've only included the first results, out of 53 processes):

  127 11460 marco    /opt/sublime_text/sublime_text
    20 11472 marco    /opt/sublime_text/plugin_host 11460
      5  2914 marco    upstart-file-bridge --daemon --user
      3  2989 marco    /usr/bin/pulseaudio --start --log-target=syslog
      2  8989 marco    compiz --replace
      2  6679 marco    megasync
      2  3966 marco    /usr/lib/unity-settings-daemon/unity-settings-daemon
      2  2718 marco    init --user
      2 22017 marco    /usr/lib/x86_64-linux-gnu/indicator-sound/indicator-sound-service
      1  6676 marco    update-notifier
      1 31171 marco    nautilus --new-window
    ...

So, as you can see the sublime text inotify count is way over the average, outdistancing any other program (even the nautilus filemanger, never uses more than one inotify instance).

Considering that in many distros the default value for fs.inotify.max_user_instances is set to 1024, sublime could cause the inotify service not to be avilable to other processes.

I've not checked the internals, but I'm quite sure there's a way to reduce the usage of inotify instances, even keeping the monitor active for many views. As the GFileMonitor (python API) allows to do.

[FichteFoll]

Do you experience any slowdowns in ST usage (potentially because of this)?

[3v1n0]

Not, really... I can't say as I don't have numbers for this. But indeed it causes problems in running other programs that need inotify instances and that are then unable to set it up because they're all taken.

[simonrussell]

I have this issue as well. Sublime Text 3 (build 3114) uses all the inotify resources available, so things like Gulp can't actually operate. And that's after I've boosted the number available by 4x.

To me it's actually a fairly major issue.

[celeste-sinead]

I just spent a couple hours tracing a mysterious "Too many open files" in another application to sublime's insane inotify instance infatuation. This behaviour is super painful because the symptoms pop up anywhere in the system without an immediately obvious connection to the root cause.

[thendrix]

Just saw this break UnrealEngine4 cook under Linux. Work around is to shutdown sublime then cook.

If I had the time to waste I'd suggest someone making a caching LD_PRELOAD lib to redirect inotify calls from sublime to one that cached overlapping instances to at least avoid the duplication. I didn't run an ltrace, but you could see how often they call inotify that way for stats. :)

[RPGillespie6]

Just ran into this issue today. I can't build Yocto Project while Sublime Text 3 is open because all of the inotifies get used up. If I close Sublime, suddenly I can build.

[RPGillespie6]

This same bug also affects angular-cli and its ability to watch files

angular/angular-cli#1610

[thendrix]

This is a real problem that should be addressed. In plain language the application is stealing "handles" from all other apps and services on the system. Running this application can break nearly anything else on a system using inotify or APIs using inotify.

Here's a bandaid for those suffering:
sudo sysctl -n -w fs.inotify.max_user_instances=512

By default I think you have 128 instances, and check out the usage from subl is 172. This means subl will actually break itself. You can see this happen when file changes do not update in the app.

      1  1934 thendrix /lib/systemd/systemd --user
      1  2098 thendrix /usr/bin/dbus-daemon --fork --print-pid 6 --print-address 9 --session
      1  2116 thendrix /usr/bin/dbus-daemon --config-file=/etc/at-spi2/accessibility.conf --nofork --print-address 3
      1 21621 thendrix hexchat
      1  2214 thendrix /usr/lib/gvfs/gvfs-udisks2-volume-monitor
      1  2229 thendrix /usr/lib/gvfs/gvfs-afc-volume-monitor
      1  2303 thendrix nemo -n
      1  2315 thendrix mono /usr/lib/docky/Docky.exe
      1  2341 thendrix /usr/lib/evolution/evolution-source-registry
      1  2385 thendrix /usr/lib/gvfs/gvfsd-trash --spawner :1.7 /org/gtk/gvfs/exec_spaw/0
      1 28163 thendrix remmina
      2 19248 thendrix /opt/google/chrome/chrome
      2  2273 thendrix cinnamon --replace
      2  2291 thendrix /usr/lib/x86_64-linux-gnu/indicator-sound/indicator-sound-service
      2  2319 thendrix mono /usr/lib/tomboy/Tomboy.exe
      2  2525 thendrix cinnamon-screensaver
      3  2139 thendrix /usr/bin/pulseaudio --start --log-target=syslog
      4  2134 thendrix /usr/lib/x86_64-linux-gnu/cinnamon-settings-daemon/cinnamon-settings-daemon
     24 13904 thendrix /opt/sublime_text/plugin_host 13888
    172 13888 thendrix /opt/sublime_text/sublime_text

[FichteFoll]

For the record, I just ran the command after 10h of "not doing a lot" in ST with 3 projects and a couple files open each, and it uses 28 instances.

[wbond]

From my understanding, inotify does not allow watching a whole directory tree, so every subfolder of the folders in your sidebar, plus every subfolder in your Packages folder must be watched by a separate inotify instance.

I think the possible solutions are:

1. Increase the number of inotify watches on your machine
2. Don't open multiple windows at a time unless you really need to
3. Tweak your settings to exclude large directory trees like node_modules from your sidebar

[3v1n0]

Mh @wbond ok, sure you need a watcher for each dir, but I guess there's a way for optimizing it... For example, I'm using syncthing-inotify, and here it currently monitors 144460 elements, ~9.23 GiB.

Well, it just creates one instance. Compare:

      1 15225 marco    /usr/bin/syncthing-inotify -logflags=0 -interval=5s
      3  5908 marco    megasync
      5  3935 marco    upstart-file-bridge --daemon --user
      7  6259 marco    /usr/lib/firefox/firefox
     11 21118 marco    /opt/sublime_text/plugin_host 21106
    105 21106 marco    /opt/sublime_text/sublime_text

[RPGillespie6]

@wbond All I know is that VSCode and Atom do not have this problem of gobbling up inotifies while Sublime does. I can use VSCode at the same time I am using Yocto Project or angular-cli - not so with Sublime 3. What is Sublime 3 even watching files for? angular-cli watches files so it knows when to automatically recompile typescript files. What is Sublime watching files for?

[wbond]

@3v1n0 I've never use syncthing. It looks like from your commandline it perhaps rescans your filesystem every 5 seconds? The readme recommends setting your machine to have 200k watches…

@RPGillespie6 From my experience (which is limited) VSCode and Atom provide a very different user experience. A few things that Sublime Text does with inotify watches:

• Updates your project index to jump to the definition of your code
• Updates the sidebar when files or folders are deleted
• Keeps the Goto anything list up-to-date
• Allows package files to be changed with hot-reload
• Notifies you if another process edits the file you are working on

I'm sure there are more, but those are just off the top of my head.

Is there an issue with increasing the number of watches on your machine? 105 doesn't seem too bad. Ubuntu 10.04 (which I hope you aren't still using) defaulted to 8k user watches. syncthing-inotify recommends 200k watches. guard recommend 500k watches.

Am I missing something here?

[celeste-sinead]

A single inotify instance can be used to subscribe to notifications in any number of directories. There is no good reason for a single process to use more than a couple instances; certainly not one per watched directory.

You are correct in that it is possible to avoid this issue by modifying the limit (and indeed, I have done so, so this issue no longer affects me). Unfortunately, the default limit is easily reached in normal use, and it is extremely difficult to correctly diagnose the problem and resolve it.

In my case, the issue manifested with intermittent EMFILE errors in my build process. I spent significant time poking around messing with that processes rlimits, system limits, trying to find a fd leak, etc. None of that lead anywhere, and the only thing I had to go on was that my build was consistently failing to create inotify instances, and not any other fds. I did some more poking, and only then discovered that notify instances were covered by a different limit (not shown in any typical tool). Further futzing (standard tools like lsof not being able to tell you about inotify instances) eventually fingered Sublime as the responsible party.

This is horrific UX! As things stand, Sublime's sloppy resource use starves random innocent processes! Worse, the resource in question is rather obscure, not exposed by typical diagnostic tools (rlimit, lsof), and generally hard to trace back. Sublime itself apparently silently ignores the errors it gets creating additional instances, and just intermittently fails to reload files, instead of some sort of error message that might tip the user off on the root cause of all the breakage.

[wbond]

I think I was imprecise with my terminology a few comments ago. I am also no expert in inotify, I am just trying to help in diagnosing if there is a real issue here, or if this is just the nature of the beast.

I am reviewing the Linux file notification code now. Here is how works:

• We utilize a single instance (via inotify_init()) for each folder in the sidebar
• inotify_add_watch() is called on all subfolders

I was interchangeably referring to instances and watches.

We utilize an instance per distinct sidebar folder, for the Packages/ folder, and the Cache/ folder. I believe there will also be a watch for the Installed Packages/ folder and the Packages/ folder next to the executable.

So, out of the box, Sublime Text will utilize 4 + n instances, where n is the number of folders open in the sidebars of all Sublime Text windows.

However, it appears that an instance is set up for the directory containing a file when it is opened. I have yet to work in this part of the codebase, so I am not familiar with exactly what it is doing and why, but I know it deals with monitoring a file for changes. I know there is a fair amount of complexity here because files can be opened from filesystems that don't support inotify, so there are fallback mechanisms to ensure that the user is notified if a file is modified while they are working on it. There is a fair amount going on in the implementation, and I don't have the time right at the moment to immerse myself in it. I'll follow up in the future to see if I find some way to reduce the number of instances.

[3v1n0]

Mh, I see... Thanks for the explaination.

Using GIO FileMonitor isn't an option here? As you'd avoid to care about different file systems too.

[celeste-sinead]

Since it sounds like this isn't exactly an easy fix, in the interim, an obvious error message along the lines of "Unable to watch directory because system watch limit has been reached. Raising the system limit is recommended. This situation may also cause other applications that use inotify to misbehave." would go a long way. (There's a specific errno for this, and there's no way Sublime isn't also getting hit by it, so it should be pretty easy to detect).

The worst part of the problem is that it manifests as intermittent misbehaviour of random applications, and is hard to diagnose. A strong indication of the root cause and corrective action would turn this into a minor annoyance.

[chubio]

I did not encounter a problem where other applications misbehaved, but I had the problem where the sidebar was not updated when new files were created in a folder in my project. I can confirm that in my case the bottleneck is definitely the number of instances and not the number of watches (where the default value is 128 instances not 8K like the watches). To make things worse, I work on a server where I don't have admin access to tweak that number and where other coworkers also use sublime, so the 128 instances are shared between all of us. I don't know inotify except for what is mentioned in this thread, but I sure hope there can be a way to make sublime use watches instead of instances to achieve the same behavior.

[fpeterschmitt]

Hello,

I raised fs.inotify.max_user_watches to 524288 on my machine so Sublime is able to handle a consequent amount of files. In fact i have exactly 101786 files under this project.

For me this does not induce a high CPU load, Sublime is very quiet in terms of resources.

But it does consumes CPU when Sublime cannot create that amount of watches. It also makes trouble at seeing new files even created from Sublime, in the project files tree.

Also, maybe I should create a separate issue for this, but could Sublime notify about the lack of watches on the system? I discovered this by using pycharm that recommends raising the amount of available watches.

Edit: i use the latest SublimeText 3 Nightly, on ArchLinux with kernel 4.11.9.

[devendranaga]

Hello,

There are too many git instances n the version Build 3143. See below ps -aex on MacOS. After a while, nothing opens up and I get "Too many processes" and all of a sudden MacOS reboots informing that there's an issue with the OS. Please let me know if you require additional debug info..

Devendras-MacBook-Pro:~ devnaga$ ps -aex | grep git | wc -l

41376 ?? 0:00.00 (git)
41383 ?? 0:00.00 (git)
41385 ?? 0:00.00 (git)
41390 ?? 0:00.00 (git)
41398 ?? 0:00.00 (git)
41400 ?? 0:00.00 (git)
41406 ?? 0:00.00 (git)
41409 ?? 0:00.00 (git)
41412 ?? 0:00.00 (git)
41414 ?? 0:00.00 (git)
41417 ?? 0:00.00 (git)
41423 ?? 0:00.00 (git)
41425 ?? 0:00.00 (git)
41427 ?? 0:00.00 (git)
41431 ?? 0:00.00 (git)
41437 ?? 0:00.00 (git)
41441 ?? 0:00.00 (git)
41443 ?? 0:00.00 (git)
41447 ?? 0:00.00 (git)
41463 ?? 0:00.00 (git)
41468 ?? 0:00.00 (git)
41473 ?? 0:00.00 (git)
41479 ?? 0:00.00 (git)
41481 ?? 0:00.00 (git)
41483 ?? 0:00.00 (git)
51793 ?? 0:00.00 (git)
51798 ?? 0:00.00 (git)
51799 ?? 0:00.00 (git)
51814 ?? 0:00.00 (git)
51824 ?? 0:00.00 (git)
51826 ?? 0:00.00 (git)
51833 ?? 0:00.00 (git)
51835 ?? 0:00.00 (git)
51837 ?? 0:00.00 (git)
51841 ?? 0:00.00 (git)
51843 ?? 0:00.00 (git)
51848 ?? 0:00.00 (git)
51850 ?? 0:00.00 (git)
51861 ?? 0:00.00 (git)
51863 ?? 0:00.00 (git)

[FichteFoll]

Sublime Text doesn't run git by itself, so this must be some plugin's work. I suggest disabling a few (git-related) packages and trying to find the one causing these many unclodes git processes.

[wbond]

This was fixed in build 3158.

[digulla]

I'm on ST3, Build 3207 and ST again eats all watchers that it can (8K on the system):

   INOTIFY
   WATCHER
    COUNT     PID     CMD
----------------------------------------
    4183     6197  /snap/sublime-text/69/opt/sublime_text/plugin_host 6182 --auto-shell-env
    4183     6182  /snap/sublime-text/69/opt/sublime_text/sublime_text
    2375     6205  /usr/bin/node $HOME/.config/sublime-text-3/Packages/TypeScript/tsserver/tsserver.js 

Is this related to the TypeScript plugin or is the bug back?

Maybe a workaround would be to add a project-specific exclude list of folders. By default, that should include:

.git
node-modules
venv

since those folders usually contain thousands of files and devs will rarely look into them.

[trwww]

ditto @digulla's comment, this is not fixed. ST3 build 3211, can't run npm start and friends because ST uses up the watch limit:

COUNT     PID     CMD
----------------------------------------
    8030    18544  /opt/sublime_text/plugin_host 18532 --auto-shell-env
    8030    18532  /opt/sublime_text/sublime_text

[melbahja]

@wbond build 3211 consume 2x time watchers, plugin_host watches the same files that sublime_text process watching...

   COUNT    PID            CMD
----------------------------------------
   1106     32961        /opt/sublime_text/plugin_host 32949 --auto-shell-env
   1106     32949        /opt/sublime_text/sublime_text

EDIT:
Watching all plugins and packages all the time make things worse... if there a setting option to disable watching plugins and packages.. and the user still have an option to reload them manually!

[tajayi]

Also seeing this problem on 3211. I tried disabling indexing and git but that didn't seem to help