core.useBuiltinFSMonitor and file locking

[johnburnett]

Hi - I was just trying the new core.useBuiltinFSMonitor feature and ran into issues with a product failing due to file locks. Specifically, we have a repo that's storing a Unity (https://unity.com/) project, and one of the things Unity does is store/update packages within a subdir of the project using its own package manager. This subdir is .gitignore'd for us. However, it appears that when Unity updates its packages it's attempting to delete some working files/dirs and was unable to due to directories being locked (I'm including the error text below for future google searches).

I'm curious if this is expected behavior, or is the file watching intended to be a bit more unobtrusive (e.g. opening with FILE_SHARE_DELETE, not setting cwd into subdirs of the repo, and anything else that might avoid locking files/dirs on Windows). Or is this an inherent limitation of how it works?

John

An error occurred while resolving packages:
  Project has invalid dependencies:
    com.unity.build-report-inspector: EBUSY: resource busy or locked, rmdir
'D:\path-to-repo\Library\PackageCache\.tmp416602wsbNnhmYaeJ\clone'
    com.unity.shadergraphpropertyrenamer: EBUSY: resource busy or locked, rmdir
'D:\path-to-repo\Library\PackageCache\.tmp41660ChXqOGABAzw9\clone'
    com.unity.ui.controls-extra: EBUSY: resource busy or locked, rmdir
'D:\path-to-repo\Library\PackageCache\.tmp41660Bz07sYNd4dFk\clone'
(For the full error message, see the editor log file located at [C:/Users/username/AppData/Local/Unity/Editor/Editor.log])

Click on Retry to relaunch Unity and reopen your project.

Click on COntinue to launch Unity.  Some or all packages may not be imported which may cause compilation errors.

[dscho]

I bet this is not just any old directory that is getting deleted, but a clone of a Git repository. The problem with that is that the fsmonitor--daemon seems not to be exiting fast enough when detecting that the .git directory was deleted.

@jeffhostetler do you think we can chdir("/") and close the file descriptor quicker upon receiving that event? Or do you have any better strategy in mind?

[johnburnett]

@dscho - yep, looks like your guess was correct. Using procmon, I'm seeing this:

git.EXE 16352   CreateFile  D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  SUCCESS Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0, OpenResult: Created
git.EXE 16352   CloseFile   D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  SUCCESS 
git.EXE 16352   CreateFile  D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\.git\info\attributes  NAME NOT FOUND  Desired Access: Generic Read, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, Write, AllocationSize: n/a
git.EXE 16352   CreateFile  D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  SUCCESS Desired Access: Read Data/List Directory, Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
git.EXE 16352   QueryDirectory  D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  SUCCESS 0: ., 1: ..
git.EXE 16352   QueryDirectory  D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  NO MORE FILES   
git.EXE 16352   CloseFile   D:\projectdir\Library\PackageCache\.tmp27560V1bsbnHt8qgU\clone\com.unity.build-report-inspector  SUCCESS 

... so it's cloning git repo into a temp dir, then moving the contents out once that's done (the final package contents don't have a .git dir in the end).

I had core.useBuiltinFSMonitor enabled globally (I just checked the checkbox in the installer when I saw it, thinking "sure, let's try this out"). This was on a freshly imaged machine, so I was installing/recloning the world. I can change the settings for the parent repo, but we have no control over the clones that appear in ../.tmp*/clone (that's all wrangled by the package manager).

I wonder if it's also worth respecting the .gitignore in the parent git repo (if that's even possible in this situation)?

[jeffhostetler]

Interesting.

Globally unset core.useBuiltinFSMonitor and stop any fsmonitor daemon processes. You can cd into each and do a git fsmonitor--daemon stop. (Or just reboot.)

Then do git config core.useBuiltinFSMonitor true in your source directory. and then git fsmonitor--daemon start (this step is optional, since the daemon should auto-start), but then you can do git fsmonitor--daemon status if you want.

WRT the .gitignore, I'm not sure that that's something I can control. Git processes started in a child workdir/repo never see/know about containing repos. It's a similar situation for submodules -- when the child is started within the submodule, it doesn't know about the super.

[dscho]

The problem is not the .gitignore'd files/directory. The problem is that the top-level directory cannot be deleted because the fsmonitor--daemon is still running in that directory.

I have a TODO item to have the daemon cd itself out of the workdir once it gets up and running.

That sounds like a good plan.

How certain are we that the handle that we need for ReadDirectoryChangesW() does not block the removal of that directory?

[johnburnett]

Globally unset core.useBuiltinFSMonitor and stop any fsmonitor daemon processes. You can cd into each and do a git fsmonitor--daemon stop. (Or just reboot.)

Already done :) (I just killed all the daemon processes, didn't know there was a daemon stop command, neat).

Then do git config core.useBuiltinFSMonitor true in your source directory. and then git fsmonitor--daemon start (this step is optional, since the daemon should auto-start), but then you can do git fsmonitor--daemon status if you want.

I'm unfamiliar with how hierarchical git configs are in this situation - will the repos that are cloned down within this parent project/repo inherit the config of the parent repo? I'm guessing based on this:

WRT the .gitignore, I'm not sure that that's something I can control. Git processes started in a child workdir/repo never see/know about containing repos.

...that the answer is "they don't"? In which case, will the behavior be that the fs monitor will spin up for any git operations in the parent repo (potentially seeing the child repos as they're cloned down), but those child repos won't spin up their own daemons for the files they track?

[dscho]

The answer is indeed "they don't".

However, the system (/etc/gitconfig) and "global" (~/.gitconfig) are respected.