File System Woes

In the past week, I've spent some time writing a Shader Daemon - a tool that usually consists of a process or thread running in the background detecting changes to shader files during development. A Shader Daemon allows our engine to trigger shader recompilations automatically during runtime when a shader file has been modified and saved. Sounds like a simple task, right?

There are a few ways that one might go about implementing such a tool. Jim Beveridge wrote a great summary on the various different methods here, but the first thing that might come to your mind would be to utilize some kind of kernel-level signal that fires when a file or directory has been updated. Indeed, WinAPI seem to have just the right function:

BOOL ReadDirectoryChangesW(
    HANDLE                          hDirectory,
    LPVOID                          lpBuffer,
    DWORD                           nBufferLength,
    BOOL                            bWatchSubtree,
    DWORD                           dwNotifyFilter,
    LPDWORD                         lpBytesReturned,
    LPOVERLAPPED                    lpOverlapped,
    LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
);

ReadDirectoryChangesW allows us to watch a specific directory for all kinds of changes - from file content modification to security policy updates. We can include or exclude notifications with dwNotifyFilter. Since we only really care about file content updates, we'll probably be interested in FILE_NOTIFY_CHANGE_LAST_WRITE.

But this is where we run into our first problem:

1) The notification is not guaranteed to be sent after the file has been unlocked

In other words if our code looks something like this:

while (ReadDirectoryChangesW(hDir, &notifyInfo, sizeof(notifyInfo), TRUE, FILE_NOTIFY_CHANGE_LAST_WRITE, &bytesReturned, NULL, NULL)) 
{
    FILE_NOTIFY_INFORMATION* info = reinterpret_cast<FILE_NOTIFY_INFORMATION*>(&notifyInfo);
    ...
    
    if (IsFileWeCareAbout(info->FileName))
    {
        m_DxcLibrary->CreateBlobFromFile(GetShaderPath(info->FileName));
        ....
    }
}

There is a very good chance that our program will fail because we're attempting to read a file that is very likely to still be locked. There is surprisingly no way around this (See: https://stackoverflow.com/a/1746823/3879417), and the consensus is that the best way forward is with a loop that checks if a handle to the file can be opened, and if not - sleep for some arbitrary time before trying to open the handle again. Once the handle has been opened, immediately close the handle and proceed with actually operating on the file.

So then our code would start to look more like this:

    ...
    if (IsFileWeCareAbout(info->FileName))
    {
        SleepUntilFileHandleCanBeOpened();
        m_DxcLibrary->CreateBlobFromFile(GetShaderPath(info->FileName));
        ....
    }

We now test out our Shader Daemon by saving a shader file in, let's say, Visual Studio, since this has been where we're doing development in the first place. This is where we see our next problem.

2) Visual Studio does not modify files directly

Instead, Visual studio saves files in a multi-step process. The gist of it is something along the lines of:

1. Save into temp file
2. Delete original file
3. Rename temp file to the original filename

So in fact, our above code will completely miss the correct shader file because info->FileName will return one or more files with unhelpful names that are something along the lines of iut23kj.tmp~.

The solution would then be to also detect file renames. We can do this by adding FILE_NOTIFY_CHANGE_FILE_NAME to our notification filter as well. We must also filter away info->Action == FILE_ACTION_RENAMED_OLD_NAME, since both FILE_ACTION_RENAMED_OLD_NAME and FILE_ACTION_RENAMED_NEW_NAME will be signalled.

I distinctly remember that, while working at ▒▒▒▒▒▒▒, the in-house Shader Daemon will not trigger shader recompilations if the file was saved within Visual studio for this exact reason. Most of us worked around this by using other text editors for shaders, which isn't ideal and also brings us to the next problem.

3) VSCode, along with a bunch of other editors, signals file write more than once

Yes, the exact same FILE_NOTIFY_CHANGE_LAST_WRITE is signaled twice in VSCode. As far as I can tell, these two events cannot be distinguished. This means that in the best-case scenario, the first compilation will fail (but the second will pass), and in the worst, you'll trigger shader compilation twice. I do not completely understand the mechanics causing this double signal, but you can refer to https://github.com/microsoft/vscode/issues/9419 and https://github.com/nodejs/node/issues/6112.

Dealing with the best case first, this happens because the file may be empty when the first signal occurs. It is very likely for DXC to throw an "Entry point not found" error when attempting to compile the shader.

In the worst case, the shader will be recompiled twice. This is fine if the shader is small, but if you were recompiling a complex shader with thousands of permutations, this can become a problem really quickly. I also recall that shaders were mysteriously compiling twice when saved with VSCode back in ▒▒▒▒▒▒▒, for this exact reason.

The only way around having the shader recompile twice may perhaps be to ignore a request for recompilation if no changes to the shader have been made. This however, will still not solve the issue of our logger printing out erroneous errors when the first save produces invalid or empty files. This is of course much less of an issue than double recompilations.