Fix the following CAS-related tests with LLVM_ENABLE_ONDISK_CAS=OFF on Windows #11140

hjyamauchi · 2025-08-06T20:47:36Z

Fix the following CAS-related tests with LLVM_ENABLE_ONDISK_CAS=OFF on Windows

Clang :: ClangScanDeps/modules-include-tree-pch-common-stale.c
Clang :: Driver/fdepscan-prefix-map-sdk.c
Clang :: Driver/fdepscan-prefix-map-toolchain.c
Clang :: Driver/fdepscan-prefix-map.c
Clang-Unit :: ./AllClangUnitTests.exe/DependencyScanningCASFilesystem/DirectiveScanFailure
Clang-Unit :: ./AllClangUnitTests.exe/DependencyScanningCASFilesystem/FilenameSpelling
Clang-Unit :: ./AllClangUnitTests.exe/IncludeTree/IncludeTreeFileSystemOverlay
Clang-Unit :: ./AllClangUnitTests.exe/IncludeTree/IncludeTreeScan
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSIteration
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSRecursiveIteration
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSRecursiveIterationNoPush
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BrokenSymlinkRealFSIteration
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BrokenSymlinkRealFSRecursiveIteration
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/ExcludeFromTacking
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/Exists
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/MultipleWorkingDirs
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccesses
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccessesExists
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccessesStack
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/caseSensitivityDir
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/caseSensitivityFile
LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/getRealPath
LLVM-Unit :: CAS/./CASTests.exe/OnDiskCASLoggerTest/MultiProcess
LLVM-Unit :: CAS/./CASTests.exe/OnDiskCASLoggerTest/MultiThread
LLVM-Unit :: Support/./SupportTests.exe/OnDiskBackendTest/OnlyIfDifferent
LLVM-Unit :: Support/./SupportTests.exe/SourceMgrTest/AddIncludedFile
LLVM-Unit :: Support/./SupportTests.exe/TreePathPrefixMapperTest/map
LLVM-Unit :: Support/./SupportTests.exe/TreePathPrefixMapperTest/mapInPlace
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/Keep/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/Keep/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepEmpty/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepEmpty/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlush/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlush/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlushProxy/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlushProxy/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk_DisableTemporaries
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk_DisableTemporaries
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk_DisableRemoveOnSignal
LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk_DisableTemporaries

hjyamauchi · 2025-08-06T20:47:55Z

@swift-ci please test llvm

hjyamauchi · 2025-08-06T22:56:19Z

@swift-ci please test llvm

hjyamauchi · 2025-08-07T00:40:48Z

@swift-ci please test llvm

clang/test/Driver/fdepscan-prefix-map-sdk.c

cachemeifyoucan · 2025-08-07T20:25:11Z

llvm/include/llvm/CAS/CASFileSystem.h

@@ -20,12 +20,14 @@ class CASID;
 // FIXME: Consider taking a "mount point". Then this could perhaps be


The FIXME in the file should be updated.

I am not sure what this comment meant. Do you have a suggestion?

Do we need a mount point for windows? I feel like multiple roots are already handled. should just remove it.

cachemeifyoucan · 2025-08-07T20:25:24Z

llvm/include/llvm/CAS/CASFileSystem.h

@@ -20,12 +20,14 @@ class CASID;
 // FIXME: Consider taking a "mount point". Then this could perhaps be
 // generalized for windows.
 Expected<std::unique_ptr<vfs::FileSystem>>
-createCASFileSystem(std::shared_ptr<ObjectStore> DB, const CASID &RootID);
+createCASFileSystem(std::shared_ptr<ObjectStore> DB, const CASID &RootID,
+                    sys::path::Style PathStyle = sys::path::Style::native);

 // FIXME: Consider taking a "mount point". Then this could perhaps be


cachemeifyoucan · 2025-08-07T20:27:33Z

llvm/include/llvm/Support/FileSystem.h

 /// @returns errc::success if result has been successfully set, otherwise a
 ///          platform-specific error_code.
 LLVM_ABI std::error_code status(const Twine &path, file_status &result,
-                                bool follow = true);
+                                bool follow = true,
+                                bool UseWinFileIndex = false);


Let's put this change onto llvm.org/main (but you can land here first if you want).

let's land here first if we really need it

I removed UseWinFileIndex

llvm/include/llvm/CAS/FileSystemCache.h

llvm/include/llvm/Testing/Support/SupportHelpers.h

llvm/unittests/CAS/CachingOnDiskFileSystemTest.cpp

cachemeifyoucan · 2025-08-07T20:40:33Z

llvm/unittests/Support/VirtualOutputBackendsTest.cpp

@@ -561,6 +561,8 @@ Error OnDiskOutputBackendProvider::checkKept(StringRef FilePath,

  sys::fs::UniqueID UID =
      shouldUseTemporaries(*Info) ? *Info->TempUID : *Info->UID;
+  // For this check to pass on Windows, we need to use the
+  // UseWinFileIndex option in fs::status() call in getCurrentUniqueID()


On windows, maybe we can just check the file content is the same? I don't see there is strict need to test for rename not changing UID.

Also, does WinFileIndex can be used always as UniqueID?

Checking the file content sounds good and I added such code here.

The file indices in Windows (that WinFileIndex uses) doesn't seem to be always unique, unfortunately, it could be reused after the file handles are closed or they aren't unique on network files, etc. according to this change:
llvm@02a3754 which changed UniqueID on Windows from file indices to hash of the canonical file paths.

So it doesn't seem like a good idea to use file indices after all

The only other places that I used WinFileIndex were the OnDiskBackendTest OnlyIfDifferent test below (around line 773 in the same file.) That test checks the identity of files across file overwrites. So, UID based on file paths won't be able to detect a difference.

Do you think we could do some alternative checks without using UniqueID in the OnlyIfDifferent test for Windows? If OnlyIfDifferent is about performance rather than correctness or a non-critical feature, could we maybe disable this on Windows?

Does CAS rely on sys::fs::UniqueID being truly unique in other places (on Mac/Linux)?

We can probably do a relax version of the check on windows (since here it is trying to test the rename behavior using UniqueID, which is probably not true for all platforms) or all the platforms.

And yes, UniqueID needs to be unique, and unfortunately it is heavily relied in some part of the LLVM (clang #pragma once include for example). However, many file systems actually doesn't guarantee that and it is causing some hard to debug issue on those file systems.

We should not change the underlying UniqueID for windows yet just because of this test.

I added file content checks as the alternate for windows in the OnlyIfDifferent test. I removed UseWinFileIndex.

…n Windows Clang :: ClangScanDeps/modules-include-tree-pch-common-stale.c Clang :: Driver/fdepscan-prefix-map-sdk.c Clang :: Driver/fdepscan-prefix-map-toolchain.c Clang :: Driver/fdepscan-prefix-map.c Clang-Unit :: ./AllClangUnitTests.exe/DependencyScanningCASFilesystem/DirectiveScanFailure Clang-Unit :: ./AllClangUnitTests.exe/DependencyScanningCASFilesystem/FilenameSpelling Clang-Unit :: ./AllClangUnitTests.exe/IncludeTree/IncludeTreeFileSystemOverlay Clang-Unit :: ./AllClangUnitTests.exe/IncludeTree/IncludeTreeScan LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSIteration LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSRecursiveIteration LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BasicRealFSRecursiveIterationNoPush LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BrokenSymlinkRealFSIteration LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/BrokenSymlinkRealFSRecursiveIteration LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/ExcludeFromTacking LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/Exists LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/MultipleWorkingDirs LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccesses LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccessesExists LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/TrackNewAccessesStack LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/caseSensitivityDir LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/caseSensitivityFile LLVM-Unit :: CAS/./CASTests.exe/CachingOnDiskFileSystemTest/getRealPath LLVM-Unit :: CAS/./CASTests.exe/OnDiskCASLoggerTest/MultiProcess LLVM-Unit :: CAS/./CASTests.exe/OnDiskCASLoggerTest/MultiThread LLVM-Unit :: Support/./SupportTests.exe/OnDiskBackendTest/OnlyIfDifferent LLVM-Unit :: Support/./SupportTests.exe/SourceMgrTest/AddIncludedFile LLVM-Unit :: Support/./SupportTests.exe/TreePathPrefixMapperTest/map LLVM-Unit :: Support/./SupportTests.exe/TreePathPrefixMapperTest/mapInPlace LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/Keep/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/Keep/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepEmpty/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepEmpty/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlush/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlush/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlushProxy/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepFlushProxy/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectory/OnDisk_DisableTemporaries LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNested/OnDisk_DisableTemporaries LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk_DisableRemoveOnSignal LLVM-Unit :: Support/./SupportTests.exe/VirtualOutput/BackendTest/KeepMissingDirectoryNoImply/OnDisk_DisableTemporaries

hjyamauchi · 2025-08-08T18:20:07Z

@swift-ci please test llvm

cachemeifyoucan

LGTM. Thanks!

hjyamauchi force-pushed the cas branch from bde5139 to 33748e3 Compare August 6, 2025 22:54

hjyamauchi force-pushed the cas branch from 33748e3 to e5d4d99 Compare August 7, 2025 00:40

hjyamauchi marked this pull request as ready for review August 7, 2025 18:27

hjyamauchi requested a review from cachemeifyoucan August 7, 2025 18:29

cachemeifyoucan reviewed Aug 7, 2025

View reviewed changes

hjyamauchi force-pushed the cas branch from e5d4d99 to 64cca37 Compare August 8, 2025 00:15

hjyamauchi force-pushed the cas branch from 64cca37 to 42732ab Compare August 8, 2025 18:17

cachemeifyoucan approved these changes Aug 8, 2025

View reviewed changes

hjyamauchi merged commit 506713c into swiftlang:next Aug 8, 2025
0 of 2 checks passed

hjyamauchi mentioned this pull request Aug 8, 2025

Fix the CAS tests on Windows when LLVM_ENABLE_ONDISK_CAS=OFF #9041

Closed

		@@ -20,12 +20,14 @@ class CASID;
		// FIXME: Consider taking a "mount point". Then this could perhaps be

Fix the following CAS-related tests with LLVM_ENABLE_ONDISK_CAS=OFF on Windows #11140

Fix the following CAS-related tests with LLVM_ENABLE_ONDISK_CAS=OFF on Windows #11140

Conversation

hjyamauchi commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hjyamauchi commented Aug 6, 2025

Uh oh!

hjyamauchi commented Aug 6, 2025

Uh oh!

hjyamauchi commented Aug 7, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hjyamauchi commented Aug 8, 2025

Uh oh!

cachemeifyoucan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hjyamauchi commented Aug 6, 2025 •

edited

Loading