[InstallAPI] Pick up input headers by directory traversal #94508
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
llvmbot
added
clang
|
@llvm/pr-subscribers-clang Author: Cyndy Ishida (cyndyishida) ChangesMatch TAPI behavior and allow input headers to be resolved via a passed directory, which is expected to be a library sitting in a build directory. Patch is 34.57 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/94508.diff 19 Files Affected:
diff --git a/clang/include/clang/Basic/DiagnosticInstallAPIKinds.td b/clang/include/clang/Basic/DiagnosticInstallAPIKinds.td
index cdf27247602f2..e10fa71011f30 100644
--- a/clang/include/clang/Basic/DiagnosticInstallAPIKinds.td
+++ b/clang/include/clang/Basic/DiagnosticInstallAPIKinds.td
@@ -26,6 +26,8 @@ def err_unsupported_environment : Error<"environment '%0' is not supported: '%1'
def err_unsupported_os : Error<"os '%0' is not supported: '%1'">;
def err_cannot_read_input_list : Error<"could not read %0 input list '%1': %2">;
def err_invalid_label: Error<"label '%0' is reserved: use a different label name for -X<label>">;
+def err_directory_scanning: Error<"could not read directory '%0': %1">;
+def err_more_than_one_library: Error<"more than one framework/dynamic library found">;
} // end of command line category.
let CategoryName = "Verification" in {
diff --git a/clang/include/clang/InstallAPI/DirectoryScanner.h b/clang/include/clang/InstallAPI/DirectoryScanner.h
new file mode 100644
index 0000000000000..803328982ec87
--- /dev/null
+++ b/clang/include/clang/InstallAPI/DirectoryScanner.h
@@ -0,0 +1,81 @@
+//===- InstallAPI/DirectoryScanner.h ----------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// The DirectoryScanner for collecting library files on the file system.
+///
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_CLANG_INSTALLAPI_DIRECTORYSCANNER_H
+#define LLVM_CLANG_INSTALLAPI_DIRECTORYSCANNER_H
+
+#include "clang/Basic/FileManager.h"
+#include "clang/InstallAPI/Library.h"
+
+namespace clang::installapi {
+
+enum ScanMode {
+ /// Scanning Framework directory.
+ ScanFrameworks,
+ /// Scanning Dylib directory.
+ ScanDylibs,
+};
+
+class DirectoryScanner {
+public:
+ DirectoryScanner(FileManager &FM, ScanMode Mode = ScanMode::ScanFrameworks)
+ : FM(FM), Mode(Mode) {}
+
+ /// Scan for all input files throughout directory.
+ ///
+ /// \param Directory Path of input directory.
+ llvm::Error scan(StringRef Directory);
+
+ /// Take over ownership of stored libraries.
+ std::vector<Library> takeLibraries() { return std::move(Libraries); };
+
+ /// Get all the header files in libraries.
+ ///
+ /// \param Libraries Reference of collection of libraries.
+ static HeaderSeq getHeaders(ArrayRef<Library> Libraries);
+
+private:
+ /// Collect files for dylibs in usr/(local)/lib within directory.
+ llvm::Error scanForUnwrappedLibraries(StringRef Directory);
+
+ /// Collect files for any frameworks within directory.
+ llvm::Error scanForFrameworks(StringRef Directory);
+
+ /// Get a library from the libraries collection.
+ Library &getOrCreateLibrary(StringRef Path, std::vector<Library> &Libs) const;
+
+ /// Collect multiple frameworks from directory.
+ llvm::Error scanMultipleFrameworks(StringRef Directory,
+ std::vector<Library> &Libs) const;
+ /// Collect files from nested frameworks.
+ llvm::Error scanSubFrameworksDirectory(StringRef Directory,
+ std::vector<Library> &Libs) const;
+
+ /// Collect files from framework path.
+ llvm::Error scanFrameworkDirectory(StringRef Path, Library &Framework) const;
+
+ /// Collect header files from path.
+ llvm::Error scanHeaders(StringRef Path, Library &Lib, HeaderType Type,
+ StringRef BasePath,
+ StringRef ParentPath = StringRef()) const;
+
+ /// Collect files from Version directories inside Framework directories.
+ llvm::Error scanFrameworkVersionsDirectory(StringRef Path,
+ Library &Lib) const;
+ FileManager &FM;
+ ScanMode Mode;
+ StringRef RootPath;
+ std::vector<Library> Libraries;
+};
+
+} // namespace clang::installapi
+
+#endif // LLVM_CLANG_INSTALLAPI_DIRECTORYSCANNER_H
diff --git a/clang/include/clang/InstallAPI/HeaderFile.h b/clang/include/clang/InstallAPI/HeaderFile.h
index c67503d4ad49e..12a87c01ad1c4 100644
--- a/clang/include/clang/InstallAPI/HeaderFile.h
+++ b/clang/include/clang/InstallAPI/HeaderFile.h
@@ -97,6 +97,14 @@ class HeaderFile {
Other.Excluded, Other.Extra,
Other.Umbrella);
}
+
+ bool operator<(const HeaderFile &Other) const {
+ if (isExtra() && Other.isExtra())
+ return std::tie(Type, Umbrella) < std::tie(Other.Type, Other.Umbrella);
+
+ return std::tie(Type, Umbrella, Extra, FullPath) <
+ std::tie(Other.Type, Other.Umbrella, Other.Extra, Other.FullPath);
+ }
};
/// Glob that represents a pattern of header files to retreive.
diff --git a/clang/include/clang/InstallAPI/Library.h b/clang/include/clang/InstallAPI/Library.h
new file mode 100644
index 0000000000000..8373d424dd364
--- /dev/null
+++ b/clang/include/clang/InstallAPI/Library.h
@@ -0,0 +1,65 @@
+//===- InstallAPI/Library.h -------------------------------------*- C++ -*-===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+///
+/// Defines the content of a library, such as public and private
+/// header files, and whether it is a framework.
+///
+//===----------------------------------------------------------------------===//
+#ifndef LLVM_CLANG_INSTALLAPI_LIBRARY_H
+#define LLVM_CLANG_INSTALLAPI_LIBRARY_H
+
+#include "clang/InstallAPI/HeaderFile.h"
+#include "clang/InstallAPI/MachO.h"
+
+namespace clang::installapi {
+
+class Library {
+public:
+ Library(StringRef Directory) : BaseDirectory(Directory) {}
+
+ /// Capture the name of the framework by the install name.
+ ///
+ /// \param InstallName The install name of the library encoded in a dynamic
+ /// library.
+ static StringRef getFrameworkNameFromInstallName(StringRef InstallName);
+
+ /// Get name of library by the discovered file path.
+ StringRef getName() const;
+
+ /// Get discovered path of library.
+ StringRef getPath() const { return BaseDirectory; }
+
+ /// Add a header file that belongs to the library.
+ ///
+ /// \param FullPath Path to header file.
+ /// \param Type Access level of header.
+ /// \param IncludePath The way the header should be included.
+ void addHeaderFile(StringRef FullPath, HeaderType Type,
+ StringRef IncludePath = StringRef()) {
+ Headers.emplace_back(FullPath, Type, IncludePath);
+ }
+
+ /// Determine if library is empty.
+ bool empty() {
+ return SubFrameworks.empty() && Headers.empty() &&
+ FrameworkVersions.empty();
+ }
+
+private:
+ std::string BaseDirectory;
+ HeaderSeq Headers;
+ std::vector<Library> SubFrameworks;
+ std::vector<Library> FrameworkVersions;
+ bool IsUnwrappedDylib{false};
+
+ friend class DirectoryScanner;
+};
+
+} // namespace clang::installapi
+
+#endif // LLVM_CLANG_INSTALLAPI_LIBRARY_H
diff --git a/clang/include/clang/InstallAPI/MachO.h b/clang/include/clang/InstallAPI/MachO.h
index 1ea544412f4cd..6036a7e5397cb 100644
--- a/clang/include/clang/InstallAPI/MachO.h
+++ b/clang/include/clang/InstallAPI/MachO.h
@@ -31,6 +31,7 @@ using RecordLinkage = llvm::MachO::RecordLinkage;
using Record = llvm::MachO::Record;
using EncodeKind = llvm::MachO::EncodeKind;
using GlobalRecord = llvm::MachO::GlobalRecord;
+using InterfaceFile = llvm::MachO::InterfaceFile;
using ObjCContainerRecord = llvm::MachO::ObjCContainerRecord;
using ObjCInterfaceRecord = llvm::MachO::ObjCInterfaceRecord;
using ObjCCategoryRecord = llvm::MachO::ObjCCategoryRecord;
diff --git a/clang/lib/InstallAPI/CMakeLists.txt b/clang/lib/InstallAPI/CMakeLists.txt
index b36493942300b..b63173bc1be3e 100644
--- a/clang/lib/InstallAPI/CMakeLists.txt
+++ b/clang/lib/InstallAPI/CMakeLists.txt
@@ -8,10 +8,12 @@ set(LLVM_LINK_COMPONENTS
add_clang_library(clangInstallAPI
DiagnosticBuilderWrappers.cpp
+ DirectoryScanner.cpp
DylibVerifier.cpp
FileList.cpp
Frontend.cpp
HeaderFile.cpp
+ Library.cpp
Visitor.cpp
LINK_LIBS
diff --git a/clang/lib/InstallAPI/DirectoryScanner.cpp b/clang/lib/InstallAPI/DirectoryScanner.cpp
new file mode 100644
index 0000000000000..ae4ba52de6ba9
--- /dev/null
+++ b/clang/lib/InstallAPI/DirectoryScanner.cpp
@@ -0,0 +1,300 @@
+//===- DirectoryScanner.cpp -----------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/InstallAPI/DirectoryScanner.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/ADT/StringSwitch.h"
+#include "llvm/TextAPI/DylibReader.h"
+
+using namespace llvm;
+using namespace llvm::MachO;
+
+namespace clang::installapi {
+
+HeaderSeq DirectoryScanner::getHeaders(ArrayRef<Library> Libraries) {
+ HeaderSeq Headers;
+ for (const Library &Lib : Libraries)
+ llvm::append_range(Headers, Lib.Headers);
+ return Headers;
+}
+
+llvm::Error DirectoryScanner::scan(StringRef Directory) {
+ if (Mode == ScanMode::ScanFrameworks)
+ return scanForFrameworks(Directory);
+
+ return scanForUnwrappedLibraries(Directory);
+}
+
+llvm::Error DirectoryScanner::scanForUnwrappedLibraries(StringRef Directory) {
+ // Check some known sub-directory locations.
+ auto GetDirectory = [&](const char *Sub) -> OptionalDirectoryEntryRef {
+ SmallString<PATH_MAX> Path(Directory);
+ sys::path::append(Path, Sub);
+ return FM.getOptionalDirectoryRef(Path);
+ };
+
+ auto DirPublic = GetDirectory("usr/include");
+ auto DirPrivate = GetDirectory("usr/local/include");
+ if (!DirPublic && !DirPrivate) {
+ std::error_code ec = std::make_error_code(std::errc::not_a_directory);
+ return createStringError(ec,
+ "cannot find any public (usr/include) or private "
+ "(usr/local/include) header directory");
+ }
+
+ Library &Lib = getOrCreateLibrary(Directory, Libraries);
+ Lib.IsUnwrappedDylib = true;
+
+ if (DirPublic)
+ if (Error Err = scanHeaders(DirPublic->getName(), Lib, HeaderType::Public,
+ Directory))
+ return Err;
+
+ if (DirPrivate)
+ if (Error Err = scanHeaders(DirPrivate->getName(), Lib, HeaderType::Private,
+ Directory))
+ return Err;
+
+ return Error::success();
+}
+
+static bool isFramework(StringRef Path) {
+ while (Path.back() == '/')
+ Path = Path.slice(0, Path.size() - 1);
+
+ return llvm::StringSwitch<bool>(llvm::sys::path::extension(Path))
+ .Case(".framework", true)
+ .Default(false);
+}
+
+Library &
+DirectoryScanner::getOrCreateLibrary(StringRef Path,
+ std::vector<Library> &Libs) const {
+ if (Path.consume_front(RootPath) && Path.empty())
+ Path = "/";
+
+ auto LibIt =
+ find_if(Libs, [Path](const Library &L) { return L.getPath() == Path; });
+ if (LibIt != Libs.end())
+ return *LibIt;
+
+ Libs.emplace_back(Path);
+ return Libs.back();
+}
+
+Error DirectoryScanner::scanHeaders(StringRef Path, Library &Lib,
+ HeaderType Type, StringRef BasePath,
+ StringRef ParentPath) const {
+ std::error_code ec;
+ auto &FS = FM.getVirtualFileSystem();
+ PathSeq SubDirectories;
+ for (vfs::directory_iterator i = FS.dir_begin(Path, ec), ie; i != ie;
+ i.increment(ec)) {
+ StringRef HeaderPath = i->path();
+ if (ec)
+ return createStringError(ec, "unable to read: " + HeaderPath);
+
+ if (sys::fs::is_symlink_file(HeaderPath))
+ continue;
+
+ // Ignore tmp files from unifdef.
+ const StringRef Filename = sys::path::filename(HeaderPath);
+ if (Filename.starts_with("."))
+ continue;
+
+ // If it is a directory, remember the subdirectory.
+ if (FM.getOptionalDirectoryRef(HeaderPath))
+ SubDirectories.push_back(HeaderPath.str());
+
+ if (!isHeaderFile(HeaderPath))
+ continue;
+
+ // Skip files that do not exist. This usually happens for broken symlinks.
+ if (FS.status(HeaderPath) == std::errc::no_such_file_or_directory)
+ continue;
+
+ auto IncludeName = createIncludeHeaderName(HeaderPath);
+ Lib.addHeaderFile(HeaderPath, Type,
+ IncludeName.has_value() ? IncludeName.value() : "");
+ }
+
+ // Go through the subdirectories.
+ // Sort the sub-directory first since different file systems might have
+ // different traverse order.
+ llvm::sort(SubDirectories);
+ if (ParentPath.empty())
+ ParentPath = Path;
+ for (const StringRef Dir : SubDirectories)
+ return scanHeaders(Dir, Lib, Type, BasePath, ParentPath);
+
+ return Error::success();
+}
+
+llvm::Error
+DirectoryScanner::scanMultipleFrameworks(StringRef Directory,
+ std::vector<Library> &Libs) const {
+ std::error_code ec;
+ auto &FS = FM.getVirtualFileSystem();
+ for (vfs::directory_iterator i = FS.dir_begin(Directory, ec), ie; i != ie;
+ i.increment(ec)) {
+ StringRef Curr = i->path();
+
+ // Skip files that do not exist. This usually happens for broken symlinks.
+ if (ec == std::errc::no_such_file_or_directory) {
+ ec.clear();
+ continue;
+ }
+ if (ec)
+ return createStringError(ec, Curr);
+
+ if (sys::fs::is_symlink_file(Curr))
+ continue;
+
+ if (isFramework(Curr)) {
+ if (FM.getOptionalDirectoryRef(Curr))
+ continue;
+ Library &Framework = getOrCreateLibrary(Curr, Libs);
+ if (Error Err = scanFrameworkDirectory(Curr, Framework))
+ return Err;
+ }
+ }
+
+ return Error::success();
+}
+
+llvm::Error
+DirectoryScanner::scanSubFrameworksDirectory(StringRef Directory,
+ std::vector<Library> &Libs) const {
+ if (FM.getOptionalDirectoryRef(Directory))
+ return scanMultipleFrameworks(Directory, Libs);
+
+ std::error_code ec = std::make_error_code(std::errc::not_a_directory);
+ return createStringError(ec, Directory);
+}
+
+/// FIXME: How to handle versions? For now scan them separately as independent
+/// frameworks.
+llvm::Error
+DirectoryScanner::scanFrameworkVersionsDirectory(StringRef Path,
+ Library &Lib) const {
+ std::error_code ec;
+ auto &FS = FM.getVirtualFileSystem();
+ for (vfs::directory_iterator i = FS.dir_begin(Path, ec), ie; i != ie;
+ i.increment(ec)) {
+ const StringRef Curr = i->path();
+
+ // Skip files that do not exist. This usually happens for broken symlinks.
+ if (ec == std::errc::no_such_file_or_directory) {
+ ec.clear();
+ continue;
+ }
+ if (ec)
+ return createStringError(ec, Curr);
+
+ if (sys::fs::is_symlink_file(Curr))
+ continue;
+
+ // Each version should be a framework directory.
+ if (!FM.getOptionalDirectoryRef(Curr))
+ continue;
+
+ Library &VersionedFramework =
+ getOrCreateLibrary(Curr, Lib.FrameworkVersions);
+ if (Error Err = scanFrameworkDirectory(Curr, VersionedFramework))
+ return Err;
+ }
+
+ return Error::success();
+}
+
+llvm::Error DirectoryScanner::scanFrameworkDirectory(StringRef Path,
+ Library &Framework) const {
+ // If the framework is inside Kernel or IOKit, scan headers in the different
+ // directories separately.
+ Framework.IsUnwrappedDylib =
+ Path.contains("Kernel.framework") || Path.contains("IOKit.framework");
+
+ // Unfortunately we cannot identify symlinks in the VFS. We assume that if
+ // there is a Versions directory, then we have symlinks and directly proceed
+ // to the Versions folder.
+ std::error_code ec;
+ auto &FS = FM.getVirtualFileSystem();
+
+ for (vfs::directory_iterator i = FS.dir_begin(Path, ec), ie; i != ie;
+ i.increment(ec)) {
+ StringRef Curr = i->path();
+ // Skip files that do not exist. This usually happens for broken symlinks.
+ if (ec == std::errc::no_such_file_or_directory) {
+ ec.clear();
+ continue;
+ }
+
+ if (ec)
+ return createStringError(ec, Curr);
+
+ if (sys::fs::is_symlink_file(Curr))
+ continue;
+
+ StringRef FileName = sys::path::filename(Curr);
+ // Scan all "public" headers.
+ if (FileName.contains("Headers")) {
+ if (Error Err = scanHeaders(Curr, Framework, HeaderType::Public, Curr))
+ return Err;
+ continue;
+ }
+ // Scan all "private" headers.
+ if (FileName.contains("PrivateHeaders")) {
+ if (Error Err = scanHeaders(Curr, Framework, HeaderType::Private, Curr))
+ return Err;
+ continue;
+ }
+ // Scan sub frameworks.
+ if (FileName.contains("Frameworks")) {
+ if (Error Err = scanSubFrameworksDirectory(Curr, Framework.SubFrameworks))
+ return Err;
+ continue;
+ }
+ // Check for versioned frameworks.
+ if (FileName.contains("Versions")) {
+ if (Error Err = scanFrameworkVersionsDirectory(Curr, Framework))
+ return Err;
+ continue;
+ }
+ }
+
+ return Error::success();
+}
+
+llvm::Error DirectoryScanner::scanForFrameworks(StringRef Directory) {
+ RootPath = "";
+
+ // Expect a certain directory structure and naming convention to find
+ // frameworks.
+ static const char *SubDirectories[] = {"System/Library/Frameworks/",
+ "System/Library/PrivateFrameworks/"};
+
+ // Check if the directory is already a framework.
+ if (isFramework(Directory)) {
+ Library &Framework = getOrCreateLibrary(Directory, Libraries);
+ if (Error Err = scanFrameworkDirectory(Directory, Framework))
+ return Err;
+ return Error::success();
+ }
+
+ // Check known sub-directory locations.
+ for (const auto *SubDir : SubDirectories) {
+ SmallString<PATH_MAX> Path(Directory);
+ sys::path::append(Path, SubDir);
+
+ if (Error Err = scanMultipleFrameworks(Path, Libraries))
+ return Err;
+ }
+
+ return Error::success();
+}
+} // namespace clang::installapi
diff --git a/clang/lib/InstallAPI/Library.cpp b/clang/lib/InstallAPI/Library.cpp
new file mode 100644
index 0000000000000..bdfa3535273e1
--- /dev/null
+++ b/clang/lib/InstallAPI/Library.cpp
@@ -0,0 +1,40 @@
+//===- Library.cpp --------------------------------------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+
+#include "clang/InstallAPI/Library.h"
+
+using namespace llvm;
+namespace clang::installapi {
+
+const Regex Rule("(.+)/(.+)\\.framework/");
+StringRef Library::getFrameworkNameFromInstallName(StringRef InstallName) {
+ assert(InstallName.contains(".framework") && "expected a framework");
+ SmallVector<StringRef, 3> Match;
+ Rule.match(InstallName, &Match);
+ if (Match.empty())
+ return "";
+ return Match.back();
+}
+
+StringRef Library::getName() const {
+ assert(!IsUnwrappedDylib && "expected a framework");
+ StringRef Path = BaseDirectory;
+
+ // Return the framework name extracted from path.
+ while (!Path.empty()) {
+ if (Path.ends_with(".framework"))
+ return sys::path::filename(Path);
+ Path = sys::path::parent_path(Path);
+ }
+
+ // Otherwise, return the name of the BaseDirectory.
+ Path = BaseDirectory;
+ return sys::path::filename(Path.rtrim("/"));
+}
+
+} // namespace clang::installapi
diff --git a/clang/test/InstallAPI/asm.test b/clang/test/InstallAPI/asm.test
index b6af7f643d72f..9df644a823909 100644
--- a/clang/test/InstallAPI/asm.test
+++ b/clang/test/InstallAPI/asm.test
@@ -3,7 +3,7 @@
// RUN: sed -e "s|DSTROOT|%/t|g" %t/inputs.json.in > %t/inputs.json
// RUN: clang-installapi -target arm64-apple-macos13.1 \
-// RUN: -I%t/usr/include \
+// RUN: -...
[truncated]
|
zixu-w
reviewed
Jun 6, 2024
Match TAPI behavior and allow input headers to be resolved via a passed directory, which is expected to be a library sitting in a build directory.
cyndyishida
force-pushed
the
eng/PR-DirectoryScanner
branch
from
d859ae6 to
95cc0c9
Compare
|
ping |
zixu-w
approved these changes
Jun 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Match TAPI behavior and allow input headers to be resolved via a passed directory, which is expected to be a library sitting in a build directory.