Skip to content

Commit

Permalink
[clangd] [C++20] [Modules] Introduce initial support for C++20 Modules (
Browse files Browse the repository at this point in the history
#66462)

Alternatives to https://reviews.llvm.org/D153114.

Try to address clangd/clangd#1293.

See the links for design ideas and the consensus so far. We want to have
some initial support in clang18.

This is the initial support for C++20 Modules in clangd.
As suggested by sammccall in https://reviews.llvm.org/D153114,
we should minimize the scope of the initial patch to make it easier
 to review and understand so that every one are in the same page:

> Don't attempt any cross-file or cross-version coordination: i.e. don't
    > try to reuse BMIs between different files, don't try to reuse BMIs
> between (preamble) reparses of the same file, don't try to persist the
> module graph. Instead, when building a preamble, synchronously scan
> for the module graph, build the required PCMs on the single preamble
> thread with filenames private to that preamble, and then proceed to
    > build the preamble.

This patch reflects the above opinions.

# Testing in real-world project

I tested this with a modularized library:
https://github.com/alibaba/async_simple/tree/CXX20Modules. This library
has 3 modules (async_simple, std and asio) and 65 module units. (Note
that a module consists of multiple module units). Both `std` module and
`asio` module have 100k+ lines of code (maybe more, I didn't count). And
async_simple itself has 8k lines of code. This is the scale of the
project.

The result shows that it works pretty well, ..., well, except I need to
wait roughly 10s after opening/editing any file. And this falls in our
expectations. We know it is hard to make it perfect in the first move.

# What this patch does in detail

- Introduced an option `--experimental-modules-support` for the support
for C++20 Modules. So that no matter how bad this is, it wouldn't affect
current users. Following off the page, we'll assume the option is
enabled.
- Introduced two classes `ModuleFilesInfo` and
`ModuleDependencyScanner`. Now `ModuleDependencyScanner` is only used by
`ModuleFilesInfo`.
- The class `ModuleFilesInfo` records the built module files for
specific single source file. The module files can only be built by the
static member function `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef
File, ...)`.
- The class `PreambleData` adds a new member variable with type
`ModuleFilesInfo`. This refers to the needed module files for the
current file. It means the module files info is part of the preamble,
which is suggested in the first patch too.
- In `isPreambleCompatible()`, we add a call to
`ModuleFilesInfo::CanReuse()` to check if the built module files are
still up to date.
- When we build the AST for a source file, we will load the built module
files from ModuleFilesInfo.

# What we need to do next

Let's split the TODOs into clang part and clangd part to make things
more clear.

The TODOs in the clangd part include:
1. Enable reusing module files across source files. The may require us
to bring a ModulesManager like thing which need to handle `scheduling`,
`the possibility of BMI version conflicts` and `various events that can
invalidate the module graph`.
2. Get a more efficient method to get the `<module-name> ->
<module-unit-source>` map. Currently we always scan the whole project
during `ModuleFilesInfo::buildModuleFilesInfoFor(PathRef File, ...)`.
This is clearly inefficient even if the scanning process is pretty fast.
I think the potential solutions include:
- Make a global scanner to monitor the state of every source file like I
did in the first patch. The pain point is that we need to take care of
the data races.
- Ask the build systems to provide the map just like we ask them to
provide the compilation database.
3. Persist the module files. So that we can reuse module files across
clangd invocations or even across clangd instances.

TODOs in the clang part include:
1. Clang should offer an option/mode to skip writing/reading the bodies
of the functions. Or even if we can requrie the parser to skip parsing
the function bodies.

And it looks like we can say the support for C++20 Modules is initially
workable after we made (1) and (2) (or even without (2)).
  • Loading branch information
ChuanqiXu9 authored Jul 18, 2024
1 parent c41fa0f commit fe6c240
Show file tree
Hide file tree
Showing 24 changed files with 1,327 additions and 4 deletions.
3 changes: 3 additions & 0 deletions clang-tools-extra/clangd/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -97,12 +97,14 @@ add_clang_library(clangDaemon
IncludeFixer.cpp
InlayHints.cpp
JSONTransport.cpp
ModulesBuilder.cpp
PathMapping.cpp
Protocol.cpp
Quality.cpp
ParsedAST.cpp
Preamble.cpp
RIFF.cpp
ScanningProjectModules.cpp
Selection.cpp
SemanticHighlighting.cpp
SemanticSelection.cpp
Expand Down Expand Up @@ -161,6 +163,7 @@ clang_target_link_libraries(clangDaemon
clangAST
clangASTMatchers
clangBasic
clangDependencyScanning
clangDriver
clangFormat
clangFrontend
Expand Down
8 changes: 8 additions & 0 deletions clang-tools-extra/clangd/ClangdLSPServer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include "Feature.h"
#include "GlobalCompilationDatabase.h"
#include "LSPBinder.h"
#include "ModulesBuilder.h"
#include "Protocol.h"
#include "SemanticHighlighting.h"
#include "SourceCode.h"
Expand Down Expand Up @@ -51,6 +52,7 @@

namespace clang {
namespace clangd {

namespace {
// Tracks end-to-end latency of high level lsp calls. Measurements are in
// seconds.
Expand Down Expand Up @@ -563,6 +565,12 @@ void ClangdLSPServer::onInitialize(const InitializeParams &Params,
Mangler.ResourceDir = *Opts.ResourceDir;
CDB.emplace(BaseCDB.get(), Params.initializationOptions.fallbackFlags,
std::move(Mangler));

if (Opts.EnableExperimentalModulesSupport) {
ModulesManager.emplace(*CDB);
Opts.ModulesManager = &*ModulesManager;
}

{
// Switch caller's context with LSPServer's background context. Since we
// rather want to propagate information from LSPServer's context into the
Expand Down
5 changes: 5 additions & 0 deletions clang-tools-extra/clangd/ClangdLSPServer.h
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,9 @@ class ClangdLSPServer : private ClangdServer::Callbacks,

/// Limit the number of references returned (0 means no limit).
size_t ReferencesLimit = 0;

/// Flag to hint the experimental modules support is enabled.
bool EnableExperimentalModulesSupport = false;
};

ClangdLSPServer(Transport &Transp, const ThreadsafeFS &TFS,
Expand Down Expand Up @@ -323,6 +326,8 @@ class ClangdLSPServer : private ClangdServer::Callbacks,
std::optional<OverlayCDB> CDB;
// The ClangdServer is created by the "initialize" LSP method.
std::optional<ClangdServer> Server;
// Manages to build module files.
std::optional<ModulesBuilder> ModulesManager;
};
} // namespace clangd
} // namespace clang
Expand Down
2 changes: 2 additions & 0 deletions clang-tools-extra/clangd/ClangdServer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ ClangdServer::ClangdServer(const GlobalCompilationDatabase &CDB,
Callbacks *Callbacks)
: FeatureModules(Opts.FeatureModules), CDB(CDB), TFS(TFS),
DynamicIdx(Opts.BuildDynamicSymbolIndex ? new FileIndex() : nullptr),
ModulesManager(Opts.ModulesManager),
ClangTidyProvider(Opts.ClangTidyProvider),
UseDirtyHeaders(Opts.UseDirtyHeaders),
LineFoldingOnly(Opts.LineFoldingOnly),
Expand Down Expand Up @@ -308,6 +309,7 @@ void ClangdServer::addDocument(PathRef File, llvm::StringRef Contents,
Inputs.Index = Index;
Inputs.ClangTidyProvider = ClangTidyProvider;
Inputs.FeatureModules = FeatureModules;
Inputs.ModulesManager = ModulesManager;
bool NewFile = WorkScheduler->update(File, Inputs, WantDiags);
// If we loaded Foo.h, we want to make sure Foo.cpp is indexed.
if (NewFile && BackgroundIdx)
Expand Down
6 changes: 6 additions & 0 deletions clang-tools-extra/clangd/ClangdServer.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#include "FeatureModule.h"
#include "GlobalCompilationDatabase.h"
#include "Hover.h"
#include "ModulesBuilder.h"
#include "Protocol.h"
#include "SemanticHighlighting.h"
#include "TUScheduler.h"
Expand Down Expand Up @@ -112,6 +113,9 @@ class ClangdServer {
/// This throttler controls which preambles may be built at a given time.
clangd::PreambleThrottler *PreambleThrottler = nullptr;

/// Manages to build module files.
ModulesBuilder *ModulesManager = nullptr;

/// If true, ClangdServer builds a dynamic in-memory index for symbols in
/// opened files and uses the index to augment code completion results.
bool BuildDynamicSymbolIndex = false;
Expand Down Expand Up @@ -477,6 +481,8 @@ class ClangdServer {
std::unique_ptr<BackgroundIndex> BackgroundIdx;
// Storage for merged views of the various indexes.
std::vector<std::unique_ptr<SymbolIndex>> MergedIdx;
// Manage module files.
ModulesBuilder *ModulesManager = nullptr;

// When set, provides clang-tidy options for a specific file.
TidyProviderRef ClangTidyProvider;
Expand Down
3 changes: 3 additions & 0 deletions clang-tools-extra/clangd/Compiler.h
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_COMPILER_H

#include "FeatureModule.h"
#include "ModulesBuilder.h"
#include "TidyProvider.h"
#include "index/Index.h"
#include "support/ThreadsafeFS.h"
Expand Down Expand Up @@ -60,6 +61,8 @@ struct ParseInputs {
TidyProviderRef ClangTidyProvider = {};
// Used to acquire ASTListeners when parsing files.
FeatureModuleSet *FeatureModules = nullptr;
// Used to build and manage (C++) modules.
ModulesBuilder *ModulesManager = nullptr;
};

/// Clears \p CI from options that are not supported by clangd, like codegen or
Expand Down
23 changes: 23 additions & 0 deletions clang-tools-extra/clangd/GlobalCompilationDatabase.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
#include "GlobalCompilationDatabase.h"
#include "Config.h"
#include "FS.h"
#include "ProjectModules.h"
#include "ScanningProjectModules.h"
#include "SourceCode.h"
#include "support/Logger.h"
#include "support/Path.h"
Expand Down Expand Up @@ -741,6 +743,20 @@ DirectoryBasedGlobalCompilationDatabase::getProjectInfo(PathRef File) const {
return Res->PI;
}

std::unique_ptr<ProjectModules>
DirectoryBasedGlobalCompilationDatabase::getProjectModules(PathRef File) const {
CDBLookupRequest Req;
Req.FileName = File;
Req.ShouldBroadcast = false;
Req.FreshTime = Req.FreshTimeMissing =
std::chrono::steady_clock::time_point::min();
auto Res = lookupCDB(Req);
if (!Res)
return {};

return scanningProjectModules(Res->CDB, Opts.TFS);
}

OverlayCDB::OverlayCDB(const GlobalCompilationDatabase *Base,
std::vector<std::string> FallbackFlags,
CommandMangler Mangler)
Expand Down Expand Up @@ -833,6 +849,13 @@ std::optional<ProjectInfo> DelegatingCDB::getProjectInfo(PathRef File) const {
return Base->getProjectInfo(File);
}

std::unique_ptr<ProjectModules>
DelegatingCDB::getProjectModules(PathRef File) const {
if (!Base)
return nullptr;
return Base->getProjectModules(File);
}

tooling::CompileCommand DelegatingCDB::getFallbackCommand(PathRef File) const {
if (!Base)
return GlobalCompilationDatabase::getFallbackCommand(File);
Expand Down
13 changes: 13 additions & 0 deletions clang-tools-extra/clangd/GlobalCompilationDatabase.h
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
#ifndef LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H
#define LLVM_CLANG_TOOLS_EXTRA_CLANGD_GLOBALCOMPILATIONDATABASE_H

#include "ProjectModules.h"
#include "support/Function.h"
#include "support/Path.h"
#include "support/Threading.h"
Expand Down Expand Up @@ -45,6 +46,12 @@ class GlobalCompilationDatabase {
return std::nullopt;
}

/// Get the modules in the closest project to \p File
virtual std::unique_ptr<ProjectModules>
getProjectModules(PathRef File) const {
return nullptr;
}

/// Makes a guess at how to build a file.
/// The default implementation just runs clang on the file.
/// Clangd should treat the results as unreliable.
Expand Down Expand Up @@ -76,6 +83,9 @@ class DelegatingCDB : public GlobalCompilationDatabase {

std::optional<ProjectInfo> getProjectInfo(PathRef File) const override;

std::unique_ptr<ProjectModules>
getProjectModules(PathRef File) const override;

tooling::CompileCommand getFallbackCommand(PathRef File) const override;

bool blockUntilIdle(Deadline D) const override;
Expand Down Expand Up @@ -122,6 +132,9 @@ class DirectoryBasedGlobalCompilationDatabase
/// \p File's parents.
std::optional<ProjectInfo> getProjectInfo(PathRef File) const override;

std::unique_ptr<ProjectModules>
getProjectModules(PathRef File) const override;

bool blockUntilIdle(Deadline Timeout) const override;

private:
Expand Down
Loading

0 comments on commit fe6c240

Please sign in to comment.