Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-37848: [C++][Gandiva] Migrate LLVM JIT engine from MCJIT to ORC v2/LLJIT #39098

Merged
merged 28 commits into from
Jan 4, 2024
Merged

GH-37848: [C++][Gandiva] Migrate LLVM JIT engine from MCJIT to ORC v2/LLJIT #39098

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
336a69d
Switch Gandiva JIT engine from MCJIT to ORC v2 with LLJIT API.
niyue Nov 26, 2023
06ed4b2
Store target machine instead of target ir analysi in the engine to av…
niyue Nov 26, 2023
5c310d0
Use a new configuration flag to indicate Gandiva engine to dump IR.
niyue Nov 26, 2023
772b5fd
Enhance module api of gandiva engine to ensure it is not called after…
niyue Nov 26, 2023
7a935f0
Add optional object cache for gandiva engine creation API.
niyue Nov 26, 2023
502bcc3
Add back the SetLLVMObjectCache API to addObjectFile to LLJIT instance.
niyue Nov 27, 2023
cb09cbf
Set generator for LLJIT so that current process symbol can be found.
niyue Nov 28, 2023
21bda88
Set module data layout correctly and add back the module parameter we…
niyue Nov 28, 2023
ffd322b
Use reinterpret_cast instead of C-style static cast for converting fu…
niyue Dec 5, 2023
6c28f46
Change LLVMGenerator and Engine's Make factory method to return arrow…
niyue Dec 6, 2023
b715b87
Make CompileFunction to return arrow Result to handle error better, a…
niyue Dec 6, 2023
37ad65a
Include inttypes.h to avoid PRIxPTR not defined issue in CI.
niyue Dec 6, 2023
cf7fb78
Hide LLJIT into implementation to try to address the compilation issue.
niyue Dec 6, 2023
6e9af35
Remove LLVM IR API from Gandiva's pyarrow binding since the configura…
niyue Dec 6, 2023
3f9fd5b
Use full qualified API to avoid compilation issue.
niyue Dec 7, 2023
9948762
Troubleshoot CI failure.
niyue Dec 7, 2023
ba5d8e3
Rename function name to be different with the module name used by LLJ…
niyue Dec 8, 2023
7c12fdf
Minor tweaks for llvm generator and engine to simplify the code.
niyue Dec 8, 2023
9f556ca
Add a minimum atexit symbol lookup test for troubleshooting.
niyue Dec 9, 2023
5627ab4
Remove the lookup test case used for troubleshooting.
niyue Dec 9, 2023
e2d89b6
Set several variables to be const.
niyue Dec 9, 2023
f5356e5
Add atexit symbol if it doesn't exist so that ASAN won't complain.
niyue Dec 13, 2023
31a4b09
Switch to use JITLink for LLVM 9+ to see if it addresses the ASAN issue.
niyue Dec 15, 2023
7462473
Make JITLink defaults to off, and add a benchmark for expression comp…
niyue Dec 15, 2023
3213dbf
Rename configuration to `dump_ir` and return Status for the SetLLVMOb…
niyue Dec 16, 2023
7177d7f
Add a gandiva Configuration class for the python binding so that dump…
niyue Dec 17, 2023
db819dc
Remove WIN64 macro since WIN32 could be enough to skip Windows.
niyue Dec 18, 2023
8b88f6d
Require error_context for AsArrowResult API.
niyue Dec 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cpp/cmake_modules/FindLLVMAlt.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,8 @@ if(LLVM_FOUND)
debuginfodwarf
ipo
linker
mcjit
native
orcjit
target)
if(LLVM_VERSION_MAJOR GREATER_EQUAL 14)
list(APPEND LLVM_TARGET_COMPONENTS passes)
Expand Down
17 changes: 15 additions & 2 deletions cpp/src/gandiva/configuration.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,12 @@ class GANDIVA_EXPORT Configuration {

explicit Configuration(bool optimize,
std::shared_ptr<FunctionRegistry> function_registry =
gandiva::default_function_registry())
gandiva::default_function_registry(),
bool dump_ir = false)
: optimize_(optimize),
target_host_cpu_(true),
function_registry_(function_registry) {}
function_registry_(std::move(function_registry)),
dump_ir_(dump_ir) {}

Configuration() : Configuration(true) {}

Expand All @@ -50,11 +52,13 @@ class GANDIVA_EXPORT Configuration {

bool optimize() const { return optimize_; }
bool target_host_cpu() const { return target_host_cpu_; }
bool dump_ir() const { return dump_ir_; }
std::shared_ptr<FunctionRegistry> function_registry() const {
return function_registry_;
}

void set_optimize(bool optimize) { optimize_ = optimize; }
void set_dump_ir(bool dump_ir) { dump_ir_ = dump_ir; }
void target_host_cpu(bool target_host_cpu) { target_host_cpu_ = target_host_cpu; }
void set_function_registry(std::shared_ptr<FunctionRegistry> function_registry) {
function_registry_ = std::move(function_registry);
Expand All @@ -65,6 +69,9 @@ class GANDIVA_EXPORT Configuration {
bool target_host_cpu_; /* set the mcpu flag to host cpu while compiling llvm ir */
std::shared_ptr<FunctionRegistry>
function_registry_; /* function registry that may contain external functions */
// flag indicating if IR dumping is needed, defaults to false, and turning it on will
// negatively affect performance
bool dump_ir_ = false;
};

/// \brief configuration builder for gandiva
Expand All @@ -83,6 +90,12 @@ class GANDIVA_EXPORT ConfigurationBuilder {
return configuration;
}

std::shared_ptr<Configuration> build_with_ir_dumping(bool dump_ir) {
std::shared_ptr<Configuration> configuration(
new Configuration(true, gandiva::default_function_registry(), dump_ir));
return configuration;
}

std::shared_ptr<Configuration> build(
std::shared_ptr<FunctionRegistry> function_registry) {
std::shared_ptr<Configuration> configuration(
Expand Down
Loading