cross-platform, cross-compile build toolchain #280
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,5 +21,11 @@ hs_err*.log | |
| *.iws | ||
| .idea | ||
|
|
||
| # python binaries | ||
| *.pyc | ||
|
|
||
| # Build input/output. | ||
| node | ||
| build.out | ||
| cmake.out | ||
| node.out | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| # Build-Steps | ||
|
|
||
| The J2V8 build-system performs several build steps in a fixed order to produce the final J2V8 packages for usage on the designated target platforms. What follows is a short summary for what each of the executed build-steps does and what output artifacts are produced by each step. | ||
|
|
||
| --- | ||
| ## Node.js | ||
|
|
||
| Builds the [Node.js](https://nodejs.org/en/) & [V8](https://developers.google.com/v8/) dependency artifacts that are later linked against by the J2V8 native bridge code. | ||
|
|
||
| __Inputs:__ | ||
| - Node.js source code | ||
| - see [Github](https://github.com/nodejs/node) | ||
| - Node.js GIT patches with customizations for integrating Node.js into J2V8 | ||
| - `./node.patches/*.diff` | ||
|
|
||
| __Artifacts:__ | ||
| - Node.js & V8 static link libraries | ||
| - `./node/out/` | ||
| - `./node/build/` | ||
| - `./node/Debug/` | ||
| - `./node/Release/` | ||
| --- | ||
| ## CMake | ||
|
|
||
| Uses [CMake](https://cmake.org/) to generate the native Makefiles / IDE project files to later build the J2V8 C++ native bridge shared libraries (.so/.dylib/.dll) | ||
|
|
||
| __Inputs__: | ||
| - Node.js / V8 static link libraries | ||
| - `./cmake/NodeJsUtils.cmake` | ||
| - CMakeLists & CMake utilities | ||
| - `CMakeLists.txt` | ||
| - `./cmake/*.cmake` | ||
|
|
||
| __Artifacts:__ | ||
| - CMake generated Makefiles / IDE Project-files | ||
|
There was a problem hiding this comment. wow, this documentation is so awesome 👍 There was a problem hiding this comment. Thanks .:heart: |
||
| - `./cmake.out/{platform}.{architecture}/` | ||
| --- | ||
| ## JNI | ||
|
|
||
| The previously generated Makefiles / IDE project files are used to compile and link the J2V8 C++ source code, which provides the JNI bridge to interop between the Java code and the C++ code of Node.js / V8. | ||
|
|
||
| __Inputs__: | ||
| - CMake generated Makefiles / IDE Project-files | ||
| - J2V8 C++ JNI source code | ||
| - `./jni/com_eclipsesource_v8_V8Impl.h` | ||
| - `./jni/com_eclipsesource_v8_V8Impl.cpp` | ||
|
|
||
| __Artifacts:__ | ||
| - J2V8 native shared libraries | ||
| - `./cmake.out/{platform}.{architecture}/libj2v8_{platform}_{abi}.{ext}` | ||
| - e.g. `./cmake.out/linux.x64/libj2v8_linux_x86_64.so` | ||
| - The built shared libraries will also be automatically copied to the required Java / Android project directories to be included in the .jar/.aar packages that will be built later. | ||
| - `./src/main/resources/` (Java) | ||
| - `./src/main/jniLibs/{abi}/libj2v8.so` (Android) | ||
| --- | ||
| ## Java / Android | ||
|
|
||
| Compiles the Java source code and packages it, including the previously built native libraries, into the final package artifacts. For the execution of this build-step [Maven](https://maven.apache.org/) (Java) or [Gradle](https://gradle.org/) (Android) are used for the respective target platforms. | ||
|
|
||
| __Inputs__: | ||
| - J2V8 native shared libraries | ||
| - J2V8 Java source code | ||
| - `./src/main/` | ||
| - J2V8 Java test source code | ||
| - `./src/test/` | ||
|
|
||
| __Artifacts:__ | ||
| - J2V8 platform-specific packages | ||
| - `./build.out/j2v8_{platform}_{abi}-{j2v8_version}.jar` | ||
| - e.g. `./build.out/j2v8_linux_x86_64-4.8.0-SNAPSHOT.jar` | ||
| - `./build/outputs/aar/j2v8-release.aar` | ||
| --- | ||
| ## JUnit | ||
|
|
||
| Runs the Java ([JUnit](http://junit.org/)) unit tests. | ||
|
|
||
| __Inputs__: | ||
| - J2V8 platform-specific packages | ||
| - J2V8 Java test source code | ||
| - `./src/test/` | ||
|
|
||
| __Artifacts:__ | ||
| - Maven Surefire test reports | ||
| - `./target/surefire-reports/` | ||
| - Gradle connected-test reports | ||
| - `./build/outputs/androidTest-results/connected/` | ||
| --- | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,183 @@ | ||
|
|
||
| # NOTE: this works around the windows docker / msbuild bug, that causes the | ||
| # VC++ linker to fail creating PDB debug files in a shared volume | ||
| # (see: https://social.msdn.microsoft.com/Forums/en-US/bd281c61-8e04-4f56-aec2-5ad565271d7f/windowsservercorelatest-vc-2015-build-tools-throws-linker-error-lnk1318-unexpected-pdb?forum=windowscontainers) | ||
| if(J2V8_CROSS_COMPILE AND J2V8_WIN32_PDB_DOCKER_FIX) | ||
| #{ | ||
| # TODO: remove this if the Win32 Docker PDB bug is fixed | ||
| # see: https://stackoverflow.com/a/39109288 | ||
| message("CMAKE-WARNING: applied workaround for windows docker PDB bug") | ||
| message("see: https://social.msdn.microsoft.com/Forums/en-US/bd281c61-8e04-4f56-aec2-5ad565271d7f/windowsservercorelatest-vc-2015-build-tools-throws-linker-error-lnk1318-unexpected-pdb?forum=windowscontainers") | ||
| set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) | ||
| #} | ||
| endif() | ||
|
|
||
| cmake_minimum_required(VERSION 3.6) | ||
| project(j2v8) | ||
|
|
||
| # adding cmake directory for includes | ||
| set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake) | ||
|
|
||
| # set up the module path | ||
| set(CMAKE_MODULE_PATH ${CMAKE_SOURCE_DIR}/cmake) | ||
|
|
||
| include(BuildUtils) | ||
| include(NodeJsUtils) | ||
| include(Policies) | ||
|
|
||
| #----------------------------------------------------------------------- | ||
| # BUILD PLATFORM SETUP & VARIABLES | ||
| #----------------------------------------------------------------------- | ||
|
|
||
| if(CMAKE_SYSTEM_NAME STREQUAL "Android") | ||
| #{ | ||
| set(JAVA_PLATFORM_NAME "android") | ||
|
|
||
| # output library filename | ||
| set(J2V8_LIB_PLATFORM_NAME "android") | ||
| set(J2V8_LIB_PREFIX "") | ||
| set(J2V8_LIB_ARCH_NAME ${CMAKE_ANDROID_ARCH_ABI}) | ||
| #} | ||
| elseif(CMAKE_SYSTEM_NAME STREQUAL "Linux") | ||
| #{ | ||
| set(JAVA_PLATFORM_NAME "linux") | ||
|
|
||
| # output library filename | ||
| set(J2V8_LIB_PLATFORM_NAME "linux") | ||
| set(J2V8_LIB_PREFIX "") | ||
| set(J2V8_LIB_ARCH_NAME "x86") | ||
| #} | ||
| elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") | ||
| #{ | ||
| set(JAVA_PLATFORM_NAME "darwin") | ||
|
|
||
| # output library filename | ||
| set(J2V8_LIB_PLATFORM_NAME "macosx") | ||
| set(J2V8_LIB_PREFIX "") | ||
| set(J2V8_LIB_ARCH_NAME "x86") | ||
| #} | ||
| elseif(CMAKE_SYSTEM_NAME STREQUAL "Windows") | ||
| #{ | ||
| set(JAVA_PLATFORM_NAME "win32") | ||
|
|
||
| # output library filename | ||
| set(J2V8_LIB_PLATFORM_NAME "win32") | ||
| set(J2V8_LIB_PREFIX "lib") | ||
| set(J2V8_LIB_ARCH_NAME "x86") | ||
| #} | ||
| endif() | ||
|
|
||
| #----------------------------------------------------------------------- | ||
| # DEPENDENCY SETTINGS / CMAKE OPTIONS | ||
| #----------------------------------------------------------------------- | ||
|
|
||
| # look for dependencies | ||
| find_package(Java) | ||
|
|
||
| # j2v8 dependency options | ||
| set(J2V8_JDK_DIR ${Java_ROOT} CACHE STRING "Path to the Java JDK dependency") | ||
| set(J2V8_NODEJS_DIR "${CMAKE_SOURCE_DIR}/node" CACHE STRING "Path to the Node.js dependency") | ||
|
|
||
| # get the required Node.js link libraries | ||
| get_njs_libs(${J2V8_NODEJS_DIR} "Debug") | ||
| get_njs_libs(${J2V8_NODEJS_DIR} "Release") | ||
|
|
||
| # j2v8 build options | ||
| option(J2V8_NODE_COMPATIBLE "Build the J2V8 native bridge with Node.js support enabled" ON) | ||
| option(J2V8_BUILD_ONLY_DEBUG_RELEASE "Generate only Debug and Release configurations (exclude RelWithDebInfo and MinSizeRel)" ON) | ||
|
|
||
| if(CMAKE_SYSTEM_NAME STREQUAL "Windows" AND MSVC) | ||
| #{ | ||
| option(J2V8_LINK_WITH_STATIC_MSVCRT "Link against the static version of the Microsoft Visual C++ Common Runtime (will link against the dynamic DLL version if this option is disabled)" ON) | ||
| #} | ||
| endif() | ||
|
|
||
| #----------------------------------------------------------------------- | ||
| # INCLUDE DIRECTORIES & SOURCE FILES | ||
| #----------------------------------------------------------------------- | ||
|
|
||
| # project include directories | ||
| set(include_dirs | ||
| ${J2V8_NODEJS_DIR} | ||
| ${J2V8_NODEJS_DIR}/src | ||
| ${J2V8_NODEJS_DIR}/deps/v8 | ||
| ${J2V8_NODEJS_DIR}/deps/v8/include | ||
| ) | ||
|
|
||
| # do not use JAVA_HOME for Java include files for Android | ||
| # (instead the Android NDK toolchain will introduce them automatically) | ||
| if(NOT CMAKE_SYSTEM_NAME STREQUAL "Android") | ||
| #{ | ||
| set(include_dirs | ||
| ${include_dirs} | ||
| ${J2V8_JDK_DIR}/include | ||
| ${J2V8_JDK_DIR}/include/${JAVA_PLATFORM_NAME} | ||
| ) | ||
| #} | ||
| endif() | ||
|
|
||
| # project source files | ||
| set(src_files | ||
| jni/com_eclipsesource_v8_V8Impl.cpp | ||
| jni/com_eclipsesource_v8_V8Impl.h | ||
| ) | ||
|
|
||
| source_group("" FILES ${src_files}) | ||
|
|
||
| #----------------------------------------------------------------------- | ||
| # BUILD SETTINGS & COMPILATION | ||
| #----------------------------------------------------------------------- | ||
|
|
||
| # tell gcc/clang to use the c++11 standard | ||
| set(CMAKE_CXX_STANDARD 11) | ||
|
|
||
| if(CMAKE_SYSTEM_NAME STREQUAL "Android") | ||
| set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -Wno-unused-function -Wno-unused-variable -O3 -funroll-loops -ftree-vectorize -ffast-math -fpermissive -fPIC ") | ||
| set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wno-unused-function -Wno-unused-variable -O3 -funroll-loops -ftree-vectorize -ffast-math -fpermissive -fPIC ") | ||
| endif() | ||
|
|
||
| # remove the MinSizeRel and RelWithDebInfo configurations | ||
| if(J2V8_BUILD_ONLY_DEBUG_RELEASE) | ||
| #{ | ||
| set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "limited configs" FORCE) | ||
| #} | ||
| endif() | ||
|
|
||
| # link against the static MS C++ runtime libraries | ||
| if(J2V8_LINK_WITH_STATIC_MSVCRT) | ||
| link_static_crt() | ||
| endif() | ||
|
|
||
| # create the j2v8 library | ||
| add_library(j2v8 SHARED ${src_files}) | ||
|
|
||
| # enable Node.js if requested by the build options above | ||
| if(J2V8_NODE_COMPATIBLE) | ||
| #{ | ||
| set_property(TARGET j2v8 PROPERTY COMPILE_DEFINITIONS ${COMPILE_DEFINITIONS} NODE_COMPATIBLE=1) | ||
| #} | ||
| endif() | ||
|
|
||
| # build output directory | ||
| set(LIBRARY_OUTPUT_DIRECTORY ${CMAKE_SOURCE_DIR}/lib) | ||
|
|
||
| # set the include directories | ||
| include_directories(${include_dirs}) | ||
|
|
||
| # link the necessary libraries | ||
| target_link_libraries(j2v8 | ||
| debug "${njs_Debug_libs}" | ||
| optimized "${njs_Release_libs}" | ||
| ) | ||
|
|
||
| #----------------------------------------------------------------------- | ||
| # OUTPUT SETTINGS & POST-BUILD | ||
| #----------------------------------------------------------------------- | ||
|
|
||
| # apply lib suffix if building a 64 bit target | ||
| if(CMAKE_CL_64 OR CMAKE_SIZEOF_VOID_P EQUAL 8) | ||
| set(J2V8_LIB_ARCH_NAME "${J2V8_LIB_ARCH_NAME}_64") | ||
| endif() | ||
|
|
||
| # set library output filename | ||
| set_target_properties(j2v8 PROPERTIES OUTPUT_NAME "${PROJECT_NAME}_${J2V8_LIB_PLATFORM_NAME}_${J2V8_LIB_ARCH_NAME}") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,58 +10,35 @@ We developed J2V8 as a high performance engine for our multi-platform mobile too | |
|
|
||
| Building J2V8 | ||
| ============= | ||
| Building J2V8 requires building both the native parts and the Java library (.jar/.aar file). To build the native parts we first build node.js as a library and then statically link J2V8 to that. The Java parts are built with maven/gradle. | ||
|
|
||
| J2V8 uses a cross-platform, cross-compiling build-system written in Python. | ||
|
|
||
| Follow these steps to build J2V8 from source: | ||
|
|
||
| 1) clone the Node.js source code | ||
| - `python prepare_build.py` | ||
|
There was a problem hiding this comment. It is worth to mention that this works only for python2 (tried with python3 but doesn't work), was there any reason why you choose python over lets say bash scripts? I am just thinking out loud here that not everyone has python2 installed. There was a problem hiding this comment. also when I already had node folder checked out the build failed: There was a problem hiding this comment. even most simple build didn't work on my latest manjaro (arch) linux: most likely I ran into this issue: gcc7 doesn't compile: this makes me think that perhaps docker should always be used... There was a problem hiding this comment.
There are several reasons why I would not use bash scripts for more complex scenarios
I targeted Python 2 primarily because you already need it if you want to build Node.js from source. But I will look into how much work it would take to make it compatible for Python 3 (but I don't think this will be a priority, since Python 2 is still very widely adopted) About the error you get from running There was a problem hiding this comment. You did so much amazing work I have to say. I think you have a point here, I was just thinking from a scenario that one does git clone and invokes command to build, the rest would happen in docker container. One of biggest advantages of docker as I understood this is to eliminate various linux distrobution flavours, e.g. gcc version, kernel version, different packages versions, python2 vs python3. In that respect it would be perhaps better to fork of python2 invoking a build inside a docker container (in theory) -> that way probability that build would work is higher. On another hand, if this is intended to be used by Travis to create releases (as most people may not care to build snapshots), then it doesn't really matter. As long as maintainers of this project know how to run this, then it is fine. |
||
| - This will download & prepare the latest compatible Node.js version for use in J2V8 | ||
| - The Node.js source code will be cloned into the local `node` sub-directory, which is the expected default location for the J2V8 build | ||
| 2) build Node.js and the J2V8 library | ||
| - `python build.py --target linux --arch x64 --node-enabled --cross-compile` | ||
| - or shorthand | ||
| - `python build.py -t linux -a x64 -ne -x` | ||
|
|
||
| For all available options, supported platforms and architectures you can consult the build-script help: | ||
|
|
||
| `python build.py --help` | ||
|
|
||
| Cross-Compiling | ||
| --------------- | ||
|
|
||
| For cross-compiling J2V8 uses [Docker](https://www.docker.com/) (android, linux, windows) and [Vagrant](https://www.vagrantup.com/) (macos). | ||
| The full source-code (of both J2V8 and Node.js) on the build-host are just shared via mounted volumes with the Docker / Vagrant machines, so you can quickly make changes and perform builds fast. | ||
|
|
||
| To invoke a cross-compile build, simply invoke the `build.py` script as usual but add the `--cross-compile`, `-x` flag. | ||
| This will automatically provision and run the necessary virtualization to run the requested build fully independent of your local environment. | ||
|
|
||
| <b>Note:</b> using Docker / Vagrant for cross-compiliation requires many gigabytes of harddrive space as well as downloading the required images & tools. | ||
|
|
||
| Tutorials | ||
| ========== | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good to know, so building with another NodeJs is not out of the box. I didn't know one needed such patches (although I believe there was a legitimate reason).
👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is more or less there for convenience / to be used if necessary ... I know there has been work in other issues / PRs to get the build working without any customization to node (which would be preferred of course), but until we get there, this was a clean way that I added to overcome any such issues in the meantime.