Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Node bindings on Windows #6334

Merged
merged 54 commits into from
Sep 20, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
e9c631d
Build Node bindings on Windows
SiarheiFedartsou Aug 25, 2022
2707eba
Build Node bindings on Windows
SiarheiFedartsou Aug 25, 2022
a36034b
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
33c9e05
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
fd7cf93
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
e32f03e
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
e177ef1
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
90e6c4b
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
28f9c46
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
3062375
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
f404d75
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
7a1a0ff
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
b6c22ac
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
232b814
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
76b671d
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
789de84
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
c5e6e02
Build Node bindings on Windows
SiarheiFedartsou Aug 26, 2022
3551ca1
wip
SiarheiFedartsou Sep 7, 2022
4c492c0
Merge branch 'master' into sf-windows-node
SiarheiFedartsou Sep 7, 2022
fb5cf31
wip
SiarheiFedartsou Sep 7, 2022
050095c
wip
SiarheiFedartsou Sep 7, 2022
7f0e5a3
wip
SiarheiFedartsou Sep 7, 2022
b3f8b06
wip
SiarheiFedartsou Sep 7, 2022
f5ffe1f
wip
SiarheiFedartsou Sep 7, 2022
f1ce6c7
wip
SiarheiFedartsou Sep 7, 2022
c53b825
wip
SiarheiFedartsou Sep 7, 2022
9b7ae7e
wip
SiarheiFedartsou Sep 8, 2022
15f8f67
wip
SiarheiFedartsou Sep 8, 2022
45ae6ea
wip
SiarheiFedartsou Sep 8, 2022
fcd03d8
wip
SiarheiFedartsou Sep 8, 2022
8961146
wip
SiarheiFedartsou Sep 8, 2022
217087f
wip
SiarheiFedartsou Sep 8, 2022
723c1f8
wip
SiarheiFedartsou Sep 8, 2022
7a359fc
wip
SiarheiFedartsou Sep 8, 2022
57c9b64
wip
SiarheiFedartsou Sep 8, 2022
075798a
wip
SiarheiFedartsou Sep 8, 2022
2c16318
wip
SiarheiFedartsou Sep 8, 2022
ff3ef89
Update osrm-backend.yml
SiarheiFedartsou Sep 8, 2022
fa1c4bb
Update osrm-backend.yml
SiarheiFedartsou Sep 8, 2022
a143711
Update node_package.sh
SiarheiFedartsou Sep 8, 2022
e738e8e
wip
SiarheiFedartsou Sep 9, 2022
8d9f4b0
wip
SiarheiFedartsou Sep 9, 2022
72bc8fd
wip
SiarheiFedartsou Sep 9, 2022
614e4b2
wip
SiarheiFedartsou Sep 9, 2022
3400be3
wip
SiarheiFedartsou Sep 9, 2022
f1ad211
wip
SiarheiFedartsou Sep 9, 2022
29c0c09
wip
SiarheiFedartsou Sep 9, 2022
467fafb
wip
SiarheiFedartsou Sep 9, 2022
ad1d4e6
wip
SiarheiFedartsou Sep 9, 2022
fd574c2
M1
SiarheiFedartsou Sep 9, 2022
75780d0
Build Node bindings on Windows
SiarheiFedartsou Sep 19, 2022
25f3f0f
Build Node bindings on Windows
SiarheiFedartsou Sep 19, 2022
b2a5fc7
Update osrm-backend.yml
SiarheiFedartsou Sep 19, 2022
62858cf
Build Node bindings on Windows
SiarheiFedartsou Sep 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@

# Declare files that will always have LF line endings on checkout.
*.sh text eol=lf

# https://eslint.org/docs/latest/rules/linebreak-style#using-this-rule-with-version-control-systems
*.js text eol=lf
38 changes: 32 additions & 6 deletions .github/workflows/osrm-backend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ jobs:
needs: format-taginfo-docs
runs-on: windows-2022
continue-on-error: false
env:
BUILD_TYPE: Release
ENABLE_APPLE_SILICON: "OFF"
steps:
- uses: actions/checkout@v3
- run: pip install conan==1.51.3
Expand All @@ -34,12 +37,38 @@ jobs:
node-version: 16
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using Node v16, not sure if we need matrix build for Windows building for older versions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may need Node v18 though too(which is currently latest)

- run: node --version
- run: npm --version
- name: Prepare environment
shell: bash
run: |
PACKAGE_JSON_VERSION=$(node -e "console.log(require('./package.json').version)")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was not able to check how it actually works with tag(because in order to do that I actually need to push a tag), but it at least sets PUBLISH=Off

echo PUBLISH=$([[ "${GITHUB_REF:-}" == "refs/tags/v${PACKAGE_JSON_VERSION}" ]] && echo "On" || echo "Off") >> $GITHUB_ENV
- run: npm install --ignore-scripts
- run: npm link --ignore-scripts
- uses: microsoft/setup-msbuild@v1.1
- name: Build
run: |
.\scripts\ci\windows-build.bat
- name: Run node tests
shell: bash
run: |
./lib/binding/osrm-datastore.exe test/data/ch/monaco.osrm
node test/nodejs/index.js
- name: Build Node package
shell: bash
run: ./scripts/ci/node_package.sh
- name: Publish Node package
if: ${{ env.PUBLISH == 'On' }}
uses: ncipollo/release-action@v1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't it check as well, but I expect that it should work as on other Linux/macOS jobs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can test it on your own fork by publishing some dummy versions. I'll give it a try on mine.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, well the Windows build was published, which is great.
https://github.com/mjjbell/osrm-backend/releases/tag/v9.90.22

But the Linux packaging failed.
https://github.com/mjjbell/osrm-backend/actions/runs/3063972998/jobs/4946640880

Not sure if it's related to these changes, looks like it might be a Conan issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I fixed it in #6360 (via RPATH changes, tbh I am not sure that it is caused by Conan, but may be)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with:
allowUpdates: true
artifactErrorsFailBuild: true
artifacts: build/stage/**/*.tar.gz
omitBody: true
omitBodyDuringUpdate: true
omitName: true
omitNameDuringUpdate: true
replacesArtifacts: true
token: ${{ secrets.GITHUB_TOKEN }}

format-taginfo-docs:
runs-on: ubuntu-20.04
Expand Down Expand Up @@ -526,15 +555,13 @@ jobs:
key: v3-test-${{ matrix.name }}-${{ github.sha }}
restore-keys: |
v3-test-${{ matrix.name }}-

- name: Prepare environment
run: |
PACKAGE_JSON_VERSION=$(node -e "console.log(require('./package.json').version)")
echo PUBLISH=$([[ "${GITHUB_REF:-}" == "refs/tags/v${PACKAGE_JSON_VERSION}" ]] && echo "On" || echo "Off") >> $GITHUB_ENV

echo "OSRM_INSTALL_DIR=${GITHUB_WORKSPACE}/install-osrm" >> $GITHUB_ENV
echo "OSRM_BUILD_DIR=${GITHUB_WORKSPACE}/build-osrm" >> $GITHUB_ENV

if [[ "$ENABLE_SANITIZER" == 'ON' ]]; then
# We can only set this after checkout once we know the workspace directory
echo "LSAN_OPTIONS=print_suppressions=0:suppressions=${GITHUB_WORKSPACE}/scripts/ci/leaksanitizer.conf" >> $GITHUB_ENV
Expand All @@ -549,12 +576,10 @@ jobs:
sudo mdutil -i off /
export MASON_OS=osx
fi

echo "MASON=${GITHUB_WORKSPACE}/scripts/mason.sh" >> $GITHUB_ENV

echo "CMAKE_URL=https://mason-binaries.s3.amazonaws.com/${MASON_OS}-x86_64/cmake/${CMAKE_VERSION}.tar.gz" >> $GITHUB_ENV
echo "CMAKE_DIR=mason_packages/${MASON_OS}-x86_64/cmake/${CMAKE_VERSION}" >> $GITHUB_ENV


- name: Install dev dependencies
run: |
python3 -m pip install conan==1.51.3
Expand Down Expand Up @@ -634,6 +659,7 @@ jobs:
else
APPLE_SILICON_FLAGS=()
fi

cmake .. -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
-DENABLE_CONAN=${ENABLE_CONAN:-OFF} \
-DENABLE_ASSERTIONS=${ENABLE_ASSERTIONS:-OFF} \
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
- FIXED: Bug in bicycle profile that caused exceptions if there is a highway=bicycle in the data. [#6296](https://github.com/Project-OSRM/osrm-backend/pull/6296)
- FIXED: Internal refactoring of identifier types used in data facade [#6044](https://github.com/Project-OSRM/osrm-backend/pull/6044)
- Build:
- ADDED: Build Node bindings on Windows. [#6334](https://github.com/Project-OSRM/osrm-backend/pull/6334)
- ADDED: Configure cross-compilation for Apple Silicon. [#6360](https://github.com/Project-OSRM/osrm-backend/pull/6360)
- CHANGED: Use apt-get to install Clang on CI. [#6345](https://github.com/Project-OSRM/osrm-backend/pull/6345)
- CHANGED: Fix TBB in case of Conan + NodeJS build. [#6333](https://github.com/Project-OSRM/osrm-backend/pull/6333)
Expand Down
38 changes: 20 additions & 18 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ cmake_minimum_required(VERSION 3.2)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED TRUE)


if(CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_CURRENT_BINARY_DIR AND NOT MSVC_IDE)
message(FATAL_ERROR "In-source builds are not allowed.
Please create a directory and run cmake from there, passing the path to this source directory as the last argument.
Expand Down Expand Up @@ -883,22 +884,23 @@ if (ENABLE_FUZZING)
add_subdirectory(fuzz)
endif ()

# add headers sanity check target that includes all headers independently
set(check_headers_dir "${PROJECT_BINARY_DIR}/check-headers")
file(GLOB_RECURSE headers_to_check
${PROJECT_BINARY_DIR}/*.hpp
${PROJECT_SOURCE_DIR}/include/*.hpp)
foreach(header ${headers_to_check})
if ("${header}" MATCHES ".*/include/nodejs/.*")
# we do not check NodeJS bindings headers
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NodeJS bindings contain nan.h, i.e. we would have to add NAN as a dependency to check-headers target.

continue()
endif()
get_filename_component(filename ${header} NAME_WE)
set(filename "${check_headers_dir}/${filename}.cpp")
if (NOT EXISTS ${filename})
file(WRITE ${filename} "#include \"${header}\"\n")
endif()
list(APPEND sources ${filename})
endforeach()
add_library(check-headers STATIC EXCLUDE_FROM_ALL ${sources})
set_target_properties(check-headers PROPERTIES ARCHIVE_OUTPUT_DIRECTORY ${check_headers_dir})
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is something I had to spend a lot of time on.

First of all as I understand the only purpose of this is to check "sanity" of headers, i.e. that every header includes everything what is needed to compile it. That's why it is not clear why we had condition to only have it if ENABLE_NODE_BINDINGS=ON. The second as it turned out we never built it due to EXCLUDE_FROM_ALL, but MSVC builds it for some reason and that's why I faced it and had to modify some headers in this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it was designed to run manually or in CI as make check-headers (#3409) and the node conditional was added later for unknown reasons.

In any case, perhaps there is a C++ tooling replacement for this nowadays.


# add headers sanity check target that includes all headers independently
# make sure we have all deps for the nodejs sub project's includes (nan, node)
if (ENABLE_NODE_BINDINGS)
set(check_headers_dir "${PROJECT_BINARY_DIR}/check-headers")
file(GLOB_RECURSE headers_to_check
${PROJECT_BINARY_DIR}/*.hpp
${PROJECT_SOURCE_DIR}/include/*.hpp)
foreach(header ${headers_to_check})
get_filename_component(filename ${header} NAME_WE)
set(filename "${check_headers_dir}/${filename}.cpp")
if (NOT EXISTS ${filename})
file(WRITE ${filename} "#include \"${header}\"\n")
endif()
list(APPEND sources ${filename})
endforeach()
add_library(check-headers STATIC EXCLUDE_FROM_ALL ${sources})
set_target_properties(check-headers PROPERTIES ARCHIVE_OUTPUT_DIRECTORY ${check_headers_dir})
endif()
1 change: 1 addition & 0 deletions include/engine/api/route_api.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
#include "util/json_util.hpp"

#include <iterator>
#include <map>
#include <vector>

namespace osrm
Expand Down
1 change: 1 addition & 0 deletions include/engine/guidance/assemble_leg.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#include "engine/guidance/route_leg.hpp"
#include "engine/guidance/route_step.hpp"
#include "engine/internal_route_result.hpp"
#include "util/coordinate_calculation.hpp"
#include "util/typedefs.hpp"

#include <boost/algorithm/string/join.hpp>
Expand Down
107 changes: 0 additions & 107 deletions include/extractor/geojson_debug_policies.hpp

This file was deleted.

4 changes: 2 additions & 2 deletions include/extractor/road_classification.hpp
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
#ifndef OSRM_EXTRACTOR_CLASSIFICATION_DATA_HPP_
#define OSRM_EXTRACTOR_CLASSIFICATION_DATA_HPP_

#include "extractor/intersection/constants.hpp"
#include <algorithm>
#include <boost/assert.hpp>
#include <cmath>
#include <cstdint>
#include <cstdlib>
#include <string>

#include "extractor/intersection/constants.hpp"

namespace osrm
{
namespace extractor
Expand Down
1 change: 1 addition & 0 deletions include/util/ieee754.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ THE SOFTWARE.

#include <assert.h>
#include <math.h>
#include <memory.h>

#if defined(_MSC_VER)
#include "rapidjson/msinttypes/stdint.h"
Expand Down
Loading