From 56ba70cea6f1f9d069f24d9b3d55d4e141fcab5e Mon Sep 17 00:00:00 2001 From: Brian Martin Date: Wed, 13 May 2020 12:56:26 -0700 Subject: [PATCH] Sync ccommon and fix the build (#285) * Squashed 'deps/ccommon/' changes from f5efe29..55769cc 55769cc restore cmake for rust libraries (#241) 6b1d8d5 Improve the Rust build story (#240) 5955d6b move nodelay setting to the server socket, and accepted connections will inherit that (#238) aafd20f formatting (#237) 3b5c069 Update ccommon_rs to use bytes v5.0 (#235) df74087 Address clippy lints in rust code (#234) 7c5bbd1 Make metrics and options Send + Sync (#233) c2e617e Ensure that manually implemented C options use the correct name (#232) 6e76741 Add some docs b5b4c5a Change check_pipe to use nanosleep instead of usleep (#231) eb0a389 Use name of field instead of description for C metrics (#230) 16ddc76 Fix broken buf impls (#229) eefcdcb Avoid redundant rebuilds of rust packages in CI (#228) 33f62a8 Update bindgen to also generate bindings recursively (#227) 27ffc7c Implement bytes::Buf and Bytes::ButMut on Buf and OwnedBuf (#226) f873930 Various small bugfixes and usability improvements for rust code (#225) 37a1ecd Port option parsing module to Rust (#224) 38f7556 Fix failed test detection (#215) 0ab1604 Conditionally use std::any::type_name if it's supported (#223) 98176d3 Backport changes from twitter/pelikan#265 (#222) ba54096 Remove test for removed rust logging functionality (#221) 519118d Rewrite cmake cargo build wrapper (#220) 5d23b3a Fix some small typos found in twitter/pelikan#263 (#219) 475dda7 Clean up logging shim (#218) 1d28dd2 Expand rust bindings to add options, metrics, and ccbox (#217) 88b5400 Upstream changes from twitter/pelikan#261 (#216) 4e99e63 fix a bug and change how check is found (#214) 54067ef slightly simplify accept error-handling logic (#210) e9fe980 Fix synchronize ccommon with pelikan deps/ccommon (#212) 7eb6424 Cleanup libcheck related code (#211) 683bc1a cc_bstring simplify and fix (#207) 8737d99 continue on server socket on non-blocking errors (#209) 2a62281 add atoi64 to bstring (#206) f71c657 cc_option simplify _allowed_in_name (#205) 24e3131 Add ITT instrumentation option (#204) 236c98d Fix docs (#200) e58f6a8 cc_array and cc_ring_array NULL fixes (#201) 1c8df42 Add basic support of build type (#199) 7107988 Fix now_ns() (#198) da240e5 cc: extend cc_util module (#196) 4846b15 Fix TAILQ_REINIT (#195) 4f5dbb0 Update Cmake version to 2.8 (#197) 2e6f78a cc_mm use OS_DARWIN macro to detect OS (#194) 57acaf6 cc: extend queue module (#193) a64ada2 cc: extend duration module (#192) b117632 reverting CMake file changes (#191) dea5bee backport changes made to ccommon in pelikan (#190) a4c0334 add linebreak to stats_log() (#188) 05eb03e fix inconsistent naming and bump version (#187) 4acc53a Stats to file (#186) 2168fec minimize osx build config (#185) 42b24de Simplify rust options, specify fewer output targets (#183) c9fa905 update CMakeRust used to latest version, tweaks to make build work (#184) 2ef0163 Reorder dependency includes in cmake, don't parallel build (#182) a6a54d9 remove endian-specific logic from str*cmp (#177) 4c0668b epoll_create* ignores size hint in newer kernels, switch to new API (#179) c9c5ee5 improve cc_bstring string literal and cstring names (#176) 0184d73 Add unit tests for buffer, fix buf/dbuf bugs and refactor (#174) d7dab43 create a .cargo/config so intellij uses the same target dir as cmake (#173) e710712 use accept4 for tcp_accept when available (#171) 21ba10e Remove cargo lock for shared lib, closes #169 (#172) 24660f1 update style guide (#170) 17baf1e Per thread logging (#168) git-subtree-dir: deps/ccommon git-subtree-split: 55769cc8ae9e7d5c9ceda0472498276aec1d1ed5 * Fixes to work with newer ccommon and cargo for rust builds Fixes various places where ccommon rust libraries were renamed or have interface changes. Changes the rust builds to use cargo instead of cmake. This made it easier to integrate the changes and makes building the rust portion more intuitive for rust developers. * add cargo build on osx to ci * fix copy paste error in travis config * fix the cmake rust build * cleanup * remove scripts folder --- .gitignore | 3 + .travis.yml | 30 ++ CMakeLists.txt | 7 +- Cargo.lock | 132 +++--- Cargo.toml | 9 +- cmake/CMakeCargo.cmake | 413 ++++++++++++++++++ cmake/CargoLink.cmake | 115 +++++ cmake/CargoTest.cmake | 107 +++++ deps/ccommon/.gitignore | 9 + deps/ccommon/.travis.yml | 21 + deps/ccommon/ci/cargo.sh | 53 +++ deps/ccommon/cmake/CMakeCargo.cmake | 38 +- deps/ccommon/cmake/CargoLink.cmake | 20 +- deps/ccommon/cmake/CargoTest.cmake | 18 +- deps/ccommon/docs/modules/cc_log.rst | 120 ++++- deps/ccommon/docs/modules/cc_metric.rst | 131 ++++-- deps/ccommon/docs/modules/cc_option.rst | 85 +++- deps/ccommon/docs/modules/cc_ring_array.rst | 47 +- deps/ccommon/docs/overview.rst | 50 ++- deps/ccommon/rust/CMakeLists.txt | 9 +- deps/ccommon/rust/cc_binding/CMakeLists.txt | 5 - deps/ccommon/rust/cc_binding/Cargo.toml | 15 - deps/ccommon/rust/ccommon-backend/Cargo.toml | 4 +- .../rust/ccommon-backend/src/c_export.rs | 4 +- .../ccommon-backend/src/option/default.rs | 5 +- .../rust/ccommon-backend/src/option/parse.rs | 7 +- .../rust/ccommon-backend/src/option/print.rs | 2 +- deps/ccommon/rust/ccommon-derive/src/lib.rs | 6 +- deps/ccommon/rust/ccommon-rs/CMakeLists.txt | 9 + .../{ccommon_rs => ccommon-rs}/Cargo.toml | 14 +- .../rust/{ccommon_rs => ccommon-rs}/build.rs | 0 .../{ccommon_rs => ccommon-rs}/src/bstring.rs | 9 +- .../{ccommon_rs => ccommon-rs}/src/buf.rs | 20 +- .../{ccommon_rs => ccommon-rs}/src/ccbox.rs | 5 +- .../{ccommon_rs => ccommon-rs}/src/error.rs | 0 .../{ccommon_rs => ccommon-rs}/src/lib.rs | 0 .../src/log/debug.rs | 2 +- .../{ccommon_rs => ccommon-rs}/src/log/mod.rs | 0 .../src/log/shim.rs | 12 +- .../src/metric/counter.rs | 2 +- .../src/metric/fpn.rs | 2 +- .../src/metric/gauge.rs | 2 +- .../src/metric/mod.rs | 4 +- .../src/option/boolean.rs | 2 +- .../src/option/fpn.rs | 2 +- .../src/option/mod.rs | 4 +- .../src/option/string.rs | 2 +- .../src/option/uint.rs | 4 +- .../{ccommon_rs => ccommon-rs}/src/ptrs.rs | 0 .../tests/derive_metrics.rs | 0 .../tests/derive_options.rs | 0 .../{ccommon_rs => ccommon-rs}/tests/log.rs | 2 +- deps/ccommon/rust/ccommon-sys/CMakeLists.txt | 5 + deps/ccommon/rust/ccommon-sys/Cargo.toml | 16 + .../rust/{cc_binding => ccommon-sys}/build.rs | 160 ++----- .../{cc_binding => ccommon-sys}/src/lib.rs | 13 +- .../{cc_binding => ccommon-sys}/src/metric.rs | 0 .../{cc_binding => ccommon-sys}/wrapper.h | 0 deps/ccommon/rust/ccommon_rs/CMakeLists.txt | 9 - deps/ccommon/rust/scripts/setup.sh | 35 -- deps/ccommon/src/channel/cc_tcp.c | 12 +- src/rust-util/pelikan-sys/CMakeLists.txt | 4 +- src/rust-util/pelikan-sys/Cargo.toml | 11 +- src/rust-util/pelikan-sys/build.rs | 36 +- src/rust-util/pelikan-sys/src/storage/slab.rs | 4 +- src/rust-util/pelikan/Cargo.toml | 8 +- src/rustcore/CMakeLists.txt | 2 +- src/rustcore/Cargo.toml | 4 +- src/server/pingserver-rs/Cargo.toml | 8 +- src/server/pingserver-rs/src/admin.rs | 4 +- src/server/pingserver-rs/src/alloc.rs | 3 +- src/server/twemcache-http/Cargo.toml | 11 +- src/server/twemcache-http/build.rs | 21 +- src/server/twemcache-http/src/admin.rs | 8 +- src/storage/cdb/cdb_rs/Cargo.toml | 9 +- src/storage/cdb/cdb_rs/build.rs | 24 +- 76 files changed, 1402 insertions(+), 567 deletions(-) create mode 100644 cmake/CMakeCargo.cmake create mode 100644 cmake/CargoLink.cmake create mode 100644 cmake/CargoTest.cmake create mode 100755 deps/ccommon/ci/cargo.sh delete mode 100644 deps/ccommon/rust/cc_binding/CMakeLists.txt delete mode 100644 deps/ccommon/rust/cc_binding/Cargo.toml create mode 100644 deps/ccommon/rust/ccommon-rs/CMakeLists.txt rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/Cargo.toml (62%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/build.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/bstring.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/buf.rs (94%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/ccbox.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/error.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/lib.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/log/debug.rs (98%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/log/mod.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/log/shim.rs (95%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/metric/counter.rs (98%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/metric/fpn.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/metric/gauge.rs (98%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/metric/mod.rs (99%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/option/boolean.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/option/fpn.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/option/mod.rs (99%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/option/string.rs (97%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/option/uint.rs (95%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/src/ptrs.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/tests/derive_metrics.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/tests/derive_options.rs (100%) rename deps/ccommon/rust/{ccommon_rs => ccommon-rs}/tests/log.rs (99%) create mode 100644 deps/ccommon/rust/ccommon-sys/CMakeLists.txt create mode 100644 deps/ccommon/rust/ccommon-sys/Cargo.toml rename deps/ccommon/rust/{cc_binding => ccommon-sys}/build.rs (56%) rename deps/ccommon/rust/{cc_binding => ccommon-sys}/src/lib.rs (82%) rename deps/ccommon/rust/{cc_binding => ccommon-sys}/src/metric.rs (100%) rename deps/ccommon/rust/{cc_binding => ccommon-sys}/wrapper.h (100%) delete mode 100644 deps/ccommon/rust/ccommon_rs/CMakeLists.txt delete mode 100755 deps/ccommon/rust/scripts/setup.sh diff --git a/.gitignore b/.gitignore index 62e4b4eb2..addf70a9e 100644 --- a/.gitignore +++ b/.gitignore @@ -52,3 +52,6 @@ CMAKE_BINARY_DIR # CMake generates a .cargo/config which is specific to the local # setup. We don't want to include it within the repo .cargo + +# Cargo build directory +/target \ No newline at end of file diff --git a/.travis.yml b/.travis.yml index 26a944005..b2eb3f1d4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -36,14 +36,44 @@ matrix: packages: - libsubunit-dev + - name: "cargo build" + language: rust + addons: + apt: + <<: *apt + packages: + - libsubunit-dev + script: + - cargo build + - cargo test + - cargo build --release + - cargo test --release + - os: osx osx_image: xcode10.1 compiler: clang + - os: osx + osx_image: xcode10.1 + language: rust + script: + - cargo build + - cargo test + - cargo build --release + - cargo test --release + allow_failures: - os: osx osx_image: xcode10.1 compiler: clang + - os: osx + osx_image: xcode10.1 + language: rust + script: + - cargo build + - cargo test + - cargo build --release + - cargo test --release before_install: - ./ci/before-install.sh diff --git a/CMakeLists.txt b/CMakeLists.txt index d18eaac9b..0793ead5b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -188,6 +188,10 @@ if(TARGET_HTTP) set(HAVE_RUST on) endif() +if(HAVE_RUST) + include(cmake/CMakeCargo.cmake) +endif(HAVE_RUST) + #set(CMAKE_INCLUDE_CURRENT_DIR) include_directories(${include_directories} "${PROJECT_BINARY_DIR}" @@ -237,9 +241,6 @@ endif() # vim:ts=4:sw=4:et if(HAVE_RUST) - include(deps/ccommon/cmake/CMakeCargo.cmake) - set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DHAVE_RUST=1") - # Create a .cargo file which sets the target directory make_directory(.cargo) file(WRITE .cargo/config "[build]\ntarget-dir=\"${CMAKE_BINARY_DIR}/target\"\n") diff --git a/Cargo.lock b/Cargo.lock index 04133cb3d..4213bbd0c 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -62,35 +62,13 @@ dependencies = [ [[package]] name = "bindgen" -version = "0.51.1" +version = "0.53.2" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "cexpr 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)", + "cexpr 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)", "cfg-if 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)", - "clang-sys 0.28.1 (registry+https://github.com/rust-lang/crates.io-index)", - "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", - "env_logger 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)", - "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", - "peeking_take_while 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)", - "proc-macro2 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)", - "quote 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "rustc-hash 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)", - "shlex 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)", - "which 3.1.0 (registry+https://github.com/rust-lang/crates.io-index)", -] - -[[package]] -name = "bindgen" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "cexpr 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)", - "cfg-if 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)", - "clang-sys 0.28.1 (registry+https://github.com/rust-lang/crates.io-index)", + "clang-sys 0.29.3 (registry+https://github.com/rust-lang/crates.io-index)", "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", "env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)", @@ -174,21 +152,12 @@ name = "cc" version = "1.0.48" source = "registry+https://github.com/rust-lang/crates.io-index" -[[package]] -name = "cc_binding" -version = "0.1.0" -dependencies = [ - "bindgen 0.51.1 (registry+https://github.com/rust-lang/crates.io-index)", - "failure 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", - "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", -] - [[package]] name = "ccommon-backend" version = "0.1.0" dependencies = [ - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", ] @@ -203,29 +172,39 @@ dependencies = [ ] [[package]] -name = "ccommon_rs" +name = "ccommon-rs" version = "0.1.0" dependencies = [ "autocfg 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", "bytes 0.5.4 (registry+https://github.com/rust-lang/crates.io-index)", - "cc_binding 0.1.0", "ccommon-backend 0.1.0", "ccommon-derive 0.1.0", + "ccommon-sys 0.1.0", "gag 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)", "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", "rusty-fork 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "ccommon-sys" +version = "0.1.0" +dependencies = [ + "bindgen 0.53.2 (registry+https://github.com/rust-lang/crates.io-index)", + "cmake 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", + "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", +] + [[package]] name = "cdb_rs" version = "0.1.0" dependencies = [ - "bindgen 0.51.1 (registry+https://github.com/rust-lang/crates.io-index)", + "bindgen 0.53.2 (registry+https://github.com/rust-lang/crates.io-index)", "bytes 0.4.12 (registry+https://github.com/rust-lang/crates.io-index)", - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", + "cmake 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", "env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)", "failure 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", "failure_derive 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)", @@ -248,10 +227,10 @@ dependencies = [ [[package]] name = "cexpr" -version = "0.3.6" +version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ - "nom 4.2.3 (registry+https://github.com/rust-lang/crates.io-index)", + "nom 5.1.1 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] @@ -273,7 +252,7 @@ dependencies = [ [[package]] name = "clang-sys" -version = "0.28.1" +version = "0.29.3" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "glob 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", @@ -303,6 +282,14 @@ dependencies = [ "bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)", ] +[[package]] +name = "cmake" +version = "0.1.42" +source = "registry+https://github.com/rust-lang/crates.io-index" +dependencies = [ + "cc 1.0.48 (registry+https://github.com/rust-lang/crates.io-index)", +] + [[package]] name = "criterion" version = "0.3.0" @@ -401,18 +388,6 @@ name = "either" version = "1.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -[[package]] -name = "env_logger" -version = "0.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -dependencies = [ - "atty 0.2.13 (registry+https://github.com/rust-lang/crates.io-index)", - "humantime 1.3.0 (registry+https://github.com/rust-lang/crates.io-index)", - "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", - "regex 1.3.1 (registry+https://github.com/rust-lang/crates.io-index)", - "termcolor 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)", -] - [[package]] name = "env_logger" version = "0.7.1" @@ -769,11 +744,11 @@ dependencies = [ [[package]] name = "nom" -version = "4.2.3" +version = "5.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" dependencies = [ "memchr 2.2.1 (registry+https://github.com/rust-lang/crates.io-index)", - "version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)", + "version_check 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)", ] [[package]] @@ -824,8 +799,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" name = "pelikan" version = "0.1.0" dependencies = [ - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", "pelikan-sys 0.0.0", @@ -835,10 +810,11 @@ dependencies = [ name = "pelikan-sys" version = "0.0.0" dependencies = [ - "bindgen 0.51.1 (registry+https://github.com/rust-lang/crates.io-index)", - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "bindgen 0.53.2 (registry+https://github.com/rust-lang/crates.io-index)", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "cdb_rs 0.1.0", + "cmake 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", "glob 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)", "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)", "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", @@ -849,8 +825,8 @@ dependencies = [ name = "pelikan_pingserver_rs" version = "0.0.0" dependencies = [ - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)", "libc 0.2.68 (registry+https://github.com/rust-lang/crates.io-index)", @@ -865,12 +841,13 @@ name = "pelikan_twemcache_http" version = "0.1.0" dependencies = [ "arrayvec 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)", - "bindgen 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)", + "bindgen 0.53.2 (registry+https://github.com/rust-lang/crates.io-index)", "bytes 0.5.4 (registry+https://github.com/rust-lang/crates.io-index)", "cc 1.0.48 (registry+https://github.com/rust-lang/crates.io-index)", - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", + "cmake 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)", "httparse 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)", "httpdate 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)", "httpencode 0.1.0", @@ -1203,8 +1180,8 @@ name = "rustcore" version = "0.1.0" dependencies = [ "bytes 0.5.4 (registry+https://github.com/rust-lang/crates.io-index)", - "cc_binding 0.1.0", - "ccommon_rs 0.1.0", + "ccommon-rs 0.1.0", + "ccommon-sys 0.1.0", "futures 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)", "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", "parking_lot 0.10.0 (registry+https://github.com/rust-lang/crates.io-index)", @@ -1437,7 +1414,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] name = "version_check" -version = "0.1.5" +version = "0.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" [[package]] @@ -1535,8 +1512,7 @@ dependencies = [ "checksum autocfg 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "1d49d90015b3c36167a20fe2810c5cd875ad504b39cff3d4eae7977e6b7c1cb2" "checksum backtrace 0.3.40 (registry+https://github.com/rust-lang/crates.io-index)" = "924c76597f0d9ca25d762c25a4d369d51267536465dc5064bdf0eb073ed477ea" "checksum backtrace-sys 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "5d6575f128516de27e3ce99689419835fce9643a9b215a14d2b5b685be018491" -"checksum bindgen 0.51.1 (registry+https://github.com/rust-lang/crates.io-index)" = "ebd71393f1ec0509b553aa012b9b58e81dadbdff7130bd3b8cba576e69b32f75" -"checksum bindgen 0.52.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f1c85344eb535a31b62f0af37be84441ba9e7f0f4111eb0530f43d15e513fe57" +"checksum bindgen 0.53.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6bb26d6a69a335b8cb0e7c7e9775cd5666611dc50a37177c3f2cedcfc040e8c8" "checksum bit-set 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)" = "e84c238982c4b1e1ee668d136c510c67a13465279c0cb367ea6baf6310620a80" "checksum bit-vec 0.5.1 (registry+https://github.com/rust-lang/crates.io-index)" = "f59bbe95d4e52a6398ec21238d31577f2b28a9d86807f06ca59d191d8440d0bb" "checksum bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693" @@ -1547,11 +1523,12 @@ dependencies = [ "checksum c2-chacha 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "214238caa1bf3a496ec3392968969cab8549f96ff30652c9e56885329315f6bb" "checksum cast 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "4b9434b9a5aa1450faa3f9cb14ea0e8c53bb5d2b3c1bfd1ab4fc03e9f33fbfb0" "checksum cc 1.0.48 (registry+https://github.com/rust-lang/crates.io-index)" = "f52a465a666ca3d838ebbf08b241383421412fe7ebb463527bba275526d89f76" -"checksum cexpr 0.3.6 (registry+https://github.com/rust-lang/crates.io-index)" = "fce5b5fb86b0c57c20c834c1b412fd09c77c8a59b9473f86272709e78874cd1d" +"checksum cexpr 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f4aedb84272dbe89af497cf81375129abda4fc0a9e7c5d317498c15cc30c0d27" "checksum cfg-if 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822" -"checksum clang-sys 0.28.1 (registry+https://github.com/rust-lang/crates.io-index)" = "81de550971c976f176130da4b2978d3b524eaa0fd9ac31f3ceb5ae1231fb4853" +"checksum clang-sys 0.29.3 (registry+https://github.com/rust-lang/crates.io-index)" = "fe6837df1d5cba2397b835c8530f51723267e16abbf83892e9e5af4f0e5dd10a" "checksum clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9" "checksum cloudabi 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "ddfc5b9aa5d4507acaf872de71051dfd0e309860e88966e1051e462a077aac4f" +"checksum cmake 0.1.42 (registry+https://github.com/rust-lang/crates.io-index)" = "81fb25b677f8bf1eb325017cb6bb8452f87969db0fedb4f757b297bee78a7c62" "checksum criterion 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "938703e165481c8d612ea3479ac8342e5615185db37765162e762ec3523e2fc6" "checksum criterion-plot 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "eccdc6ce8bbe352ca89025bee672aa6d24f4eb8c53e3a8b5d1bc58011da072a2" "checksum crossbeam-deque 0.7.2 (registry+https://github.com/rust-lang/crates.io-index)" = "c3aa945d63861bfe624b55d153a39684da1e8c0bc8fba932f7ee3a3c16cea3ca" @@ -1561,7 +1538,6 @@ dependencies = [ "checksum csv 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "37519ccdfd73a75821cac9319d4fce15a81b9fcf75f951df5b9988aa3a0af87d" "checksum csv-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "9b5cadb6b25c77aeff80ba701712494213f4a8418fcda2ee11b6560c3ad0bf4c" "checksum either 1.5.3 (registry+https://github.com/rust-lang/crates.io-index)" = "bb1f6b1ce1c140482ea30ddd3335fc0024ac7ee112895426e0a629a6c20adfe3" -"checksum env_logger 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "aafcde04e90a5226a6443b7aabdb016ba2f8307c847d524724bd9b346dd1a2d3" "checksum env_logger 0.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "44533bbbb3bb3c1fa17d9f2e4e38bbbaf8396ba82193c4cb1b6445d711445d36" "checksum failure 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "b8529c2421efa3066a5cbd8063d2244603824daccb6936b079010bb2aa89464b" "checksum failure_derive 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "030a733c8287d6213886dd487564ff5c8f6aae10278b3588ed177f9d18f8d231" @@ -1603,7 +1579,7 @@ dependencies = [ "checksum mio-uds 0.6.7 (registry+https://github.com/rust-lang/crates.io-index)" = "966257a94e196b11bb43aca423754d87429960a768de9414f3691d6957abf125" "checksum miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f2f3b1cf331de6896aabf6e9d55dca90356cc9960cca7eaaf408a355ae919" "checksum net2 0.2.33 (registry+https://github.com/rust-lang/crates.io-index)" = "42550d9fb7b6684a6d404d9fa7250c2eb2646df731d1c06afc06dcee9e1bcf88" -"checksum nom 4.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "2ad2a91a8e869eeb30b9cb3119ae87773a8f4ae617f41b1eb9c154b2905f7bd6" +"checksum nom 5.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0b471253da97532da4b61552249c521e01e736071f71c1a4f7ebbfbf0a06aad6" "checksum num-traits 0.2.10 (registry+https://github.com/rust-lang/crates.io-index)" = "d4c81ffc11c212fa327657cb19dd85eb7419e163b5b076bede2bdb5c974c07e4" "checksum num_cpus 1.11.1 (registry+https://github.com/rust-lang/crates.io-index)" = "76dac5ed2a876980778b8b85f75a71b6cbf0db0b1232ee12f826bccb00d09d72" "checksum parking_lot 0.10.0 (registry+https://github.com/rust-lang/crates.io-index)" = "92e98c49ab0b7ce5b222f2cc9193fc4efe11c6d0bd4f648e374684a6857b1cfc" @@ -1674,7 +1650,7 @@ dependencies = [ "checksum unicode-width 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "caaa9d531767d1ff2150b9332433f32a24622147e5ebb1f26409d5da67afd479" "checksum unicode-xid 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "826e7639553986605ec5979c7dd957c7895e93eabed50ab2ffa7f6128a75097c" "checksum vec_map 0.8.1 (registry+https://github.com/rust-lang/crates.io-index)" = "05c78687fb1a80548ae3250346c3db86a80a7cdd77bda190189f2d0a0987c81a" -"checksum version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "914b1a6776c4c929a602fafd8bc742e06365d4bcbe48c30f9cca5824f70dc9dd" +"checksum version_check 0.9.1 (registry+https://github.com/rust-lang/crates.io-index)" = "078775d0255232fb988e6fccf26ddc9d1ac274299aaedcedce21c6f72cc533ce" "checksum wait-timeout 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "9f200f5b12eb75f8c1ed65abd4b2db8a6e1b138a20de009dacee265a2498f3f6" "checksum walkdir 2.2.9 (registry+https://github.com/rust-lang/crates.io-index)" = "9658c94fa8b940eab2250bd5a457f9c48b748420d71293b165c8cdbe2f55f71e" "checksum wasi 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b89c3ce4ce14bdc6fb6beaf9ec7928ca331de5df7e5ea278375642a2f478570d" diff --git a/Cargo.toml b/Cargo.toml index 96f0d4845..01521810e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,18 +1,15 @@ [workspace] members = [ - "deps/ccommon/rust/cc_binding", - "deps/ccommon/rust/ccommon_rs", + "deps/ccommon/rust/ccommon-sys", + "deps/ccommon/rust/ccommon-rs", "deps/ccommon/rust/ccommon-derive", - "src/server/pingserver-rs", - "src/server/twemcache-http", - "src/storage/cdb/cdbgen", - "src/storage/cdb/cdb_rs", "src/rust-util/httpencode", "src/rust-util/pelikan", "src/rust-util/pelikan-sys", "src/rustcore", "src/server/pingserver-rs", + "src/server/twemcache-http", "src/storage/cdb/cdbgen", "src/storage/cdb/cdb_rs", "test/server/pingserver-rs", diff --git a/cmake/CMakeCargo.cmake b/cmake/CMakeCargo.cmake new file mode 100644 index 000000000..4dff90c99 --- /dev/null +++ b/cmake/CMakeCargo.cmake @@ -0,0 +1,413 @@ +# ccommon - a cache common library. +# Copyright (C) 2019 Twitter, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Hack to ensure we have the directory of this cmake file when +# the functions contained within are called. +set(FILE_LIST_DIR ${CMAKE_CURRENT_LIST_DIR} CACHE INTERNAL "") + +# Get the target that we want to pass to cargo +function(cargo_build_private_get_target TARGET_VAR) + if(WIN32) + if(CMAKE_SIZEOF_VOID_P EQUAL 8) + set(LIB_TARGET "x86_64-pc-windows-msvc") + else() + set(LIB_TARGET "i686-pc-windows-msvc") + endif() + elseif(ANDROID) + if(ANDROID_SYSROOT_ABI STREQUAL "x86") + set(LIB_TARGET "i686-linux-android") + elseif(ANDROID_SYSROOT_ABI STREQUAL "x86_64") + set(LIB_TARGET "x86_64-linux-android") + elseif(ANDROID_SYSROOT_ABI STREQUAL "arm") + set(LIB_TARGET "arm-linux-androideabi") + elseif(ANDROID_SYSROOT_ABI STREQUAL "arm64") + set(LIB_TARGET "aarch64-linux-android") + endif() + elseif(IOS) + set(LIB_TARGET "universal") + elseif(CMAKE_SYSTEM_NAME STREQUAL Darwin) + set(LIB_TARGET "x86_64-apple-darwin") + else() + if(CMAKE_SIZEOF_VOID_P EQUAL 8) + set(LIB_TARGET "x86_64-unknown-linux-gnu") + else() + set(LIB_TARGET "i686-unknown-linux-gnu") + endif() + endif() + + set(${TARGET_VAR} ${LIB_TARGET} PARENT_SCOPE) +endfunction() + +# Get whether to build in release or debug mode +function(cargo_build_private_get_build_type TARGET_VAR) + if(NOT CMAKE_BUILD_TYPE) + set(${TARGET_VAR} "debug" PARENT_SCOPE) + elseif(${CMAKE_BUILD_TYPE} STREQUAL "Release") + set(${TARGET_VAR} "release" PARENT_SCOPE) + else() + set(${TARGET_VAR} "debug" PARENT_SCOPE) + endif() +endfunction() + +# Create a set of targets to invoke cargo and pass in +# the required linker flags. This is meant to be run +# in the same directory as the Cargo.toml file for the +# library. +# +# In all cases, the target can be used as if it was defined +# through the add_executable or add_binary. In addition, +# library dependencies will be properly passed through. +# +# Arguments: +# NAME +# The library package name as specified in Cargo.toml. +# This argument is required. +# TARGET_DIR +# Overrides the default directory for the cargo target +# directory which is ${CMAKE_BINARY_DIR}/target. +# BIN +# Indicates that this crate generated a binary and to +# expose that binary file as a target. +# STATIC +# Indicates this target generates a static library that +# can be consumed by other cmake targets. +# NO_TEST +# Disables generation of the test target. +# USE_CMAKE_LINK +# Pass link flags through from cmake through RUSTFLAGS. This +# is necessary if the build scripts for the rust code don't +# account for all the necessary linker dependencies. +# +# Notes: +# - If neither BIN or STATIC is defined then it is assumed +# that the target exports no artifacts (i.e. it is only used +# by other rust targets within the build) +# - Build and exported are currently mutually exclusive. If you +# want to have multiple targets like this then call cargo_build +# multiple times. +# +# Additional Things: +# - This also sets up the required cmake properties to delete the +# target directory when `make clean` or equivalent is run. +# +# How it Works: +# In general, cmake has a hard time integrating with external build +# systems within the same directory. However, cmake has extensive +# support for custom linkers and also allows you to define interface +# libraries when no artifact is created. Together, we can use these +# to trick cmake into thinking it is building the executable while +# actually using cargo to build them. This means that things like +# linker flags and dependencies should mostly "just work" (hopefully). +# +# What we have to do depends on what type of library we are compiling: +# - For pure rust libraries that don't export any C API, we define +# an interface library. This means that any linker flags are +# properly passed on to downstream dependencies. In addition, we +# define a custom target so the library is built. +# - For static libraries, we define a library that's built using a +# custom linker language. The custom linker command is really just +# a `cargo build` in disguise, but one that passes the correct +# flags in. +# - For binaries, we define an executable target also using a custom +# linker language similar to a static library. +# +# To pass the proper linker flags to the rust process, we have a special +# target .linkflags.txt that echoes the linker flags into a known +# file. These are then picked up by the import-link-flags crate which +# uses a build script to pass them to rustc. +function(cargo_build) + cmake_parse_arguments( + CARGO + "BIN;STATIC;NO_TEST;USE_CMAKE_LINK" + "NAME;TARGET_DIR;COPY_TO" + "" + ${ARGN} + ) + + set(CARGO_FORCE_CMAKE_LINK OFF CACHE BOOL "Pass link flags from cmake to cargo - this will slow down builds considerably") + set(CARGO_CI OFF CACHE BOOL "Optimize build for from-scratch build time instead of incremental rebuild") + + mark_as_advanced(CARGO_FORCE_CMAKE_LINK CARGO_CI) + + if(CARGO_FORCE_CMAKE_LINK) + set(${CARGO_USE_CMAKE_LINK} ON) + endif() + + string(REPLACE "-" "_" LIB_NAME ${CARGO_NAME}) + + if(NOT DEFINED CARGO_TARGET_DIR) + if(NOT CARGO_USE_CMAKE_LINK AND NOT DEFINED CARGO_TARGET_DIR) + set(CARGO_TARGET_DIR "${CMAKE_BINARY_DIR}/target") + elseif(CARGO_CI) + set(CARGO_TARGET_DIR "${CMAKE_BINARY_DIR}/target") + else() + set(CARGO_TARGET_DIR "${CMAKE_CURRENT_BINARY_DIR}/target") + endif() + endif() + + if(CARGO_BIN AND CARGO_STATIC) + message( + FATAL_ERROR + "Cannot create a cargo target that has " + "both a binary and a static library. Use multiple " + "targets instead." + ) + endif() + + cargo_build_private_get_target(CRATE_TARGET) + cargo_build_private_get_build_type(CRATE_BUILD_TYPE) + + # The CONFIGURE_DEPENDS flag will rerun the glob at build time if the + # the build system supports it. + file( + GLOB_RECURSE CRATE_SOURCES + CONFIGURE_DEPENDS "*.rs" + ) + + list(APPEND CRATE_SOURCES Cargo.toml) + + # Clean the target directory when make clean is run + set_directory_properties(PROPERTIES + ADDITIONAL_CLEAN_FILES + ${CARGO_TARGET_DIR} + ) + + if(CARGO_BIN OR CARGO_STATIC) + set(LINK_FLAGS_FILE $.linkflags.txt) + else() + set(LINK_FLAGS_FILE $) + endif() + + set(FORWARDED_VARS + # So that internal invocations of cmake are consistent + "CMAKE=${CMAKE_COMMAND}" + # So that build scripts can configure themselves based + # on whether cmake is driving the build or not + "CCOMMON_CMAKE_IS_DRIVING_BUILD=1" + # Needed to configure the correct target directory + "CARGO_TARGET_DIR=${CARGO_TARGET_DIR}" + # Needed for build scripts + "CMAKE_BINARY_DIR=${CMAKE_BINARY_DIR}" + ) + + if(CARGO_BIN) + set(OUTPUT_FILE_NAME ${LIB_NAME}${CMAKE_EXECUTABLE_SUFFIX}) + set(OUTPUT_FILE ${CARGO_TARGET_DIR}/${CRATE_TARGET}/${CRATE_BUILD_TYPE}/${OUTPUT_FILE_NAME}) + elseif(CARGO_STATIC) + set(OUTPUT_FILE_NAME ${CMAKE_STATIC_LIBRARY_PREFIX}${LIB_NAME}${CMAKE_STATIC_LIBRARY_SUFFIX}) + set(OUTPUT_FILE ${CARGO_TARGET_DIR}/${CRATE_TARGET}/${CRATE_BUILD_TYPE}/${OUTPUT_FILE_NAME}) + endif() + + if(IOS) + # Since we're going through cargo rustc to pass linker flags + # this won't work. The previous build script used cargo lipo + # here. However, the likelyhood of someone wanting to use + # a library for cache servers on IOS is low at this time so + # this is OK. + message(FATAL_ERROR "Compiling for IOS is not supported") + endif() + + # Arguments to cargo + set(CRATE_ARGS "") + list(APPEND CRATE_ARGS "--target" ${CRATE_TARGET}) + + if(${CRATE_BUILD_TYPE} STREQUAL "release") + list(APPEND CRATE_ARGS "--release") + endif() + + # Convert CRATE_ARGS from a list to a string + string(REPLACE ";" " " CRATE_ARGS_STR "${CRATE_ARGS}") + + set(LINK_FLAGS_FILE "${CMAKE_CURRENT_BINARY_DIR}/${CARGO_NAME}.linkflags.txt") + + # The following is a hack. It takes advantage of the fact that + # cmake allows us to define arbitrary linker languages in order + # to invoke cargo as our linker command. To do this, we define + # a custom language specific to only our target that also takes + # in the required environment variables we want to set and the + # output file generated by cargo. + # + # This works by running a cmake script during build-time which + # parses the link flags and sends them off to cargo as -Clink-flag=... + # For the full details in what's done during build see CargoLink.cmake. + # + # The upside of this hack is that it allows libraries and binaries + # generated by cargo to be used as normal cmake targets. This + # means that stuff like target_link_libraries works properly. + # + # Extra Note: The variables that look like in the string + # below are called expansion variables by the community wiki. They + # don't seem to be documented anywhere in the official docs but + # are hopefully stable. + # + # Another Note: Since these are required to be global variables we + # push them in the cache as hidden variables. + # + # Note 3: There is a expansion rule in the docs which I would + # expect to expand to what was passed in to target_compile_flags. + # Unfortunately, it doesn't, so I've instead just shoved the + # compile flags into linker command template directory. + # + # The inspiration for this hack came from this SO answer + # https://stackoverflow.com/questions/34165365/retrieve-all-link-flags-in-cmake + # + # "Docs" for the expansion rules can be found here + # https://gitlab.kitware.com/cmake/community/wikis/doc/cmake/Build-Rules + set(LINK_COMMAND " -P ${FILE_LIST_DIR}/CargoLink.cmake 'TARGET=' 'LINK_FLAGS=' 'LINK_LIBRARIES=' 'FLAGS=${CRATE_ARGS_STR}' 'OUTPUT=${OUTPUT_FILE}' 'CMAKE_CURRENT_SOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR}' 'LINK_FLAGS_FILE=${LINK_FLAGS_FILE}' 'USE_CMAKE_LINK=${USE_CMAKE_LINK}' -- ") + + foreach(VAR ${FORWARDED_VARS}) + string(APPEND LINK_COMMAND " ${VAR}") + endforeach() + + # TODO(sean): disambiguate this based on bin/lib so that multiple targets in + # the same directory don't clash. + set(CMAKE_${CARGO_NAME}_LINK_EXECUTABLE ${LINK_COMMAND} CACHE INTERNAL "") + set(CMAKE_${CARGO_NAME}_CREATE_STATIC_LIBRARY ${LINK_COMMAND} CACHE INTERNAL "") + set(CMAKE_${CARGO_NAME}_CREATE_SHARED_LIBRARY ${LINK_COMMAND} CACHE INTERNAL "") + set(CMAKE_${CARGO_NAME}_LINK_FLAGS ${CMAKE_C_LINK_FLAGS} CACHE INTERNAL "") + set(CMAKE_${CARGO_NAME}_LINK_DIRECTORIES ${CMAKE_C_LINK_DIRECTORIES} CACHE INTERNAL "") + + # Needed to build when we have test targets + set( + CMAKE_ECHO_LINK_EXECUTABLE + "bash -c \" -E echo_append > \"" + CACHE INTERNAL "" + ) + set(CMAKE_ECHO_LINK_FLAGS ${CMAKE_C_LINK_FLAGS} CACHE INTERNAL "") + set(CMAKE_ECHO_LINK_DIRECTORIES ${CMAKE_C_LINK_DIRECTORIES} CACHE INTERNAL "") + + set_source_files_properties( + ${CRATE_SOURCES} + PROPERTY + HEADER_FILE_ONLY ON + ) + + # CMake doesn't seem to add dependencies on the crate + # sources. This means that making changes to a rust + # target is rather painful. + # + # We work around this by creating a stamp file that depends + # on all the source files in the crate. When a source file + # is modified, the stamp file is updated, and this is picked + # up since we set the stamp file as a link dependency of + # library/binary. + add_custom_command( + OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + DEPENDS ${CRATE_SOURCES} + COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + COMMENT "checking whether crate sources are up-to-date" + ) + + # Targets + if(CARGO_BIN) + # We are building a binary executable + add_executable( + ${CARGO_NAME} + ${CRATE_SOURCES} + # Needed so that this file is usable in LINK_DEPENDENCY + ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + ) + + # Ensure that we use our custom "linker" + set_target_properties( + ${CARGO_NAME} PROPERTIES + LINKER_LANGUAGE ${CARGO_NAME} + LINK_DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + ) + + target_compile_options(${CARGO_NAME} PRIVATE ${CRATE_ARGS}) + elseif(CARGO_STATIC) + # We are building a static library that will be used from C code + add_library( + ${CARGO_NAME} + STATIC + ${CRATE_SOURCES} + ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + ) + + # Ensure that we use our custom "linker" + set_target_properties( + ${CARGO_NAME} PROPERTIES + LINKER_LANGUAGE ${CARGO_NAME} + # Needed so that this file is usable in LINK_DEPENDENCY + LINK_DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/up-to-date.stamp + ) + + target_compile_options(${CARGO_NAME} PRIVATE ${CRATE_ARGS}) + else() + set(CARGO_ENV_COMMAND ${CMAKE_COMMAND} -E env ${FORWARDED_VARS}) + + # We are building a rust-only library. Define it as a interface library + add_library( + ${CARGO_NAME} + INTERFACE + ) + + if (${CMAKE_VERSION} VERSION_GREATER_EQUAL "3.13.0") + # This is required for dependency detection to work correctly. + # However, nothing breaks on a clean build if you don't have it + # so we don't make it required. + set_target_properties( + ${CARGO_NAME} PROPERTIES + INTERFACE_LINK_DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/build.stamp + ) + endif() + + # However, we still want to build the library, so define a custom command for that + add_custom_command( + OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/build.stamp + COMMAND ${CARGO_ENV_COMMAND} cargo build ${CRATE_ARGS} + COMMAND ${CMAKE_COMMAND} -E touch ${CMAKE_CURRENT_BINARY_DIR}/build.stamp + DEPENDS ${CRATE_SOURCES} + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + COMMENT "running cargo for target ${CARGO_NAME}" + ) + + add_custom_target( + ${CARGO_NAME}-build + ALL DEPENDS + ${CMAKE_CURRENT_BINARY_DIR}/build.stamp + ) + + add_dependencies(${CARGO_NAME} ${CARGO_NAME}-build) + add_dependencies(${CARGO_NAME} ${CARGO_NAME}-link-export) + + add_executable(${CARGO_NAME}-link-export Cargo.toml) + set_target_properties( + ${CARGO_NAME}-link-export PROPERTIES + LINKER_LANGUAGE ECHO + SUFFIX ".txt" + TARGET_MESSAGES OFF + ) + target_link_libraries(${CARGO_NAME}-link-export $) + + set(LINK_FLAGS_FILE $) + endif() + + if(NOT CARGO_NO_TEST) + add_test( + NAME test-${CARGO_NAME} + COMMAND ${CMAKE_COMMAND} -P "${FILE_LIST_DIR}/CargoTest.cmake" + "LINK_FLAGS_FILE=${LINK_FLAGS_FILE}" + "FLAGS=${TEST_ARGS_STR}" + "CMAKE_CURRENT_SOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR}" + "CMAKE_CURRENT_BINARY_DIR=${CMAKE_CURRENT_BINARY_DIR}" + "USE_CMAKE_LINK=${CARGO_USE_CMAKE_LINK}" + -- + ${FORWARDED_VARS} + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + ) + endif() +endfunction() diff --git a/cmake/CargoLink.cmake b/cmake/CargoLink.cmake new file mode 100644 index 000000000..e8ff1b7d8 --- /dev/null +++ b/cmake/CargoLink.cmake @@ -0,0 +1,115 @@ +# ccommon - a cache common library. +# Copyright (C) 2019 Twitter, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Ensure that empty elements in lists aren't deleted +cmake_policy(SET CMP0007 NEW) + +# Ignore the first 3 arguments since they will always be cmake -P /LinkRust.cmake +set(ARGI 3) +# This flips once we see -- in the arguments +set(PARSING_ENV_VARS OFF) +# Arguments to be passed directly to the build command +set(PASSTHROUGH_VARS ) + +# Split up the command-line arguments to this script +# into two groups +# +# Arguments before '--' are cmake variables that we set +# in this script. These are parameters which control the +# behaviour here. +# +# Arguments after '--' are environment variables to pass +# through to the cargo invocation, they are used by build +# scripts and to control cargo behaviour. +while(ARGI LESS ${CMAKE_ARGC}) + set(CURRENT_ARG ${CMAKE_ARGV${ARGI}}) + + if(NOT PARSING_ENV_VARS) + if(CURRENT_ARG STREQUAL "--") + set(PARSING_ENV_VARS ON) + else() + string(REPLACE "=" ";" ARGLIST "${CURRENT_ARG}") + + list(GET ARGLIST 0 VAR) + list(REMOVE_AT ARGLIST 0) + string(REPLACE ";" "=" VALUE "${ARGLIST}") + + set(${VAR} "${VALUE}") + endif() + else() + list(APPEND PASSTHROUGH_VARS "${CURRENT_ARG}") + endif() + + math(EXPR ARGI "${ARGI} + 1") +endwhile() + +file( + WRITE + "${LINK_FLAGS_FILE}" + ${LINK_FLAGS} ${LINK_LIBRARIES} +) + +# This converts a space-delimited string to a cmake list +string(REPLACE " " ";" LINK_FLAGS_LIST "${LINK_LIBRARIES}" "${LINK_FLAGS}") +set(LINK_FLAGS ) + +# To pass linker args through cargo we need to use +# the -Clink-arg= syntax. +foreach(FLAG ${LINK_FLAGS_LIST}) + if(EXISTS "${FLAG}") + get_filename_component(FLAG "${FLAG}" ABSOLUTE) + endif() + + list(APPEND LINK_FLAGS "-Clink-arg=${FLAG}") +endforeach() + +string(REPLACE ";" " " LINK_FLAGS "${LINK_FLAGS}") +string(REPLACE " " ";" FLAGS "${FLAGS}") + +# TODO(sean): We don't always want to colour the output. Is +# there a way to autodetect this properly? +set(CARGO_COMMAND cargo build --color always ${FLAGS}) + +if(USE_CMAKE_LINK) + execute_process( + COMMAND ${CMAKE_COMMAND} -E env ${PASSTHROUGH_VARS} "RUSTFLAGS=${LINK_FLAGS}" ${CARGO_COMMAND} + WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" + RESULT_VARIABLE STATUS + ) +else() + execute_process( + COMMAND ${CMAKE_COMMAND} -E env ${PASSTHROUGH_VARS} ${CARGO_COMMAND} + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + RESULT_VARIABLE STATUS + ) +endif() + +# Ensure that our script exits with the correct error code. +# The only way to get a cmake script to exit with an error +# code is to print a message so that's what we do here. +if(NOT STATUS EQUAL 0) + message(STATUS "${PASSTHROUGH_VARS}") + + message(FATAL_ERROR "Cargo build failed") +endif() + +# Get the directory above TARGET since file(COPY ...) +# uses a directory as the destination +get_filename_component( + TARGET "${TARGET}/.." + ABSOLUTE +) + +file(COPY "${OUTPUT}" DESTINATION "${TARGET}") diff --git a/cmake/CargoTest.cmake b/cmake/CargoTest.cmake new file mode 100644 index 000000000..5e4de5064 --- /dev/null +++ b/cmake/CargoTest.cmake @@ -0,0 +1,107 @@ +# ccommon - a cache common library. +# Copyright (C) 2019 Twitter, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Ensure that empty elements in lists aren't deleted +cmake_policy(SET CMP0007 NEW) + +# Ignore the first 3 arguments since they will always be cmake -P /LinkRust.cmake +set(ARGI 3) +# This flips once we see -- in the arguments +set(PARSING_ENV_VARS OFF) +# Arguments to be passed directly to the build command +set(PASSTHROUGH_VARS ) + +# Split up the command-line arguments to this script +# into two groups +# +# Arguments before '--' are cmake variables that we set +# in this script. These are parameters which control the +# behaviour here. +# +# Arguments after '--' are environment variables to pass +# through to the cargo invocation, they are used by build +# scripts and to control cargo behaviour. +while(ARGI LESS ${CMAKE_ARGC}) + set(CURRENT_ARG ${CMAKE_ARGV${ARGI}}) + + if(NOT PARSING_ENV_VARS) + if(CURRENT_ARG STREQUAL "--") + set(PARSING_ENV_VARS ON) + else() + string(REPLACE "=" ";" ARGLIST "${CURRENT_ARG}") + + list(GET ARGLIST 0 VAR) + list(REMOVE_AT ARGLIST 0) + string(REPLACE ";" "=" VALUE "${ARGLIST}") + + set(${VAR} "${VALUE}") + endif() + else() + list(APPEND PASSTHROUGH_VARS "${CURRENT_ARG}") + endif() + + math(EXPR ARGI "${ARGI} + 1") +endwhile() + +file( + READ + "${LINK_FLAGS_FILE}" + LINK_FLAGS +) + +# This converts a space-delimited string to a cmake list +string(REPLACE " " ";" LINK_FLAGS_LIST "${LINK_FLAGS}") +set(LINK_FLAGS ) + +# To pass linker args through cargo we need to use +# the -Clink-arg= syntax. +foreach(FLAG ${LINK_FLAGS_LIST}) + if(EXISTS "${CMAKE_CURRENT_BINARY_DIR}/${FLAG}") + get_filename_component(FLAG "${CMAKE_CURRENT_BINARY_DIR}/${FLAG}" ABSOLUTE) + endif() + + list(APPEND LINK_FLAGS "-Clink-arg=${FLAG}") +endforeach() + +string(REPLACE ";" " " LINK_FLAGS "${LINK_FLAGS}") +string(REPLACE " " ";" FLAGS "${FLAGS}") + +# TODO(sean): We don't always want to colour the output. Is +# there a way to autodetect this properly? +set(CARGO_COMMAND cargo test --color always ${FLAGS}) + +if (USE_CMAKE_LINK) + execute_process( + COMMAND ${CMAKE_COMMAND} -E env ${PASSTHROUGH_VARS} "RUSTFLAGS=${LINK_FLAGS}" ${CARGO_COMMAND} + WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" + RESULT_VARIABLE STATUS + ) +else() + execute_process( + COMMAND ${CMAKE_COMMAND} -E env ${PASSTHROUGH_VARS} ${CARGO_COMMAND} + WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" + RESULT_VARIABLE STATUS + ) +endif() + +# Ensure that our script exits with the correct error code. +# The only way to get a cmake script to exit with an error +# code is to print a message so that's what we do here. +if(NOT STATUS EQUAL 0) + # Dump some script state to help with debugging + message(STATUS "PASSTHROUGH_VARS=${PASSTHROUGH_VARS}") + + message(FATAL_ERROR "cargo test failed") +endif() diff --git a/deps/ccommon/.gitignore b/deps/ccommon/.gitignore index 781ba3860..0682f193d 100644 --- a/deps/ccommon/.gitignore +++ b/deps/ccommon/.gitignore @@ -44,3 +44,12 @@ cscope.* lcov CMAKE_BINARY_DIR + +# Generated Rust Bindings +bindings.rs + +# Cargo lock file +Cargo.lock + +# Cargo build path +target \ No newline at end of file diff --git a/deps/ccommon/.travis.yml b/deps/ccommon/.travis.yml index 26a944005..0286b3a85 100644 --- a/deps/ccommon/.travis.yml +++ b/deps/ccommon/.travis.yml @@ -40,10 +40,31 @@ matrix: osx_image: xcode10.1 compiler: clang + - name: "cargo build" + language: rust + addons: + apt: + <<: *apt + packages: + - libsubunit-dev + script: + - ./ci/cargo.sh + + - os: osx + osx_image: xcode10.1 + language: rust + script: + - ./ci/cargo.sh + allow_failures: - os: osx osx_image: xcode10.1 compiler: clang + - os: osx + osx_image: xcode10.1 + language: rust + script: + - ./ci/cargo.sh before_install: - ./ci/before-install.sh diff --git a/deps/ccommon/ci/cargo.sh b/deps/ccommon/ci/cargo.sh new file mode 100755 index 000000000..a24b2dcfa --- /dev/null +++ b/deps/ccommon/ci/cargo.sh @@ -0,0 +1,53 @@ +#!/bin/bash + +set -uo pipefail +IFS=$'\n\t' + +# Cargo currently does not allow nested workspaces. Since this repo is vendored +# into github.com/twitter/pelikan which is itself a workspace, we cannot commit +# a workspace manifest for this repository. Cargo tracking issue: +# https://github.com/rust-lang/cargo/issues/5042 + +# As a workaround, we can construct a workspace manifest before running the +# build. This allows the crates within this repo to share build artifacts. + +### +# Create workspace manifest +### + +cat > Cargo.toml <
> Cargo.toml +done + +cat >> Cargo.toml <