Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruby: support conan v2 and expose more configuration options for native extension gems #18338

Closed
wants to merge 31 commits into from

Conversation

jmarrec
Copy link
Contributor

@jmarrec jmarrec commented Jul 5, 2023

Specify library name and version: ruby/3.1.0

Ruby: support conan v2 and expose more configuration options for native extension gems

Reboots:

Both PRs were opened more than a year ago and got stale and closed.

Changelog (old commit messages):

post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/ruby/3.1.0/x86_64-linux/pty.so'
 links to system library 'util' but it is not in cpp_info.system_libs.
  • Use --with-opt-dir instead of --with-xx-dir that ruby isn't respecting.

conanio/gccXX (eg 10) removed the libxxx-dev (eg libgmp-dev) from the image. This made me realize that the conan deps weren't being picked up by the build. The openssl was, because --with-openssl-dir is explicitly used in ruby config.

So here, we rely on --with-opt-dir.

cf: https://bugs.ruby-lang.org/issues/19014#change-99241 (--with-gmp-dir was added on September 14, 2022, wrote this in Sep 22, 2022)

  • Pass both --with-opt-dir and --with-xxx-dir

https://gist.github.com/mrkn/6647630


@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@stale
Copy link

stale bot commented Oct 15, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@jmarrec jmarrec force-pushed the 12455_ruby_v2_more_options branch 2 times, most recently from ffed62a to 5e00d0e Compare December 19, 2023 13:55
@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

Comment on lines 22 to 34
# when --static-linked-ext is used, ruby defines EXTSTATIC as 1
# But when ruby itself is static there's nothing, so:
# We define RUBY_STATIC_RUBY when ruby itself is static
# We define RUBY_STATIC_LINKED_EXT when the ruby extensions are static (same as EXTSTATIC but clearer)
defs = {}
tested_options = self.dependencies[self.tested_reference_str].options
if not tested_options.shared:
defs["RUBY_STATIC_RUBY"] = 1
if tested_options.with_static_linked_ext:
defs["RUBY_STATIC_LINKED_EXT"] = 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these be added to self.cpp_info.defines in package_info() instead?

Copy link
Contributor Author

@jmarrec jmarrec Dec 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@valgur is right, these would be better placed in the package_info() of the upstream recipe if this is something that is expected for consumer use - consumers shouldn't have to define the variables themselves if they're provided by upstream.

Having said that, if upstream doesn't provide these and you want to use dependency options to do this kind of thing, you shouldn't be trying to parse these in the build() method anyway. Populating cmake with defines needs to happen in the generate() method. To illustrate (for your awareness), passing defines to cmake should be done as follows:

def generate(self):
    ruby_opts = self.dependencies[self.tested_reference_str].options
    tc = CMakeToolchain(self)
    tc.preprocessor_definitions["RUBY_STATIC_RUBY"] = not ruby_opts.shared
    tc.preprocessor_definitions["RUBY_STATIC_LINKED_EXT"] = not ruby_opts.shared and ruby_opts.with_static_linked_ext

def build(self):
    cmake = CMake(self)
    cmake.configure()
    cmake.build()

See the docs for CMakeToolchain for more info.

Note: I've assumed that you want to pass these as defines to the consumer build - as far as I can see, these might actually be better as cmake variables rather than defines, since I can't see a reason why a consumer library would ever want to do the following - I would imagine in most cases the code base itself wouldn't care whether the symbols were resolved statically or dynamically.

#ifdef RUBY_STATIC_RUBY
// do some stuff
#endif

If these don't need to be defines, then the above can change to the following to pass these options as cmake variables instead

def generate(self):
    ruby_opts = self.dependencies[self.tested_reference_str].options
    tc = CMakeToolchain(self)
    tc.variables["RUBY_STATIC_RUBY"] = not ruby_opts.shared
    tc.variables["RUBY_STATIC_LINKED_EXT"] = not ruby_opts.shared and ruby_opts.with_static_linked_ext

Note: ruby has a `--with-destdir=DESTDIR  specify default directory to install` option, maybe I should use that?
For libffi and libyaml, starting at 3.2.x ruby supports:
* Downloading the SOURCES to libyaml / libffi
* Configure with --with-libyaml-source-dir / --with-libffi-source-dir
* It then builds the libffi / libyaml itself

https://bugs.ruby-lang.org/issues/18571
> post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libruby.3.1.dylib, libruby.dylib

conan-io#18338 (comment)
During the build, miniruby is built, then it is actually used to build the extension libs.

The c3i v2 runner on macOS is failing

```
linking miniruby
dyld[16179]: Library not loaded: @rpath/libgmp.10.dylib
  Referenced from: <BC0A8868-CFAE-3B40-A112-FCAC741169E4> /Users/jenkins/w/prod-v2/bsr@2/80054/defbf/p/b/ruby1af0f3d020f64/b/build-release/miniruby
  Reason: tried: '/usr/local/lib/libgmp.10.dylib' (no such file), '/usr/lib/libgmp.10.dylib' (no such file, not in dyld cache)
make: *** [exe/ruby] Abort trap: 6
make: *** Deleting file `exe/ruby'
```
@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ✔️

All green in build 26 (7ddb85e1c871647d4ba8d10bdfbb59f2fc6cbde3):

  • ruby/3.1.0:
    All packages built successfully! (All logs)

Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

See details:

Failure in build 26 (7ddb85e1c871647d4ba8d10bdfbb59f2fc6cbde3):

  • ruby/3.1.0:
    CI failed to create some packages (All logs)

    Logs for packageID 12a7fecab21700c447590b654aefc8eb4a721158:
    [settings]
    arch=armv8
    build_type=Release
    compiler=apple-clang
    compiler.cppstd=17
    compiler.libcxx=libc++
    compiler.version=13
    os=Macos
    [options]
    */*:shared=True
    
    [...]
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/scheduler.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/signal.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/sprintf.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/st.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/strftime.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/string.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/struct.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/symbol.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/thread.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/time.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/transcode.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/transient_heap.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/util.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/variable.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/vm.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/vm_backtrace.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/vm_dump.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/vm_sync.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/vm_trace.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/yjit.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/missing/explicit_bzero.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/missing/setproctitle.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/addr2line.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/dmyenc.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/dln.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/localeinit.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/array.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/mjit.c
    config.status: creating ruby-runner.h
    assembling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/coroutine/arm64/Context.S
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/enc/ascii.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/enc/us_ascii.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/enc/unicode.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/enc/utf_8.c
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/enc/trans/newline.c
    /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/revision.h unchanged
    compiling /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/src/version.c
    linking miniruby
    dyld[77971]: Library not loaded: @rpath/libgmp.10.dylib
      Referenced from: <CF7DBBF0-FBB4-38DB-9A80-39B5E7551078> /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/build-release/miniruby
      Reason: tried: '/usr/local/lib/libgmp.10.dylib' (no such file), '/usr/lib/libgmp.10.dylib' (no such file, not in dyld cache)
    make: *** [exe/ruby] Abort trap: 6
    make: *** Deleting file `exe/ruby'
    
    ruby/3.1.0: ERROR: 
    Package '12a7fecab21700c447590b654aefc8eb4a721158' build failed
    ruby/3.1.0: WARN: Build folder /Users/jenkins/w/prod-v2/bsr/84037/cbdec/p/b/rubyd4c006a583fe1/b/build-release
    ERROR: ruby/3.1.0: Error in build() method, line 215
    	autotools.make()
    	ConanException: Error 2 while executing
    

Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.

@jmarrec
Copy link
Contributor Author

jmarrec commented Feb 2, 2024

Re the macOS V2 error that fails when shared: so we build miniruby, then we need to execute miniruby. Which is going to search for the dylibs.

The conanrunenv does have the LD_LIBRARY_PATH and DYLD_LIBRARY_PATH, but we don't have it activated. Not sure how to ensure these are defined when I build.

Copy link
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 28, 2024
Copy link
Contributor

This pull request has been automatically closed because it has not had recent activity. Thank you for your contributions.

@github-actions github-actions bot closed this Apr 27, 2024
valgur pushed a commit to valgur/conan-center-index that referenced this pull request May 21, 2024
valgur pushed a commit to valgur/conan-center-index that referenced this pull request May 21, 2024
> post_package(): WARN: [APPLE RELOCATABLE SHARED LIBS (KB-H077)] install_name dir of these shared libs is not @rpath: libruby.3.1.dylib, libruby.dylib

conan-io#18338 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants