librdkafka and cppkafka recipes #324

hmenn · 2021-03-26T09:13:45Z

I couldn't find kafka related recipes in meta-oe and opened this pr.

librdkafka is widely used in new IOT projects

kraj · 2021-03-26T17:38:12Z

meta-oe/recipes-support/librdkafka/librdkafka_1.6.1.bb

+SRCREV = "1a722553638bba85dbda5050455f7b9a5ef302de"
+
+DEPENDS = "zlib openssl zstd"
+RDEPENDS_${PN} = ""


why do we need this ?

meta-oe/recipes-support/libcppkafka/libcppkafka_git.bb

kraj · 2021-03-26T17:40:59Z

meta-oe/recipes-support/libcppkafka/libcppkafka_git.bb

+SRCREV = "5e4b350806d561473138ce7a982e8f6cf2e77733"
+
+DEPENDS = "librdkafka boost"
+RDEPENDS_${PN} = "librdkafka"


is this needed ? or can shlibs code automatically detect this dependency and add it to rdeps

hmenn

done

meta-oe/recipes-support/libcppkafka/libcppkafka_git.bb

kraj · 2021-03-30T16:16:21Z

in master-next now

kraj · 2021-03-30T19:38:16Z

Fails to configure see

https://errors.yoctoproject.org/Errors/Details/575207/

hmenn · 2021-03-31T13:13:35Z

@kraj Strange. I started a new build on dunfell and couldn't get any error.

kraj · 2021-03-31T14:17:39Z

@kraj Strange. I started a new build on dunfell and couldn't get any error.

Please test against master since it has to apply to master and then backported to dunfell if needed.

hmenn · 2021-04-02T08:49:43Z

@kraj I compiled again on master. Here is the logs.

NOTE: Direct dependencies are ['virtual:native:/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', 'virtual:native:/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/ninja/ninja_1.10.2.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/cmake/cmake-native_3.19.5.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-support/boost/boost_1.75.0.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/meta-openembedded/meta-oe/recipes-extended/librdkafka/librdkafka_1.6.1.bb:do_populate_sysroot', 'virtual:native:/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/patch/patch_2.7.6.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-core/glibc/glibc_2.33.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/quilt/quilt-native_0.66.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/gcc/gcc-runtime_10.2.bb:do_populate_sysroot', 'virtual:native:/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/chrpath/chrpath_0.16.bb:do_populate_sysroot', '/home/hmenn/Workplace/opensource/yocto/poky/meta/recipes-devtools/gcc/gcc-cross_10.2.bb:do_populate_sysroot']
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: ['pseudo-native', 'ninja-native', 'cmake-native', 'boost', 'librdkafka', 'patch-native', 'glibc', 'quilt-native', 'gcc-runtime', 'chrpath-native', 'gcc-cross-x86_64', 're2c-native', 'bzip2-native', 'curl-native', 'xz-native', 'zlib-native', 'ncurses-native', 'zlib', 'bzip2', 'icu', 'python3', 'openssl', 'zstd', 'automake-native', 'attr-native', 'libtool-native', 'autoconf-native', 'linux-libc-headers', 'libgcc', 'binutils-cross-x86_64', 'gmp-native', 'libmpc-native', 'texinfo-dummy-native', 'flex-native', 'mpfr-native', 'gnu-config-native', 'pkgconfig-native', 'openssl-native', 'gettext-minimal-native', 'opkg-utils', 'readline', 'libtirpc', 'libffi', 'libxcrypt', 'sqlite3', 'util-linux', 'libnsl2', 'gdbm', 'xz', 'm4-native', 'unzip-native', 'ncurses', 'libcap-ng', 'bash-completion', 'util-linux-libuuid']
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing shell function do_configure
-- The C compiler identification is GNU 10.2.0
-- The CXX compiler identification is GNU 10.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/hmenn/Workplace/opensource/yocto/build/tmp/work/core2-64-poky-linux/libcppkafka/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/hmenn/Workplace/opensource/yocto/build/tmp/work/core2-64-poky-linux/libcppkafka/git-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux/x86_64-poky-linux-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Build will generate a shared library. Use CPPKAFKA_BUILD_SHARED=0 to perform a static build
-- Found ZSTD: /home/hmenn/Workplace/opensource/yocto/build/tmp/work/core2-64-poky-linux/libcppkafka/git-r0/recipe-sysroot/usr/lib/libzstd.so
-- RdKafka configuration file found: /home/hmenn/Workplace/opensource/yocto/build/tmp/work/core2-64-poky-linux/libcppkafka/git-r0/recipe-sysroot/usr/lib/cmake/RdKafka/RdKafkaConfig.cmake
-- Disabling tests because submodule Catch2 isn't checked out
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_EXPORT_NO_PACKAGE_REGISTRY
    LIB_SUFFIX
    PYTHON_EXECUTABLE
    Python3_EXECUTABLE
    Python_EXECUTABLE


-- Build files have been written to: /home/hmenn/Workplace/opensource/yocto/build/tmp/work/core2-64-poky-linux/libcppkafka/git-r0/build
DEBUG: Shell function do_configure finished
DEBUG: Executing python function do_qa_configure
DEBUG: Python function do_qa_configure finished

This configurations were used.

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.2+snapshot-7d0988966cc1fd9e93549ca2ec21ac3bc765ceb6"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:7d0988966cc1fd9e93549ca2ec21ac3bc765ceb6"
meta-oe              
meta-networking      
meta-python          = "master:ba2576ed79a09879f714ccf7f4fa32259c18ab8c"

kraj · 2021-04-02T15:40:43Z

the failure is seen with. master-next of poky

hmenn · 2021-04-02T19:39:58Z

@kraj Here is the results for master-next branchs.

➜  yocto bitbake libcppkafka                       
Loading cache: 100% |##################################################################################################################################################################################################################################| Time: 0:00:01
Loaded 3532 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.2+snapshot-f5579cd98822c27c3c7414c9e03465f1f54f0637"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master-next:f5579cd98822c27c3c7414c9e03465f1f54f0637"
meta-oe              
meta-networking      
meta-python          = "master-next:a49892059b8cdfa8e3e5ca8de8e7957355aaa85d"

Initialising tasks: 100% |#############################################################################################################################################################################################################################| Time: 0:00:01
Sstate summary: Wanted 371 Local 0 Network 0 Missed 371 Current 237 (0% match, 38% complete)
Removing 10 stale sstate objects for arch allarch: 100% |##############################################################################################################################################################################################| Time: 0:00:00
Removing 220 stale sstate objects for arch core2-64: 100% |############################################################################################################################################################################################| Time: 0:00:00
Removing 96 stale sstate objects for arch qemux86_64: 100% |###########################################################################################################################################################################################| Time: 0:00:00
Removing 1 stale sstate objects for arch x86_64_ubuntu-18.04: 100% |###################################################################################################################################################################################| Time: 0:00:00
Removing 18 stale sstate objects for arch x86_64: 100% |###############################################################################################################################################################################################| Time: 0:00:00
NOTE: Executing Tasks
NOTE: Tasks Summary: Attempted 1955 tasks of which 1624 didn't need to be rerun and all succeeded.

kraj · 2021-04-03T01:43:00Z

I have restaged it in master-next and will see if it passes this time.

kraj · 2021-04-03T13:36:25Z

it still shows up, I think the reason is that librdkafka generates cmake with absolute paths into its build directory
and when the component build directory is deleted when using rm_work then this issue appears

I see

set_target_properties(RdKafka::rdkafka PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "/mnt/b/yoe/master/build/tmp/work/cortexa55-yoe-linux/librdkafka/1.6.1
-r0/recipe-sysroot/usr/include;/mnt/b/yoe/master/build/tmp/work/cortexa55-yoe-linux/librdkafka/1.6.1-r
0/recipe-sysroot/usr/include;/mnt/b/yoe/master/build/tmp/work/cortexa55-yoe-linux/librdkafka/1.6.1-r0/
recipe-sysroot/usr/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "m;ZLIB::ZLIB;/mnt/b/yoe/master/build/tmp/work/cortexa55-yoe-linux/librdkaf
ka/1.6.1-r0/recipe-sysroot/usr/lib/libzstd.so;OpenSSL::SSL;OpenSSL::Crypto;Threads::Threads;dl"
)

in ./build/CMakeFiles/Export/lib/cmake/RdKafka/RdKafkaTargets.cmake

and this .cmake file is exported by librdkafka package

these absolute paths is the problem,

jenkins builders user rm_work so you can reproducre it like below
add inherit rm_work to librdkafka recipe and repeat these steps

bitbake librdkafka libcppkafka -ccleansstate 
bitbake librdkafka
bitbake libcppkafka

and you should see the issue

kraj · 2021-04-04T17:56:46Z

meta-oe/recipes-support/librdkafka/librdkafka/0001_fix_absolute_path_usage.patch

+From: hasan.men <hasan.men@bosphorusiss.com>
+Date: Sun, 4 Apr 2021 17:20:32 +0200
+Subject: [PATCH] fix absolute path problem
+


please submit this patch upstream as well. If its not worthy of upstream then added the status to patch comment so we know we have to carry it for a reason.

@kraj I've already pushed. Just waiting for response. Thanks

confluentinc/librdkafka#3327

kraj · 2021-04-04T18:46:25Z

still fails :( with latest patch, here are steps.

diff --git a/meta-oe/recipes-support/librdkafka/librdkafka_1.6.1.bb b/meta-oe/recipes-support/librdkafka/librdkafka_1.6.1.bb
index 21a130e82b..f6be5cf1d5 100644
--- a/meta-oe/recipes-support/librdkafka/librdkafka_1.6.1.bb
+++ b/meta-oe/recipes-support/librdkafka/librdkafka_1.6.1.bb
@@ -19,3 +19,5 @@ inherit cmake
 S = "${WORKDIR}/git"

 FILES_${PN} += "${datadir}"
+
+inherit rm_work

bitbake -ccleansstate librdkafka libcppkafka
bitbake librdkafka ; bitbake libcppkafka

* A patch is added to fix absolute path usage in cmake.

hmenn · 2021-04-04T22:40:27Z

@kraj After lots of debugging I found problems with include path's of external packages. I changed their dependency because we don't them public to use librdkafka unless their FindPackage changed. I'll also push those changes to mainstream. I'm able to build with rm_work now.

For my previous commit, I don't know how build passed with rm_work. I'm really sorry for that mistake.

hmenn · 2021-04-09T10:46:42Z

Hello @kraj, thank you for your support. Now, it's merged with master. What should I do in order to get these in dunfell? Which documentation should I read to learn meta-openembedded merge mechanism?

kraj · 2021-04-09T16:10:52Z

Hello @kraj, thank you for your support. Now, it's merged with master. What should I do in order to get these in dunfell? Which documentation should I read to learn meta-openembedded merge mechanism?

usually new recipes are not often backported but you can make a case by creating a pull request against dunfell banch and stable maintainer will advice you accordingly.

kraj · 2021-04-13T19:25:47Z

its committed with 3e5209b and 285e612

Changelog: ========== Add use_location_hints argument to document API for giving the option of ignoring XSI schema locations hints Fix import from locations hints with namespace mismatch (issue openembedded#324) Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>

@yozachar

License-Update: Rename LICENSE to LICENSE.txt and update copyright year * Ptest and library example tested on qemux86-64 and qemuarm64 * Add ptest into PTESTS_FAST_META_PYTHON qemux86-64: Testsuite summary TOTAL: 599 PASS: 599 SKIP: 0 XFAIL: 0 FAIL: 0 XPASS: 0 ERROR: 0 DURATION: 19 END: /usr/lib/python3-validators/ptest 2024-03-30T05:07 STOP: ptest-runner TOTAL: 1 FAIL: 0 qemuarm64: Testsuite summary TOTAL: 599 PASS: 599 SKIP: 0 XFAIL: 0 FAIL: 0 XPASS: 0 ERROR: 0 DURATION: 18 END: /usr/lib/python3-validators/ptest 2024-03-30T05:10 STOP: ptest-runner TOTAL: 1 FAIL: 0 Changelog: 0.24.0 =========== Features feat: conditionally raises ValidationError; bump version by @yozachar in openembedded#343 Maintenance patch: domain & url modules by @yozachar in openembedded#339 fix: domain name not confirming to rfc_2782 by @yozachar in openembedded#341 maint: update dev dependencies; adds favicon to docs by @yozachar in openembedded#342 0.23.2 =========== Maintenance maint: rectifies changelog by @yozachar in openembedded#336 fix: packaging as well as rST & md document generation by @yozachar in openembedded#337 0.23.1 =========== Maintenance maint: fix between & length validators by @yozachar in openembedded#334 fix: manual nav reference for mkdocs; bumps version by @yozachar in openembedded#335 0.23.0 =========== Features feat: add french i18n validation by @imperosol in openembedded#308 Maintenance fix: Valid URLs failing validation - query and fragment parts by @danherbriley in openembedded#297 fix: bug in between module by @yozachar in openembedded#301 chore: update dependencies, improve packaging by @yozachar in openembedded#304 Fix fragment check by @darkdragon-001 in openembedded#305 build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /package by @dependabot in openembedded#310 fix: allow pct-encoded entities in fragments by @conitrade-as in openembedded#317 chore: update dev dependencies by @yozachar in openembedded#318 build(deps): bump gitpython from 3.1.37 to 3.1.41 in /package by @dependabot in openembedded#321 build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /package by @dependabot in openembedded#322 chore: monthly updates for Jan'24 by @yozachar in openembedded#324 maint: adds versiond docs; update copyright year by @yozachar in openembedded#329 chore: update dev dependencies by @yozachar in openembedded#330 build(deps): bump gitpython from 3.1.37 to 3.1.41 in /package by @dependabot in openembedded#331 build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /package by @dependabot in openembedded#332 build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /package by @dependabot in openembedded#319 Signed-off-by: alperak <alperyasinak1@gmail.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>

@yozachar

License-Update: Rename LICENSE to LICENSE.txt and update copyright year * Ptest and library example tested on qemux86-64 and qemuarm64 * Add ptest into PTESTS_FAST_META_PYTHON qemux86-64: Testsuite summary TOTAL: 599 PASS: 599 SKIP: 0 XFAIL: 0 FAIL: 0 XPASS: 0 ERROR: 0 DURATION: 19 END: /usr/lib/python3-validators/ptest 2024-03-30T05:07 STOP: ptest-runner TOTAL: 1 FAIL: 0 qemuarm64: Testsuite summary TOTAL: 599 PASS: 599 SKIP: 0 XFAIL: 0 FAIL: 0 XPASS: 0 ERROR: 0 DURATION: 18 END: /usr/lib/python3-validators/ptest 2024-03-30T05:10 STOP: ptest-runner TOTAL: 1 FAIL: 0 Changelog: 0.24.0 =========== Features feat: conditionally raises ValidationError; bump version by @yozachar in #343 Maintenance patch: domain & url modules by @yozachar in #339 fix: domain name not confirming to rfc_2782 by @yozachar in #341 maint: update dev dependencies; adds favicon to docs by @yozachar in #342 0.23.2 =========== Maintenance maint: rectifies changelog by @yozachar in #336 fix: packaging as well as rST & md document generation by @yozachar in #337 0.23.1 =========== Maintenance maint: fix between & length validators by @yozachar in #334 fix: manual nav reference for mkdocs; bumps version by @yozachar in #335 0.23.0 =========== Features feat: add french i18n validation by @imperosol in #308 Maintenance fix: Valid URLs failing validation - query and fragment parts by @danherbriley in #297 fix: bug in between module by @yozachar in #301 chore: update dependencies, improve packaging by @yozachar in #304 Fix fragment check by @darkdragon-001 in #305 build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /package by @dependabot in #310 fix: allow pct-encoded entities in fragments by @conitrade-as in #317 chore: update dev dependencies by @yozachar in #318 build(deps): bump gitpython from 3.1.37 to 3.1.41 in /package by @dependabot in #321 build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /package by @dependabot in #322 chore: monthly updates for Jan'24 by @yozachar in #324 maint: adds versiond docs; update copyright year by @yozachar in #329 chore: update dev dependencies by @yozachar in #330 build(deps): bump gitpython from 3.1.37 to 3.1.41 in /package by @dependabot in #331 build(deps): bump jinja2 from 3.1.2 to 3.1.3 in /package by @dependabot in #332 build(deps): bump urllib3 from 2.0.6 to 2.0.7 in /package by @dependabot in #319 Signed-off-by: alperak <alperyasinak1@gmail.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>

Changelog: =========== - Fixes openembedded#409: IndexError on empty strong mark. - Fix openembedded#332: Insert at most one space for multiple emphasis - Feature openembedded#318: Make padded tables more similar to pandoc's pipe_tables. - Add support for Python 3.9. - Fix extra line breaks inside html link text (between '[' and ']') - Fix openembedded#344: indent <ul> inside <ol> three spaces instead of two to comply with CommonMark, GFM, etc. - Fix openembedded#324: unnecessary spaces around <b>, <em>, and strike tags. - Don't wrap tables by default and add a --wrap-tables config option. - Feature openembedded#198: Ignore <p> tags inside table rows. - Don't wrap tables by default and add a --wrap-tables config option - Remove support for Python <= 3.5. Now requires Python 3.6+. - Support for Python 3.10+. - Fix openembedded#320 padding empty tables and tables with no </tr> tags. - Add ignore_mailto_links config option to ignore mailto: style links. - Feature openembedded#407: Support the superscript and subscript tags. - Fix openembedded#373: \n inside text of a Markdown link. - Feature openembedded#406: Improve support for null atttibute values. Signed-off-by: Wang Mingyu <wangmy@fujitsu.com> Signed-off-by: Khem Raj <raj.khem@gmail.com>

kraj requested changes Mar 26, 2021

View reviewed changes

hmenn force-pushed the dunfell-next branch from 433d38e to 0b4524a Compare March 30, 2021 09:45

hmenn commented Mar 30, 2021

View reviewed changes

meta-oe/recipes-support/libcppkafka/libcppkafka_git.bb Show resolved Hide resolved

hmenn force-pushed the dunfell-next branch from 0b4524a to e2ec09f Compare April 4, 2021 15:32

kraj reviewed Apr 4, 2021

View reviewed changes

hmenn added 2 commits April 5, 2021 00:24

librdkafka: Add initial recipe v1.6.1

993194e

* A patch is added to fix absolute path usage in cmake.

libcppkafka: Add initial recipe for cppkafka wrapper

daa6db2

hmenn force-pushed the dunfell-next branch from e2ec09f to daa6db2 Compare April 4, 2021 22:35

hmenn mentioned this pull request Apr 5, 2021

cmake: Fix using absolute path in include directories confluentinc/librdkafka#3329

Merged

kraj closed this Apr 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

librdkafka and cppkafka recipes #324

librdkafka and cppkafka recipes #324

hmenn commented Mar 26, 2021

kraj Mar 26, 2021

kraj Mar 26, 2021

hmenn left a comment

kraj commented Mar 30, 2021

kraj commented Mar 30, 2021

hmenn commented Mar 31, 2021

kraj commented Mar 31, 2021

hmenn commented Apr 2, 2021 •

edited

Loading

kraj commented Apr 2, 2021

hmenn commented Apr 2, 2021

kraj commented Apr 3, 2021

kraj commented Apr 3, 2021

kraj Apr 4, 2021

hmenn Apr 4, 2021

kraj commented Apr 4, 2021

hmenn commented Apr 4, 2021

hmenn commented Apr 9, 2021

kraj commented Apr 9, 2021

kraj commented Apr 13, 2021

librdkafka and cppkafka recipes #324

librdkafka and cppkafka recipes #324

Conversation

hmenn commented Mar 26, 2021

kraj Mar 26, 2021

Choose a reason for hiding this comment

kraj Mar 26, 2021

Choose a reason for hiding this comment

hmenn left a comment

Choose a reason for hiding this comment

kraj commented Mar 30, 2021

kraj commented Mar 30, 2021

hmenn commented Mar 31, 2021

kraj commented Mar 31, 2021

hmenn commented Apr 2, 2021 • edited Loading

kraj commented Apr 2, 2021

hmenn commented Apr 2, 2021

kraj commented Apr 3, 2021

kraj commented Apr 3, 2021

kraj Apr 4, 2021

Choose a reason for hiding this comment

hmenn Apr 4, 2021

Choose a reason for hiding this comment

kraj commented Apr 4, 2021

hmenn commented Apr 4, 2021

hmenn commented Apr 9, 2021

kraj commented Apr 9, 2021

kraj commented Apr 13, 2021

hmenn commented Apr 2, 2021 •

edited

Loading