Install librdkafka dependency on Mac OSX #6

mikesparr · 2016-05-26T20:41:29Z

I was unable to install this via PIP on my Mac given the latest Homebrew formula for librdkafka was only up to 0.9 and I saw errors during PIP install. To solve I edited the Homebrew formula for librdkafka to download the latest version. After that, the PIP install worked without errors.

https://gist.github.com/mikesparr/164c4dec9e37ad78d3b0ffcbad12909c

You can type:

brew edit librdkafka and then paste in content above
you might have to brew unlink librdkafka before installing new version
afterwords in your setup.py file add dependency confluent-kafka and then run
- python setup.py install (or with sudo)

Good luck!

The text was updated successfully, but these errors were encountered:

ewencp · 2016-05-26T20:49:55Z

@mikesparr Thanks for the note. I'm going to close this since what we need is a change in homebrew, but the comment will still be discoverable by search for anyone who runs into the same issue. In fact, I'd suggest just submitting that patch to homebrew if you haven't already!

mikesparr · 2016-05-26T21:41:47Z

Thanks @ewencp and yes I figure just in case folks need to search for it. Possibly will share in Stack Overflow just in case. I'm unsure how to generate the bottles and checksums for the different Mac OS versions like in original so was hesitant to commit to the repo given I commented out that section. I'll research how to contribute though indeed.

ksachdeva11 · 2017-02-08T06:20:46Z

@mikesparr can you tell me the sequence of commands to install librdkafka on Mac ? Thanks!

mikesparr · 2017-02-08T07:01:05Z

I believe I used Homebrew so brew install librdkafka. If you are a Docker user, you could spin up an image with Ubuntu latest and install that way with apt-get install librdkafka as well. It's been many months and I'm on other projects so I don't recall exactly but I try to utilize Homebrew on Mac whenever possible.

@ksachdeva11 if you see my Gist link above, that is my update to the Homebrew formulae that you run locally ( brew edit [formulaname] and replace old contents with mine before running brew install [formulaname] where [formulaname] in this case is librdkafka )

ksachdeva11 · 2017-02-08T07:56:57Z

@mikesparr Is there a way to uninstall librdkafka and install it again? I actually installed without editing the file and didn't make the changes mentioned in your Gist..

ksachdeva11 · 2017-02-08T08:01:13Z

Installing librdkafka initially and then editing brew and then installing brew again gave me this error

Kunals-Mac:~ ksachdeva$ brew install librdkafka
Updating Homebrew...
==> Auto-updated Homebrew!
Updated Homebrew from 9ff22c7 to bef2c6c.
No changes to formulae.

Error: undefined method `endclass' for #<Class:0x007f904bab4fb8>
Please report this bug:
  http://docs.brew.sh/Troubleshooting.html
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/librdkafka.rb:37:in `<class:Librdkafka>'
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/librdkafka.rb:1:in `load_formula'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:26:in `module_eval'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:26:in `load_formula'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:46:in `load_formula_from_path'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:101:in `load_file'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:92:in `klass'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:88:in `get_formula'
/usr/local/Homebrew/Library/Homebrew/formulary.rb:259:in `factory'
/usr/local/Homebrew/Library/Homebrew/extend/ARGV.rb:43:in `block in formulae'
/usr/local/Homebrew/Library/Homebrew/extend/ARGV.rb:41:in `map'
/usr/local/Homebrew/Library/Homebrew/extend/ARGV.rb:41:in `formulae'
/usr/local/Homebrew/Library/Homebrew/cmd/install.rb:104:in `install'
/usr/local/Homebrew/Library/Homebrew/brew.rb:94:in `<main>`
```

mikesparr · 2017-02-08T08:08:18Z

Usually it's a typo or indentation issue. I'd recommend downloading the RAW format of Gist and editing and retrying.

Otherwise, every environment is unique so you'll have to trial/error. I'm on a new Macbook Pro w/ Touchbar and Sierra 10.12.3 but I did this on my prior computer 9 months ago so unsure what may have changed. My current env I'm using Docker containers so I'm not testing brew formulae at the moment. Google each error until you get through them all.

ksachdeva11 · 2017-02-08T08:46:07Z

@mikesparr Do I need to edit sha256 with sha256 "63b97ce272e07632f10e33a7963f291bf1a0457f0823d4ab4bdce01eb3e65bbe" => :el_capitan for El Capitan or the default value holds good ?

mikesparr · 2017-02-08T16:30:34Z

Google is your friend. ;-) If you use Homebrew you can brew remove any installed formula and various flags for forcing cleanup.

http://brew.sh/ (check their documentation)
- http://docs.brew.sh/

Most of us learn through trial/error and Google searching for answers and I apologize but I don't have time to teach 3rd-party tool usage. If something is not working, comment it out and try again and if error messages, Google for them and you'll find most answers on Stack Overflow or similar sites where people have faced same thing.

rjurney · 2017-08-14T19:05:08Z

Unable to install confluent-kafka on OS X because it doesn't detect librdkafka as installed by homebrew :(

edenhill · 2017-08-14T19:11:59Z

@rjurney Can you provide the output of the pip install .. or other means of installation?

Also paste the output from brew info librdkafka

Thanks

pranny · 2017-08-24T11:18:08Z

I am using librdkafka 0.11 and still unable to get this issue resolved. The details of my brew info librdkafka goes below

(venv) Pranavs-MacBook-Pro-2:thin-worker1 pranav$ brew info librdkafka
librdkafka: stable 0.11.0 (bottled), HEAD
The Apache Kafka C/C++ library
https://github.com/edenhill/librdkafka
/usr/local/Cellar/librdkafka/0.11.0 (14 files, 1.9MB) *
  Poured from bottle on 2017-08-24 at 16:38:14
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/librdkafka.rb
==> Dependencies
Build: pkg-config ✔
Required: lzlib ✔, openssl ✔
Recommended: lz4 ✔
==> Options
--without-lz4
	    Build without lz4 support
--HEAD
    	Install HEAD version

The error line is

In file included from confluent_kafka/src/confluent_kafka.c:17:
confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found

edenhill · 2017-08-24T11:20:06Z

That's odd, can you paste the full output from the confluent_kafka python installation?

pranny · 2017-08-24T11:34:41Z

My version is Python 2.7.10 and pip 9.0.1. Here is the full log

(venv) Pranavs-MacBook-Pro-2:thin-worker1 pranav$ pip install confluent_kafka
Collecting confluent_kafka
  Using cached confluent-kafka-0.11.0.tar.gz
Building wheels for collected packages: confluent-kafka
  Running setup.py bdist_wheel for confluent-kafka ... error
  Complete output from command /Users/pranav/workspace/quartic.ai/thin-worker1/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-build-a5hdLm/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/tmpk07rEApip-wheel- --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.12-intel-2.7
  creating build/lib.macosx-10.12-intel-2.7/confluent_kafka
  copying confluent_kafka/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka
  creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
  copying confluent_kafka/avro/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
  copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
  copying confluent_kafka/avro/error.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
  copying confluent_kafka/avro/load.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
  creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
  creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.macosx-10.12-intel-2.7
  creating build/temp.macosx-10.12-intel-2.7/confluent_kafka
  creating build/temp.macosx-10.12-intel-2.7/confluent_kafka/src
  cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.12-intel-2.7/confluent_kafka/src/confluent_kafka.o
  In file included from confluent_kafka/src/confluent_kafka.c:17:
  confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
  #include <librdkafka/rdkafka.h>
           ^
  1 error generated.
  error: command 'cc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
Failed to build confluent-kafka
Installing collected packages: confluent-kafka
  Running setup.py install for confluent-kafka ... error
    Complete output from command /Users/pranav/workspace/quartic.ai/thin-worker1/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-build-a5hdLm/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-x84TPg-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/pranav/workspace/quartic.ai/thin-worker1/venv/include/site/python2.7/confluent-kafka:
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.12-intel-2.7
    creating build/lib.macosx-10.12-intel-2.7/confluent_kafka
    copying confluent_kafka/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka
    creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
    copying confluent_kafka/avro/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
    copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
    copying confluent_kafka/avro/error.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
    copying confluent_kafka/avro/load.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro
    creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
    copying confluent_kafka/kafkatest/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
    copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
    copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
    copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/kafkatest
    creating build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
    copying confluent_kafka/avro/serializer/__init__.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
    copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.macosx-10.12-intel-2.7/confluent_kafka/avro/serializer
    running build_ext
    building 'confluent_kafka.cimpl' extension
    creating build/temp.macosx-10.12-intel-2.7
    creating build/temp.macosx-10.12-intel-2.7/confluent_kafka
    creating build/temp.macosx-10.12-intel-2.7/confluent_kafka/src
    cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c confluent_kafka/src/confluent_kafka.c -o build/temp.macosx-10.12-intel-2.7/confluent_kafka/src/confluent_kafka.o
    In file included from confluent_kafka/src/confluent_kafka.c:17:
    confluent_kafka/src/confluent_kafka.h:21:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^
    1 error generated.
    error: command 'cc' failed with exit status 1

    ----------------------------------------
Command "/Users/pranav/workspace/quartic.ai/thin-worker1/venv/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-build-a5hdLm/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-x84TPg-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/pranav/workspace/quartic.ai/thin-worker1/venv/include/site/python2.7/confluent-kafka" failed with error code 1 in /private/var/folders/4t/8q620bfn1b34s9wyzm1hp1qm0000gn/T/pip-build-a5hdLm/confluent-kafka/
(venv) Pranavs-MacBook-Pro-2:thin-worker1 pranav$

edenhill · 2017-08-26T07:58:03Z

Looks like Python is not making use of the homebrew installation location (which should be linked to /usr/local/), try fixing the homebrew package with brew link --overwrite librdkafka and if that doesn't work you can force it in the right direction by:

C_INCLUDE_PATH=/usr/local/Cellar/librdkafka/0.11.0/include LIBRARY_PATH=/usr/local/Cellar/librdkafka/0.11.0/lib pip install confluent_kafka

pranny · 2017-08-28T10:36:10Z

Including the right paths did help me install, although in a virtualenv only. While trying to install it system wide I did face permissions issue, but neverthless, this helped me move past the issue. Thanks a lot.

BerghmansNathan · 2021-12-16T15:08:14Z

Looks like Python is not making use of the homebrew installation location (which should be linked to /usr/local/), try fixing the homebrew package with brew link --overwrite librdkafka and if that doesn't work you can force it in the right direction by:

C_INCLUDE_PATH=/usr/local/Cellar/librdkafka/0.11.0/include LIBRARY_PATH=/usr/local/Cellar/librdkafka/0.11.0/lib pip install confluent_kafka

Including the paths was the only thing that worked for me in the virtualenv !

jensneuhaus · 2022-02-24T11:43:16Z

Adding the environment variables also worked for me. On macOS Monterey, they might be a little different though.

Following worked for me:

brew install librdkafka

# check the outputed path and adapt with your version
export C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/include
export LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.8.2/lib

# Now it should work
pip install confluent-kafka

haf · 2022-04-16T00:07:17Z

@jensneuhaus That's not a very good solution since you'll have to do that for every single version and library you install. Here's the generic solution:

export LIBRARY_PATH="$HOMEBREW_PREFIX/lib:/usr/lib"
export CPATH="$HOMEBREW_PREFIX/include:$(xcrun --show-sdk-path)/usr/include"

As configured by https://github.com/haf/macos

edenhill closed this as completed Jun 23, 2016

drice mentioned this issue Aug 4, 2016

Producer not thread-safe #19

Closed

tejasbehra mentioned this issue Nov 28, 2016

Seg-fault in running integration test #89

Closed

tejas3190 mentioned this issue Apr 2, 2017

Segmentation fault on using consumer #167

Closed

tejas3190 mentioned this issue Apr 9, 2017

Consumer fails to consume on broker restart #173

Closed

edenhill added the installation label Aug 26, 2017

ghost mentioned this issue Oct 16, 2017

SIGSEGV on program exit when RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS #265

Closed

7 tasks

danyc mentioned this issue May 19, 2020

SIGSEGV, Objects/unicodeobject.c: No such file or directory. randomly on consumer re-connecting to broker #874

Closed

7 tasks

odemkovych mentioned this issue Jun 8, 2023

Consumer close() function hungs and prevent app exit #1573

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Install librdkafka dependency on Mac OSX #6

Install librdkafka dependency on Mac OSX #6

mikesparr commented May 26, 2016

ewencp commented May 26, 2016

mikesparr commented May 26, 2016

ksachdeva11 commented Feb 8, 2017

mikesparr commented Feb 8, 2017 •

edited

Loading

ksachdeva11 commented Feb 8, 2017

ksachdeva11 commented Feb 8, 2017 •

edited

Loading

mikesparr commented Feb 8, 2017

ksachdeva11 commented Feb 8, 2017

mikesparr commented Feb 8, 2017 •

edited

Loading

rjurney commented Aug 14, 2017

edenhill commented Aug 14, 2017

pranny commented Aug 24, 2017

edenhill commented Aug 24, 2017

pranny commented Aug 24, 2017

edenhill commented Aug 26, 2017

pranny commented Aug 28, 2017

BerghmansNathan commented Dec 16, 2021

jensneuhaus commented Feb 24, 2022

haf commented Apr 16, 2022

Install librdkafka dependency on Mac OSX #6

Install librdkafka dependency on Mac OSX #6

Comments

mikesparr commented May 26, 2016

ewencp commented May 26, 2016

mikesparr commented May 26, 2016

ksachdeva11 commented Feb 8, 2017

mikesparr commented Feb 8, 2017 • edited Loading

ksachdeva11 commented Feb 8, 2017

ksachdeva11 commented Feb 8, 2017 • edited Loading

mikesparr commented Feb 8, 2017

ksachdeva11 commented Feb 8, 2017

mikesparr commented Feb 8, 2017 • edited Loading

rjurney commented Aug 14, 2017

edenhill commented Aug 14, 2017

pranny commented Aug 24, 2017

edenhill commented Aug 24, 2017

pranny commented Aug 24, 2017

edenhill commented Aug 26, 2017

pranny commented Aug 28, 2017

BerghmansNathan commented Dec 16, 2021

jensneuhaus commented Feb 24, 2022

haf commented Apr 16, 2022

mikesparr commented Feb 8, 2017 •

edited

Loading

ksachdeva11 commented Feb 8, 2017 •

edited

Loading

mikesparr commented Feb 8, 2017 •

edited

Loading