pip install of 0.9.15 = No $GRAPHITE_ROOT/webapp/graphite exists

[jblaine]

I'm at a loss as to what is going on here. I'm expecting /opt/graphite/webapp/graphite to exist but it does not.

Here's a fresh virtualenv + pip install of 0.9.15:

etc-metrics:etc-metrics# virtualenv /opt/graphite
New python executable in /opt/graphite/bin/python
Installing setuptools, pip, wheel...done.
etc-metrics:etc-metrics# cd /opt/graphite
graphite:etc-metrics# ls
bin  include  lib  lib64  pip-selfcheck.json
graphite:etc-metrics# source bin/activate
(graphite) graphite:etc-metrics# pip install whisper==0.9.15
Collecting whisper==0.9.15
Installing collected packages: whisper
Successfully installed whisper-0.9.15
(graphite) graphite:etc-metrics# pip install carbon==0.9.15
Collecting carbon==0.9.15
Collecting txamqp (from carbon==0.9.15)
Collecting twisted (from carbon==0.9.15)
Collecting zope.interface>=3.6.0 (from twisted->carbon==0.9.15)
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages (from zope.interface>=3.6.0->twisted->carbon==0.9.15)
Installing collected packages: zope.interface, twisted, txamqp, carbon
Successfully installed carbon-0.9.15 twisted-16.1.1 txamqp-0.6.2 zope.interface-4.1.3
(graphite) graphite:etc-metrics# pip install graphite-web==0.9.15
Collecting graphite-web==0.9.15
Installing collected packages: graphite-web
Successfully installed graphite-web-0.9.15
(graphite) graphite:etc-metrics# pwd
/opt/graphite
(graphite) graphite:etc-metrics# ls webapp/
content
(graphite) graphite:etc-metrics#

[jblaine]

As additional info, everything is correct when not using pip and instead building from source.

[deniszh]

@jblaine,
I saw similar issue before, I suspect it's because of last changes in pip.
Could you please test is adding location options helps, like in https://graphite.readthedocs.org/en/0.9.15/install-pip.html#installing-carbon-in-a-custom-location but for /opt/graphite instead of /srv/graphite, e.g.
pip install whisper --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/lib"
pip install carbon --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/lib"
pip install graphite-web --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/webapp"
If yes - then we need to update docs.

[jblaine]

That works.

(graphite) webapp:etc-metrics# pwd
/opt/graphite/webapp
(graphite) webapp:etc-metrics# rm -rf graphite
(graphite) webapp:etc-metrics# pip install graphite-web --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/webapp"
/opt/graphite/lib/python2.7/site-packages/pip/commands/install.py:180: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting graphite-web
  Downloading graphite-web-0.9.15.tar.gz (2.3MB)
    100% |████████████████████████████████| 2.3MB 540kB/s
Skipping bdist_wheel for graphite-web, due to binaries being disabled for it.
Installing collected packages: graphite-web
  Running setup.py install for graphite-web ... done
Successfully installed graphite-web
(graphite) webapp:etc-metrics# ls graphite
account           composer      local_settings.py.example  metrics             settings.pyc  url_shortener  version
app_settings.py   dashboard     logger.py                  remote_storage.py   storage.py    urls.py        views.py
app_settings.pyc  events        logger.pyc                 remote_storage.pyc  storage.pyc   urls.pyc       views.pyc
browser           __init__.py   manage.py                  render              templates     util.py        whitelist
cli               __init__.pyc  manage.pyc                 settings.py         thirdparty    util.pyc
(graphite) webapp:etc-metrics#

[jblaine]

FWIW

(graphite) webapp:etc-metrics# pip --version
pip 8.1.1 from /opt/graphite/lib/python2.7/site-packages (python 2.7)
(graphite) webapp:etc-metrics#

[deniszh]

Cool! Thanks a lot for testing!
We have another pip install issue, it looks like pip didn't follow any deps and requirements automatically. Will try to find out what's going on.

[Aldian-fr]

I just encountered a similar problem trying to install graphite with pip and solved it by downgrading pip the following way, then retrying pip install pip==1.2.1

[bigunyak]

Have just encountered the same problem.
Looks like the problem has been known for at least 4 months now but the installation documentation under http://graphite.readthedocs.io/en/latest/install-pip.html still hasn't been fixed.
I use pip-8.1.2 and it doesn't install graphite under /opt/graphite, using /usr/local/lib/python2.7/dist-packages/opt/graphite/ instead.
What are my options here? Would using --install-option pip option solve this problem?

[obfuscurity]

I can confirm this behavior, and it started with pip version 7.0.0. This can be worked around with the --no-binary=:all: pip argument (to disable wheels).

Related, from the pip version 7.0.0 release notes:

• Support --install-option and --global-option per requirement in requirement files (PR #2537)
• The use of --install-option, --global-option or --build-option disable the use of wheels, and the autobuilding of wheels. (PR #2711) Fixes #2677

At the very least we should mention this issue in our install docs.

[obfuscurity]

@jblaine So we can't "fix" the docs for 0.9.15 since they're associated with the tag, which is associated with the pypi package, etc. Yay for python packaging once again. For the time being please make sure to use --no-binary=:all: when installing 0.9.15 and pip >= 7.0.0.

[Ajedi32]

It seems that installing from GitHub behaves as expected regardless of pip version even not on the master branch.

For example:

pip install https://github.com/graphite-project/graphite-web/tarball/0.9.15

works as expected, but

pip install graphite-web==0.9.15

does not.

So it seems like this problem has not been fixed on master; the only reason it currently seems to work is that you're not installing the master branch from PyPi. Even once the next version of graphite-web is released, the problem will continue for anyone installing from PyPi. IMO this issue should be reopened.

Anyone have any idea why pip behaves so differently based where it fetches the package from?

Edit: Link to relevant question on Launchpad: https://answers.launchpad.net/graphite/+question/401544

[Ajedi32]

It seems the problem isn't just related to where you got the tarball from (as in, the install mechanism used). The tarball on GitHub is actually different from the one hosted on PyPI! Compare the contents of https://github.com/graphite-project/graphite-web/tarball/0.9.15 with that of https://pypi.python.org/packages/b6/f3/7e4bae02f1a21cc29e9e9205bbc01aa29cdc6c696a996d41c1143e8935e3/graphite-web-0.9.15.tar.gz#md5=f4e80ba810fa83f57a62a2b8dd4e3545

Even if both packages are moved to a different web server and installed with pip install <url>, the one from GitHub gets installed to /opt/graphite/webapp/graphite_web-0.9.15-py2.7.egg-info/ while the one from PyPi ends up in /usr/local/lib/python2.7/dist-packages/graphite_web-0.9.15.dist-info/. (Edit: This is not the case, see my later comment below.)

It's still unclear as to what specific difference between the two files is causing this.

[deniszh]

Yep, pypi version (sic!) has following in setup.cfg

[install]
prefix = /opt/graphite
install-lib = %(prefix)s/webapp

Not sure how that section can do exactly opposite thing and bring package to /usr/local...

[Ajedi32]

Actually, it looks like the contents don't really matter after all. While those two packages do have slightly different contents, whether they get installed to /opt/graphite/webapp/graphite_web-0.9.15-py2.7.egg-info/ or /opt/graphite/webapp/graphite_web-0.9.15-py2.7.egg-info/ seems to depend entirely on the filename of the package when it gets installed by pip! 😱

I conducted 4 tests, each with a freshly-built, identically configured VM, using pip 8.1.1 from the official Ubuntu Xenial apt repository. In both cases where I named the file graphite-web-0.9.15.tar.gz and installed with pip install ./graphite-web-0.9.15.tar.gz (regardless of whether I used the file from GitHub or the file from PyPI), Graphite was installed to /usr/local/lib/python2.7/dist-packages/graphite_web-0.9.15.dist-info/ and I got this output:

Processing ./graphite-web-0.9.15.tar.gz
Building wheels for collected packages: graphite-web
  Running setup.py bdist_wheel for graphite-web ... done
  Stored in directory: /root/.cache/pip/wheels/4c/f2/34/33e2da2971584766ab6c6f9d7cb2b7167b00d3ce9d9ef85104
Successfully built graphite-web
Installing collected packages: graphite-web
Successfully installed graphite-web-0.9.15
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

In both cases where I named the file test.tar.gz and installed with pip install ./test.tar.gz (regardless of whether I used the file from GitHub or the file from PyPI), Graphite was installed to /opt/graphite/webapp/graphite_web-0.9.15-py2.7.egg-info/ and I got the following output:

Installing collected packages: graphite-web
  Running setup.py install for graphite-web ... done
Successfully installed graphite-web
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

So... yeah. What the heck, pip?!?! 😦

[Ajedi32]

After running a few more experiments, it looks like it always installs under /user/local/lib/ if I include the characters g, r, a, p, h, i, t, e, -, w, and b anywhere in the name of the file, in any order, otherwise it installs under /opt (seriously, what the heck pip?):

pip install $x	Installs to	Other notes
./graphite-web.tar.gz	/user/local/lib/...
./graphite-web	N/A	Fails with: "Invalid requirement: './graphite-web'\nIt looks like a path. Does it exist ?" (Even though the file does, in fact, exist at that location. Maybe it was expecting a directory?)
./graphite_web.tar.gz	/user/local/lib/...
./graphite\ web-0.9.15.tar.gz	/user/local/lib/...
./graphitexweb.tar.gz	/opt/...
./graphite.asdf	N/A	Fails with: "Invalid requirement: './graphite.asdf'\nIt looks like a path[...]"
./graphite.gz	N/A	Fails with: "Invalid requirement[...]"
./graphite.tar	/opt/...
http://server/0.9.15	/opt/...
./graphite-namehack-web-0.9.15.tar.gz	/user/local/lib/...
./namehackgraphite-web-0.9.15.tar.gz	/user/local/lib/...
./graphitename-hackweb-0.9.15.tar.gz	/user/local/lib/...
./graphitexweb-0.9.15.tar.gz	/user/local/lib/...
./x-webgraphite.tar.gz	/user/local/lib/...
./x\ webgraphite.tar.gz	/opt/...
./web-hitegrap.tar.gz	/user/local/lib/...
./grap-hitewb.tar.gz	/user/local/lib/...

pip 9.0.1 also appears to behave similarly.

[deniszh]

I'm wondering what's result has '--no-binary' flag on pip.

[deniszh]

@Ajedi32 - I mean, it's known issue, see previous comment - #1508 (comment)

[Ajedi32]

@deniszh I'm dealing with some rather unique circumstances which make using the --no-binary flag especially difficult (see echocat/puppet-graphite#309 (comment)), so I'm exploring alternative options.

[snctfd]

This still seems to be happening with the most recent version of Graphite. Running Python 3.8.2, pip 20.0.2 inside a virtualenv. These are the commands I'm using to install, taken straight from docs:

mkdir -p /opt/graphite
cd /opt/graphite
virtualenv .
source bin/activate

export PYTHONPATH="/opt/graphite/lib/:/opt/graphite/webapp/"
pip install --no-binary=:all: https://github.com/graphite-project/whisper/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/carbon/tarball/master
pip install --no-binary=:all: https://github.com/graphite-project/graphite-web/tarball/master

After the install, webapp/ only contains the content directory. If I instead use the commands from this comment, the install works as expected.

[deniszh]

@snctfd : I'm suspecting that's because of virtualenv, it should be 16.x and not 20.x