Cannot install Bioconductor package ShortRead #74
Comments
|
I just tested with an intermediate build of This version requires |
|
@brendanf I'm not sure what is causing the issue. But to better understand your use case, what is your motivation for installing ShortRead from source instead of from the bioconda channel? The commands below install ShortRead 1.40.0, which is the current release. Note that including If you need to install a bleeding edge version of ShortRead, you could modify the existing bioconda recipe for bioconductor-shortread to point to the devel version of ShortRead (1.41.0), build the recipe with |
|
@jdblischak Thanks for the reply. I'm using packrat to manage my R packages, because I am using some which can only be installed from GitHub (or where I need a feature that hasn't made it to CRAN/Bioconductor yet). However, |
|
I just downloaded the source of diff --git a/configure.ac b/configure.ac
index d9111f7..c6bcd2b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,6 @@
AC_INIT("DESCRIPTION")
-AC_CHECK_LIB([z], [gzeof], , AC_ERROR([zlib not found]))
+#AC_CHECK_LIB([z], [gzeof], , AC_ERROR([zlib not found]))
AC_CHECK_SIZEOF([unsigned long])
AC_OUTPUT(src/Makevars)I then installed it in a Conda environment using the latest $ conda list
# packages in environment at /home/brendan/miniconda3/envs/test1:
#
# Name Version Build Channel
_r-mutex 1.0.0 anacondar_1
binutils_impl_linux-64 2.31.1 h6176602_1 conda-forge
binutils_linux-64 2.31.1 h6176602_3 conda-forge
bwidget 1.9.11 1
bzip2 1.0.6 h14c3975_1002 conda-forge
ca-certificates 2018.11.29 ha4d7672_0 conda-forge
cairo 1.14.12 h80bd089_1005 conda-forge
curl 7.64.0 h646f8bb_2 conda-forge
fontconfig 2.13.1 h2176d3f_1000 conda-forge
freetype 2.9.1 h94bbf69_1005 conda-forge
gcc_impl_linux-64 7.3.0 habb00fd_1 conda-forge
gcc_linux-64 7.3.0 h553295d_3 conda-forge
gettext 0.19.8.1 h9745a5d_1001 conda-forge
gfortran_impl_linux-64 7.3.0 hdf63c60_1 conda-forge
gfortran_linux-64 7.3.0 h553295d_3 conda-forge
glib 2.56.2 had28632_1001 conda-forge
graphite2 1.3.13 hf484d3e_1000 conda-forge
gxx_impl_linux-64 7.3.0 hdf63c60_1 conda-forge
gxx_linux-64 7.3.0 h553295d_3 conda-forge
harfbuzz 1.9.0 he243708_1001 conda-forge
icu 58.2 hf484d3e_1000 conda-forge
jpeg 9c h14c3975_1001 conda-forge
krb5 1.16.3 h05b26f9_1001 conda-forge
libcurl 7.64.0 h541490c_2 conda-forge
libedit 3.1.20170329 hf8c457e_1001 conda-forge
libffi 3.2.1 hf484d3e_1005 conda-forge
libgcc-ng 7.3.0 hdf63c60_0 conda-forge
libgfortran-ng 7.3.0 hdf63c60_0
libiconv 1.15 h14c3975_1004 conda-forge
libpng 1.6.36 h84994c4_1000 conda-forge
libsodium 1.0.16 h14c3975_1001 conda-forge
libssh2 1.8.0 h90d6eec_1004 conda-forge
libstdcxx-ng 7.3.0 hdf63c60_0 conda-forge
libtiff 4.0.10 h648cc4a_1001 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxml2 2.9.8 h143f9aa_1005 conda-forge
make 4.2.1 h14c3975_2004 conda-forge
ncurses 6.1 hf484d3e_1002 conda-forge
openssl 1.1.1b h14c3975_0 conda-forge
pango 1.40.14 hf0c64fd_1003 conda-forge
pcre 8.41 hf484d3e_1003 conda-forge
pixman 0.34.0 h14c3975_1003 conda-forge
pthread-stubs 0.4 h14c3975_1001 conda-forge
r-base 3.5.1 he45234b_1005 conda-forge
readline 7.0 hf8c457e_1001 conda-forge
tk 8.6.9 h84994c4_1000 conda-forge
tktable 2.10 h14c3975_0
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.9 h14c3975_1004 conda-forge
xorg-libsm 1.2.3 h4937e3b_1000 conda-forge
xorg-libx11 1.6.7 h14c3975_1000 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxdmcp 1.1.2 h14c3975_1007 conda-forge
xorg-libxext 1.3.3 h14c3975_1004 conda-forge
xorg-libxrender 0.9.10 h14c3975_1002 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.4 h14c3975_1001 conda-forge
zeromq 4.3.1 hf484d3e_1000 conda-forge
zlib 1.2.11 h14c3975_1004 conda-forge
$ R
> install.packages(devtools)
> devtools::install_deps()
> devtools::install()Just for good measure, I also opened a |
@brendanf OK. That is going to be tough to manage for some edge cases, as you are already finding out. When I want to use GitHub-only R packages in a conda environment, I create a conda recipe for it (the only requirement is that the GitHub repo has at least one tag/release), build it, and then upload it to my personal Anaconda channel. That looks something like this:
Due to the practical constraints of time (both maintainers and CI servers) and space (for tarballs on Anaconda Cloud), conda-forge builds R packages for the first patch of each minor release of R (e.g. R 3.x.1). Each successive patched version of R only makes very minimal changes, so you shouldn't notice any difference in performance.
Glad you found a workaround for your particular use case! |
|
Still an issue with r-base=3.6.0 and packages depending on zlib. Since many Bioconductor packages are likely to depend on some package interfacing with zlib, this is likely to affect many users. For example, GenomicAlignments is #20 in the Bioconductor package download rankings, and depends on Rhtslib, which fails to install because of zlib. |
|
On a side-note, do you think there is any possibility of creating a "bridge" between Conda and (CRAN/Bioconductor/Github, etc.)?
I imagine this has already been explored and ruled out or else issue wouldn't exist, but I'm curious as to whether this is something that is not likely to ever happen, or if it could be accomplished with sufficient resources?.. My current approach is attempt to use conda to control the version of R installed, and renv to manage R packages, however, compatibility issues like these limit the effectiveness of that approach.. |
@khughitt
Conda users can install the Bioconductor packages from the bioconda channel: I tested and confirmed that this works.
What did you have in mind?
As you know, managing dependencies is a hard problem. Different package managers each have their own solutions. The conda packages in the conda-forge and bioconda channels should all be interoperable. But if you install R packages (or other software) using a different method, there is no guarantee that this will work. |
|
@jdblischak That's fair -- I realize my points (and also the original zlib issue, I believe) are not at all conda-forge specific. In general, I really don't like the idea that the only solution to using conda to manage an R installation is to go "all-in" and use it to manage R packages as well. It seems like it should be possible to use conda to install a specific version of R, and then let R (or devtools/BiocManager/renv/etc.) handle the R packages themselves. With respect to a bridge, I don't really have anything specific in mind, and I'm afraid I don't know enough about conda internals to offer any meaningful insights.. One approach might be to create one or more pseudo-channels (e.g. 'r', 'bioconductor') to interface with those repositories. When a user searches for some package name with those channels enabled, conda could query those repositories behind the scene to check for relevant results. When a user then attempts to do a "conda install" for an R package, conda could then use R's I do appreciate the difficulty of managing dependencies, and perhaps this would just be too complex to implement. Regardless, I am grateful for the efforts of all of the conda (and conda-forge and bioconda) devs and packagers, and for the significant improvements to cross-platform package management and reproducibility that these have made possible. |
|
@khughitt I've decided to go "all-in" with conda, as you put it. In the cases where the package/version I want is not available, I've started packaging it myself on a personal conda channel, as suggested by @jdblischak above. |
@khughitt You don't have to go "all-in" with conda or any other package manager. You can install R with conda, APT, Homebrew, etc., and then install your packages with devtools/BiocManager/renv/etc.. This will work much of the time. But if a particular package relies on system libraries (e.g. ShortRead, rJava, etc.), then it might be more difficult to install. This is where a package manager is helpful. It will ensure that you have the necessary system libraries installed and that the new software you are installing will be able to link to them.
That exact implementation wouldn't work because But even if manually installed R packages were given more support, they would still have the same installation problems.
@brendanf That's awesome! And if you manage to create a working conda recipe for an R package released on CRAN (or Bioconductor) that isn't available on conda-forge (or bioconda), please consider submitting it. |
|
@jdblischak Thanks for taking the time to respond and for clarifying with regard to conda's implementation. Based on this discussion and my experience working with other methods for capturing the R environment (really just singularity / docker, if you include the version of R itself), it seems like the best approach is probably to just do the same and start building recipes for packages that aren't already on the main channels. It might take a little bit of time, but most of the the packages are simple enough that they shouldn't be too difficult to port. |
Second that and still an issue with r-base=3.6.1 and getting zlib.h not found error while installing Rhtslib. I do not see r-essential package for R 3.6.1 in conda-forge. No rush but any ETA on that? For those looking to install zlib dependent R packages, I was able to install Rhtslib by editing Makefiles and specifying CPPFLAGS and LDFLAGS. via Bioconductor/Rhtslib#9 (comment) cd ~/Downloads && \
wget wget https://bioconductor.org/packages/release/bioc/src/contrib/Rhtslib_1.16.1.tar.gz && \
tar xvzf Rhtslib_1.16.1.tar.gz && \
cd Rhtslib/src/htslib-1.7
cd ~/Downloads/Rhtslib && \
R CMD INSTALL .Done! conda infosessionInfo() |
|
We (bioconda) are likely to start building the bioconductor packages for R 3.6 early this week. |
I'm curious why there is no interest in ensuring that R installed from conda is properly configured to install compiled packages, including finding system libraries in their standard (according to conda) locations. It's one of the core functions of the software, and it would be nice if the software packaged by conda-forge was functional. In the original post for this issue, I pointed out that variables like Of course I understand that, if I try to install R packages directly, I would have to be sure to install the proper system libraries also; ideally from As I said in a previous comment, this issue has led me to start managing all my R packages through conda, building private conda packages as necessary. This works, but it is much slower and more painful than installing them from within R using, e.g., As @khughitt said above, I do appreciate all the work that goes in to maintaining conda-forge, and I am grateful for that -- after all, when I realized that I had to choose between |
Because it's not really possible? I could go into the technical details in excruciating detail if you wish. It involves intimate knowledge of how the completely different OSes load shared libraries and more importantly, how they decided not to. |
|
@mingwandroid Ok, then it is at least clear to me that something along the lines of "something that was done intentionally to streamline the build process for R packages" is the case. Thanks for the reply. @khughitt Thanks, I'll check it out. |
|
For my own clarification, could you be explicit in what you mean by system libraries? Do you mean exactly the shared library files in /usr/lib{64} Thing is everyone has their own definition of that round here and it makes communication difficult (and the R people have their own formalisation of that too). The only ways I can think of to make this work (I'd like to believe me!) would cause more trouble than they'd solve. It'd involve redirecting /usr to your conda env inside our compilers |
|
@mingwandroid Sorry for the confusion. When I said "system libraries" in my previous post, I was using a very R-centric definition: "any shared library not packaged inside an R package". I understand that it's not reasonable for |
@brendanf I agree this is a reasonable use case. IIRC this used to be possible, as you also recall. I am pretty sure the change occurred during the Migration to conda-build 3 and new compilers from Anaconda. This solved a lot of problems, but apparently this was a negative side effect. And I also trust @mingwandroid when he says this isn't easily possible. And for future testing, I tested the current behavior below: |
You save my life! |
Issue:
Installing the Bioconductor package ShortRead from inside R (not from conda) fails with the error:
This has been previously reported here, here and here, without any truly satisfactory resolution, except for the observation that the difference was introduced with
zlib 1.2.11.I tried this:
but none of the results seemed to explain any possible difference.
I then tried
This yielded, unsurprisingly, a LOT of results. Using GUI diff tools and grep guesswork, I finally found this:
I am a bit out of my depth here, and maybe I'm on the wrong track entirely, but I can see that the version of
r-baseinstalled withzlib=1.2.11references directories which are not in my conda installation, and don't exist on my machine.Here are the environments:
Test1 with
zlib=1.2.11(not working)test2withzlib=1.2.8(working)And my conda info:
(edit to fix details formatting)
The text was updated successfully, but these errors were encountered: