Error when calling insertPackage: permission issue ?

[kabarigou]

Hello,

Thanks for this great package.

After forking the drat repository, cloning the folder locally, when I try to insert a binary windows R package into drat, I get the following error (under Windows):

> drat::insertPackage(file="StanMoMo-win.zip", repodir="./drat")
Error in read.dcf(path, "Built") : cannot open the connection
In addition: Warning message:
In read.dcf(path, "Built") :
  cannot open compressed file 'C:\Users\KARIMB~1\AppData\Local\Temp\RtmpiUbONP/StanMoMo-win.zip/DESCRIPTION', probable reason 'No such file or directory'

For information, my working directory and temporary file are

> getwd()
[1] "C:/Users/Karim Barigou/Dropbox/Package StanMoMo"
> tempfile()
[1] "C:\\Users\\KARIMB~1\\AppData\\Local\\Temp\\RtmpiUbONP\\file3fc017d6e3f"

I don't know how I can solve this. Is it a permission issue ? I tried to run R as administrator but I still get the same error.
It seems that the function is searching for the file in the temporary directory not in the working directory.
I tried under Mac and got the same kind of error:

> drat::insertPackage(file="StanMoMo-win.zip", repodir="./drat")
Error in read.dcf(path, "Built") : cannot open the connection
In addition: Warning message:
In read.dcf(path, "Built") :
  cannot open compressed file '/var/folders/7c/5xb6gq6d739b_rs0hm2d2_w00000gn/T//RtmphkVdLN/StanMoMo-win.zip/DESCRIPTION', probable reason 'No such file or directory'

For information:

• I use the drat version 0.1.8.1 installed via

remotes::install_github("eddelbuettel/drat")

• I have forked the drat-base/drat repository from https://github.com/drat-base/drat

Any help is much appreciated,
Thank you very much in advance.

[eddelbuettel]

Hi there and glad you like it. (We're in the middle of a small update switching to also enabling serving GitHub Pages from the docs/ folder so documentation and code may be a little out of sync. Please edit your question and add a) which drat version you use and where you installed it from, and b) how you bootstrapped you drat repo: fork of which repo, or dratInit() or ...)

That looks like a permissions issue indeed, and as I longer use windows I am not in the best position to offer tips. I presume
paths like C:\Users\KARIMB~1\AppData\Local\Temp\RtmpO09aLu are what you see when you call tempfile() in R on your machine?

If that is your normal temp.file, why would it refuse to let you open and unpack the package source so that drat can read the DESCRIPTION file. That is a little odd.

[kabarigou]

Thank you for the quick reply! I have updated my question with some additional information.

[eddelbuettel]

It looks like a logic error I may have made but I am not seeing it. Both for Windows and on the mac, we construct a path ... which then does not exist.

The simplest answer may have to do with 0.1.8.1 and the "in-progress" changes I made (and that @RomanHornung and I are working on some documentation for) which has to do with using docs/ rather than gh-pages -- so please try options(dratBranch="docs") (and/or set location="docs/" when you call insertPackage`). This may just be it.

Otherwise I'd be stumped. Can you maybe help debug a local copy of drat at your end (the old way via print is what I often do, or browser() or other R tricks to debug?

[kabarigou]

Thank you for the quick reply!

Problem solved! In fact, I produced the binaries via Github Actions. This produces zip files containing the binary windows zip file and the binary Mac gz file. What I was trying to do was inserting the zip file containg the binary, not the binary itself.

To be more explicit, Github Actions produces

• windows-latest-release.zip containing StanMoMo_1.0.0.zip
• macOS-latest-release.zip containing StanMoMo_1.0.0.gz

I was calling the function drat::insertPackage on windows-latest-release.zip rather than StanMoMo_1.0.0.zip...

I have just called drat::insertPackage on StanMoMo_1.0.0.zip and StanMoMo_1.0.0.gz and the binaries were included perfectly, as expected.

Maybe for future development of the package, it could be interesting to include drat in the Github Actions Workflow. At the moment, Github Actions allows the user to build binaries every time there is a Github push. Is it possible to update the binaries in the drat folder every time there is a push in a given repository ? I don't know.

[eddelbuettel]

Yay! Very nice.

In favour of GitHub Actions too -- I use them quite a bit, I personally do not bother so much with artifact creation and storage but other people. As ... this seems to be your itch to scratch, do you want to maybe start by putting some bullet points together and what is there, what is missing, how we could get there etc pp? Maybe it is "just" a need for more documentation and walkthrough (as I am currently writing one along with @RomanHornung) ?

[kabarigou]

I would be happy to help improve the documentation. Below you can find a first draft 😉

Hereafter are the main steps to build binary packages via Github Actions for Windows and Mac and insert them into a drat repository. For the rest, I only assume that the repository package is a standard github repository.

1. If this is the first time you use the CI Github Workflow, one can call

usethis::use_github_action("check-release")

to initialize Continuous Integration in the Github repository. From now on, the R package will be checked for every push.
2. Create the following yaml file in .github/workflows/. This will build Windows, Mac and Ubuntu binaries of your package for every push and pull request (the file can be simplified if Ubuntu is not required or if the binaries should not be built for every push).

on:
  push:
    branches:
      - main
      - master
  pull_request:
    branches:
      - main
      - master

name: Package builder

jobs:
  binary:
    runs-on: ${{ matrix.config.os }}

    name: ${{ matrix.config.os }} (${{ matrix.config.r }})

    strategy:
      fail-fast: false
      matrix:
        config:
          - {os: macOS-latest,   r: 'release'}
          - {os: windows-latest, r: 'release'}
          - {os: ubuntu-latest, r: 'release'}


    env:
      R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
      RSPM: ${{ matrix.config.rspm }}
      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}

    steps:

      - uses: actions/checkout@v2
        with:
          repository: ${{ github.event.inputs.package_repo }}
          ref: ${{ github.event.inputs.package_branch }}

      - uses: r-lib/actions/setup-r@v1
        with:
          r-version: ${{ matrix.config.r }}
          http-user-agent: ${{ matrix.config.http-user-agent }}

      - uses: r-lib/actions/setup-pandoc@v1

      - name: Query dependencies
        run: |
          install.packages('remotes')
          saveRDS(remotes::dev_package_deps(dependencies = TRUE), ".github/depends.Rds", version = 2)
          writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), ".github/R-version")
        shell: Rscript {0}

      - name: Cache R packages
        if: runner.os != 'Windows'
        uses: actions/cache@v2
        with:
          path: ${{ env.R_LIBS_USER }}
          key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }}
          restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-

      - name: Install system dependencies
        if: runner.os == 'Linux'
        run: |
          while read -r cmd
          do
            eval sudo $cmd
          done < <(Rscript -e 'writeLines(remotes::system_requirements("ubuntu", "16.04"))')
      - name: Install dependencies
        run: |
          remotes::install_deps(dependencies = TRUE)
          remotes::install_cran(c("rcmdcheck", "caret", "gbm", "randomForest", "e1071"))
        shell: Rscript {0}

      - name: Session info
        run: |
          options(width = 100)
          pkgs <- installed.packages()[, "Package"]
          sessioninfo::session_info(pkgs, include_base = TRUE)
        shell: Rscript {0}

      - name: Build source package
        if: matrix.config.os == 'ubuntu-latest'
        run: |
          src <- pkgbuild::build(".", dest_path = tempdir(), vignettes = FALSE, manual = FALSE)
          dir.create("build")
          file.copy(src, "build")
        shell: Rscript {0}

      - name: Build binary
        if: matrix.config.os != 'ubuntu-latest'
        env:
          _R_CHECK_CRAN_INCOMING_: false
        run: |
          bin <- pkgbuild::build(".", dest_path = tempdir(), vignettes = FALSE, manual = FALSE, binary = TRUE)
          dir.create("build_bin")
          file.copy(c(bin), "build_bin")
        shell: Rscript {0}

      - name: Upload source package
        if: matrix.config.os == 'ubuntu-latest'
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.config.os }}-${{ matrix.config.r }}
          path: build/

      - name: Upload binary packages
        if: matrix.config.os != 'ubuntu-latest'
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.config.os }}-${{ matrix.config.r }}
          path: build_bin/

3. If the construction of binaries is successful, you should get three artifacts which look like this:
   
4. Download the zip artifacts and make sure that you unzip the zip artifacts after download.
5. After unzipping, you should normally get two binaries of the form: nameofpackage_VersionNumber.zip (for Windows) and nameofpackage_VersionNumber.tgz (for Mac)
6. Now, you can proceed by following the standard vignette of drat. For instance, assuming that your drat repository is in your working directory with your binaries, you can simply call:

drat::insertPackage(file="nameofpackage_VersionNumber.zip", repodir="./drat")
drat::insertPackage(file="nameofpackage_VersionNumber.tgz", repodir="./drat")

7. Your drat repository should be updated. Now, you just need to commit and push, that's it!

Hope it helps!

[eddelbuettel]

It's good. It's a bit heavy on the whole usethis / r-lib nexus I do not use myself which makes it a little hard for me to maintain or say anything. That said, I am sure some people may find this helpful. Let me ponder this for a bit...

[kabarigou]

To continue my "drat experience", I was able to download the binary package by calling

install.packages("nameofpackage", repos="https://username.github.io/drat",type="binary")

Important to note that type="binary" is necessary. Otherwise, there will be some warning and package is not installed.
Also important to note that the dependencies are not installed by default and the argument dependencies=TRUE does not work. Instead, one should then call:

devtools::install_deps("nameofpackage")

to install dependencies.
I think it is worth including these two lines of code somewhere in the vignette 'Drat Basics for Package Users: Installing or Updating Packages'

Now, that everything is set and done, I really appreciate the package but I think that at the present, the documentation is a bit diluted. It took me some time and trials/errors to find the information I wanted. The post of Roman greatly helped in that respect. I think it is definitely worth writing a quick start guide along the lines of Roman but probably much shorter like:

1. Fork this repo.
2. Enable Github pages through Settings -> gh-pages -> docs (as you already showed)
3. Git clone the forked drat repo via Git commands or e.g. Rstudio (New Project -> Version Control -> Git -> Enter https://username.github.io/drat in Repository URL).
4. Install drat via remotes::install_github("eddelbuettel/drat")
5. Assuming that your drat repo and binaries are in your working directory, call

options(dratBranch="docs")
drat::insertPackage(file="nameofpackage.zip", repodir="./drat") #for Windows binary
drat::insertPackage(file="nameofpackage.tgz", repodir="./drat") #for Mac binary

6. The drat repository should be updated. Commit and push the changes.
7. Now, all your users can install the binaries of your package via

install.packages("nameofpackage", repos="https://username.github.io/drat",type="binary")
devtools::install_deps("nameofpackage")

Concerning Github Actions and drat, this is indeed maybe outside the scope of the drat package. However, I think that a vignette discussing how to check and build R binary packages via CI and share the binaries via the drat package can attract more R users to the drat package, in my opinion.

[eddelbuettel]

One at a time:

Important to note that type="binary" is necessary.

Yes, known and documented, applies for the OSs that have binaries.

the dependencies are not installed by default and the argument dependencies=TRUE does not work.

I do not think this is generally true. Recall we use the standard R command install.packages() here. It installs dependencies. If you spot something "odd", can you analyse it and may suggest a ticket? I think there may be an (also known?) issue of mixing source and binaries but I not the persion to ask as I am on source-only OS.

the documentation is a bit diluted.

Over time, kind users like yourself came forward and either suggested expanded documentation, offered PRs for documentation, or even contributed some. That naturally expands and "dilutes" and there isn't really a way around it. What seems best for you may drive someone else to suggested the (near-)opposite. Documentation is hard and I have found in 20+ years of offering open source software that there may not be one size that fits all.

The post of Roman greatly helped

Sit tight. We have a better version coming. In-progress preview is here.

GH Actions

Can be argued both ways. Adding it here "dilutes" :) May still be of help. I am still working with @RomanHornung on the other one.

[kabarigou]

I do not think this is generally true. Recall we use the standard R command install.packages() here. It installs dependencies. If you spot something "odd", can you analyse it and may suggest a ticket? I think there may be an (also known?) issue of mixing source and binaries but I not the persion to ask as I am on source-only OS.

At least, in my case, the dependencies of the binary are not automatically installed:

> install.packages("StanMoMo", repos = "https://kabarigou.github.io/drat", type = "binary",dependencies = TRUE)
Installing package into ‘C:/Users/Karim/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)
Warning in install.packages :
  dependencies ‘Rcpp’, ‘fanplot’ are not available
trying URL 'https://kabarigou.github.io/drat/bin/windows/contrib/4.0/StanMoMo_1.0.0.zip'
Content type 'application/zip' length 3037530 bytes (2.9 MB)
downloaded 2.9 MB

package ‘StanMoMo’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
	C:\Users\Karim Barigou\AppData\Local\Temp\RtmpQj1IhN\downloaded_packages

To install the dependencies, I need to call:

> devtools::install_deps("StanMoMo")
Installing 2 packages: Rcpp, fanplot
Installing packages into ‘C:/Users/Karim Barigou/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)
trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/Rcpp_1.0.6.zip'
Content type 'application/zip' length 3253408 bytes (3.1 MB)
downloaded 3.1 MB

trying URL 'https://cran.rstudio.com/bin/windows/contrib/4.0/fanplot_3.4.2.zip'
Content type 'application/zip' length 1913608 bytes (1.8 MB)
downloaded 1.8 MB

package ‘Rcpp’ successfully unpacked and MD5 sums checked
package ‘fanplot’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
	C:\Users\Karim Barigou\AppData\Local\Temp\RtmpQj1IhN\downloaded_packages

From my understanding, this is a known issue. Dependencies are installed if the package is from CRAN but if the URL is outside CRAN, the argument dependencies is not used. That is a very minor issue. I don't mind writing an extra line of code.

Over time, kind users like yourself came forward and either suggested expanded documentation, offered PRs for documentation, or even contributed some. That naturally expands and "dilutes" and there isn't really a way around it. What seems best for you may drive someone else to suggested the (near-)opposite. Documentation is hard and I have found in 20+ years of offering open source software that there may not be one size that fits all.

I totally agree. Hard to propose a documentation that fits everyone. Some want short and to-the-point documentation and others extended and step-by-step documentation.

Sit tight. We have a better version coming. In-progress preview is here.

Nice vignette !

[eddelbuettel]

install.packages("StanMoMo", repos = "https://kabarigou.github.io/drat", type = "binary",dependencies = TRUE)

You are replacing repos with a single archive not containing the other packages. Unsurprisingly, they are not found.

See the drat documentation on adding to an existing repos selection, or, if you must use install.packages do at least something like

install.packages("StanMoMo", repos = c("https://cloud.r-project.org",             # all of R
                                       "https://kabarigou.github.io/drat"),       # plus drat
                 type = "binary",dependencies = TRUE)

Should work much better and have the magic of R plus the enhancement you worked hard to add.

[kabarigou]

Perfect, that was exactly what I was looking for! It works like a charm ! Thanks a lot !

[FelixErnst]

Hi @kabarigou

the last issue was about setting the repos to allow dependencies to be resolved correctly. For this purpose drat::addRepo exists, so that you don't have to set your repos "manually".

So maybe you can help with the following question:
At which point of the documentation should a reference to drat::addRepo be added, so that the pitfall you encountered is made clearly visible to a user?

I think this might help everyone get it right from the beginning.

[kabarigou]

Hi @FelixErnst

Concerning the last point, it was not clear for me in the documentation 'Drat Basics for Package Users: Installing or Updating Packages' what the users need to call to install my binary package StanMoMo from my drat repository and make sure all the dependencies are installed as well.

As explained by @eddelbuettel , one solution is

install.packages("StanMoMo", repos = c("https://cloud.r-project.org",             # all of R
                                       "https://kabarigou.github.io/drat"),       # plus drat
                 type = "binary")

Now, I have just understood that another solution is

drat::addRepo("kabarigou")
install.packages("StanMoMo",type = "binary")

Maybe providing an example of the two ways to install a package from a drat repository (with and without drat) can make the documentation more clear.

[eddelbuettel]

@kabarigou Please look into addRepo. @FelixErnst was just politely explaining to you what it does. You should have used it as from a single repo you cannot resolve dependencies beyond that repo and R, as the drat docs note, is and always has been capable of using multiple repos (withness CRAN and BioC being "on" by default for many).

[kabarigou]

Thank you very much for all the explanations in this long issue. All this discussion helped me better understand the drat functionalities and how the install.packages handles repositories and dependencies. 😉

[bryanhanson]

For interested parties, we wanted to do something similar, but to also automatically deploy (source files) to a different repo. Described in this blog post.