Fix HTTP(S) git repo installation #603

niheaven · 2021-04-22T16:03:31Z

Fix Multiple private git repo dependencies got error #597

Now for git repo, remotes use git archive to get DESCRIPTION file and package name (remote_package_name.git2r_remote and remote_package_name.xgit_remote). But for HTTP(S) protocol, git archive just say fatal: operation not supported by protocol and cannot work.

git archive --remote https://github.com/r-lib/httr.git
# fatal: operation not supported by protocol

This lead to some problem when users use HTTP(S) protocol with any git server, e.g. GitHub.com or private hosted server (Gogs, Gitea, etc.). Here is a example:

Just add following lines in devtools' DESCRIPTION file:

Remotes:
    git::https://github.com/r-hub/rversions.git,
    git::https://github.com/r-lib/httr.git

In this way we install rversions and httr from GitHub.com using HTTPS protocol.
2. Remove these two packages from local lib
3. Check for dependencies

remotes::dev_package_deps("E:/Projects/Repos/devtools")
#> Needs update -----------------------------
#>  package   installed available    is_cran remote
#>  NA        NA        e55b558e6... FALSE   Git   
#>  NA        NA        21ff69f21... FALSE   Git   
#>  httr      NA        1.4.2        TRUE    CRAN  
#>  rversions NA        2.0.2        TRUE    CRAN

Install missing deps for this hacked devtools

remotes::install_deps('E:/Projects/Repos/devtools')
#> NA        (NA -> e55b558e6...) [Git]
#> NA        (NA -> 21ff69f21...) [Git]
#> httr      (NA -> 1.4.2.9000  ) [CRAN]
#> rversions (NA -> 2.0.2       ) [CRAN]
#> Downloading git repo https://github.com/r-hub/rversions.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpUNqzQG\file37384de81e63/DESCRIPTION' (1.7s)
#> -  preparing 'rversions': (354ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (426ms)
#> -  checking for empty or unneeded directories
#> -  building 'rversions_2.0.2.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-lib/httr.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpUNqzQG\file3738f2e5671/DESCRIPTION' (1.5s)
#> -  preparing 'httr': (1s)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (1.3s)
#> -  checking for empty or unneeded directories
#> -  building 'httr_1.4.2.9000.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Installing 2 packages: httr, rversions
#> Installing packages into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> package 'httr' successfully unpacked and MD5 sums checked
#> package 'rversions' successfully unpacked and MD5 sums checked
#> 
#> The downloaded binary packages are in
#>  C:\Users\zhangxn\AppData\Local\Temp\RtmpUNqzQG\downloaded_packages
#> Downloading git repo https://github.com/r-hub/rversions.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpUNqzQG\file37383b945679/DESCRIPTION' (1.6s)
#> -  preparing 'rversions': (345ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (478ms)
#> -  checking for empty or unneeded directories
#> -  building 'rversions_2.0.2.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-lib/httr.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpUNqzQG\file37385268658b/DESCRIPTION' (1.5s)
#> -  preparing 'httr': (926ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (956ms)
#> -  checking for empty or unneeded directories
#> -  building 'httr_1.4.2.9000.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)

You can see above that remotes installs rversions and httr three times: two from GitHub.com and one from CRAN.
Now check missing dependencies again:

remotes::dev_package_deps("E:/Projects/Repos/devtools")
#> Needs update -----------------------------
#>  package installed available    is_cran remote
#>  NA      NA        e55b558e6... FALSE   Git   
#>  NA      NA        21ff69f21... FALSE   Git

Oh no they think we still don't have rersions and httr!

Try to install missing deps again:

remotes::install_deps("E:/Projects/Repos/devtools")
#> NA (NA -> e55b558e6...) [Git]
#> NA (NA -> 21ff69f21...) [Git]
#> Downloading git repo https://github.com/r-hub/rversions.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpAhwdeL\file652c3c1a2086/DESCRIPTION' (1.3s)
#> -  preparing 'rversions':
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (412ms)
#> -  checking for empty or unneeded directories
#> -  building 'rversions_2.0.2.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-lib/httr.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpAhwdeL\file652c12dde59/DESCRIPTION' (1.8s)
#> -  preparing 'httr': (882ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (881ms)
#> -  checking for empty or unneeded directories
#> -  building 'httr_1.4.2.9000.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-hub/rversions.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpAhwdeL\file652c64196293/DESCRIPTION' (1.5s)
#> -  preparing 'rversions':
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (399ms)
#> -  checking for empty or unneeded directories
#> -  building 'rversions_2.0.2.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-lib/httr.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpAhwdeL\file652cb492e21/DESCRIPTION' (1.9s)
#> -  preparing 'httr': (860ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (871ms)
#> -  checking for empty or unneeded directories
#> -  building 'httr_1.4.2.9000.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)

Okay, this time, we install these packages, TWICE!

This error is introduced from NA_character_ return of remote_package_name, then parse_one_extra, then extra_deps, then dev_package_deps, and finally install_deps.

When remote_package_name could give correct package name, there will be no NA packages in dev_package_deps, and upgradable_packages will produce correct result:

remotes::dev_package_deps("E:/Projects/Repos/devtools")
#> Needs update -----------------------------
#>  package   installed available    is_cran remote
#>  rversions NA        e55b558e6... FALSE   Git   
#>  httr      NA        21ff69f21... FALSE   Git
remotes::install_deps("E:/Projects/Repos/devtools")
#> rversions (NA -> e55b558e6...) [Git]
#> httr      (NA -> 21ff69f21...) [Git]
#> Downloading git repo https://github.com/r-hub/rversions.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpWaJcAL\file17ac6f23707d/DESCRIPTION' (1.8s)
#> -  preparing 'rversions':
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (388ms)
#> -  checking for empty or unneeded directories
#> -  building 'rversions_2.0.2.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Downloading git repo https://github.com/r-lib/httr.git
#> √  checking for file 'C:\Users\zhangxn\AppData\Local\Temp\RtmpWaJcAL\file17ac3ef9477c/DESCRIPTION' (1.4s)
#> -  preparing 'httr': (970ms)
#> √  checking DESCRIPTION meta-information
#> -  checking for LF line-endings in source and make files and shell scripts (1.1s)
#> -  checking for empty or unneeded directories
#> -  building 'httr_1.4.2.9000.tar.gz'
#> 
#> Installing package into 'E:/Documents/R/win-library/4.0'
#> (as 'lib' is unspecified)
#> Skipping install of 'rversions' from a git2r remote, the SHA1 (e55b558e) has not changed since last install.
#>   Use `force = TRUE` to force installation
#> Skipping install of 'httr' from a git2r remote, the SHA1 (21ff69f2) has not changed since last install.
#>   Use `force = TRUE` to force installation
remotes::dev_package_deps("E:/Projects/Repos/devtools")

Main fixes are in R/install-git.R L82-L83 and L148-L152, other diff is made by styler. I've also added some necessary tests and modified vignette.

Signed-off-by: Hsiao-nan Cheung <niheaven@gmail.com>

niheaven · 2021-04-22T17:45:47Z

devtools::test(pkg = "E:/Projects/Repos/remotes", filter = "install-github")
#> i<U+00A0>Loading remotes
#> i<U+00A0>Testing remotes
#> √ |  OK F W S | Context
#> √ |  26       | Install from GitHub [170.5 s]
#> 
#> == Results =====================================================================
#> Duration: 170.6 s
#> 
#> [ FAIL 0 | WARN 0 | SKIP 0 | PASS 26 ]

Session info:

devtools::session_info()
#> - Session info ---------------------------------------------------------------
#>  setting  value                         
#>  version  R version 4.0.5 (2021-03-31)  
#>  os       Windows 10 x64                
#>  system   x86_64, mingw32               
#>  ui       RTerm                         
#>  language (EN)                          
#>  collate  Chinese (Simplified)_China.936
#>  ctype    Chinese (Simplified)_China.936
#>  tz       Asia/Taipei                   
#>  date     2021-04-23                    
#> 
#> - Packages -------------------------------------------------------------------
#>  package     * version     date       lib source        
#>  backports     1.2.1       2020-12-09 [1] CRAN (R 4.0.3)
#>  cachem        1.0.4.9000  2021-03-13 [1] local         
#>  callr         3.7.0.9000  2021-04-20 [1] local         
#>  cli           2.4.0.9000  2021-04-05 [1] local         
#>  crayon        1.4.1.9000  2021-03-13 [1] local         
#>  desc          1.3.0.9000  2021-03-13 [1] local         
#>  devtools      2.4.0.9000  2021-04-19 [1] local         
#>  digest        0.6.27      2020-10-24 [1] CRAN (R 4.0.3)
#>  ellipsis      0.3.1.9000  2021-03-20 [1] local         
#>  evaluate      0.14.1      2021-03-13 [1] local         
#>  fansi         0.4.2       2021-01-15 [1] CRAN (R 4.0.3)
#>  fastmap       1.1.0       2021-01-25 [1] CRAN (R 4.0.3)
#>  fs            1.5.0.9000  2021-03-23 [1] local         
#>  glue          1.4.2       2020-08-27 [1] CRAN (R 4.0.2)
#>  highr         0.9         2021-04-16 [1] CRAN (R 4.0.5)
#>  htmltools     0.5.1.9002  2021-04-21 [1] local         
#>  knitr         1.32        2021-04-14 [1] CRAN (R 4.0.5)
#>  lifecycle     1.0.0.9000  2021-03-23 [1] local         
#>  magrittr      2.0.1.9000  2021-03-25 [1] local         
#>  memoise       2.0.0.9000  2021-03-16 [1] local         
#>  pillar        1.6.0.9001  2021-04-18 [1] local         
#>  pkgbuild      1.2.0       2020-12-15 [1] CRAN (R 4.0.3)
#>  pkgconfig     2.0.3       2019-09-22 [1] CRAN (R 4.0.2)
#>  pkgload       1.2.1.9000  2021-04-06 [1] local         
#>  prettyunits   1.1.1.9000  2021-03-19 [1] local         
#>  processx      3.5.1.9000  2021-04-11 [1] local         
#>  ps            1.6.0.9000  2021-03-13 [1] local         
#>  purrr         0.3.4.9000  2021-03-23 [1] local         
#>  R6            2.5.0.9000  2021-03-13 [1] local         
#>  remotes       2.3.0.9000  2021-04-22 [1] local         
#>  reprex        1.0.0.9002  2021-03-15 [1] local         
#>  rlang         0.4.10.9002 2021-04-21 [1] local         
#>  rmarkdown     2.7.4       2021-03-24 [1] local         
#>  rprojroot     2.0.2.9000  2021-03-18 [1] local         
#>  sessioninfo   1.1.1.9000  2021-03-14 [1] local         
#>  stringi       1.5.3       2020-09-09 [1] CRAN (R 4.0.2)
#>  stringr       1.4.0.9000  2021-03-20 [1] local         
#>  styler        1.4.1.9000  2021-04-16 [1] local         
#>  testthat      3.0.2.9000  2021-03-16 [1] local         
#>  tibble        3.1.1.9000  2021-04-18 [1] local         
#>  usethis       2.0.1.9000  2021-04-19 [1] local         
#>  utf8          1.2.1       2021-03-12 [1] CRAN (R 4.0.4)
#>  vctrs         0.3.7.9000  2021-03-30 [1] local         
#>  withr         2.4.2       2021-04-16 [1] local         
#>  xfun          0.22        2021-03-11 [1] CRAN (R 4.0.4)
#>  yaml          2.2.1       2020-02-01 [1] CRAN (R 4.0.2)
#> 
#> [1] E:/Documents/R/win-library/4.0
#> [2] D:/Scoop/apps/r/4.0.5/library

jimhester · 2021-06-02T17:02:31Z

niheaven added 3 commits April 22, 2021 20:36

Fix HTTP(S) git repo installation

27c1965

Signed-off-by: Hsiao-nan Cheung <niheaven@gmail.com>

Fix 'install_git' help

0a70f56

Signed-off-by: Hsiao-nan Cheung <niheaven@gmail.com>

Update 'NEWS.md'

e99cad0

Signed-off-by: Hsiao-nan Cheung <niheaven@gmail.com>

jimhester merged commit a44d96b into r-lib:master Jun 2, 2021

jimhester mentioned this pull request Jun 15, 2021

install_git() Assumes github-style urls and credential-less web server access #625

Closed

dakep mentioned this pull request Jun 20, 2021

install_git() fails to install package from AzureDevOps with command status 128 #621

Closed

niheaven mentioned this pull request Jun 23, 2021

Fix regex that broke git@xxx in install-git #630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix HTTP(S) git repo installation #603

Fix HTTP(S) git repo installation #603

niheaven commented Apr 22, 2021

niheaven commented Apr 22, 2021

jimhester commented Jun 2, 2021

Fix HTTP(S) git repo installation #603

Fix HTTP(S) git repo installation #603

Conversation

niheaven commented Apr 22, 2021

niheaven commented Apr 22, 2021

jimhester commented Jun 2, 2021