[R] arrow violates the CRAN policy on downloading binary software

[rafapereirabr]

Describe the bug, including details regarding any error messages, version, and platform.

Hi all. Several package developers who maintain packages that depend on arrow received this email below from CRAN:

Package 'arrow' violates the CRAN policy on downloading binary software
and continued to do so in multiple re-submissions, each taking hours of
CRAN time. The CRAN team is not prepared to spend any more time on it,
so it will be removed on Feb 9.

That will make the packages requiring it (directly or indirectly),

APCalign ClickHouseHTTP DominoDataR IGoRRR MolgenisArmadillo SQL brclimr
censobr diffdfs disk.frame ebirdst foundry gbifdb geslaR nhdplusTools
parqr parquetize radiant.data receptiviti sfarrow starvz strand TKCat
elfgen gwavr radiant radiant.basics radiant.design radiant.model
radiant.multivariate

uninstallable, (They already show failures for their fedora-clang check
results.) To retain your package on CRAN we need an update not depending
on arrow.

arrow is a game changer in R, so I'm opening this issue hopping the developer community can find a solution to this. Thank you for a fantastic arrow package in R.

Component(s)

R

[thisisnic]

Thanks for letting us know about this @rafapereirabr. Please bear with us for the moment; we are working to get this resolved before their deadline. We’ll let you know as soon as we have an update.

[cboettig]

Thanks to the arrow team for dealing with this.

You probably know all this already but fwiw, CRAN policy has always allowed binary installs "with permission" (from CRAN) and continues to host packages that install software binaries.

If the sources are too large, it is acceptable to download them as part of installation, but do ensure that the download is of a fixed version rather than the latest. Only as a last resort and with the agreement of the CRAN team should a package download pre-compiled software. (See also Using Rust.)

[gilbertocamara]

@rafapereirabr The torch package has a similar problem, which has been solved in a better way. It downloads the torch and lantern libraries in run-time, thus avoiding problems with CRAN.

[rafapereirabr]

Thanks for sharing this info, @gilbertocamara . I hope the @apache team will find a similar solution to this issue.

[arnaud-feldmann]

Hi,

I wish you a lot of strength @thisisnic . Your package is awesome, it has become a must-have for R. I use it everyday. Thank you for your precious work.

[thisisnic]

Thanks @arnaud-feldmann, though the arrow R package is a team effort and definitely not just me! Things are progressing anyway - we'll update the community in due course.

[thisisnic]

Hi folks,

We’re still working to respond to CRAN’s evolving requirements on external C++ libraries, and while we believe we know what we need to do to meet the new requirements, it’s not clear that we can resolve everything before February 9.

As an interim measure, we are starting to submit PRs to packages that import arrow to make sure they have the updates they need to stay on CRAN while arrow is removed. It won’t be possible for us to get to all of the packages that rely on arrow, but if we haven’t gotten to yours, feel free to ping us here and we can make suggestions and offer code review.

For packages that Suggest arrow, it should be sufficient to add Additional_repositories: https://p3m.dev/cran/2024-02-02 to your DESCRIPTION to indicate that arrow can be installed from the snapshot in the Posit Package Manager repo (package version 14.0.0.2). Alternatively, you can instead get the latest pre-release version of arrow by instead using Additional_repositories: https://apache.r-universe.dev or Additional_repositories: https://nightlies.apache.org/arrow/r.

We are committed to doing what it takes to maintaining arrow on CRAN. If arrow gets removed this week, we will work to resubmit it as soon as possible.

Nic and the arrow team

[mrcaseb]

Are there any updates regarding the Feb 9 deadline? Arrow is obviously not archived yet, so has CRAN postponed the deadline?

[thisisnic]

We've submitted another version of the R package and also emailed CRAN to discuss some details, but have not heard back from them yet. The new version of the R package is currently pending manual inspection. We haven't had any further interaction with CRAN regarding the deadline.

[mrcaseb]

We've submitted another version of the R package and also emailed CRAN to discuss some details, but have not heard back from them yet. The new version of the R package is currently pending manual inspection. We haven't had any further interaction with CRAN regarding the deadline.

Thank you for the update @thisisnic! And thank you for working through this!

[eddelbuettel]

@thisisnic Just a note / guess / question from afar (while having a package that "Suggests: arrow" on hold at CRAN in 'inspect'): the issues seen at the CRAN results page also bite. Does the pending 14.0.2.1 overcome those clang issues?

[thisisnic]

@eddelbuettel Yeah, thanks for pointing that out, but we do keep an eye on that stuff as it's pretty essential.

[eddelbuettel]

I am sure you do @thisisnic -- and thanks for looking after it.

Any idea if/when the clang impasse at CRAN may get resolved?

(FWIW the Vienna machine had an impasse where it didn't build Rcpp either, which was resolved AFAIK by rolling back at their end. I have less insight into the Fedora issue. It all looks a little like an upstream clang problem?)

[assignUser]

It's resolved on our end, as mentioned above we are pending manual inspection and that hasn't changed. You'd have to ask CRAN 🤷

[eddelbuettel]

Sounds good, and thanks! I am in the same boat but behind you 'waiting on your issue to resolved'. It is always a little hard to tell from the outside, but it did look like we are currently waiting only on them so thanks for confirming, and let's hope they get their clang17 setup sorted out.

[thisisnic]

Right, that makes sense! I think we ended up with a slightly different fix to you - we needed to update one of our C++ headers to include something we weren't including before and that resolved the issue, so we didn't end up asking them to change anything about their setup.

[thisisnic]

@eddelbuettel In case you're curious: #40010

[eddelbuettel]

You love to see it:

Congrats for seeing it through.

[thisisnic]

Dammit Dirk, I wanted to post the update 😂

Yep, thanks for bearing with us folks, version 14.0.2.1 is now on CRAN, with the previous issues resolved!

[rafapereirabr]

Thanks @thisisnic and the {arrow} team !