Use Rf_charIsASCII for IS_ASCII instead of testing LEVELS on R >= 4.5 #6422
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Since data.table now depends on R >= 3.3, the backports are no longer needed. Moreover, MAYBE_SHARED is currently a function, while MAYBE_REFERENCED expands to !NO_REFERENCES (which is a function). In debugging output, show MAYBE_REFERENCED (NAMED > 0) instead of NAMED.
getCharCE appeared in R-2.7, making it possible to check for strings _marked_ as UTF-8 or Latin-1. There is no marking as ASCII, so fixing IS_ASCII will have to wait for R >= 4.5.
There's no explicit encoding code for ASCII, so use charIsASCII()
("eapi", expected to appear in R-4.5.0).
|
Generated via commit 090dc37 Download link for the artifact containing the test results: ↓ atime-results.zip
|
Don't mind that, we maintainers will take care of it :) |
|
set #6420 as the target of this PR to make the chain clearer |
|
Good catch! It's a bit worse: LEVELS(x) & 64 returns 0 or 64.
Here are the only users of IS_ASCII:
data.table.h:
#define NEED2UTF8(s) !(IS_ASCII(s) || (s)==NA_STRING || IS_UTF8(s))
fwriteR.c:
#define TO_NATIVE(s) (native && (s)!=NA_STRING && !IS_ASCII(s))
forder.c:
if (!anynotutf8 && /*...*/ !IS_ASCII(s)) {
They all use the result of the macro in boolean context (if / || / &&),
so any non-zero result is interpreted as true.
|
MichaelChirico
changed the title
| @@ -42,7 +42,11 @@ | |||
| /* we mean the encoding bits, not CE_NATIVE in a UTF-8 locale */ | |||
| #define IS_UTF8(x) (getCharCE(x) == CE_UTF8) | |||
| #define IS_LATIN(x) (getCharCE(x) == CE_LATIN1) | |||
| #define IS_ASCII(x) (LEVELS(x) & 64) // API expected in R >= 4.5 | |||
| #if R_VERSION < R_Version(4, 5, 0) || R_SVN_REVISION < 86789 | |||
There was a problem hiding this comment.
Keeping this here even though there's another (4,5,0) check above at L21
- The
R_SVN_REVISIONs are different - The
IS_UTF8()andIS_LATIN()macros are also right here
MichaelChirico
approved these changes
Mar 4, 2025
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6422 +/- ##
=======================================
Coverage 98.59% 98.59%
=======================================
Files 79 79
Lines 14661 14661
=======================================
Hits 14455 14455
Misses 206 206 ☔ View full report in Codecov by Sentry. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This part is split from #6420 (but includes #6420; can rebase if needed) because the R version containing
Rf_isCharASCIIis not currently released. The overall issue is #6180. Not yet sure how to changeNEWS.mdin case we're expectingdata.tablereleases in the meantime.