Query on possibilities for fread enhancements for highly structured files

[mike-lawrence]

Sorry if this is the wrong venue for this type of thing, happy to move it elsewhere if there's a more appropriate place for discussions. I'm a contributor (n.b. very peripheral though) to the Stan project, where the cmdstanr package uses fread to import csv files produced by cmdstan. These files have very predictable structure whereby outside of some metadata indicated by lines starting with # characters (so we get those with cmd = "grep '^[#a-zA-Z]' out.csv and the header-included data with cmd = "grep -v '^#' out.csv) it's a matrix of comma separated numeric values with no missing values and the column and row counts are available in the metadata. Unfortunately there's a bit of variability in the precise number of characters encoding the numeric values, but it's possible that could be made truly constant (I'd have to look at cmdstan to see how easy this would be).

fread() already provides a huge speed boost to our importing all by itself, but I wonder if there might be any further improvements if could be toggled into a less-flexible mode for such regular input (assuming the fread's current flexibility even has any kind of performance cost)?

Another feature that would be helpful is the ability to skip columns by name; tossing columns is obviously easy after-the-fact, but if it can improve performance at all by not encoding them in the first place, that'd be a useful feature too.

[MichaelChirico]

I see two specific things in what you wrote

• FR Implement comment.char argument in fread #856 -- fread gets comment.char argument
• For skipping columns, fread already has the drop argument, e.g. fread('a,b\n1,2\n3,4\n', drop='b'). I'm not 100% sure but I think that's happening in C, so R never sees the dropped columns.

I think more generally, using hints to improve the performance of fread falls under what I think we could eventually accomplish with CSVY, e.g., specifying the data schema (as you allude to) & using that to skip some overhead in fread.