Add ignoreZeroDecimal to ReadOptions

[larshelge]

• Tick to sign-off your agreement to the Developer Certificate of Origin (DCO) 1.1

Description

Introduces new option ignoreZeroDecimal to ReadOptions. This option controls whether a numeric value ending with ".0" may be considered an integer (or short, long). Default value is true and retains the current behavior. If set to false, values ending with ".0" will be not be considered an integer and instead be considered a floating point.

Since different software/systems have different opinions on this matter, an option to control this behavior could be justified. Will be useful when using TableSaw together with other systems.

The logic could be centralized in AbstractColumnParser if that is preferred. Open for other variable naming suggestions.

Relates to issue #747.

Testing

Tests added to CsvReaderTest and StringUtilsTest.

[larshelge]

Fully appreciating that people are busy, I was just wondering whether you may consider including this patch, or if I should start working on an external work-around?

[benmccann]

It's a little weird that it only works with a single 0. What if it was .00? I know you didn't write that part, but it might be a good opportunity to clean it up

[larshelge]

Agreed. I have updated the PR now.

I added a function to StringUtils so that we can centralize the logic. I opted for a regex with a compiled pattern for performance reasons. I combined the check for zero decimal with the remove operation as it is simpler and performance will be the same. I can put this method in another util class if that is better.

This made the FixedWidthReaderTest#testDataTypeDetection test fail which is legitimate as the test data contains data with two zero decimals, so I updated the test.

I plan to add another test once the approach is considered acceptable.

[lwhite1]

This seems fine. Would you mind adding the test?

[larshelge]

Many thanks @lwhite1. I have added tests now. These cover both the column type detection feature and getting columns and their types after reading the data into a table. There are tests asserting behavior when ignoreZeroDecimal is disabled as well as enabled. Let me know if the tests look appropriate.

[larshelge]

Would it be possible to kindly get a timeline for when a review of this PR could take place? I am happy to change the approach to what the reviewer find most appropriate.

[lwhite1]

I guess I don't see why you can't read the file as a double or float column and then convert it to an int column. The asIntColumn() method uses a cast

  public IntColumn asIntColumn() {
    IntColumn result = IntColumn.create(name());
    for (double d : data) {
      if (DoubleColumnType.valueIsMissing(d)) {
        result.appendMissing();
      } else {
        result.append((int) d);
      }
    }
    return result;
  }

DoubleColumn also implements asShortColumn and asLongColumn.

One advantage to doing this work here as opposed to in the reader is that most numeric operators return a double column, which you may want to convert to an integral value.

[benmccann]

Yeah, I'd debated that as well. You could generally convert after the fact, but it's probably enough extra work that I didn't think this approach was too bad.

Also, there might be circumstances where the current approach makes you lose information. Right now we always strip ".0" and treat as an int. But what if you had a data source where some columns had trailing zeros and others didn't and you want to retain that information? Then I think you'd need to turn off the automatic stripping of ".0". I think that's sort of what was being suggested in #747

[larshelge]

Many thanks for the comments, really appreciate it.

Just wanted to mention that the original use-case for #747 is data type detection, where for various reasons one would like to consider numbers ending in zero-decimals only as type double.

Converting the column to int after the fact is not really an option as we do not know the data type of the data files/columns up front, as they are uploaded by users. We are using TableSaw to
inform us what data type each column has.

Note that this PR will also coincidentally solve the problem of #732.

[larshelge]

I have added tests. @lwhite1 would it be possible to kindly ask for a review?