invalid dates in a test suite cause a warning when writing to disk

[goodmami]

The make_item script for creating test suites from glossed examples (part of the Matrix) writes the dates in a unsupported format for [incr tsdb()], such as "february 29, 2020". art, logon/bin/tsdb, and PyDelphin seem to ignore these for most operations, but PyDelphin raises a Warning when attempting to compare the column or process a profile:

$ delphin select 'i-id where i-date > 01-jan-2006' mrs
/home/mwg/repos/pydelphin/delphin/tsdb.py:630: TSDBWarning: Invalid date field: 'oct 15, 2006'
  warnings.warn(f'Invalid date field: {s!r}', TSDBWarning)

When this happens, PyDelphin outputs an empty field for the date.

LOGON also does this when comparing with the column:

$ ~/logon/bin/tsdb -home mrs/ -query 'select i-id where i-date > 01-jan-2006'
parse_date(): invalid date expression `oct 15, 2006'.

An operation like delphin process does not really need to parse that date, so it might be better to ignore such warnings unless it's actually needed. The counterargument is that PyDelphin should not produce profiles that are not syntactically ill-formed (where we consider date formats part of the syntax).

@fcbond, what's your take?

[goodmami]

I'm leaning toward saying this is a bug with the make_item script while grumbling about the strange format requirements of [incr tsdb()], but I'll entertain convincing arguments for making PyDelphin more forgiving.

[goodmami]

@emilymbender, I bring this issue to your attention for your reference for future 567 classes. Francis dug around and found that the make_item script is just inserting the date string that the user provided, but the 567 instructions suggest a date like "April 7, 2006", which is not supported by [incr tsdb()]. In PyDelphin's documentation for tsdb.cast() I have outlined the acceptable date formats. (Aside: PyDelphin also accepts the more reasonable YYYY-MM-DD format, but in order to be compatible with [incr tsdb()] it always writes a DD-MM-YYYY format).

This may not have come up in the past because [incr tsdb()] and art mostly ignore the date field unless you need to sort by or filter on it, e.g., with TSQL or in the podium. PyDelphin now warns on bad date formats when it writes a profile, but in trying to avoid writing a syntactically ill-formed test suite, it drops the date altogether.

I suggest that the make_item script validate the date field and warn the user so they can correct it. I think @fcbond may have volunteered to do this.