Global async query - Unserializable object {} of type <class 'datetime.time'>

[jerwen]

Impossible to render a chart, say a TableView, presenting a Time column when the global async query feature is enabled.

Expected results

We expected the chart to be rendered with the data on display.

Actual results

Error message on display.

Screenshots

How to reproduce the bug

1. Enable Global Async Query feature
2. Make sure to have a Redis configured for caching the result
3. Open a dashboard with a table view presenting a column Time
4. See error
   From the logs we have the following message
   {"asctime": "2021-07-27 13:46:01,004", "name": "superset.views.base", "levelname": "WARNING", "process": 18, "message": "Unserializable object 10:30:00 of type <class 'datetime.time'>"}

Environment

(please complete the following information):

• superset version: 1.2
• python version: 3.7.9
• node.js version: Unknown

Checklist

Make sure to follow these steps before submitting your issue - thank you!

• I have checked the superset logs for python stacktraces and included it here as text if there are any.
• I have reproduced the issue with at least the latest released version of superset.
• I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

I looked up in the code base to see where the message was coming from. I found out two different methods aiming to perform the same job which is to json serialize dates

def json_iso_dttm_ser(obj: Any, pessimistic: bool = False) -> str:
    """
    json serializer that deals with dates

    >>> dttm = datetime(1970, 1, 1)
    >>> json.dumps({'dttm': dttm}, default=json_iso_dttm_ser)
    '{"dttm": "1970-01-01T00:00:00"}'
    """
    val = base_json_conv(obj)
    if val is not None:
        return val
    if isinstance(obj, (datetime, date, time, pd.Timestamp)):
        obj = obj.isoformat()
    else:
        if pessimistic:
            return "Unserializable [{}]".format(type(obj))

        raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
    return obj


def json_int_dttm_ser(obj: Any) -> float:
    """json serializer that deals with dates"""
    val = base_json_conv(obj)
    if val is not None:
        return val
    if isinstance(obj, (datetime, pd.Timestamp)):
        obj = datetime_to_epoch(obj)
    elif isinstance(obj, date):
        obj = (obj - EPOCH.date()).total_seconds() * 1000
    else:
        raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
    return obj

It seems to me that the second might be at fault here.

[rubypollev]

This bug is preventing us from rolling out this feature.

[frafra]

I can confirm that json_int_dttm_ser is the problem. This is my log:

superset_app            | Traceback (most recent call last):
superset_app            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
superset_app            |     rv = self.dispatch_request()
superset_app            |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
superset_app            |     return self.view_functions[rule.endpoint](**req.view_args)
superset_app            |   File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 61, in wraps
superset_app            |     return f(self, *args, **kwargs)
superset_app            |   File "/app/superset/views/base_api.py", line 85, in wraps
superset_app            |     raise ex
superset_app            |   File "/app/superset/views/base_api.py", line 82, in wraps
superset_app            |     duration, response = time_function(f, self, *args, **kwargs)
superset_app            |   File "/app/superset/utils/core.py", line 1461, in time_function
superset_app            |     response = func(*args, **kwargs)
superset_app            |   File "/app/superset/utils/log.py", line 242, in wrapper
superset_app            |     value = f(*args, **kwargs)
superset_app            |   File "/app/superset/charts/api.py", line 726, in data
superset_app            |     return self.get_data_response(command)
superset_app            |   File "/app/superset/charts/api.py", line 543, in get_data_response
superset_app            |     return self.send_chart_response(result, form_data)
superset_app            |   File "/app/superset/charts/api.py", line 522, in send_chart_response
superset_app            |     ignore_nan=True,
superset_app            |   File "/usr/local/lib/python3.7/site-packages/simplejson/__init__.py", line 412, in dumps
superset_app            |     **kw).encode(obj)
superset_app            |   File "/usr/local/lib/python3.7/site-packages/simplejson/encoder.py", line 296, in encode
superset_app            |     chunks = self.iterencode(o, _one_shot=True)
superset_app            |   File "/usr/local/lib/python3.7/site-packages/simplejson/encoder.py", line 378, in iterencode
superset_app            |     return _iterencode(o, 0)
superset_app            |   File "/app/superset/utils/core.py", line 614, in json_int_dttm_ser
superset_app            |     raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
superset_app            | TypeError: Unserializable object 14:06:00 of type <class 'datetime.time'>

My setup is a standard Superset started using docker-compose, connected to a mssql database containing a table with a TIME column.

[frafra]

There are various places in the code where such method is used as default when dumping the JSON (look for default=json_int_dttm_ser):

superset/superset/charts/api.py

Lines 518 to 523 in 1669947

	if result_format == ChartDataResultFormat.JSON:
	response_data = simplejson.dumps(
	{"result": result["queries"]},
	default=json_int_dttm_ser,
	ignore_nan=True,
	)

superset/superset/utils/core.py

Lines 618 to 619 in 0f16177

	def json_dumps_w_dates(payload: Dict[Any, Any]) -> str:
	return json.dumps(payload, default=json_int_dttm_ser)

...and others.

Here is the problematic function:

superset/superset/utils/core.py

Lines 604 to 615 in 0f16177

	def json_int_dttm_ser(obj: Any) -> float:
	"""json serializer that deals with dates"""
	val = base_json_conv(obj)
	if val is not None:
	return val
	if isinstance(obj, (datetime, pd.Timestamp)):
	obj = datetime_to_epoch(obj)
	elif isinstance(obj, date):
	obj = (obj - EPOCH.date()).total_seconds() * 1000
	else:
	raise TypeError("Unserializable object {} of type {}".format(obj, type(obj)))
	return obj

It was meant to handle date, datetime, but not time. It actually uses a generic function before trying to dealing with date and datetime, such base_json_conv:

superset/superset/utils/core.py

Lines 550 to 573 in 0f16177

	def base_json_conv(obj: Any,) -> Any: # pylint: disable=inconsistent-return-statements
	if isinstance(obj, memoryview):
	obj = obj.tobytes()
	if isinstance(obj, np.int64):
	return int(obj)
	if isinstance(obj, np.bool_):
	return bool(obj)
	if isinstance(obj, np.ndarray):
	return obj.tolist()
	if isinstance(obj, set):
	return list(obj)
	if isinstance(obj, decimal.Decimal):
	return float(obj)
	if isinstance(obj, uuid.UUID):
	return str(obj)
	if isinstance(obj, timedelta):
	return format_timedelta(obj)
	if isinstance(obj, bytes):
	try:
	return obj.decode("utf-8")
	except Exception: # pylint: disable=broad-except
	return "[bytes]"
	if isinstance(obj, LazyString):
	return str(obj)

It would be enough to add a specific case:

> python3
Python 3.8.10 (default, May 05 2021, 15:36:36) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datetime
>>> str(datetime.time())
'00:00:00'

The function json_iso_dttm_ser is almost identical, but it tries to convert to ISO format instead of an integer, and it has support for time:

superset/superset/utils/core.py

Lines 587 to 588 in 27a40d2

	if isinstance(obj, (datetime, date, time, pd.Timestamp)):
	obj = obj.isoformat()

In case of time, .isoformat() is equivalent to str(...): __str__ = isoformat (https://github.com/python/cpython/blob/84975146a7ce64f1d50dcec8311b7f7188a5c962/Lib/datetime.py#L1434)

As a time without a date cannot be converted to an integer with respect to EPOCH, we should decide if it would make more sense to convert it to seconds (from the beginning of the midnight) or to a string.

datetime.time has no total_seconds() method or similar, so I would suggest converting it to a string, which seems the safest solution.