Timestamp parsing issue

[blaze225]

Environment

Delta-rs version: 0.9.0

Binding: Python

Environment:

• Cloud provider: Corporate S3 implementation based on Ceph
• OS: Ubuntu

---

Bug

What happened:
Schema error for timestamp column when trying to create a delta table from a pyarrow dataframe, read from a csv file: deltalake.PyDeltaTableError: Schema error: Invalid data type for Delta Lake: Timestamp(Millisecond, Some("UTC"))

How to reproduce it:

1. Have a csv file with a timestamp column with the format: 2023-05-11 18:00:00+03:00 (Timezone information explicitly mentioned)
2. Read it using pyarrow: pyarrow.csv.read_csv(<file>)
3. Try to write to a delta table: deltalake.write_deltalake(<path>, storage_options=<storage_options>)

More details:

• Repeating the same with Pandas works without any issues.
• Pyarrow parses it as timestamp[s, tz=UTC]
• Pandas parses it as an Object

[wjones127]

Hi @blaze225. Delta lake only supports microsecond timestamps, so you need to cast the datatype to the correct resolution.

We have an open issue to support automatic casting of timestamps and other common data types: #686

I'm going to close this issue as a duplicate of that one. This seems to be a high demand feature, so I'll see if we can get that issue prioritized.