Skip to content

Commit

Permalink
convert datetime to str
Browse files Browse the repository at this point in the history
  • Loading branch information
erfannariman committed Jun 4, 2024
1 parent 9de29ae commit 482778e
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions df_to_azure/export.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,14 @@ def upload_to_blob(self):
blob=f"{self.table_name}/{self.table_name}.parquet",
)

# This is needed because ADF converts datetime to Unix Epoch
# resulting in INT64 type,
# which conflicts with our Datetime column in the database
# https://shorturl.at/dtSm6
datetime_dtypes = self.df.select_dtypes("datetime")
if datetime_dtypes.empty is False:
for col in datetime_dtypes.columns:
self.df[col] = self.df[col].astype(str)
data = self.df.to_parquet(index=False)
blob_client.upload_blob(data, overwrite=True)

Expand Down

0 comments on commit 482778e

Please sign in to comment.