data types update - destinations

[edgao]

(this does not depend on #17639, because it's purely about consuming the new protocol rather than forcing anyone to use it)

Once #17486 is approved and merged, we need to make these changes:

• Update normalization to handle the new $ref data types (including binary data) #17642
• Update the destination data type tests to use the new data types #17644

Once those tasks are complete, we can start upgrading connectors to the new protocol. Specifically, these changes need to happen in each destination connector that cares about data types:

1. check the catalog for $ref: well_known_types....Foo rather than checking {type: foo} (/ airbyte_type / format)
2. handle stringified numeric values (int/float/etc)
3. handle binary data (a v0 implementation is to just treat it as a string, since the JSON blob will contain the base64-encoded data rather than raw binary)

Affected destinations:

• Update destination-s3 to handle the new data types protocol #17645
• Update destination-bigquery to handle the new data types protocol #17647
  • (this relies on the same JsonToAvroSchemaConverter, but might have some extra logic surrounding it)
• Update destination-gcs to handle the new data types protocol #17646
  • (should be a simple re-publish after completing dest-s3, since they share the underlying avro/parquet code)
• Update destination-aws-datalake to handle the new data types protocol #17648
  • (written in python; maybe uses stream schema for something

    airbyte/airbyte-integrations/connectors/destination-aws-datalake/destination_aws_datalake/destination.py

    Line 54 in b4fd1ea

    schema=s.stream.json_schema["properties"],

    )
• Update destination-databricks to handle the new data types protocol #18548
• Update destination-azure-blob-storage to handle the new data types protocol #18549

[edgao]

@edgao check if databricks needs to be updated

edit: #18548