[SNOW-172] Add filter to file_latest and add comments

[danlu1]

Problem:

The original file_latest dynamic table does not have filter since the interpretation of CHANGE_TYPE='DELETE' vs STATUS != "AVAILABLE". Platform team confirmed that CHANGE_TYPE!='DELETE' can be used to filter out file handles which have been deleted.

Solution:

• Replace file_latest by adding the filter CHANGE_TYPE!='DELETE' and NOT IS_PREVIEW
• Add table and column comments again since comments will be wiped out when replacing the table

[philerooski]

There's one other filter which is missing. The original upsert_to_file_latest_task had this filter before QUALIFYing the results.

... WHERE NOT IS_PREVIEW

See these lines: https://github.com/Sage-Bionetworks/snowflake/blob/main/synapse_data_warehouse/synapse_raw/tasks/V1.14.0__merge_into_latest_tables.sql#L149-L150

[danlu1]

There's one other filter which is missing. The original upsert_to_file_latest_task had this filter before QUALIFYing the results.

... WHERE NOT IS_PREVIEW

See these lines: https://github.com/Sage-Bionetworks/snowflake/blob/main/synapse_data_warehouse/synapse_raw/tasks/V1.14.0__merge_into_latest_tables.sql#L149-L150

Thanks for catching. I forgot to add this one back when removing the status filter.

[philerooski]

@danlu1 There's an optimization we can do here, since a file handle which is a preview file handle will never NOT be a preview, we don't need to consider whether this is the latest snapshot of that file. That means we can move the WHERE filter up into the WITH... AS... statement.

The WHERE filtering will be done before QUALIFY as documented here.

[philerooski]

@danlu1 Do you want to move the WHERE NOT IS_PREVIEW statement? I'd like to approve and validate this today so that we can make a new release.

[danlu1]

@philerooski Done.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud