[SNOW-175] Convert node_latest table -> dynamic table

[jaymedina]

problem

Currently the node_latest table is a regular table that gets updated using tasks and streams. We can simplify our data warehouse by turning this into a dynamic table that updates itself without requiring tasks & streams.

solution

Following the instructions of this SOP:

• Add a script to create a backup of the original node_latest table in case of issues
• Add a script to drop the original node_latest table so we can create a dynamic table with the same name
• Create node_latest dynamic table that introduces some post-processing of the RAW data
• Add a script to delete the scheduled task + stream that were for the original node_latest table

testing

1. Comparing the results between the new and old node_latest tables

I found a discrepancy between the results from the new node_latest dynamic table query and the current node_latest table. There are 1,413,032 rows missing from the new node_latest dynamic table:

As suspected, the discrepancies are due to the added time window of 14 days in the dynamic table query:

✅

2. Confirm there are no duplicate ids

✅

3. Confirm that all non-deleted nodes are accounted for

For this test I created a temp table that shows the IDs missing from the latest table that exist in the snapshots table. From visual inspection, it looks like these IDs are missing either because their change_type is set to DELETE, or because their last snapshot was taken outside of the 2-week window, in which case they are misrepresentations of the current state of the nodes.

✅

4. Confirm that the LATEST version of each node is represented in the new node_latest

✅

[thomasyu888]

🔥 Awesome work here - I'll leave it to Phil for a final review. I'm excited about the transition to dynamic tables!

[thomasyu888]

@jaymedina / @philerooski I wouldn't merge this just yet, because if it is merged then the highest V scripts would be recognized as 2.26 and skip over the work you're currently working to fix, Phil.

Done here: https://github.com/Sage-Bionetworks/snowflake/actions/runs/12266497862

[sonarqubecloud]

Quality Gate failed

Failed conditions
B Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[thomasyu888]

🔥 LGTM! I'll leave it to Phil for the final review so he can learn about all the changes.

For testing, it's a bit nuanced since when we clone the databases, the cloned tasks are in an auto suspended state. We don't necessarily want to resume them, but just something we have to keep in mind.

[jaymedina]

For testing, it's a bit nuanced

I'll add this to our CI/CD design doc so we can explore potential test cases.