bugfix/timestamp-without-time-zone

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -4,48 +4,27 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present)
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
-- [ ] BuildKite integration tests are passing
+- [ ] The appropriate issue has been linked, tagged, and properly assigned.
+- [ ] All necessary documentation and version upgrades have been applied.
+    <!--- Be sre to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates).
+- [ ] BuildKite integration tests are passing.
+- [ ] Detailed validation steps have been provided below.
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+💃

.github/pull_request_template.md

@@ -1,52 +1,33 @@
-**Are you a current Fivetran customer?** 
-<!--- Please tell us your name, title and company -->
+**Please provide your name and company**
 
-**What change(s) does this PR introduce?** 
-<!--- Describe what changes your PR introduces to the package and how to leverage this new feature. -->
+**Link the issue/feature request which this PR is meant to address**
+<!--- If an issue was not created, please create one first so we may discuss the PR prior to opening one. -->
 
-**Did you update the CHANGELOG?** 
+**Detail what changes this PR introduces and how this addresses the issue/feature request linked above.**
+
+**How did you validate the changes introduced within this PR?**
+
+**Which warehouse did you use to develop these changes?**
+
+**Did you update the CHANGELOG?**
 <!--- Please update the new package version’s CHANGELOG entry detailing the changes included in this PR. -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Does this PR introduce a breaking change?**
-<!--- Does this PR introduce changes that will cause current package users' jobs to fail or require a `--full-refresh`? -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes (please provide breaking change details below.)
-- [ ] No  (please provide an explanation as to how the change is non-breaking below.)
-
-**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)** 
+**Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)**
 <!--- The dbt_project.yml and the integration_tests/dbt_project.yml files contain the version number. Be sure to upgrade it accordingly -->
 <!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
 - [ ] Yes
 
-**Is this PR in response to a previously created Bug or Feature Request**
-<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Yes, Issue/Feature [link bug/feature number here]
-- [ ] No 
-
-**How did you test the PR changes?** 
-<!--- Proof of testing is required in order for the PR to be approved. -->
-<!--- To check a box, remove the space and insert an x in the box (eg. [x] Buildkite). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] Buildkite <!--- Buildkite testing is only applicable to Fivetran employees. --> 
-- [ ] Local (please provide additional testing details below)
-
-**Select which warehouse(s) were used to test the PR**
-<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). --> 
-<!--- To select a checkbox you simply need to add an "x" with no spaces between the brackets (eg. [x] Yes). -->
-- [ ] BigQuery
-- [ ] Redshift
-- [ ] Snowflake
-- [ ] Postgres
-- [ ] Databricks
-- [ ] Other (provide details below)
-
 **Provide an emoji that best describes your current mood**
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
 :dancer:
 
 **Feedback**
 
 We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.
+
+**PR Template** 
+- [Community Pull Request Template](?expand=1&template=pull_request_template.md) (default)
+
+- [Maintainer Pull Request Template](?expand=1&template=maintainer_pull_request_template.md) (to be used by maintainers)

.gitignore

@@ -2,5 +2,6 @@
 target/
 dbt_modules/
 logs/
-
+env/
+package-lock.yml
 dbt_packages/

CHANGELOG.md

@@ -1,3 +1,53 @@
+# v0.2.0 dbt_qualtrics_source
+
+[PR #5](https://github.com/fivetran/dbt_qualtrics_source/pull/5) includes the following updates: 
+
+## 🚨 Breaking Changes: Bug Fixes 🚨
+- Casted the following timestamp fields in the below models using the `dbt.type_timestamp()` macro. This is necessary to ensure all timestamps are consistently casted and do not experience datatype mismatches in downstream transformations.
+    - stg_qualtrics__contact_mailing_list_membership
+        - unsubscribed_at
+    - stg_qualtrics__directory_contact
+        - created_at
+        - unsubscribed_from_directory_at
+        - last_modified_at
+    - stg_qualtrics__directory_mailing_list
+        - created_at
+        - last_modified_at
+    - stg_qualtrics__distribution_contact
+        - opened_at
+        - response_completed_at
+        - response_started_at
+        - sent_at
+    - stg_qualtrics__distribution
+        - created_at
+        - last_modified_at
+        - send_at
+        - survey_link_expires_at
+    - stg_qualtrics__survey_response
+        - finished_at
+        - is_finished
+        - last_modified_at
+        - recorded_date
+        - started_at
+    - stg_qualtrics__survey_version
+        - created_at
+    - stg_qualtrics__survey
+        - last_accessed_at
+        - last_activated_at
+        - last_modified_at
+    - stg_qualtrics__user
+        - account_created_at
+        - account_expires_at
+        - last_login_at
+        - password_expires_at
+        - password_last_changed_at
+
+> Please note: this update will likely only impact Redshift destinations as it was found the connector synced these fields as `timestamp with time zone` when in fact they were without. Most users will not see any changes following this release. But we marked this as breaking to ensure no possible datatype conflicts downstream.
+
+## Under the Hood
+- Updated the maintainer PR template to resemble the most up to date format.
+- Added the auto release GitHub Action for easier deployment.
+
 # dbt_qualtrics_source v0.1.1
 [PR #4](https://github.com/fivetran/dbt_qualtrics_source/pull/4) includes the following update:
 ## Bug fix

README.md

@@ -45,7 +45,7 @@ If you  are **not** using the [Qualtrics transformation package](https://github.
 ```yml
 packages:
   - package: fivetran/qualtrics_source
-    version: [">=0.1.0", "<0.2.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 ## Step 3: Define database and schema variables

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'qualtrics_source'
-version: '0.1.1'
+version: '0.2.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'qualtrics_source_integration_tests'
-version: '0.1.1'
+version: '0.2.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -21,6 +21,16 @@ seeds:
   qualtrics_source_integration_tests:
     +column_types:
       _fivetran_synced: timestamp
+      directory_unsubscribe_date: timestamp
+      response_completed_at: timestamp
+      last_accessed: timestamp
+      account_creation_date: timestamp
+      response_started_at: timestamp
+      account_expiration_date: timestamp
+      last_login_date: timestamp
+      password_expiration_date: timestamp
+      password_last_changed_date: timestamp
+      unsubscribe_date: timestamp
     directory_contact:
       +column_types:
         phone: "{{ 'string' if target.type in ['bigquery','spark','databricks'] else 'varchar' }}"

models/stg_qualtrics__contact_mailing_list_membership.sql

@@ -32,7 +32,7 @@ final as (
         mailing_list_id,
         name,
         owner_id as owner_user_id,
-        unsubscribe_date as unsubscribed_at,
+        cast(unsubscribe_date as {{ dbt.type_timestamp() }}) as unsubscribed_at,
         unsubscribed as is_unsubscribed,
         _fivetran_synced,
         source_relation

models/stg_qualtrics__directory_contact.sql

@@ -26,9 +26,9 @@ fields as (
 final as (
 
     select 
-        creation_date as created_at,
+        cast(creation_date as {{ dbt.type_timestamp() }}) as created_at,
         directory_id,
-        directory_unsubscribe_date as unsubscribed_from_directory_at,
+        cast(directory_unsubscribe_date as {{ dbt.type_timestamp() }}) as unsubscribed_from_directory_at,
         directory_unsubscribed as is_unsubscribed_from_directory,
         lower(email) as email,
         lower(email_domain) as email_domain,
@@ -38,7 +38,7 @@ final as (
         REGEXP_REPLACE(phone, '[^0-9]', '') AS phone, -- remove any non-numeric chars
         id as contact_id,
         language,
-        last_modified as last_modified_at,
+        cast(last_modified as {{ dbt.type_timestamp() }}) as last_modified_at,
         _fivetran_synced,
         source_relation
 

models/stg_qualtrics__directory_mailing_list.sql

@@ -26,10 +26,10 @@ fields as (
 final as (
 
     select 
-        creation_date as created_at,
+        cast(creation_date as {{ dbt.type_timestamp() }}) as created_at,
         directory_id,
         id as mailing_list_id,
-        last_modified_date as last_modified_at,
+        cast(last_modified_date as {{ dbt.type_timestamp() }}) as last_modified_at,
         name,
         owner_id as owner_user_id,
         _fivetran_deleted as is_deleted,

models/stg_qualtrics__distribution.sql

@@ -26,7 +26,7 @@ fields as (
 final as (
 
     select 
-        created_date as created_at,
+        cast(created_date as {{ dbt.type_timestamp() }}) as created_at,
         header_from_email,
         header_from_name,
         header_reply_to_email,
@@ -35,7 +35,7 @@ final as (
         message_library_id,
         message_message_id as message_id,
         message_message_text as message_text,
-        modified_date as last_modified_at,
+        cast(modified_date as {{ dbt.type_timestamp() }}) as last_modified_at,
         organization_id,
         owner_id as owner_user_id,
         parent_distribution_id,
@@ -45,8 +45,8 @@ final as (
         recipient_sample_id,
         request_status,
         request_type,
-        send_date as send_at,
-        survey_link_expiration_date as survey_link_expires_at,
+        cast(send_date as {{ dbt.type_timestamp() }}) as send_at,
+        cast(survey_link_expiration_date as {{ dbt.type_timestamp() }}) as survey_link_expires_at,
         survey_link_link_type as survey_link_type,
         survey_link_survey_id as survey_id,
         _fivetran_deleted as is_deleted,

models/stg_qualtrics__distribution_contact.sql

@@ -30,11 +30,11 @@ final as (
         contact_id,
         contact_lookup_id,
         distribution_id,
-        opened_at,
-        response_completed_at,
+        cast(opened_at as {{ dbt.type_timestamp() }}) as opened_at,
+        cast(response_completed_at as {{ dbt.type_timestamp() }}) as response_completed_at,
         response_id,
-        response_started_at,
-        sent_at,
+        cast(response_started_at as {{ dbt.type_timestamp() }}) as response_started_at,
+        cast(sent_at as {{ dbt.type_timestamp() }}) as sent_at,
         status,
         survey_link,
         survey_session_id,

models/stg_qualtrics__survey.sql

@@ -46,9 +46,9 @@ final as (
         scoring_summary_after_questions,
         scoring_summary_after_survey,
         scoring_summary_category,
-        last_accessed as last_accessed_at,
-        last_activated as last_activated_at,
-        last_modified as last_modified_at,
+        cast(last_accessed as {{ dbt.type_timestamp() }}) as last_accessed_at,
+        cast(last_activated as {{ dbt.type_timestamp() }}) as last_activated_at,
+        cast(last_modified as {{ dbt.type_timestamp() }}) as last_modified_at,
         _fivetran_deleted as is_deleted,
         _fivetran_synced,
         source_relation

models/stg_qualtrics__survey_response.sql

@@ -28,19 +28,19 @@ final as (
     select 
         distribution_channel,
         duration_in_seconds,
-        end_date as finished_at,
+        cast(end_date as {{ dbt.type_timestamp() }}) as finished_at,
         cast(finished as {{ dbt.type_boolean() }}) as is_finished,
         id as response_id,
         ip_address,
-        last_modified_date as last_modified_at,
+        cast(last_modified_date as {{ dbt.type_timestamp() }}) as last_modified_at,
         location_latitude,
         location_longitude,
         progress,
         lower(recipient_email) as recipient_email,
         recipient_first_name,
         recipient_last_name,
-        recorded_date,
-        start_date as started_at,
+        cast(recorded_date as {{ dbt.type_timestamp() }}) as recorded_date,
+        cast(start_date as {{ dbt.type_timestamp() }}) as started_at,
         status,
         survey_id,
         user_language,

models/stg_qualtrics__survey_version.sql

@@ -26,7 +26,7 @@ fields as (
 final as (
 
     select 
-        creation_date as created_at,
+        cast(creation_date as {{ dbt.type_timestamp() }}) as created_at,
         description as version_description,
         id as version_id,
         published as is_published,

models/stg_qualtrics__user.sql

@@ -26,19 +26,19 @@ fields as (
 final as (
 
     select 
-        account_creation_date as account_created_at,
-        account_expiration_date as account_expires_at,
+        cast(account_creation_date as {{ dbt.type_timestamp() }}) as account_created_at,
+        cast(account_expiration_date as {{ dbt.type_timestamp() }}) as account_expires_at,
         account_status,
         division_id,
         email,
         first_name,
         id as user_id,
         language,
-        last_login_date as last_login_at,
+        cast(last_login_date as {{ dbt.type_timestamp() }}) as last_login_at,
         last_name,
         organization_id,
-        password_expiration_date as password_expires_at,
-        password_last_changed_date as password_last_changed_at,
+        cast(password_expiration_date as {{ dbt.type_timestamp() }}) as password_expires_at,
+        cast(password_last_changed_date as {{ dbt.type_timestamp() }}) as password_last_changed_at,
         response_count_auditable,
         response_count_deleted,
         response_count_generated,