Allow different skey definitions by data standard version (and & this…

… for grading periods v5.0) (#99) * draft of grading_period key change for v5.0 * update from main (#106) * New DS5 optional fields for `Section`, `CourseTranscript`, and `StudentSectionAssociation` (#104) * add section_type * bring into line with correct json names * Update CHANGELOG.md --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * fix typo (#90) * fix typo * Update CHANGELOG.md --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * Feature/Contacts and StudentContactAssociations (#96) * redraft * add student contact associations * switch from parent to contact * add comments about name change * can't use gsn_skey * add note about new fields * update changelog * use dbt utils surrogate key * use parents if contacts are not enabled * add docs * remove configurability * extract extenstions configured with parent naming too * fix bad syntax --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * Bugfix/Updated Surrogate Keys to include lower() in student_unique_id inputs (#100) * added lower to student_unique_id in surrogate keys * Update stg_ef3__student_objective_assessments.sql * Update CHANGELOG.md --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * Feature/new DS5 optional fields on staff (#103) * add new optional fields * fix typo * Update CHANGELOG.md --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * flatten student cohort year (#92) * flatten student cohort year * grab school year and term * rename fields * Update CHANGELOG.md --------- Co-authored-by: rlittle08 <rlittle@edanalytics.org> * explicitly call extract macro from this package (#98) * explicitly call extract macro from this package * Update CHANGELOG.md --------- Co-authored-by: John C. Merfeld <John.Merfeld@gmail.com> Co-authored-by: ejoranlienea <ejoranlien@edanalytics.org> Co-authored-by: Samantha LeBlanc <56237580+sleblanc23@users.noreply.github.com> Co-authored-by: Angelica Lastra <107369214+AngelicaLastra@users.noreply.github.com> Co-authored-by: keen <51969358+keenzarate213@users.noreply.github.com> * add ods & dm version to every base table --------- Co-authored-by: John C. Merfeld <John.Merfeld@gmail.com> Co-authored-by: ejoranlienea <ejoranlien@edanalytics.org> Co-authored-by: Samantha LeBlanc <56237580+sleblanc23@users.noreply.github.com> Co-authored-by: Angelica Lastra <107369214+AngelicaLastra@users.noreply.github.com> Co-authored-by: keen <51969358+keenzarate213@users.noreply.github.com>
edanalytics · Sep 20, 2024 · 31f041b · 31f041b
1 parent 938eed3
commit 31f041b
Show file tree

Hide file tree

Showing 60 changed files with 163 additions and 13 deletions.
diff --git a/macros/gen_skey.sql b/macros/gen_skey.sql
@@ -62,11 +62,19 @@
         },
         'k_grading_period': {
             'reference_name': 'grading_period_reference',
-            'col_list': ['gradingPeriodDescriptor',
-                         'periodSequence',
-                         'schoolId',
-                         'schoolYear'],
-            'annualize': False
+            'annualize': False,
+            'diff_by_data_standard': True,
+            'ds_specific_col_lists': {
+                "< '5.0'": ['gradingPeriodDescriptor',
+                                'periodSequence',
+                                'schoolId',
+                                'schoolYear'
+                        ],
+                ">= '5.0'": ['gradingPeriodDescriptor',
+                                'gradingPeriodName',
+                                'schoolId',
+                                'schoolYear']
+            }
         },
         'k_session': {
             'reference_name': 'session_reference',
@@ -207,8 +215,9 @@
     {#- retrieve key def for then decompose parts -#}
     {% set skey_def = skey_defs[k_name] %}
     {% set skey_ref = skey_def['reference_name'] %}
-    {% set skey_vars = skey_def['col_list'] %}
-
+    {% if not skey_def['diff_by_data_standard'] %}
+        {% set skey_vars = skey_def['col_list'] %}
+    {% endif %}
     {#- deal with special case: references embedded in unusual cases
         Note: we still want the same values, we just pull them from another place
         Example: getting k_section out of studentSectionAssociationReference instead of sectionReference
@@ -217,11 +226,22 @@
       {% set skey_ref = alt_ref %}
     {%- endif -%}
 
-    iff(
-        {{ skey_ref }} is not null, 
-        {{ dbt_utils.surrogate_key(edu_edfi_source.gen_key_list(skey_def, skey_ref, skey_vars, extras=extras)) }}, 
-        null
-    )::varchar(32) as {{ alt_k_name or k_name }}
+    {# handle cases where skey is different depending on the data standard version (need a case when to apply different rules by row) #}
+    {% if skey_def['diff_by_data_standard'] %}
+        (case
+          when {{ skey_ref }} is null then null
+        {% for ds_version in skey_def['ds_specific_col_lists'] %}
+            when data_model_version {{ ds_version }}
+                then {{ dbt_utils.surrogate_key(edu_edfi_source.gen_key_list(skey_def, skey_ref, skey_def['ds_specific_col_lists'][ds_version], extras=extras)) }} 
+        {% endfor %}
+        end)::varchar(32) as {{ alt_k_name or k_name }}
+    {% else %}
+        iff(
+            {{ skey_ref }} is not null, 
+            {{ dbt_utils.surrogate_key(edu_edfi_source.gen_key_list(skey_def, skey_ref, skey_vars, extras=extras)) }}, 
+            null
+        )::varchar(32) as {{ alt_k_name or k_name }}
+    {% endif %}
 {%- endmacro -%}
 
 {% macro gen_key_list(skey_def, skey_ref, skey_vars, extras=None) %}

diff --git a/models/staging/edfi_3/base/base_ef3__assessments.sql b/models/staging/edfi_3/base/base_ef3__assessments.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
 
         v:id::string                   as record_guid,
+        ods_version,
+        data_model_version,
         v:assessmentIdentifier::string as assessment_identifier,
         v:namespace::string            as namespace,
         v:assessmentTitle::string      as assessment_title,

diff --git a/models/staging/edfi_3/base/base_ef3__bell_schedules.sql b/models/staging/edfi_3/base/base_ef3__bell_schedules.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                              as record_guid,
+        ods_version,
+        data_model_version,
         v:bellScheduleName::string                as bell_schedule_name,
         v:schoolReference:schoolId::int           as school_id,
         v:alternateDayName::string                as alternate_day_name,

diff --git a/models/staging/edfi_3/base/base_ef3__calendar_dates.sql b/models/staging/edfi_3/base/base_ef3__calendar_dates.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                             as record_guid,
+        ods_version,
+        data_model_version,
         v:date::date                             as calendar_date,
         v:calendarReference:calendarCode::string as calendar_code,
         v:calendarReference:schoolId::integer    as school_id,

diff --git a/models/staging/edfi_3/base/base_ef3__calendars.sql b/models/staging/edfi_3/base/base_ef3__calendars.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                  as record_guid,
+        ods_version,
+        data_model_version,
         v:calendarCode::string                        as calendar_code,
         v:schoolReference:schoolId::integer           as school_id,
         v:schoolYearTypeReference:schoolYear::integer as school_year,

diff --git a/models/staging/edfi_3/base/base_ef3__class_periods.sql b/models/staging/edfi_3/base/base_ef3__class_periods.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                        as record_guid,
+        ods_version,
+        data_model_version,
         v:schoolReference:schoolId::integer as school_id,
         v:classPeriodName::string           as class_period_name,
         v:officialAttendancePeriod::boolean as is_official_attendance_period,

diff --git a/models/staging/edfi_3/base/base_ef3__cohorts.sql b/models/staging/edfi_3/base/base_ef3__cohorts.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                                  as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,
         v:cohortDescription::string                                   as cohort_description,

diff --git a/models/staging/edfi_3/base/base_ef3__contacts.sql b/models/staging/edfi_3/base/base_ef3__contacts.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
 
         v:id::string                                                                     as record_guid,
+        ods_version,
+        data_model_version,
         v:contactUniqueId::string                                                        as contact_unique_id,
         v:personReference:personId::string                                               as person_id,
         v:firstName::string                                                              as first_name,

diff --git a/models/staging/edfi_3/base/base_ef3__course_offerings.sql b/models/staging/edfi_3/base/base_ef3__course_offerings.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                   as record_guid,
+        ods_version,
+        data_model_version,
         v:courseReference:educationOrganizationId::int as ed_org_id,
         v:courseReference:courseCode::string           as course_code,
         v:schoolReference:schoolId::int                as school_id,

diff --git a/models/staging/edfi_3/base/base_ef3__course_transcripts.sql b/models/staging/edfi_3/base/base_ef3__course_transcripts.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
 				--identity components
 				{{ extract_descriptor('v:courseAttemptResultDescriptor::string') }} as course_attempt_result,
         v:courseReference:educationOrganizationId::int                      as course_ed_org_id,

diff --git a/models/staging/edfi_3/base/base_ef3__courses.sql b/models/staging/edfi_3/base/base_ef3__courses.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,
         v:courseCode::string                    as course_code,

diff --git a/models/staging/edfi_3/base/base_ef3__descriptors.sql b/models/staging/edfi_3/base/base_ef3__descriptors.sql
@@ -11,6 +11,8 @@ renamed as (
         is_deleted,
         name                           as descriptor_name, 
         v:id::string                   as record_guid,
+        ods_version,
+        data_model_version,
         v:codeValue::string            as code_value,
         v:namespace::string            as namespace,
         v:description::string          as description,

diff --git a/models/staging/edfi_3/base/base_ef3__discipline_actions.sql b/models/staging/edfi_3/base/base_ef3__discipline_actions.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         v:disciplineActionIdentifier::string          as discipline_action_id,
         v:disciplineDate::date                        as discipline_date,
         v:assignmentSchoolReference:schoolId::int     as assignment_school_id,

diff --git a/models/staging/edfi_3/base/base_ef3__discipline_incidents.sql b/models/staging/edfi_3/base/base_ef3__discipline_incidents.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                  as record_guid,
+        ods_version,
+        data_model_version,
         v:incidentIdentifier::string                  as incident_id,
         v:schoolReference:schoolId::int               as school_id,
         v:incidentDate::date                          as incident_date,

diff --git a/models/staging/edfi_3/base/base_ef3__education_organization_network_associations.sql b/models/staging/edfi_3/base/base_ef3__education_organization_network_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                            as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationNetworkReference:educationOrganizationNetworkId::int as network_id,
         v:memberEducationOrganizationReference:educationOrganizationId::int         as ed_org_id,
         v:beginDate::date                       as begin_date,

diff --git a/models/staging/edfi_3/base/base_ef3__education_organization_networks.sql b/models/staging/edfi_3/base/base_ef3__education_organization_networks.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                              as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationNetworkId::int     as network_id,
         v:nameOfInstitution::string               as network_name,
         v:shortNameOfInstitution::string          as network_short_name,

diff --git a/models/staging/edfi_3/base/base_ef3__education_service_centers.sql b/models/staging/edfi_3/base/base_ef3__education_service_centers.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                            as record_guid,
+        ods_version,
+        data_model_version,
         v:educationServiceCenterId::string                      as esc_id,
         v:nameOfInstitution::string                             as esc_name,
         v:shortNameOfInstitution::string                        as esc_short_name,

diff --git a/models/staging/edfi_3/base/base_ef3__grades.sql b/models/staging/edfi_3/base/base_ef3__grades.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         -- section key
         v:studentSectionAssociationReference:beginDate::date         as section_begin_date,
         v:studentSectionAssociationReference:localCourseCode::string as local_course_code,

diff --git a/models/staging/edfi_3/base/base_ef3__grading_periods.sql b/models/staging/edfi_3/base/base_ef3__grading_periods.sql
@@ -11,7 +11,10 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         {{ extract_descriptor('v:gradingPeriodDescriptor::string') }} as grading_period,
+        v:gradingPeriodName::string               as grading_period_name,
         v:periodSequence::int                     as period_sequence,
         v:schoolReference:schoolId::int           as school_id,
         v:schoolYearTypeReference:schoolYear::int as school_year,

diff --git a/models/staging/edfi_3/base/base_ef3__graduation_plans.sql b/models/staging/edfi_3/base/base_ef3__graduation_plans.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int           as ed_org_id,
         v:educationOrganizationReference:link:rel::string                       as ed_org_type,
         v:graduationSchoolYearTypeReference:schoolYear::int                     as graduation_school_year,

diff --git a/models/staging/edfi_3/base/base_ef3__learning_standards.sql b/models/staging/edfi_3/base/base_ef3__learning_standards.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
         -- ids
         v:id::string                              as record_guid,
+        ods_version,
+        data_model_version,
         v:learningStandardId::string              as learning_standard_id,
         v:identificationCodes                     as v_learning_standard_identification_codes,
         -- descriptions

diff --git a/models/staging/edfi_3/base/base_ef3__local_education_agencies.sql b/models/staging/edfi_3/base/base_ef3__local_education_agencies.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                      as record_guid,
+        ods_version,
+        data_model_version,
         v:localEducationAgencyId::int     as lea_id,
         v:nameOfInstitution::string       as lea_name,
         v:shortNameOfInstitution::string  as lea_short_name,

diff --git a/models/staging/edfi_3/base/base_ef3__locations.sql b/models/staging/edfi_3/base/base_ef3__locations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                          as record_guid,
+        ods_version,
+        data_model_version,
         v:classroomIdentificationCode::string as classroom_id_code,
         v:schoolReference:schoolId::int       as school_id,
         v:maximumNumberOfSeats::int           as maximum_seating,

diff --git a/models/staging/edfi_3/base/base_ef3__objective_assessments.sql b/models/staging/edfi_3/base/base_ef3__objective_assessments.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                       as record_guid,
+        ods_version,
+        data_model_version,
         v:assessmentReference:assessmentIdentifier::string as assessment_identifier,
         v:assessmentReference:namespace::string            as namespace,
         v:identificationCode::string                       as objective_assessment_identification_code,

diff --git a/models/staging/edfi_3/base/base_ef3__parents.sql b/models/staging/edfi_3/base/base_ef3__parents.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
 
         v:id::string                                                                     as record_guid,
+        ods_version,
+        data_model_version,
         v:parentUniqueId::string                                                         as parent_unique_id,
         v:personReference:personId::string                                               as person_id,
         v:firstName::string                                                              as first_name,

diff --git a/models/staging/edfi_3/base/base_ef3__programs.sql b/models/staging/edfi_3/base/base_ef3__programs.sql
@@ -11,6 +11,8 @@ renamed as (
         file_row_number,
         is_deleted,
         v:id::string                                                  as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,
         v:programId::string                                           as program_id,

diff --git a/models/staging/edfi_3/base/base_ef3__schools.sql b/models/staging/edfi_3/base/base_ef3__schools.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                     as record_guid,
+        ods_version,
+        data_model_version,
         v:schoolId::integer              as school_id,
         v:nameOfInstitution::string      as school_name,
         v:shortNameOfInstitution::string as school_short_name,

diff --git a/models/staging/edfi_3/base/base_ef3__sections.sql b/models/staging/edfi_3/base/base_ef3__sections.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                as record_guid,
+        ods_version,
+        data_model_version,
         v:sectionIdentifier::string as section_id,
         v:sectionName::string       as section_name,
         -- course offering key

diff --git a/models/staging/edfi_3/base/base_ef3__sessions.sql b/models/staging/edfi_3/base/base_ef3__sessions.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                  as record_guid,
+        ods_version,
+        data_model_version,
         v:schoolReference:schoolId::integer           as school_id,
         v:schoolYearTypeReference:schoolYear::integer as school_year,
         v:beginDate::date                             as session_begin_date,

diff --git a/...ls/staging/edfi_3/base/base_ef3__staff_education_organization_assignment_associations.sql b/...ls/staging/edfi_3/base/base_ef3__staff_education_organization_assignment_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,
         v:staffReference:staffUniqueId::string                        as staff_unique_id,

diff --git a/...ls/staging/edfi_3/base/base_ef3__staff_education_organization_employment_associations.sql b/...ls/staging/edfi_3/base/base_ef3__staff_education_organization_employment_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,
         v:staffReference:staffUniqueId::string                        as staff_unique_id,

diff --git a/models/staging/edfi_3/base/base_ef3__staff_school_associations.sql b/models/staging/edfi_3/base/base_ef3__staff_school_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                              as record_guid,
+        ods_version,
+        data_model_version,
         v:staffReference:staffUniqueId::string    as staff_unique_id,
         v:schoolReference:schoolId::int           as school_id,
         v:calendarReference:calendarCode::string  as calendar_code,

diff --git a/models/staging/edfi_3/base/base_ef3__staff_section_associations.sql b/models/staging/edfi_3/base/base_ef3__staff_section_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                 as record_guid,
+        ods_version,
+        data_model_version,
         -- section key
         v:sectionReference:localCourseCode::string   as local_course_code,
         v:sectionReference:schoolId::int             as school_id,

diff --git a/models/staging/edfi_3/base/base_ef3__staffs.sql b/models/staging/edfi_3/base/base_ef3__staffs.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
         -- fields
         v:id::string                                 as record_guid,
+        ods_version,
+        data_model_version,
         v:staffUniqueId::string                      as staff_unique_id,
         v:loginId::string                            as login_id,
         v:firstName::string                          as first_name,

diff --git a/models/staging/edfi_3/base/base_ef3__state_education_agencies.sql b/models/staging/edfi_3/base/base_ef3__state_education_agencies.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
         -- fields
         v:id::string                     as record_guid,
+        ods_version,
+        data_model_version,
         v:stateEducationAgencyId::int    as sea_id,
         v:nameOfInstitution::string      as sea_name,
         v:shortNameOfInstitution::string as sea_short_name,

diff --git a/models/staging/edfi_3/base/base_ef3__student_academic_records.sql b/models/staging/edfi_3/base/base_ef3__student_academic_records.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string as record_guid,
+        ods_version,
+        data_model_version,
         -- identity components
         v:educationOrganizationReference:educationOrganizationId::int as ed_org_id,
         v:educationOrganizationReference:link:rel::string             as ed_org_type,

diff --git a/models/staging/edfi_3/base/base_ef3__student_assessments.sql b/models/staging/edfi_3/base/base_ef3__student_assessments.sql
@@ -12,6 +12,8 @@ renamed as (
         is_deleted,
 
         v:id::string                                       as record_guid,
+        ods_version,
+        data_model_version,
         v:assessmentReference:assessmentIdentifier::string as assessment_identifier,
         v:assessmentReference:namespace::string            as namespace,
         v:schoolYearTypeReference:schoolYear::int          as school_year,

diff --git a/models/staging/edfi_3/base/base_ef3__student_cohort_associations.sql b/models/staging/edfi_3/base/base_ef3__student_cohort_associations.sql
@@ -11,6 +11,8 @@ renamed as (
         filename,
         is_deleted,
         v:id::string                                       as record_guid,
+        ods_version,
+        data_model_version,
         v:studentReference:studentUniqueId::string         as student_unique_id,
         v:cohortReference:educationOrganizationId::integer as cohort_ed_org_id,
         v:cohortReference:cohortIdentifier::string         as cohort_id,