Add a data dictionary #315

jeancochrane · 2025-01-08T21:34:09Z

This PR turns the "Features Used" table in the README into a proper data dictionary by making two changes:

Add a new column Variable Name that lists the name of the variable as it appears in the model code
Save a version of the table to CSV in docs/data-dict.csv

We also rename the Notes column to Description for clarity, and move it to the left in the table so that it's more prominent.

Note that this PR does not create historical dictionaries for past models. My expectation is that we will keep docs/data-dict.csv up to date with the most recent version of the parameter file, and then in the future we can back out the data dict that we used for past models by referencing the version of docs/data-dict.csv that existed at the time of the yearly model tag.

If this change looks good, I'll go ahead and copy it to the condo model to address ccao-data/model-condo-avm#72.

Closes #300.

…dability

README.Rmd

jeancochrane · 2025-01-08T21:40:51Z

README.Rmd

+# Write machine-readable version of the table to file
+param_tbl_fmt %>%
+  write_csv("docs/data-dict.csv")


This seemed to me like the simplest way to keep the data dict up to date: Any time we render the README, we'll save the data dict to the file. If model parameters haven't changed, the data dict file won't change, and there won't be a diff; otherwise, there will be a diff and the code author will be prompted to commit it. Not the most airtight system, but I figure it's probably a good enough starting place. Let me know if you have other ideas!

Perhaps out of scope for now, but we could also consider adding a pre-commit check similar to readme-rmd-rendered that compares the params in params.yml to the params in this file to make sure they match. I'm happy to take a crack at that now if you think it's a good idea.

The pre-commit hook was pretty straightforward so I went ahead and implemented it in 46c163b.

I don't love that we've ended up with a system where we need to sync four separate things: ccao::vars_dict, params.yaml, docs/data-dict.csv, and the README. I agree this is a good simple solution for now though. Let's roll with it and worry about something better/more long-term once 2025 modeling is finished.

I agree, it's confusing and brittle to maintain. I opened #324 to keep track of this work so that we can pick it up once we're done with modeling.

README.md

wrridgeway · 2025-01-08T22:41:14Z

This is awesome. Question for anyone - if someone wanted to grab training data and use it to feed a python model, would they have the info they need to recode features/apply variable labels currently?

…rams

jeancochrane · 2025-01-09T19:27:15Z

if someone wanted to grab training data and use it to feed a python model, would they have the info they need to recode features/apply variable labels currently?

Yes, the development version of the Python ccao package has both vars_recode and vars_rename. However, it's under active development and we haven't used it in production yet, so I wouldn't recommend this path to anyone quite yet.

…a-dictionary-in-readme

.pre-commit-config.yaml

jeancochrane · 2025-01-10T17:36:01Z

Currently blocked by ccao-data/data-architecture#704. Once that comes in I'll do a final pass at rendering the README and data dict and re-request review.

…oblems" This reverts commit 95db04c.

jeancochrane · 2025-01-10T22:36:51Z

Now blocked by ccao-data/actions#36.

This reverts commit 87ba713.

jeancochrane · 2025-01-13T15:51:33Z

Alright @dfsnow, this should finally be ready for review!

dfsnow · 2025-01-13T20:01:42Z

R/hooks/check-data-dict.R

suggestion (non-blocking): Love the simplicity of this hook! My only suggestion would be to add a call to action on failure i.e. run the README.rmd.

Good point, done in dc1ce6e!

Data dictionary must be up to date with params file.......................Failed - hook id: check-data-dict - exit code: 1 Error: Params are not present in both params.yaml and docs/data-dict.csv: foo, bar. Did you forget to reknit README.Rmd after updating params.yaml?

dfsnow · 2025-01-13T20:05:53Z

README.Rmd

-The residential model uses a variety of individual and aggregate features to determine a property's assessed value. We've tested a long list of possible features over time, including [walk score](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_walkscore.html), [crime rate](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/chicago_crimerate.html), [school districts](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_school_boundaries_mean_encoded.html), and many others. The features in the table below are the ones that made the cut. They're the right combination of easy to understand and impute, powerfully predictive, and well-behaved. Most of them are in use in the model as of `r Sys.Date()`.
+The residential model uses a variety of individual and aggregate features to determine a property's assessed value. We've tested a long list of possible features over time, including [walk score](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_walkscore.html), [crime rate](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/chicago_crimerate.html), [school districts](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_school_boundaries_mean_encoded.html), and many others. The features in the table below are the ones that made the cut. They're the right combination of easy to understand and impute, powerfully predictive, and well-behaved.
+
+For a machine-readable version of this data dictionary, see [`docs/data-dict.csv`](./docs/data-dict.csv).


suggestion (non-blocking): Let's add some callouts here to other data documentation e.g. ccao::vars_dict and the dbt catalog.

Sounds reasonable, I expanded this section in e3eca6c and moved it below the table.

dfsnow · 2025-01-13T20:13:16Z

README.Rmd

  )) %>%
  pivot_wider(
-    id_cols = `Feature Name`:`Notes`,
+    id_cols = `feature_name`:`category`,
    names_from = row,
    values_from = var_value


suggestion: We may want to use var_code here, since those are the values actually recorded in the data.

That makes sense, I went ahead and added a new column for this in b8684fb to make it clear that the encoded values are different from the semantic values. Let me know if you think there's a clearer way to express this!

dfsnow · 2025-01-13T20:15:32Z

README.Rmd

+# Write machine-readable version of the table to file
+param_tbl_fmt %>%
+  write_csv("docs/data-dict.csv")


I don't love that we've ended up with a system where we need to sync four separate things: ccao::vars_dict, params.yaml, docs/data-dict.csv, and the README. I agree this is a good simple solution for now though. Let's roll with it and worry about something better/more long-term once 2025 modeling is finished.

…e README

jeancochrane · 2025-01-13T21:50:43Z

I made some major changes based on your comments @dfsnow, mind taking one more look?

dfsnow

This looks good to me @jeancochrane! Can you put up a PR for condos as well?

jeancochrane added 2 commits January 8, 2025 18:15

Add variable names and reformat README data dictionary for better rea…

8328e9c

…dability

Add data dict to repo

65cd2eb

jeancochrane linked an issue Jan 8, 2025 that may be closed by this pull request

Missing Data Dictionary in readme #300

Closed

jeancochrane commented Jan 8, 2025

View reviewed changes

jeancochrane marked this pull request as ready for review January 8, 2025 21:48

jeancochrane requested review from dfsnow and wrridgeway as code owners January 8, 2025 21:48

Add pre-commit hook to check that the data dict is up to date with pa…

46c163b

…rams

jeancochrane added 4 commits January 9, 2025 19:58

Better error msg and dependencies in check-data-dict.R pre-commit hook

7994e4f

Merge branch '2025-assessment-year' into jeancochrane/300-missing-dat…

3f33931

…a-dictionary-in-readme

Update data dict and README to use latest feature info

5fb7b56

Merge branch '2025-assessment-year' into jeancochrane/300-missing-dat…

69d176e

…a-dictionary-in-readme

jeancochrane mentioned this pull request Jan 10, 2025

Clean up descriptions for parcel shape features so they are readable in model feature table ccao-data/data-architecture#704

Merged

Remove corner lot indicator from README and data dict

ffbc73d

jeancochrane commented Jan 10, 2025

View reviewed changes

.pre-commit-config.yaml Show resolved Hide resolved

jeancochrane marked this pull request as draft January 10, 2025 17:34

jeancochrane added 4 commits January 10, 2025 20:22

Fix descriptions for parcel shape characteristics in docs and data dict

2e300ec

Set up tmate session in pre-commit workflow to debug cache problems

95db04c

Revert "Set up tmate session in pre-commit workflow to debug cache pr…

ad157a1

…oblems" This reverts commit 95db04c.

Pin to branch of pre-commit action to test improved R caching

87ba713

jeancochrane mentioned this pull request Jan 10, 2025

Update pre-commit action to cache R additional_dependencies ccao-data/actions#36

Merged

Revert "Pin to branch of pre-commit action to test improved R caching"

7a88504

This reverts commit 87ba713.

jeancochrane marked this pull request as ready for review January 13, 2025 15:51

dfsnow approved these changes Jan 13, 2025

View reviewed changes

Add call to action to error message in check-data-dict pre-commit hook

dc1ce6e

Add links to more useful resources in the Features Used section of th…

e3eca6c

…e README

jeancochrane force-pushed the jeancochrane/300-missing-data-dictionary-in-readme branch from 551e49b to e3eca6c Compare January 13, 2025 21:24

Add encoded values to Features Used table in README and data dict

b8684fb

jeancochrane mentioned this pull request Jan 13, 2025

Streamline our assets containing information about features #324

Open

jeancochrane requested a review from dfsnow January 13, 2025 22:44

dfsnow approved these changes Jan 14, 2025

View reviewed changes

dfsnow merged commit d0cf3e0 into 2025-assessment-year Jan 14, 2025
4 checks passed

dfsnow deleted the jeancochrane/300-missing-data-dictionary-in-readme branch January 14, 2025 16:31

This was referenced Jan 14, 2025

Add a data dictionary and remove duplicate features from it ccao-data/model-condo-avm#86

Open

Add missing type column back to data dict and README feature table #327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a data dictionary #315

Add a data dictionary #315

jeancochrane commented Jan 8, 2025 •

edited

Loading

jeancochrane Jan 8, 2025 •

edited

Loading

jeancochrane Jan 9, 2025

dfsnow Jan 13, 2025

jeancochrane Jan 13, 2025

wrridgeway commented Jan 8, 2025

jeancochrane commented Jan 9, 2025

jeancochrane commented Jan 10, 2025

jeancochrane commented Jan 10, 2025

jeancochrane commented Jan 13, 2025

dfsnow Jan 13, 2025

jeancochrane Jan 13, 2025

dfsnow Jan 13, 2025

jeancochrane Jan 13, 2025

dfsnow Jan 13, 2025

jeancochrane Jan 13, 2025

dfsnow Jan 13, 2025

jeancochrane commented Jan 13, 2025

dfsnow left a comment •

edited

Loading

Add a data dictionary #315

Add a data dictionary #315

Conversation

jeancochrane commented Jan 8, 2025 • edited Loading

jeancochrane Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wrridgeway commented Jan 8, 2025

jeancochrane commented Jan 9, 2025

jeancochrane commented Jan 10, 2025

jeancochrane commented Jan 10, 2025

jeancochrane commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeancochrane commented Jan 13, 2025

dfsnow left a comment • edited Loading

Choose a reason for hiding this comment

jeancochrane commented Jan 8, 2025 •

edited

Loading

jeancochrane Jan 8, 2025 •

edited

Loading

dfsnow left a comment •

edited

Loading