Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactoring scrapers/package.py #81

Merged
merged 13 commits into from
Nov 30, 2024
Merged

Conversation

duck-master
Copy link
Contributor

This does three things:

  • Refactoring scrapers/package.py extensively by splitting off the run() function into two helper functions, namely load_json_data and merge_datasets.
  • Creating an empty overrides.json file. (Previously, in Move overrides from scrapers/package.py into a separate file, scrapers/overrides.json #76 there had been a substantial set of overrides, but since the resulting code didn't work due to further commits upstream, I abandoned it.)
  • Rewriting README.md accordingly

@duck-master
Copy link
Contributor Author

also happy thanksgiving btw :P

Copy link
Member

@psvenk psvenk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me; thanks for the PR (and sorry for breaking your previous one :/).

I've come around to agreeing with you that it would be good to split off the overrides into a separate file that is in an actual data format rather than being a Python module.

Part of me still does want to use TOML instead of JSON, though, to make the overrides file more easily human-writable. Since Python 3.11 comes with a TOML reader, I feel like TOML is mainstream enough now that we wouldn't have to worry about it being too bespoke (but we would need to use an external package to maintain compatibility with older Python versions). What are your thoughts?

Happy Thanksgiving to you too!

@psvenk psvenk merged commit c128a18 into sipb:main Nov 30, 2024
3 checks passed
@duck-master duck-master deleted the json-refactor2 branch December 3, 2024 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants