Simplify organisation lookups -- DO NOT MERGE UNTIL PASSING TESTS #89
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We envisaged collections having pipeline specific code in separate collections plulled in as a plugin. this only proved useful for brownfield-land, and the organisation lookup and date fields that plugin provided can be replaced by core config in the collector which we plan to centralise.
This is because resources with an end date < today will not be present on file system
…nd-start-date-in-resourcecsv-data Port integration test for collection_save_csv from digital-land-airflow
…to 2283-resolve-missing-bytes-and-start-date-in-resourcecsv-data-implementation
…nd-start-date-in-resourcecsv-data-implementation Resolve missing bytes in resource.csv
Don't split references with a space after the colon
Prompted by not processing flipped northings and eastings
We envisaged collections having pipeline specific code in separate collections plulled in as a plugin. this only proved useful for brownfield-land, and the organisation lookup and date fields that plugin provided can be replaced by core config in the collector which we plan to centralise.
…nd/digital-land-python into simplify-organisation-lookups
|
Updated the branch. We need to remove the listed-building data and construct better acceptance tests as that collection is being restructured and their size makes it almost impossible to spot why they're breaking. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is quite a large refactoring of the harmonise phase, and removal of the plugin architecture which isn't needed as we centralise our processing.
At this point the integration tests are failing because the pipeline now provides the organisation-entity, not the assembling of the dataset, and the test data needs updating to reflect that change.