EES-4950 Re-model data set version change sets #5089
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
src/GovUk.Education.ExploreEducationStatistics.Common/Comparers/ListValueComparer.cs
Outdated
Show resolved
Hide resolved
...GovUk.Education.ExploreEducationStatistics.Common/Converters/EnumToEnumValueListConverter.cs
Show resolved
Hide resolved
...oreEducationStatistics.Public.Data.Model.Tests/Fixtures/DataSetVersionGeneratorExtensions.cs
Outdated
Show resolved
Hide resolved
src/GovUk.Education.ExploreEducationStatistics.Public.Data.Model/Change.cs
Show resolved
Hide resolved
src/GovUk.Education.ExploreEducationStatistics.Public.Data.Model/Change.cs
Outdated
Show resolved
Hide resolved
src/GovUk.Education.ExploreEducationStatistics.Public.Data.Model/Change.cs
Show resolved
Hide resolved
ntsim
force-pushed
the
ees-4950
branch
2 times, most recently
from
ec5a882 to
6908f05
Compare
duncan-at-hiveit
approved these changes
Jul 25, 2024
This re-models the changelog data models by: - Removing `ChangeSets` - Removing JSONB columns in favour of a standard relational model using foreign keys (i.e. previous / current states). We've moved towards this approach as it allows us to avoid duplicating the entire meta domain in JSONB columns (and respective models). Using normal foreign key references means we can keep things relatively simple. The only issue is that we don't know if this approach will perform well at scale, but we believe it'd be possibel to remedy this with blob caching (or similar). EES-4921 Test filter / location option changes using owned entities
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR re-models changelogs for data set versions (i.e. v1.1+) towards a more traditional normalised schema (instead of using JSONB). This provides the usual benefits of normalisation (referential integrity, de-duplication) and produces less overhead in terms of the code that needs to be written.
The downside is that larger changelogs can have many changes, potentially resulting in poorer performance than de-normalised JSONB (due to joining across potentially hundreds / thousands of rows).
After a little discussion with @duncan-at-hiveit, we agreed on balance, the benefits of the normalised approach outweighed the downsides. Additionally, any potential performance issues are addressable via caching (e.g. blob cache) of the entire changelog.