Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Purge more bloat from registry.json #2564

Merged
merged 2 commits into from
Nov 4, 2018

Conversation

HebaruSan
Copy link
Member

@HebaruSan HebaruSan commented Nov 2, 2018

Motivation

The registry.json file nowadays is about 20 MB, 10% of which is repetitions of the following lines:

Line Occurrences
"ksp_version_strict": false, 14817
"download_content_type": "application/zip", 14764
"find_matches_files": false, 13504
"install_to": "GameData" (without a comma) 8378
"install_to": "GameData", (with a comma) 1725

A further 3.5 MB is tab characters indenting the JSON. While it can be useful to inspect this file from time to time, its primary function is to be read by code, so this formatting is not necessary.

This takes up disk space and increases the time to save and load the registry.

Changes

Now the above values of the above properties are treated as the default; when the property has that value, it will be left out of the serialization, and if the property is missing when de-serializing, it will be set to the default. There will be no change in runtime behavior because the de-serializer will still set the properties as needed.

Similarly, the indenting is removed (but the line breaks remain).

This reduces the size of registry.json by about 5.5 MB, similar to #2179.

-rw-r--r-- 1 20846338 Nov  2 18:42  OLD-registry.json
-rw-r--r-- 1 15320905 Nov  2 19:02  NEW-registry.json

$ echo $((20846338-15320905))
5525433

@HebaruSan HebaruSan added Enhancement New features or functionality Easy This is easy to fix Core (ckan.dll) Issues affecting the core part of CKAN Pull request Registry Issues affecting the registry labels Nov 2, 2018
@Olympic1
Copy link
Member

Olympic1 commented Nov 3, 2018

We can save another 5MB if we remove the indentation of the json.

Downside is that it will be very hard to read as everything is set on 1 line.

Code that indents the json:

writer.Formatting = Formatting.Indented;
writer.Indentation = 1;
writer.IndentChar = '\t';

@Olympic1 Olympic1 closed this Nov 3, 2018
@Olympic1 Olympic1 reopened this Nov 3, 2018
@HebaruSan
Copy link
Member Author

Hmm, it's not too bad if we keep the line breaks:

image

@HebaruSan HebaruSan changed the title Purge 2 more MB of bloat from registry.json Purge more bloat from registry.json Nov 3, 2018
@Olympic1
Copy link
Member

Olympic1 commented Nov 4, 2018

Doesn't look bad. We can always remove the identation completely if the file grows too big again.

@Olympic1 Olympic1 merged commit eed7b6b into KSP-CKAN:master Nov 4, 2018
Olympic1 added a commit that referenced this pull request Nov 4, 2018
@HebaruSan HebaruSan deleted the feature/registry-defaulting branch November 4, 2018 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core (ckan.dll) Issues affecting the core part of CKAN Easy This is easy to fix Enhancement New features or functionality Registry Issues affecting the registry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants