Change warehouse raw data endpoint to respond with JSON text sequence. #1858
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aaronweeden
added
Category: Data Analytics Framework
php
aaronweeden
force-pushed
the
raw-data-json-seq
branch
from
4daf0c2
to
6190107
Compare
8 tasks
aaronweeden
force-pushed
the
raw-data-json-seq
branch
from
b4b8544
to
2af3dc8
Compare
aaronweeden
force-pushed
the
raw-data-json-seq
branch
from
2af3dc8
to
93ae936
Compare
eiffel777
approved these changes
May 29, 2024
jpwhite4
approved these changes
May 29, 2024
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR changes the
rest/warehouse/raw-dataendpoint to respond with a JSON text sequence rather than one large JSON object. Each row of raw data in the response is now its own JSON array.Motivation and Context
This is to prevent a bug in which, if the
xdmod-dataPython API was used to fetch a large number of rows (e.g., 5 months of jobs data),json.loads()would fail to decode the response text as JSON due to running out of memory. With the change in this PR,json.loads()can be called on just a single row of raw data at a time.xdmod-datais updated to handle this change in ubccr/xdmod-data#19.Tests performed
I updated my port on
xdmod-devwith the changes from this PR and tested using thexdmod-dataPython API with the changes from ubccr/xdmod-data#19 to fetch raw data.Checklist: