Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added JSONDecodeError for rapidjson >= 0.7.1 with fallback to ValueError. #199

Merged
merged 7 commits into from
Jul 3, 2019

Conversation

bdice
Copy link
Member

@bdice bdice commented May 26, 2019

Motivation and Context

Resolves #172.

Types of Changes

  • Bug fix

Checklist:

If necessary:

@bdice bdice requested review from csadorf and a team as code owners May 26, 2019 19:26
# Defined for rapidjson >= 0.7.1
from rapidjson import JSONDecodeError
except ImportError:
JSONDecodeError = ValueError
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before 0.7.1, the error thrown was a ValueError. This should ensure that the error in contrib.filterparse still gets caught.

@bdice
Copy link
Member Author

bdice commented May 26, 2019

I'd like to add a step to CI that runs tests with rapidjson if we're committed to supporting it as a primary JSON backend, going forward.

Edit: oh wait, we are testing this already. I will try to add a test that fails without this PR.

@codecov
Copy link

codecov bot commented May 26, 2019

Codecov Report

Merging #199 into master will decrease coverage by 0.86%.
The diff coverage is 71.42%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
- Coverage    64.9%   64.04%   -0.87%     
==========================================
  Files          37       37              
  Lines        5557     5562       +5     
==========================================
- Hits         3607     3562      -45     
- Misses       1950     2000      +50
Impacted Files Coverage Δ
signac/contrib/filterparse.py 76.19% <0%> (ø) ⬆️
signac/core/json.py 81.39% <83.33%> (-13.35%) ⬇️
signac/common/errors.py 72.22% <0%> (-11.12%) ⬇️
signac/contrib/job.py 86.11% <0%> (-4.37%) ⬇️
signac/contrib/indexing.py 73.77% <0%> (-1.48%) ⬇️
signac/contrib/collection.py 87.59% <0%> (-1.32%) ⬇️
signac/core/jsondict.py 93.86% <0%> (-1.23%) ⬇️
signac/core/dict_manager.py 88.37% <0%> (-1.17%) ⬇️
signac/contrib/project.py 89.58% <0%> (-1.05%) ⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bbf2b74...681b483. Read the comment docs.

@codecov
Copy link

codecov bot commented May 26, 2019

Codecov Report

Merging #199 into master will increase coverage by 0.08%.
The diff coverage is 80%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
+ Coverage    64.9%   64.99%   +0.08%     
==========================================
  Files          37       37              
  Lines        5557     5565       +8     
==========================================
+ Hits         3607     3617      +10     
+ Misses       1950     1948       -2
Impacted Files Coverage Δ
signac/contrib/filterparse.py 82.53% <100%> (+6.34%) ⬆️
signac/core/json.py 91.3% <77.77%> (-3.44%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01fc155...23cef1b. Read the comment docs.

@bdice bdice self-assigned this May 28, 2019
@bdice bdice added the bug Something isn't working label May 28, 2019
@bdice bdice added this to the v1.1.1 milestone May 28, 2019
@vyasr
Copy link
Contributor

vyasr commented May 28, 2019

fwiw I think when we last discussed this we came down on not knowing whether rapidjson was really the right JSON backend to choose, but we needed to do a more comprehensive survey of libraries to know what to support and we hadn't assigned that to anyone.

@vyasr
Copy link
Contributor

vyasr commented May 28, 2019

I'm not sure I follow exactly what additional test you want to add, but this PR looks good to me right now. One minor suggestion, for redirecting stderr you might want to just copy over the functions we use in testing signac-flow.

@bdice
Copy link
Member Author

bdice commented May 28, 2019

fwiw I think when we last discussed this we came down on not knowing whether rapidjson was really the right JSON backend to choose, but we needed to do a more comprehensive survey of libraries to know what to support and we hadn't assigned that to anyone.

@vyasr I did some reading on that subject while working on this PR. Seeing this comment on rapidjson's issue tracker suggests to me that we would be better served by ujson or simplejson or another alternative to rapidjson that provides direct support for the standard library's json API.

@bdice
Copy link
Member Author

bdice commented May 28, 2019

I'm not sure I follow exactly what additional test you want to add, but this PR looks good to me right now. One minor suggestion, for redirecting stderr you might want to just copy over the functions we use in testing signac-flow.

@vyasr I added the test already, so that's done. I also made the change you suggested to use signac-flow's redirection.

I think this PR is ready to merge.

@vyasr
Copy link
Contributor

vyasr commented May 29, 2019

I agree, should be ready for merge.

And regarding choice of library, to my knowledge ujson provides the best balance of compatibility with the Python library's json module and performance, but I haven't done exhaustive research into that. We might just need to make a quick test where replace the code in signac's json module with the relevant import and see how performance improves/degrades. Actually changing which modules we support should not be too hard once we know the correct solution that provides the closest mirror to the existing functionality we expect.

@vyasr
Copy link
Contributor

vyasr commented May 29, 2019

I just went to look at issues and found #171 which I made :D good to know we're rehashing and coming to the same conclusions, it looks like your benchmarks agree with ujson as the best alternative. @csadorf if you want to assign that issue to a particular benchmark we can assign it to someone to work on.

@csadorf csadorf merged commit 48ab755 into master Jul 3, 2019
@csadorf csadorf deleted the fix/issue-172 branch July 3, 2019 17:39
@csadorf csadorf modified the milestones: v1.1.1, v1.2.0 Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug in JSON error checking
3 participants