-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Validation requests have a maximum size ~20 MB #193
Comments
I have no objection to switching to gson if it gets us over this. I have no loyalty to Jackson. There are annotations that you can use to override the default field names in gson. I can't recall what they are right now, but I did exactly that ages ago in another project... |
Probably this: https://howtodoinjava.com/gson/gson-serializedname/ I will give that a shot. Jackson and Gson can live in harmony with regard to annotations, if I remember correctly, so we can keep both in case we need to jump back for some reason. |
What's the limit on Gson with regard to size? If it's unlimited, that's asking for DOS attacks. Better we allow a configuration for something reasonable, both here, and on the raw request. |
I don't think Gson imposes any size limits of its own. org.hl7.fhir.validator-wrapper/src/jvmMain/kotlin/controller/validation/ValidationModule.kt Line 27 in 8823e1c
and then it just returns HTTP 400, no body or anything explaining what's wrong. |
@dehall I've got a working configuration for gson, which I think is the first step. It works for me, manually and via the smoke tests I've included here: https://github.com/hapifhir/org.hl7.fhir.validator-wrapper/tree/master/http-client-tests Could you take a look at it, and ensure that it works for you as well? You'll have to use the following PRs for core and for the validator-wrapper: |
Closing this as completed by #195 |
We're seeing validation requests larger than ~20 MB fail with an HTTP 400 error. These kinds of requests are uncommon but they can happen with large Bundles. In particular on Inferno we saw this reported for our Service Base URL test kit where the publication Bundle can be very large: https://chat.fhir.org/#narrow/channel/179309-inferno/topic/Inferno.20Service.20Base.20URL.20Test.20Kit.20-.20Validator.20Error.3F
It appears that the issue is with the Jackson JSON parser, which by default has a limit of 20,000,000 characters - see FasterXML/jackson-core#1014
In that thread is a suggestion on how to configure the maximum length, which we can kind of get at via the koin config in
src/jvmMain/kotlin/Module.kt
: (though note thejsonFactory
call there is deprecated)An alternative would be to switch to Gson as the json handler, which theoretically would be as easy as the below change (also in Module.kt), since the requisite libraries are already defined in build.gradle. This change also enabled validation of the nearly 50 MB file noted in that zulip thread but it breaks the API unfortunately - this field in ValidationOutcome is serialized as "issues" by Jackson based on the annotation but as "messages" by Gson based on the field name.
https://github.com/hapifhir/org.hl7.fhir.core/blob/master/org.hl7.fhir.validation/src/main/java/org/hl7/fhir/validation/cli/model/ValidationOutcome.java#L14
The text was updated successfully, but these errors were encountered: