Reference within a bundle could not be resolved with entry request conditional update

[qing-xia-snkeos]

Describe the bug
References within a transaction bundle could not be resolved, when:

• "fullUrl" of the referred resource is "urn:uuid:[id]" and "reference" field is the same "urn:uuid:[id]"
• "request" is conditional update, search parameter is "identifier"

Environment
Which version of IBM FHIR Server?
4.9.0

To Reproduce
Steps to reproduce the behavior:

1. Upload the following bundle to base URL, e.g. "https://localhost:9443/fhir-server/api/v4/"

{
    "resourceType":"Bundle",
    "type":"transaction",
    "entry":[
       {
          "fullUrl":"urn:uuid:patient",
          "resource":{
             "resourceType":"Patient",
             "text":{
                "status":"generated",
                "div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Test Patient</div>"
             },
             "identifier":[
                {
                   "system":"https://github.com/synthetichealth/synthea",
                   "value":"e531c09f-6887-6aba-af17-cbc521900b87"
                }
             ],
             "name":[
                {
                   "use":"official",
                   "family":"Rath779",
                   "given":[
                      "Iliana226"
                   ],
                   "prefix":[
                      "Ms."
                   ]
                }
             ],
             "gender":"female",
             "birthDate":"1992-08-02"
          },
          "request":{
             "method":"PUT",
             "url":"Patient?identifier=https://github.com/synthetichealth/synthea|e531c09f-6887-6aba-af17-cbc521900b87"
          }
       },
       {
          "fullUrl":"urn:uuid:observation",
          "resource":{
             "resourceType":"Observation",
             "text":{
                "status":"generated",
                "div":"<div xmlns=\"http://www.w3.org/1999/xhtml\">Test Observation</div>"
             },
             "identifier":[
                {
                   "use":"official",
                   "system":"http://www.bmc.nl/zorgportal/identifiers/observations",
                   "value":"0516"
                }
             ],
             "status":"final",
             "code":{
                "coding":[
                   {
                      "system":"http://loinc.org",
                      "code":"55233-1",
                      "display":"Genetic analysis master panel-- This is the parent OBR for the panel holding all of the associated observations that can be reported with a molecular genetics analysis result."
                   }
                ]
             },
             "subject":{
                "reference":"urn:uuid:patient",
                "display":"Ms. Iliana226 Rath779"
             }
          },
          "request":{
             "method":"PUT",
             "url":"Observation?identifier=http://www.bmc.nl/zorgportal/identifiers/observations|0516"
          }
       }
    ]
 }

2. Get the uploaded Observation and check the "subject" field

Expected behavior
After uploading the bundle, the reference field in Observation.subject is expected to be resolved, e.g. updated by the new location of the Patient, but it stays like "urn:uuid:[id]".

Additional context
The following requests are also tested:

• upload a bundle with a QuestionnaireResponse and a Patient (the QuestionnaireResponse is referring the Patient as "subject"), using the same kind of reference, it works perfectly fine. The reference in QuestionnaireResponse then will be updated by "Patient/[new_id]".
• upload a bundle with other types of resources referring a Patient, e.g. Practitioner, Encounter, DiagnosticReport.
• upload a bundle with a QuestionnaireResponse referring a Patient, a Observation referring a Patient, then only the reference in QuestionnaireResponse could be resolved.

In the log of IBM FHIR server, a successful case and a failed case have different order of creating these two resources are different:

• in the bundle with a QuestionnaireResponse and a Patient (the QuestionnaireResponse is referring the Patient as "subject"), the QuestionnaireResponse is created after the Patient, then the references are successfully resolved
• in the bundle with a Observation and a Patient (the Observation is referring the Patient as "subject"), the Observation is created before the Patient, then the references are wrongly resolved

I suppose it is expected: first the patient is created, then the Observation should be created, so that the reference could be resolved.

[lmsurpre]

So the first update is a conditional update Patient?identifier=https://github.com/synthetichealth/synthea|e531c09f-6887-6aba-af17-cbc521900b87.

So there's actually 5 cases there:

No matches, no id provided: The server creates the resource.
No matches, id provided: The server treats the interaction as an Update as Create interaction (or rejects it, if it does not support Update as Create)
One Match, no resource id provided OR (resource id provided and it matches the found resource): The server performs the update against the matching resource
One Match, resource id provided but does not match resource found: The server returns a 400 Bad Request error indicating the client id specification was a problem preferably with an OperationOutcome
Multiple matches: The server returns a 412 Precondition Failed error indicating the client's criteria were not selective enough preferably with an OperationOutcome

Which of these are working / aren't working with the local reference replacement stuff?

I think what you're saying is in the case of One Match, no resource id provided, you're looking for us to replace the "reference":"urn:uuid:patient" in the Observation with the id that we get from the result of the conditional update in the first entry.
That is pretty advanced and I don't think I'd call that a bug on any server that doesn't support it.

Would you mind trying something on your end?
See what happens if you make the reference from Observation.subject a conditional reference instead of a bundle-local reference.
See Conditional References under https://www.hl7.org/fhir/http.html#transaction for more info.
Please give it a shot and let us know how it goes!

[qing-xia-snkeos]

Thank you for the quick response!

First question:

Which of these are working / aren't working with the local reference replacement stuff?

What I tested was the first case: "No matches, no id provided: The server creates the resource." The resources were created, but unfortunately partially with the wrong reference. Actually all the tests mentioned were executed to a fresh empty server.
I chose the label "bug", as the references were partially resolved - I thought the server might have the ability to completely resolve it.

Second question:

See what happens if you make the reference from Observation.subject a conditional reference instead of a bundle-local reference

I tried the conditional reference in the Observation.subject field, too. This works if "One Match, no resource id provided". But if there is no match, the error message is "Error resolving conditional reference: search returned no results"

[lmsurpre]

Thanks for the additional info. Of the five cases in conditional update, I assume our behavior is right on the last two failure cases (should fail the transaction and return an OperationOutcome).

I do agree that it would be nice if the server could handle all of the 3 success cases of conditional update together with our bundle-local reference replacement feature, so thank you for opening this issue.

Until we support such functionality, I think the workaround would be to perform the search from the client before invoking the transaction bundle request. If the resources exist, you can do a normal update in the bundle. If they don't, then you can do a normal create in the bundle.
It doesn't sound ideal, but its actually not much different than our current conditional update implementation anyway...in either case there is a chance that someone sneaks in and creates a resource with the same identifier in between the search and the create. See #2051 for more info on that and #2050 on one idea we had to improve the situation. Please let us know if you think that one could help here (or not).

[punktilious]

In #1869 all the ids and references will be resolved prior to any persistence operations (we do a prepare phase followed by a persist phase). This will probably need to be expanded to properly handle the issue outlined here. I'll take a look at this once the big refactor related to #1869 is working.

[punktilious]

We already have a test case FHIRRestHelperTest#testTransactionBundlePostWithBackwardsConditionalDependency which replicates this. The test will need to be updated if we change the bundle processing behavior.

[michaelwschroeder]

Running with the latest codebase on my local FHIR server, I validated the following 5 conditional update cases:

No matches, no id provided: The server creates the resource.
No matches, id provided: The server treats the interaction as an Update as Create interaction (or rejects it, if it does not support Update as Create)
One Match, no resource id provided OR (resource id provided and it matches the found resource): The server performs the update against the matching resource
One Match, resource id provided but does not match resource found: The server returns a 400 Bad Request error indicating the client id specification was a problem preferably with an OperationOutcome
Multiple matches: The server returns a 412 Precondition Failed error indicating the client's criteria were not selective enough preferably with an OperationOutcome

All ran successfully, returning the expected results. For the first case I used the bundle data provided in the issue description, running with the Patient resource first in the bundle, and running with the Observation resource first in the bundle. I also ran this case with a QuestionnaireResponse/Patient combination and an Encounter/Patient combination.