bulkdata patient/group exports are slow

[albertwang-ibm]

currently, the codes get the patients first, and then do compartment search for the patients one by one on for the required resource type, so this means for each page of patients(e.g, 200), 200 search queries have to be made to the database to get all those required resources for the patients.

we should consider to enhance the search API to allow it to be able to get compartment resources for multiple patients in one request, this will greatly improve the performance of patient/group exports.

[prb112]

could you use an _include? like this... but this one is EOB "_include", "ExplanationOfBenefit:patient"

[albertwang-ibm]

not really, this is different, this is to get patient resources for the found ExplanationOfBenefits whose subject is the patients.

[albertwang-ibm]

what we need is getting ExplanationOfBenefits for all found patients.

[albertwang-ibm]

maybe can use _revinclude ...

[prb112]

Right. It was just an example, you can flip it around

[albertwang-ibm]

actually, I'm thinking that using _revinclude for the patient search could cause huge join which may also cause bad performance ... maybe we can achieve better performance by passing multiple id to compartment search instead of one and use "in" instead of "=" in the sql layer, this hopefully can help us to achieve better performance and avoid the potential time out issue. e.g, if we want to export all patients with all their ExplainationOfBenefits, then this will cause huge join in the db side which could cause db timeout that we have seen before ...

[albertwang-ibm]

and seems _revinclude for patients may not work for group export ... because we don't have something like \patient?groupid="xxxxx"

[albertwang-ibm]

ideally, we could have api like \patient?groupid="xxxxx", not matter if the group is members based or characteritics based. then we can use _revinclude based on it ....