Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulk export fails when payload offloading is configured #3874

Closed
lmsurpre opened this issue Aug 11, 2022 · 2 comments
Closed

Bulk export fails when payload offloading is configured #3874

lmsurpre opened this issue Aug 11, 2022 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@lmsurpre
Copy link
Member

lmsurpre commented Aug 11, 2022

Describe the bug

[8/11/22, 17:55:29:709 UTC] 00000054 FHIRPersisten E   Unexpected error while processing token value records.
                                 java.lang.NullPointerException
	at java.base/java.util.zip.InflaterInputStream.<init>(Unknown Source)
	at java.base/java.util.zip.GZIPInputStream.<init>(Unknown Source)
	at java.base/java.util.zip.GZIPInputStream.<init>(Unknown Source)
	at org.linuxforhealth.fhir.persistence.jdbc.dao.impl.FetchResourcePayloadsDAO.run(FetchResourcePayloadsDAO.java:132)
	at org.linuxforhealth.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.fetchResourcePayloads(FHIRPersistenceJDBCImpl.java:3168)
	at org.linuxforhealth.fhir.bulkdata.jbatch.export.fast.ResourcePayloadReader.readItem(ResourcePayloadReader.java:316)
	at com.ibm.jbatch.container.artifact.proxy.ItemReaderProxy.readItem(ItemReaderProxy.java:77)
	at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.readItem(ChunkStepControllerImpl.java:329)
	at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.readAndProcess(ChunkStepControllerImpl.java:245)
	at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeChunk(ChunkStepControllerImpl.java:626)
	at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeCoreStep(ChunkStepControllerImpl.java:795)
	at com.ibm.jbatch.container.controller.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:295)
	at com.ibm.jbatch.container.controller.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:118)
	at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeCoreTransitionLoop(WorkUnitThreadControllerImpl.java:96)
	at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeWorkUnit(WorkUnitThreadControllerImpl.java:178)
	at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl$AbstractControllerHelper.runExecutionOnThread(WorkUnitThreadControllerImpl.java:503)
	at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.runExecutionOnThread(WorkUnitThreadControllerImpl.java:92)
	at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:113)
	at com.ibm.ws.context.service.serializable.ContextualRunnable.run(ContextualRunnable.java:79)
	at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:245)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

When I tried configuring it locally, I got this one (which makes more sense to me since fhir-persistence-blob is not a dependency of our fhir-bulkdata-webapp.

An error occurred while trying to load FHIR persistence factory class 'org.linuxforhealth.fhir.persistence.blob.FHIRPersistenceJDBCBlobFactory': java.lang.ClassNotFoundException: org.linuxforhealth.fhir.persistence.blob.FHIRPersistenceJDBCBlobFactory

Environment
main

To Reproduce
Steps to reproduce the behavior:

  1. start a fhir server that is configured with payload offloading (e.g. via build/pre-integration-test-docker.sh azurite)
  2. add a resource
  3. invoke export
  4. note that the export fails and check the server log

Expected behavior
Either:
A. Bulk export should work irregardless of the payload offload configuration;
B. this should be clearly documented as a limitation of the current payload offload documentation

Additional context
At the very least, I think it should be possible to fall back to our legacy "search-based" export implementation.

@lmsurpre lmsurpre added the bug Something isn't working label Aug 11, 2022
lmsurpre added a commit that referenced this issue Aug 11, 2022
A cleaner approach would be to add a method to FHIRPersistence so that
implementations can indicate whether they support fetchResourcePayloads
which is what we use for the fast export. Then we could pass that into
BulkDataClient.submitExport where we decide which export implementation
to use.

However, ideally we'd just add this support...then we probably wouldn't
need those changes.

Therefor, for now, I opted for the quick and dirty fix.

Signed-off-by: Lee Surprenant <lmsurpre@merative.com>
lmsurpre added a commit that referenced this issue Aug 12, 2022
issue #3874 - use legacy export when payload offloading is configured
@lmsurpre
Copy link
Member Author

I took the easy way out and just had it fall back to our search-based export.
I considered opening a new "tech debt" issue for properly supporting fast-export, but after talking with robin we think export should be totally different in case of offloading because all the resources are already external.

@lmsurpre lmsurpre self-assigned this Aug 12, 2022
@lmsurpre
Copy link
Member Author

I worked with Prasanna to verify this one. We followed the steps from the "To Reproduce" section and we successfully exported some patients to azurite. We execed onto the azurite container and saw what looked like the expected output. Seems to be working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant