Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Truncate String parameter value before persisting to the database #4021

Closed
PrasannaHegde1 opened this issue Oct 13, 2022 · 4 comments
Closed

Truncate String parameter value before persisting to the database #4021

PrasannaHegde1 opened this issue Oct 13, 2022 · 4 comments
Assignees
@PrasannaHegde1
Labels
bug Something isn't working P1 Priority 1 - Must Have

Comments

@PrasannaHegde1
Copy link
Collaborator

Describe the bug
The string parameter value should be truncated before persisting to the database.
The size of string columns in the search tables(_str_values) is limited to 1024 bytes.

Environment
Which version of LinuxForHealth FHIR Server? 5.0

To Reproduce

  1. Go to R4ExampleServerTest.main() method
  2. Pass "json/spec/medicationstatement.profile.json" as the JSON example in driver.processExample(...) .
  3. Run the main method.
  4. See error
[13/10/22, 14:10:42:079 UTC] 00000034 ParameterTran E   Failed persisting parameter transaction data. Marking transaction for rollback
                                 org.linuxforhealth.fhir.persistence.exception.FHIRPersistenceException: pushBatch failed for 'StructureDefinition'  [probeId=c0-a8-1-f-c4963745-837e-4ee4-aba7-5a8e9c31ad09]
	at org.linuxforhealth.fhir.persistence.params.database.PlainBatchParameterProcessor.pushBatch(PlainBatchParameterProcessor.java:117)
	at org.linuxforhealth.fhir.persistence.params.database.PlainParamValueProcessor.pushBatch(PlainParamValueProcessor.java:110)
	at org.linuxforhealth.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.flush(FHIRPersistenceJDBCImpl.java:3124)
	at org.linuxforhealth.fhir.persistence.jdbc.impl.FHIRPersistenceJDBCImpl.onCommit(FHIRPersistenceJDBCImpl.java:3155)
	at org.linuxforhealth.fhir.persistence.jdbc.impl.ParameterTransactionDataImpl.persist(ParameterTransactionDataImpl.java:70)
	at org.linuxforhealth.fhir.persistence.jdbc.dao.impl.TransactionDataImpl.lambda$persist$0(TransactionDataImpl.java:46)
	at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
	at org.linuxforhealth.fhir.persistence.jdbc.dao.impl.TransactionDataImpl.persist(TransactionDataImpl.java:46)
	at org.linuxforhealth.fhir.persistence.jdbc.impl.CacheTransactionSync.beforeCompletion(CacheTransactionSync.java:74)
	at com.ibm.tx.jta.impl.RegisteredSyncs.coreDistributeBefore(RegisteredSyncs.java:238)
	at com.ibm.tx.jta.impl.RegisteredSyncs.distributeBefore(RegisteredSyncs.java:146)
	at com.ibm.tx.jta.impl.TransactionImpl.prePrepare(TransactionImpl.java:1404)
	at com.ibm.tx.jta.impl.TransactionImpl.stage1CommitProcessing(TransactionImpl.java:814)
	at com.ibm.tx.jta.impl.TransactionImpl.processCommit(TransactionImpl.java:770)
	at com.ibm.tx.jta.impl.TransactionImpl.commit(TransactionImpl.java:713)
	at com.ibm.tx.jta.impl.TranManagerImpl.commit(TranManagerImpl.java:153)
	at com.ibm.tx.jta.impl.TranManagerSet.commit(TranManagerSet.java:110)
	at com.ibm.tx.jta.impl.UserTransactionImpl.commit(UserTransactionImpl.java:145)
	at com.ibm.tx.jta.embeddable.impl.EmbeddableUserTransactionImpl.commit(EmbeddableUserTransactionImpl.java:101)
	at com.ibm.ws.transaction.services.UserTransactionService.commit(UserTransactionService.java:72)
	at org.linuxforhealth.fhir.persistence.jdbc.connection.FHIRUserTransactionAdapter.end(FHIRUserTransactionAdapter.java:140)
	at org.linuxforhealth.fhir.persistence.helper.FHIRTransactionHelper.commit(FHIRTransactionHelper.java:43)
	at org.linuxforhealth.fhir.server.util.FHIRRestHelper.doCreate(FHIRRestHelper.java:254)
	at org.linuxforhealth.fhir.server.spi.operation.FHIRResourceHelpers.doCreate(FHIRResourceHelpers.java:117)
	at org.linuxforhealth.fhir.server.resources.Create.create(Create.java:76)
	at org.linuxforhealth.fhir.server.resources.Create$Proxy$_$$_WeldClientProxy.create(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor874.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.ibm.ws.jaxrs20.cdi.component.JaxRsFactoryImplicitBeanCDICustomizer.serviceInvoke(JaxRsFactoryImplicitBeanCDICustomizer.java:350)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:641)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:101)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:273)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:213)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:444)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:112)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:277)
	at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:137)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:146)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.doPost(IBMRestServlet.java:104)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:96)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1258)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:746)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:443)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:193)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:98)
	at org.linuxforhealth.fhir.server.filter.rest.FHIRRestServletFilter.doFilter(FHIRRestServletFilter.java:177)
	at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:127)
	at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
	at com.ibm.ws.security.jaspi.JaspiServletFilter.doFilter(JaspiServletFilter.java:56)
	at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:201)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:1002)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1140)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5058)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:316)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1007)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:281)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1239)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:468)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:427)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:566)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:500)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:360)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:327)
	at com.ibm.ws.channel.ssl.internal.SSLConnectionLink.determineNextChannel(SSLConnectionLink.java:1100)
	at com.ibm.ws.channel.ssl.internal.SSLConnectionLink$MyReadCompletedCallback.complete(SSLConnectionLink.java:675)
	at com.ibm.ws.channel.ssl.internal.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1824)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:514)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:584)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:968)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1057)
	at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:245)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.BatchUpdateException: A truncation error was encountered trying to shri[13/10/22, 14:10:42:102 UTC] 00000034 TransactionDa 1   persisted transaction data [took 8.944 s]
nk VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
	at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(EmbedStatement.java:1113)
	at org.apache.derby.impl.jdbc.EmbedStatement.executeBatch(EmbedStatement.java:1045)
	at org.apache.derby.iapi.jdbc.BrokeredStatement.executeBatch(BrokeredStatement.java:84)
	at com.ibm.ws.rsadapter.jdbc.WSJdbcStatement.executeBatch(WSJdbcStatement.java:495)
	at org.linuxforhealth.fhir.persistence.params.database.PlainPostgresParameterBatch.pushBatch(PlainPostgresParameterBatch.java:80)
	at org.linuxforhealth.fhir.persistence.params.database.PlainBatchParameterProcessor.pushBatch(PlainBatchParameterProcessor.java:115)
	... 81 more
Caused by: java.sql.SQLDataException: A truncation error was encountered trying to shrink VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:84)
	at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Util.java:230)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(TransactionResourceImpl.java:431)
	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(TransactionResourceImpl.java:360)
	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(EmbedConnection.java:2405)
	at org.apache.derby.impl.jdbc.ConnectionChild.handleException(ConnectionChild.java:88)
	at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1436)
	at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(EmbedPreparedStatement.java:1065)
	at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(EmbedStatement.java:1087)
	... 86 more
Caused by: ERROR 22001: A truncation error was encountered trying to shrink VARCHAR 'A record of a medication that is being consumed by a patient&' to length 1024.
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:300)
	at org.apache.derby.shared.common.error.StandardException.newException(StandardException.java:295)
	at org.apache.derby.iapi.types.SQLChar.hasNonBlankChars(SQLChar.java:1846)
	at org.apache.derby.iapi.types.SQLVarchar.normalize(SQLVarchar.java:181)
	at org.apache.derby.iapi.types.SQLVarchar.normalize(SQLVarchar.java:158)
	at org.apache.derby.iapi.types.DataTypeDescriptor.normalize(DataTypeDescriptor.java:656)
	at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeColumn(NormalizeResultSet.java:332)
	at org.apache.derby.impl.sql.execute.NormalizeResultSet.normalizeRow(NormalizeResultSet.java:376)
	at org.apache.derby.impl.sql.execute.NormalizeResultSet.getNextRowCore(NormalizeResultSet.java:191)
	at org.apache.derby.impl.sql.execute.DMLWriteResultSet.getNextRowCore(DMLWriteResultSet.java:148)
	at org.apache.derby.impl.sql.execute.InsertResultSet.getNextRowCore(InsertResultSet.java:1082)
	at org.apache.derby.impl.sql.execute.InsertResultSet.open(InsertResultSet.java:451)
	at org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(GenericPreparedStatement.java:472)
	at org.apache.derby.impl.sql.GenericPreparedStatement.execute(GenericPreparedStatement.java:351)
	at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(EmbedStatement.java:1344)
	... 88 more

Expected behavior
The string parameter value should be truncated before persisting to the database.

Additional context
This can probably be done in SearchParametersTransportAdapter.stringValue() method.
@PrasannaHegde1 PrasannaHegde1 added bug Something isn't working P1 Priority 1 - Must Have labels Oct 13, 2022
@punktilious
Copy link
Collaborator

This is a regression/change in behavior from 4.11.1. Previously, string search parameter values were truncated in ParameterVisitorBatchDAO:

    public void visit(StringParmVal param) throws FHIRPersistenceException {

This class is no longer present in 5.0.0 after the parameter persistence was refactored.

@PrasannaHegde1 PrasannaHegde1 self-assigned this Oct 13, 2022
This was referenced Oct 13, 2022
Closed
Merged
@lmsurpre
Copy link
Member

lmsurpre commented Oct 20, 2022 •
edited
Loading

#4024 fixed our tests so that we actually exercise the insert parameters logic as part of the persistence tests. After that change, the CI pipeline was properly failing due to this issue.

After merging Prasanna's fix, that pull request is now passing which helps to confirm that this is working as expected now.

@lmsurpre
Copy link
Member

lmsurpre commented Nov 8, 2022 •
edited
Loading

I created a sample test patient with a name like the following:

"name": [
  {
    "family": "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz WAS IT TRUNCATED !?",
    "given": [
      "Lee"
    ]
  }
],

I then confirmed that I am able to search this patient by

  • Patient?name=Lee
  • Patient?name=abcdefg
  • Patient?name:contains=hijklmnop

Further, I verified that I cannot search it by

  • Patient?name:contains=TRUNCATED

I think it would be nice to provide a warning back to the user if they submit a searchable string value that has been truncated on the server, but that is separate from this issue so lets call this one done.

@lmsurpre lmsurpre closed this as completed Nov 8, 2022
@lmsurpre
Copy link
Member

lmsurpre commented Nov 9, 2022

linking #1638 which is somewhat related to that comment about providing a warning in the case of truncation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@PrasannaHegde1 PrasannaHegde1

Labels
bug Something isn't working P1 Priority 1 - Must Have
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lmsurpre @punktilious @PrasannaHegde1