Skip to content

Commit

Permalink
Merge pull request #1740 from IBM/robin-proto
Browse files Browse the repository at this point in the history 
issue #1708 store compartment search parameters for faster compartment search queries
  • Loading branch information
@punktilious
punktilious authored Nov 23, 2020
2 parents bdc34dd + 92bd1a7 commit a02a4d2
Show file tree
Hide file tree
Showing 35 changed files with 2,091 additions and 472 deletions.
36 changes: 26 additions & 10 deletions docs/src/pages/guides/FHIRServerUsersGuide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ This FHIR server is intended to be a common component for providing FHIR capabil
## 2.1 Installing a new server
0. Prereqs: The IBM FHIR Server requires Java 8 or higher and has been tested with OpenJDK 8, OpenJDK 11, and the IBM SDK, Java Technology Edition, Version 8. To install Java on your system, we recommend downloading and installing OpenJDK 8 from https://adoptopenjdk.net/.

1. To install the FHIR server, build or download the `fhir-install` zip installer (e.g. `fhir-server-distribution.zip` or `fhir-install-4.0.0-rc1-20191014-1610`).
1. To install the IBM FHIR Server, build or download the `fhir-install` zip installer (e.g. `fhir-server-distribution.zip` or `fhir-install-4.0.0-rc1-20191014-1610`).
The Maven build creates the zip package under `fhir-install/target`. Alternatively, releases will be made available from the [Releases tab](https://github.com/ibm/fhir/releases).

2. Unzip the `.zip` package into a clean directory (referred to as `fhir-installer` here):
Expand All @@ -55,17 +55,17 @@ The Maven build creates the zip package under `fhir-install/target`. Alternative
unzip fhir-server-distribution.zip
```
3. Determine an install location for the OpenLiberty server and the FHIR server webapp. Example: `/opt/ibm/fhir-server`
3. Determine an install location for the OpenLiberty server and the IBM FHIR Server webapp. Example: `/opt/ibm/fhir-server`
4. Run the `install.sh/.bat` script to install the server:
```
./fhir-server-dist/install.sh /opt/ibm/fhir-server
```
This step installs the OpenLiberty runtime and the FHIR server web application. The Liberty runtime is installed in a directory called `wlp` within the installation directory that you specify. For example, in the preceding command, the root directory of the Liberty server runtime would be `/opt/ibm/fhir-server/wlp`.
This step installs the OpenLiberty runtime and the IBM FHIR Server web application. The Liberty runtime is installed in a directory called `wlp` within the installation directory that you specify. For example, in the preceding command, the root directory of the Liberty server runtime would be `/opt/ibm/fhir-server/wlp`.
5. Configure the fhir-server's `server.xml` file as needed by completing the following steps:
* Configure the ports that the server listen on. The server is installed with only port 9443 (HTTPS) enabled by default. To change the port numbers, modify the values in the `httpEndpoint` element.
* Configure a server keystore and truststore. The FHIR server is installed with a default keystore file that contains a single self-signed certificate for localhost. For production use, you must create and configure your own keystore and truststore files for the FHIR server deployment (that is, generate your own server certificate or obtain a trusted certificate, and then share the public key certificate with API consumers so that they can insert it into their client-side truststore). The keystore and truststore files are used along with the server's HTTPS endpoint and the FHIR server's client-certificate-based authentication protocol to secure the FHIR server's endpoint. For more information, see [Section 5.2 Keystores, truststores, and the FHIR server](#52-keystores-truststores-and-the-fhir-server).
* Configure a server keystore and truststore. The IBM FHIR Server is installed with a default keystore file that contains a single self-signed certificate for localhost. For production use, you must create and configure your own keystore and truststore files for the FHIR server deployment (that is, generate your own server certificate or obtain a trusted certificate, and then share the public key certificate with API consumers so that they can insert it into their client-side truststore). The keystore and truststore files are used along with the server's HTTPS endpoint and the FHIR server's client-certificate-based authentication protocol to secure the FHIR server's endpoint. For more information, see [Section 5.2 Keystores, truststores, and the FHIR server](#52-keystores-truststores-and-the-fhir-server).
* Configure an appropriate user registry. The FHIR server is installed with a basic user registry that contains a single user named `fhiruser`. For production use, it's best to configure your own user registry. For more information about configuring user registries, see the [OpenLiberty documentation](https://openliberty.io/guides/security-intro.html#configuring-the-user-registry).
To override the default fhiruser's password, one may set an Environment variable `FHIR_USER_PASSWORD` and for the fhiradmin's password one may set an Environment variable `FHIR_ADMIN_PASSWORD`.
Expand Down Expand Up @@ -112,9 +112,9 @@ The Maven build creates the zip package under `fhir-install/target`. Alternative
For more information about the capabilities of the implementation, see [Conformance](https://ibm.github.io/FHIR/Conformance).
## 2.2 Upgrading an existing server
The FHIR server does not include an upgrade installer. To upgrade a server to the next version, you can run the installer on a separate server, and then copy the resulting configuration files over to the existing server.
The IBM FHIR Server does not include an upgrade installer. To upgrade a server to the next version, you can run the installer on a separate server, and then copy the resulting configuration files over to the existing server.
To manage database updates over time, the FHIR server uses custom tools from the `fhir-database-utils` project. Through the use of a metadata table, the database utilities can detect the currently installed version of the database and apply any new changes that are needed to bring the database to the current level.
To manage database updates over time, the IBM FHIR Server uses custom tools from the `fhir-database-utils` project. Through the use of a metadata table, the database utilities can detect the currently installed version of the database and apply any new changes that are needed to bring the database to the current level.
Complete the following steps to upgrade the server:
Expand All @@ -131,7 +131,7 @@ The IBM FHIR Server includes a Docker image [ibmcom/ibm-fhir-server](https://hub
Note, logging for the IBM FHIR Server docker image is to stderr and stdout, and is picked up by Logging agents.
# 3 Configuration
This chapter contains information about the various ways in which the FHIR server can be configured by users.
This chapter contains information about the various ways in which the IBM FHIR Server can be configured by users.
## 3.1 Encoded passwords
In the examples within the following sections, you'll see the default password `change-password`. In order to secure your server, these values should be changed.
Expand Down Expand Up @@ -159,7 +159,7 @@ Configuration properties stored within a `fhir-server-config.json` file are stru
Throughout this document, we use a path notation to refer to property names. For example, the name of the `defaultPrettyPrint` property in the preceding example would be `fhirServer/core/defaultPrettyPrint`.
## 3.3 Tenant-specific configuration properties
The FHIR server supports certain multi-tenant features. One such feature is the ability to set certain configuration properties on a per-tenant basis.
The IBM FHIR server supports certain multi-tenant features. One such feature is the ability to set certain configuration properties on a per-tenant basis.
In general, the configuration properties for a particular tenant are stored in the `<WLP_HOME>/usr/servers/fhir-server/config/<tenant-id>/fhir-server-config.json` file, where `<tenant-id>` refers to the tenant's “short name” or tenant id.
Expand All @@ -171,10 +171,25 @@ Search parameters are handled like a single configuration properly; providing a
More information about multi-tenant support can be found in [Section 4.9 Multi-tenancy](#49-multi-tenancy).
## 3.3.1 Compartment Search Performance
The IBM FHIR Server supports the ability to compute and store compartment membership values during ingestion. Once stored, these values can help accelerate compartment-related search queries. To use this feature, update the IBM FHIR Server to at least version 4.5.1 and run a reindex operation as described in the [fhir-bucket](https://github.com/IBM/FHIR/tree/master/fhir-bucket) project [README](https://github.com/IBM/FHIR/blob/master/fhir-bucket/README.md). The reindex operation reprocesses the resources stored in the database, computing and storing the new compartment reference values. After the reindex operation has completed, add the `useStoredCompartmentParam` configuration element to the relevant tenant fhir-server-config.json file to allow the search queries to use the pre-computed values:
```
{
"fhirServer": {
"search": {
"useStoredCompartmentParam": true
}
}
}
```
Note that this parameter only enables or disables the compartment search query optimization feature. The compartment membership values are always computed and stored during ingestion or reindexing, regardless of the setting of this value. After the reindex operation is complete, it is recommended to set `useStoredCompartmentParam` to true. No reindex is required if this value is subsequently set to false.
## 3.4 Persistence layer configuration
The IBM FHIR server allows deployers to select a persistence layer implementation that fits their needs. Currently, the server includes a JDBC persistence layer which supports Apache Derby, IBM Db2, and PostgreSQL. However, Apache Derby is not recommended for production usage.
The IBM FHIR Server allows deployers to select a persistence layer implementation that fits their needs. Currently, the server includes a JDBC persistence layer which supports Apache Derby, IBM Db2, and PostgreSQL. However, Apache Derby is not recommended for production usage.
The FHIR server is delivered with a default configuration that is already configured to use the JDBC persistence layer implementation with an Embedded Derby database. This provides the easiest out-of-the-box experience since it requires very little setup. The sections that follow in this chapter will focus on how to configure the JDBC persistence layer implementation with either Embedded Derby or Db2.
The IBM FHIR Server is delivered with a default configuration that is already configured to use the JDBC persistence layer implementation with an Embedded Derby database. This provides the easiest out-of-the-box experience since it requires very little setup. The sections that follow in this chapter will focus on how to configure the JDBC persistence layer implementation with either Embedded Derby or Db2.
### 3.4.1 Configuring the JDBC persistence layer
#### 3.4.1.1 Database preparation
Expand Down Expand Up @@ -1850,6 +1865,7 @@ This section contains reference information about each of the configuration prop
|`fhirServer/audit/serviceProperties/geoState`|string|The Geo State configure for CADF audit logging service.|
|`fhirServer/audit/serviceProperties/geoCounty`|string|The Geo Country configure for CADF audit logging service.|
|`fhirServer/search/useBoundingRadius`|boolean|True, the bounding area is a Radius, else the bounding area is a box.|
|`fhirServer/search/useStoredCompartmentParam`|boolean|False, Compute and store parameter to accelerate compartment searches. Requires reindex using at least IBM FHIR Server version 4.5.1 before this feature is enabled |
|`fhirServer/bulkdata/applicationName`| string|Fixed value, always set to fhir-bulkimportexport-webapp |
|`fhirServer/bulkdata/moduleName`|string| Fixed value, always set to fhir-bulkimportexport.war |
|`fhirServer/bulkdata/jobParameters/cos.bucket.name`|string|Object store bucket name |
Expand Down
2 changes: 2 additions & 0 deletions fhir-bucket/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -377,3 +377,5 @@ java \
```

The format of the reindex timestamp can be a date `YYYY-MM-DD` representing midnight UTC on the given day, or an ISO timestamp `YYYY-MM-DDThh:mm:ssZ`.

Values for `--reindex-resource-count` larger than 1000 will be clamped to 1000 to ensure that the `$reindex` server calls return within a reasonable time.
58 changes: 29 additions & 29 deletions fhir-bucket/src/main/java/com/ibm/fhir/bucket/reindex/DriveReindexOperation.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,39 +22,39 @@
import com.ibm.fhir.model.resource.OperationOutcome;
import com.ibm.fhir.model.resource.OperationOutcome.Issue;
import com.ibm.fhir.model.resource.Parameters;
import com.ibm.fhir.model.resource.Resource;
import com.ibm.fhir.model.resource.Parameters.Parameter;
import com.ibm.fhir.model.resource.Resource;

/**
* Drives the $reindex custom operation in parallel. Each thread keeps running
* until the OperationOutcome indicates that no work remains to be processed.
*/
public class DriveReindexOperation {
private static final Logger logger = Logger.getLogger(DriveReindexOperation.class.getName());

// the maximum number of requests we permit
private final int maxConcurrentRequests;

// flag to indicate if we should be running
private volatile boolean running = true;

private volatile boolean active = false;

// count of how many threads are currently running
private AtomicInteger currentlyRunning = new AtomicInteger();

// thread pool for processing requests
private final ExecutorService pool = Executors.newCachedThreadPool();

private final FHIRBucketClient fhirClient;

private final String url = "$reindex";

// The serialized Parameters resource sent with each POST
private final String requestBody;

private Thread monitorThread;

/**
* Public constructor
* @param client the FHIR client
Expand All @@ -65,13 +65,13 @@ public DriveReindexOperation(FHIRBucketClient fhirClient, int maxConcurrentReque
this.maxConcurrentRequests = maxConcurrentRequests;

Parameters parameters = Parameters.builder()
.parameter(Parameter.builder().name(str("_tstamp")).value(str(tstampParam)).build())
.parameter(Parameter.builder().name(str("_resourceCount")).value(intValue(resourceCountParam)).build())
.parameter(Parameter.builder().name(str("tstamp")).value(str(tstampParam)).build())
.parameter(Parameter.builder().name(str("resourceCount")).value(intValue(resourceCountParam)).build())
.build();

// Serialize into the requestBody string used by all the threads
this.requestBody = FHIRBucketClientUtil.resourceToString(parameters);

if (logger.isLoggable(Level.FINE)) {
logger.fine("Reindex request parameters: " + requestBody);
}
Expand All @@ -85,7 +85,7 @@ public DriveReindexOperation(FHIRBucketClient fhirClient, int maxConcurrentReque
private static com.ibm.fhir.model.type.String str(String str) {
return com.ibm.fhir.model.type.String.of(str);
}

private static com.ibm.fhir.model.type.Integer intValue(int val) {
return com.ibm.fhir.model.type.Integer.of(val);
}
Expand All @@ -97,7 +97,7 @@ public void init() {
if (!running) {
throw new IllegalStateException("Already shutdown");
}

// Initiate the monitorThread. This will fill the pool
// with worker threads, and monitor for completion or failure
logger.info("Starting monitor thread");
Expand All @@ -106,7 +106,7 @@ public void init() {
}

/**
* The main monitor loop.
* The main monitor loop.
*/
public void monitorLoop() {
while (this.running) {
Expand All @@ -121,11 +121,11 @@ public void monitorLoop() {
// should be OK now to fill the pool with workers
logger.info("Test probe successful - filling worker pool");
this.active = true;

for (int i=0; i<this.maxConcurrentRequests && this.running && this.active; i++) {
this.currentlyRunning.addAndGet(1);
pool.execute(() -> callReindexOperation());

// Slow down the ramp-up so we don't hit a new server with
// hundreds of requests in one go
safeSleep(1000);
Expand All @@ -140,7 +140,7 @@ public void monitorLoop() {
logger.info("Waiting for current threads to complete before restart: " + currentThreadCount);
safeSleep(5000);
}

} else { // active
// worker threads are active, so sleep for a bit before we check again
safeSleep(5000);
Expand All @@ -165,25 +165,25 @@ protected void safeSleep(long ms) {
*/
public void signalStop() {
this.running = false;

// make sure the pool doesn't start new work
pool.shutdown();
}

/**
* Wait until things are stopped
*/
public void waitForStop() {
if (this.running) {
signalStop();
}

try {
pool.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException x) {
logger.warning("Wait for pool shutdown interrupted");
}

try {
// break any sleep inside the monitorThread
this.monitorThread.interrupt();
Expand All @@ -205,25 +205,25 @@ private void callReindexOperation() {
this.active = false;
}
}

this.currentlyRunning.decrementAndGet();
}

/**
* Make one call to the FHIR server $reindex operation
* @return true if the call was successful (200 OK)
*/
private boolean callOnce() {
boolean result = false;

// tell the FHIR Server to reindex a number of resources
long start = System.nanoTime();
FhirServerResponse response = fhirClient.post(url, requestBody);
long end = System.nanoTime();

double elapsed = (end - start) / 1e9;
logger.info(String.format("called $reindex: %d %s [took %5.3f s]", response.getStatusCode(), response.getStatusMessage(), elapsed));

if (response.getStatusCode() == HttpStatus.SC_OK) {
Resource resource = response.getResource();
if (resource != null) {
Expand All @@ -243,7 +243,7 @@ private boolean callOnce() {
// Stop as soon as we hit an error
logger.severe("FHIR Server reindex operation returned an error: " + response.getStatusCode() + " " + response.getStatusMessage());
}

return result;
}

Expand All @@ -257,7 +257,7 @@ private void checkResult(OperationOutcome result) {
Issue one = issues.get(0);
if ("Reindex complete".equals(one.getDiagnostics().getValue())) {
logger.info("Reindex - all done");

// tell all the running threads they can stop now
this.running = false;
}
Expand Down
1 change: 1 addition & 0 deletions fhir-config/src/main/java/com/ibm/fhir/config/FHIRConfiguration.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ public class FHIRConfiguration {
public static final String PROPERTY_FIELD_RESOURCES_SEARCH_PARAMETER_COMBINATIONS = "searchParameterCombinations";
public static final String PROPERTY_FIELD_RESOURCES_PROFILES = "profiles";
public static final String PROPERTY_FIELD_RESOURCES_PROFILES_AT_LEAST_ONE = "atLeastOne";
public static final String PROPERTY_USE_STORED_COMPARTMENT_PARAM = "fhirServer/search/useStoredCompartmentParam";

// Auth and security properties
public static final String PROPERTY_SECURITY_CORS = "fhirServer/security/cors";
Expand Down
Loading

0 comments on commit a02a4d2

Please sign in to comment.