Skip to content

Commit

Permalink
Merge pull request #2502 from IBM/tbieste-issue-2195-reindex
Browse files Browse the repository at this point in the history 
Issue #2195 - Enable client checkpoint driven reindex
  • Loading branch information
@tbieste
tbieste authored Jun 28, 2021
2 parents 45ba58d + 1331d4d commit ff5a4ef
Show file tree
Hide file tree
Showing 32 changed files with 1,553 additions and 401 deletions.
10 changes: 5 additions & 5 deletions docs/src/pages/guides/FHIRPerformanceGuide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ The following table summarizes how the transaction timeout is used for different
| POST/PUT | Single transaction scope for entire request |
| Batch Bundle | Transaction per bundle entry. Request processing time can therefore exceed totalTranLifetimeTimeout |
| Transaction Bundle | Single transaction scope for entire request |
| $reindex | One HTTP call can request multiple resources to be reindexed. Each resource is reindexed in the scope of its own transaction. Reindexing is a relatively quick operation per resource - usually well under 1s - so transaction timeouts are unlikely. Reduce the number of resources processed per reindex operation to avoid read timeouts. Use concurrent requests to increase overall throughput. |
| $reindex | One HTTP call can request multiple resources to be reindexed. By default, each resource is reindexed in the scope of its own transaction. Reindexing is a relatively quick operation per resource - usually well under 1s - so transaction timeouts are unlikely. However, if a list of index IDs is specified, all those resources will be reindexed within a single tranaction, so reduce the number index IDs specified if transaction timeouts occur. Use concurrent requests to increase overall throughput. |

Because some requests use multiple transactions under the covers, the overall request response time can sometimes be greater than the transaction timeout. There is no server-side tuneable property for the overall request processing time. Tuning of the client read timeout and/or network configuration may be required when extending the maximum transaction time to more than 2 minutes, or supporting multi-transaction requests which also exceed 2 minutes.

Expand Down Expand Up @@ -356,11 +356,11 @@ See the [PostgreSQL Query Planning](https://www.postgresql.org/docs/12/runtime-c

### 4.1.1. Fillfactor

In PostgreSQL, the default `fillfactor` for each table is 100 - no room is reserved for updates. This maximizes storage utilization, but impacts performance for updates which occur when new versions of a resource are ingested. Update statements are also used frequently during the reindex process.
In PostgreSQL, the default `fillfactor` for each table is 100 - no room is reserved for updates. This maximizes storage utilization, but impacts performance for updates which occur when new versions of a resource are ingested. Update statements are also used frequently during the reindex process, if index IDs are not specified.

To provide space for updates, all the `<resourceType>_logical_resources` should be configured with a `fillfactor` of 80 as a starting point. DBAs may specify their own `fillfactor` values based on their own knowledge and understanding of the system.

The `fillfactor` for the `logical_resources` table may benefit from an even lower value to support the heavy update load during a reindex operation. This is a special case due to the fact that every row in the table is updated once.
The `fillfactor` for the `logical_resources` table may benefit from an even lower value to support the heavy update load during a reindex operation, if index IDs are not specified. This is a special case due to the fact that every row in the table is updated once.

To change the fillfactor for existing data, a `VACUUM FULL` operation is required:

Expand All @@ -374,7 +374,7 @@ This should only be performed during a maintenance window when there is no load

### 4.1.2. Tuning Auto-vacuum

When running reindex operations (after a search parameter configuration change, for example), the `logical_resources` table undergoes frequent updates to an indexed column. Due to the nature of how PostgreSQL handles updates, this results in a significant amount of old index blocks which slows progress. The table storage parameters may need to be tuned to vacuum the `logical_resources` table more aggressively. To address this, tune the storage parameters for this table as follows:
When running reindex operations (after a search parameter configuration change, for example) without specifying index IDs, the `logical_resources` table undergoes frequent updates to an indexed column. Due to the nature of how PostgreSQL handles updates, this results in a significant amount of old index blocks which slows progress. The table storage parameters may need to be tuned to vacuum the `logical_resources` table more aggressively. To address this, tune the storage parameters for this table as follows:

``` sql
-- Lower the trigger threshold for starting work
Expand All @@ -384,7 +384,7 @@ alter table fhirdata.logical_resources SET (autovacuum_vacuum_scale_factor = 0.0
alter table fhirdata.logical_resources SET (autovacuum_vacuum_cost_limit=2000);
```

The default value for autovacuum_vacuum_cost_limit is likely too restrictive for a system with good IO performance. Increasing the value to 2000 increases the throttling threshold 10x, significantly improving throughput and helping the `logical_resources` vacuuming to be completed before it negatively impacts reindexing performance.
The default value for autovacuum_vacuum_cost_limit is likely too restrictive for a system with good IO performance. Increasing the value to 2000 increases the throttling threshold 10x, significantly improving throughput and helping the `logical_resources` vacuuming to be completed before it negatively impacts performance of the reindex operation (when index IDs are not specified on the reindex operation).

See the [PostSQL VACUUM documentation](https://www.postgresql.org/docs/12/sql-vacuum.html) for more details.

Expand Down
71 changes: 26 additions & 45 deletions docs/src/pages/guides/FHIRSearchConfiguration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -265,62 +265,43 @@ Reindexing is implemented as a custom operation that tells the IBM FHIR Server t

The `$reindex` operation can be invoked via an HTTP(s) POST to `[base]/$reindex`, `[base]/[type]/$reindex`, or `[base]/[type]/[instance]/$reindex`. By default, the operation at the System-level or Type-level selects 10 resources and re-extract their search parameters values based on the current configuration of the server. The operation supports the following parameters to control the behavior:

Reindexing is resource-intensive and can take several hours or even days to complete depending on the approach used, the number of resources currently in the system, and the capability of the hosting platform.

### 2.1 Server-side-driven approach
By default, the operation will select 10 resources and re-extract their search parameters values based on the current configuration of the server. The operation supports the following parameters to control the behavior:

|name|type|description|
|----|----|-----------|
|`tstamp`|string|Reindex any resource not previously reindexed before this timestamp. Format as a date YYYY-MM-DD or time YYYY-MM-DDTHH:MM:DDZ.|
|`tstamp`|string|Reindex only resources not previously reindexed since this timestamp. Format as a date YYYY-MM-DD or time YYYY-MM-DDTHH:MM:SSZ.|
|`resourceCount`|integer|The maximum number of resources to reindex in this call. If this number is too large, the processing time might exceed the transaction timeout and fail.|
|`resourceLogicalId`|string|The ResourceType or the ResourceType/Logical id for targetted reindexing, only valid at System-level|

An example request is:

``` sh
curl --location --request POST 'https://localhost:9443/fhir-server/api/v4/$reindex' \
--header 'X-FHIR-TENANT-ID: default' \
--header 'Content-Type: application/fhir+json' \
-u 'fhiruser:change-password' \
--data-raw '{
"resourceType": "Parameters",
"parameter": [
{
"name": "resourceCount",
"valueInteger": 100
},
{
"name": "tstamp",
"valueString": "2021-01-01"
}
]
}'
```

An example response when processing in a loop:
The IBM FHIR Server tracks when a resource was last reindexed and only resources with a reindex_tstamp value less than the given tstamp parameter will be processed. When a resource is reindexed, its reindex_tstamp is set to the given tstamp value. In most cases, using the current date (for example "2020-10-27") is the best option for this value.

``` json
{
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"code": "informational",
"diagnostics": "Processed Patient/1795df2b501-04f88a35-9f2f-4871-a05e-ba8090fa18f5"
}
]
}
```
### 2.2 Client-side-driven approach
Another option is to have the client utilize the `$retrieve-index` and `$reindex` in parallel to drive the processing.

An example response when processing is complete:
The `$retrieve-index` operation is called to retrieve index IDs of resources available for reindexing. This can be done repeatedly by using the `_count` and `afterIndexId` parameters for pagination. The operation supports the following parameters to control the behavior:

``` json
{"resourceType":"OperationOutcome","issue":[{"severity":"information","code":"informational","diagnostics":"Reindex complete"}]}
```
|name|type|description|
|----|----|-----------|
|`_count`|string|The maximum number of index IDs to retrieve. This may not exceed 1000. If not specified, the maxinum number retrieved is 1000.|
|`notModifiedAfter`|string|Only retrieve index IDs for resources not last updated after this timestamp. Format as a date YYYY-MM-DD or time YYYY-MM-DDTHH:MM:SSZ.|
|`afterIndexId`|string|Retrieve index IDs starting with the first index ID after this index ID. If this parameter is not specified, the retrieved index IDs start with the first index ID.|

The IBM FHIR Server tracks when a resource was last reindexed and only resources with a reindex_tstamp value less than the given tstamp parameter will be processed. When a resource is reindexed, its reindex_tstamp is set to the given tstamp value. In most cases, using the current date (for example "2020-10-27") is the best option for this value.
The `$retrieve-index` operation can be invoked via an HTTP(s) POST to `[base]/$retrieve-index` or `[base]/[type]/$retrieve-index`. Invoking this operation at the type-level only retrieves indexIDs for resources of that type.

To aid in the re-indexing process, the IBM FHIR Server team has expanded the fhir-bucket resource-loading tool to support driving the reindex. The fhir-bucket tool uses a thread-pool to make concurrent POST requests to the IBM FHIR Server `$reindex` custom operation.
The `$reindex` operation is called to reindex the resources with index IDs in the specified list. The operation supports the following parameters to control the behavior:

For more information on driving the reindex operation from fhir-bucket, see https://github.com/IBM/FHIR/tree/main/fhir-bucket#driving-the-reindex-custom-operation.
|name|type|description|
|----|----|-----------|
|`indexIds`|string|Reindex only resources with an index ID in the specified list, formatted as a comma-delimited list of strings. If number of index IDs in the list is too large, the processing time might exceed the transaction timeout and fail.|

By specifying the index IDs on the `$reindex` operation, the IBM FHIR Server avoids the database overhead of choosing the next resource to reindex and updating the reindex_tstamp. Though it requires the client side to track the reindex progress, it should allow for an overall faster reindex.

Reindexing is resource-intensive and can take several hours or even days to complete depending on the number of resources currently in the system and the capability of the hosting platform.
### 2.3 fhir-bucket
To aid in the re-indexing process, the IBM FHIR Server team has expanded the fhir-bucket resource-loading tool to support driving the reindex, with the option of using either the server-side-driven or client-side-driven approach. The fhir-bucket tool uses a thread-pool to make concurrent POST requests to the IBM FHIR Server `$retrieve-index` and `$reindex` custom operations.

For more information on driving the reindex operation from fhir-bucket, see https://github.com/IBM/FHIR/tree/main/fhir-bucket#driving-the-reindex-custom-operation.

---
FHIR® is the registered trademark of HL7 and is used with the permission of HL7.
2 changes: 1 addition & 1 deletion docs/src/pages/guides/FHIRServerUsersGuide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -511,7 +511,7 @@ In addition to the standard REST API (create, update, search, and so forth), the
### 4.1.1 Packaged operations
The FHIR team provides implementations for the standard `$validate`, `$document`, `$everything`, `$expand`, `$lookup`, `$subsumes`, `$closure`, `$export`, `$import`, `$convert`, `$apply` and `$translate` operations, as well as a custom operation named `$healthcheck`, which queries the configured persistence layer to report its health.

The server also bundles `$reindex` to reindex instances of Resources so they are searchable, and `$erase` to hard delete instances of Resources. To learn more about the $erase operation, read the [design document](https://github.com/IBM/FHIR/tree/main/operation/fhir-operation-erase/README.md).
The server also bundles `$reindex` to reindex instances of Resources so they are searchable, `$retrieve-index` to retrieve lists of resources available to be reindexed, and `$erase` to hard delete instances of Resources. To learn more about the $erase operation, read the [design document](https://github.com/IBM/FHIR/tree/main/operation/fhir-operation-erase/README.md).

To extend the server with additional operations, see [Section 4.1.2 Custom operations](#412-custom-operations)

Expand Down
11 changes: 7 additions & 4 deletions fhir-bucket/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -355,13 +355,15 @@ java.util.logging.FileHandler.pattern=fhirbucket-%u-%g.log

When the IBM FHIR Server stores a FHIR resource, it extracts a configurable set of searchable parameter values and stores them in specially indexed tables which are used to support search queries. When the search parameter configuration is changed (perhaps because a profile has been updated), users may want to apply this new configuration to resources already stored. By default, such configuration changes only apply to new resources.

The IBM FHIR Server supports a custom operation to rebuild or "reindex" the search parameters extracted from resources currently stored. The user selects a date or timestamp as the reindex "marker". This value is used to determine which resources have been reindexed, and which still need to be reindexed. When a resource is successfully reindexed, it is marked with this user-selected timestamp. Each reindex REST call will process up to the requested number of resources and return an OperationOutcome resource containing issues describing which resources were processed. When there are no resources left to update, the call returns an OperationOutcome with one issue indicating that the reindex is complete.
The IBM FHIR Server supports a custom operation to rebuild or "reindex" the search parameters extracted from resources currently stored. There are two approaches for driving the reindex, server-side-driven or client-side-driven. Using server-side-driven is the default; to use client-side-driven, include the `--index-client-side-driven` parameter.

To avoid read timeouts, the number of resources processed in a single reindex call can be limited. Reindex calls can be made in parallel to increase throughput. The best number for concurrent requests depends on the capabilities of the underlying platform and any desire to balance load with other users. Concurrency up to 200 threads have been tested. Monitor the IBM FHIR Server response times when increasing concurrency. Also, make sure that the connection pool configured in the FHIR server cluster can support the required number of threads. This also means that the database needs to be configured to support this number of connections (sessions) plus any overhead.
With server-side-driven, the fhir-bucket will repeatedly call the `$reindex` operation. The user selects a date or timestamp as the reindex "marker", which is used to determine which resources have been reindexed, and which still need to be reindexed. When a resource is successfully reindexed, it is marked with this user-selected timestamp. Each reindex REST call will process up to the requested number of resources and return an OperationOutcome resource containing issues describing which resources were processed. When there are no resources left to update, the call returns an OperationOutcome with one issue, with an issue diagnostic value "Reindex complete", indicating that the reindex is complete.

With client-side-driven, the fhir-bucket will repeatedly call two operations in parallel; the `$retrieve-index` operation to determine the list of resources available to reindex, and the `$reindex` operation with a list of resources to reindex. Driving the reindex this way avoids database contention associated with updating the reindex timestamp of each resource with reindex "marker", which is used by the server-side-driven approach to keep track of the next resource to reindex.

The fhir-bucket main app has been extended to support driving a reindex operation with high concurrency. Processing will stop once the FHIR server returns the OperationOutcome with an issue diagnostic value "Reindex complete".
To avoid read timeouts, the number of resources processed in a single reindex call can be limited. Reindex calls can be made in parallel to increase throughput. The best number for concurrent requests depends on the capabilities of the underlying platform and any desire to balance load with other users. Concurrency up to 200 threads have been tested. Monitor the IBM FHIR Server response times when increasing concurrency. Also, make sure that the connection pool configured in the FHIR server cluster can support the required number of threads. This also means that the database needs to be configured to support this number of connections (sessions) plus any overhead.

The fhir-bucket main app has been extended to support driving a reindex operation with high concurrency.

```
java \
Expand All @@ -373,7 +375,8 @@ java \
--no-scan \
--reindex-tstamp 2020-12-01T00:00:00Z \
--reindex-resource-count 50 \
--reindex-concurrent-requests 20
--reindex-concurrent-requests 20 \
--reindex-client-side-driven
```

The format of the reindex timestamp can be a date `YYYY-MM-DD` representing `00:00:00` UTC on the given day, or an ISO timestamp `YYYY-MM-DDThh:mm:ssZ`.
Expand Down
Loading

0 comments on commit ff5a4ef

Please sign in to comment.