Smile CDR v2024.05.PRE
On this page:

42.13.1smileutil: Synchronize FHIR Servers

 

The synchronize-fhir-servers command may be used to keep one server in sync with another by copying all resources from a source to a target, and deleting any resources on the target which no longer exist on the source.

42.13.2Usage

 
bin/smileutil synchronize-fhir-servers -s "http://source.example.com:8000" -t "http://target.example.com:8000"

42.13.3Options

 
  • -v [version] (or --fhir-version [version]) – The version of FHIR supported by the server.
  • -s [base URL or pattern] (or --source [base URL or pattern]) – This argument should point to the base URL of the source FHIR server endpoint.
  • --source-basic-auth [username:password](optional) This argument may provide a set of credentials to be sent to the source server via HTTP Basic Auth, in the form 'username:password'.
  • --source-bearer-token [token](optional) This argument provides a bearer token that will be supplied to the source server in an Authorization header.
  • --source-tls-auth [filename](optional) If specified, this parameter supplies a path and filename for a Smile Util TLS JSON Authentication File that will be used to authenticate the source FHIR server endpoint secured through HTTPS.
  • -t [base URL] (or --target [base URL]) – This argument should point to the base URL of the target FHIR server endpoint. In mode COPY or mode SYNTHEA it is possible to specify this argument more than once, in which case requests will be spread round-robin across all supplied targets.
  • --target-basic-auth [username:password](optional) This argument may provide a set of credentials to be sent to the target server via HTTP Basic Auth, in the form 'username:password'.
  • --target-bearer-token [token](optional) This argument provides a bearer token that will be supplied to the target server in an Authorization header.
  • --target-tls-auth [filename](optional) If specified, this parameter supplies a path and filename for a Smile Util TLS JSON Authentication File that will be used to authenticate the target FHIR server endpoint secured through HTTPS.
  • --target-retry-count [count](optional) This argument provides the maximum number of times a failed write attempt on the target server should be retried before giving up. Must be a positive integer. If not supplied, failed write attempts will not be retried.
  • -r [resource types] (or --resource-types) – (optional) This argument accepts a comma-separated list of resource types to synchronize. See selecting resources to synchronize below for more information.
  • --resource-ids(optional) This argument accepts a comma-separated list of resource IDs to synchronize. See selecting resources to synchronize below for more information.
  • --thread-count [count](optional) This argument may provide a number of threads to use for processing (default is 10).
  • --prefix-numeric-ids [prefix](optional) This argument may provide a prefix to prepend to any purely numeric resource IDs found on the source server when transmitting to the target server. See Prefixing Numeric IDs for more information.
  • -m [mode] (or --mode [mode]) – (optional) This argument specifies the synchronization mode to use (see Synchronization Modes to learn more about available modes).
  • --upload-bundles-as-transactions(optional) When used with the COPY synchronization mode, any Bundle resources found in the source will be treated as FHIR transaction Bundles and transmitted to the target server using the FHIR Transaction operation. See Uploading Bundles as Transactions below for information on this option. Note this option will modify the value of Bundle.type to be transaction but otherwise make no other changes to Bundle resources found in the source (i.e. request elements will not be automatically added); as such, synchronization will likely fail for other types of Bundle resources found in the source (i.e. of type collection and document).
  • --request-id(optional) This argument populates the value for the X-Request-Id header, and can be used for matching server logs against a specific smileutil invocation.
  • --semaphore-path [fhirpath](optional) If specified, the given FHIRPath expression will be queried for each resource to be uploaded, and the value will be used as a semaphore to avoid multiple threads uploading a resource with the same value at this path concurrently. For example, when uploading Bundle resources containing a transaction, the following path could be used to provide a semaphore on the ID of the Patient resource within each bundle: Bundle.entry.resource.as(Patient).id. Note that this item only applies to mode COPY and must not be used otherwise. For multiple resources: --semaphore-path "Patient.id|Practitioner.id"

42.13.4Selecting Resource Types to Synchronize

 

By default, the synchronize-fhir-servers command will synchronize all non-infrastructure resources on the server. This means that all resources aside from the following set will be synchronized:

  • CapabilityStatement
  • CodeSystem
  • EventDefinition
  • OperationDefinition
  • SearchParameter
  • Subscription
  • StructureDefinition
  • ConceptMap
  • ValueSet

The list of resources to synchronize (and the order in which to synchronize them) may be adjusted using the -r or --resource-types parameter. An example is shown below.

--resource-types Patient,Encounter,Observation

Alternately, a list of resources may be specified explicitly using the --resource-ids parameter. An example is shown below.

---resource-ids Patient/123,Patient/456,Encounter/777

Note that resource-types and resource-ids can not be combined.

42.13.5Using a Search as the Source

 

It is possible to use a FHIR search as the source. This takes the following form:

-s "http://example.org/fhir/Patient?name=smith"

When using this style, all resources are fetched from the search and then uploaded to the target server if necessary (when using COMPARE mode as described below, not all resources will necessarily be uploaded).

42.13.6Using a Directory as the Source

 

It is possible to use a local directory as the source. This takes the following form:

Linux/OSX:

-s "file:///home/test/resources/"

Windows:

-s "file:/C:/Path/to/files/"

When using this style, all files in the given directory are loaded and parsed, and then uploaded to the target server.

When using COMPARE mode as described below, some resources may not be uploaded if they already exist on the target server. When using COPY mode, all resources will be uploaded.

42.13.6.1Uploading NDJSON

If files are found with a filename ending in .ndjson, the file will be processed as Newline Delimited JSON, where each line is expected to contain a new FHIR resource. This is the format specified in the FHIR Bulk Data Access (Flat FHIR) implementation guide.

This means that a set of NDJSON files produced by a Bulk Export operation may be placed in a directory, and that directory can then be used as the source directory for a synchronize operation. The net result is that all resources in the NDJSON files within the directory will be uploaded to the target server (provided that they match the resource types specified in the selected resource types / -r argument described above).

42.13.7Uploading Bundles as Transactions

 

If your source data consists of FHIR Transaction Bundles you can use the Synchronize FHIR Servers command to upload these Bundles directly to the server as FHIR transactions.

Note that if your Bundles were created using the Synthea tool, it is better to use Synthea Mode.

The following example shows how to upload a directory containing FHIR Transaction Bundles to a server:

bin/smileutil synchronize-fhir-servers --source "file:some/directory" --target "http://target.example.com:8000" --fhir-version R4 --mode COPY --upload-bundles-as-transactions

42.13.8Using Inline Match URLs

 

The synchronize-fhir-servers command supports the use of inline match URLs to auto-create placeholder reference targets where only an identifier is available. See Auto-Create Placeholder Reference Targets with Identifier for more information regarding this feature.

42.13.9Prefixing Numeric IDs

 

When synchronizing between two Smile CDR (or HAPI FHIR) servers, one challenge is that these servers do not normally allow clients to specify IDs for client-created resources where the ID is purely numeric. For example, on such a server, a client is allowed to create a resource and specify the ID Patient/ABC but the ID Patient/123 is reserved for assignment by the server and may not be client assigned.

In this situation, the --prefix-numeric-ids parameter may be used to specify an alphanumeric prefix. For example, if the command is executed as shown below, a resource on the source server with the ID Patient/123 will be saved on the target server as Patient/PFX123, and any other resources linking to this patient will have their references adjusted during the sync as well.

bin/smileutil synchronize-fhir-servers --prefix-numeric-ids PFX -s "http://source.example.com:8000" -t "http://target.example.com:8000"

42.13.10Synchronization Modes

 

There are different mechanisms for synchronizing one server to another, each with its own characteristics.

42.13.10.1Mode: `COPY`

The copy synchronization mode replicates resources using brute force by copying resources one-by-one.

When using a FHIR server as the source, this mode is fast for small servers with a smaller number of resources but is generally inefficient for servers with a large numbers of resources as it requires many individual HTTP transactions. When using a directory as the source, this mode maybe the fastest option, as it uploads all files without checking whether they already exist on the target server. If many files are expected to already exist, it may be faster to use the COMPARE mode.

This mode works by following the two steps below.

  1. If the source is a base URL, a series of FHIR searches are performed on the source server (one for each resource type being synchronized), downloading all resources that should be copied. If the results include multiple pages, all pages are loaded. If the source is a FHIR search or a directory, all matching resources are downloaded.
  2. After each download is complete, resources are uploaded one-by-one.

Note the following considerations:

  • The copy synchronization does not delete any resources on the target server that do not exist on the source server (no attempts are made to read or search for data on the target server).
  • Both downloads from the source server and uploads to the target server are multithreaded for better performance. Adjusting the --thread-count parameter may improve performance.
  • If the copy synchronization appears to be a taking a very long time to complete and is repeatedly showing a message indicating that it's waiting on jobs to complete but the number of jobs does not change, this may be an indication that the --thread-count parameter needs to increased.

42.13.10.2Mode: `COMPARE`

The compare synchronization mode replicates resources by downloading all resources from both the source and target systems, and then comparing them locally before replicating.

This mode is often faster for larger numbers of resources, although it requires enough free disk space to hold a copy of all resources from both the source and target servers.

This mode works via the following steps:

  1. If the source is a base URL, a series of queries is made to the source system to fetch all resources for all resource types being synchronized. If the source is a FHIR search or a directory, all matching resources are downloaded.
  2. A series of queries is made to the target system to fetch all resources for all resource types being synchronized.
  3. All resources from the previous two steps are stored locally in a pair of temporary directories.
  4. All resources are hashed.
  5. Hashes are compared to come up with a delta between the two systems.
  6. Any resources that exist on the source server and do not exist on the target server or are not up-to-date on the target server are copied via a FHIR update operation.
  7. Any resources that exist on the target server but do not exist on the source server are deleted.

Note the following considerations:

  • The compare synchronization will delete any resources on the target server that do not exist on the source server if the resource no longer exists on the source server.
  • Resource tags, security labels, and profile declarations are also compared. If necessary they will be added or removed from the target server copy of the resource.

42.13.10.3Mode: `SYNTHEA`

See Synthea Mode below for information on this mode.

42.13.11Uploading Synthea (Synthetic Generated) Resources

 

The SYNTHEA mode is used to upload a standard set of synthetic data generated by Synthea to a FHIR server. This command assumes that the generated files are FHIR Transaction Bundles, and uploads them using a standard transaction operation.

The command uploads files in the correct order, uploading Bundles with supporting data (Practitioners, Organizations, etc.) before uploading individual Patient data.

To use this command, first you must execute Synthea to generate a set of Patients. The following example generates 100 patients.

./run_synthea -p 100

Then, execute the Synchronize FHIR Servers command as follows:

bin/smileutil synchronize-fhir-servers -v r4 -m SYNTHEA --source /path/to/synthea/output/fhir --target http://localhost:8000

42.13.12Setting up a Resource Gold Master

 

One common pattern for test systems involves creating a standard set of test resources that a repository can be regularly reset to. Under this scenario, there will be two repositories:

  • A staging repository, which is secured so that only administrative users may read/write to it. This repository stores a complete Gold Master set of resources. This collection is curated by an admin.
  • A public repository, which is publically accessible. This repository is regularly reset to the contents of the staging repository, meaning that public users may make modifications to the test data, but these modifications will be short lived since the repository is regularly reset.

The folowing command may be executed regularly (e.g. using a cron job) in order to replace the contents of the public repository with the contents of the staging repository.

bin/smileutil synchronize-fhir-servers \
  -s http://localhost:8000 \
  -t http://localhost:18000 \
  -v r4 \
  --source-basic-auth 'user:password'

You may also want to regularly use the expunge operation to clean up deleted resources on the public server:

curl -X POST "http://localhost:18000/\$expunge" -H "Content-Type: application/json" --data '{"resourceType":"Parameters","parameter":[{"name":"limit","valueInteger":50000},{"name":"expungePreviousVersions","valueBoolean":true},{"name":"expungeDeletedResources","valueBoolean":true}]}'

42.13.13Load Balancing Target Servers

 

If you have multiple target servers and wish to spread the load across all of them, you may specify multiple target options. Note that this is not possible in COMPARE mode. It is also important to note that this does not send a full copy of your source data to all target servers. Instead, it spreads the resources among the targets, in round-robin fashion.

If credentials are specified (e.g. using the --target-bearer-token argument), the same credentials will be sent to all targets.

For example:

bin/smileutil synchronize-fhir-servers \
  --source http://localhost:8000 \
  --target http://example1.com:8000 \
  --target http://example2.com:8000 \
  --fhir-version R4