The synchronize-fhir-servers command may be used to keep one server in sync with another by copying all resources from a source to a target, and deleting any resources on the target which no longer exist on the source.
bin/smileutil synchronize-fhir-servers -s "http://source.example.com:8000" -t "http://target.example.com:8000"
-v [version] (or --fhir-version [version]) – The version of FHIR supported by the server.-s [base URL or pattern] (or --source [base URL or pattern]) – This argument should point to the base URL of the source FHIR server endpoint.--source-basic-auth [username:password] – (optional) This argument may provide a set of credentials to be sent to the source server via HTTP Basic Auth, in the form 'username:password'.--source-bearer-token [token] – (optional) This argument provides a bearer token that will be supplied to the source server in an Authorization header.--source-tls-auth [filename] – (optional) If specified, this parameter supplies a path and filename for a Smile Util TLS JSON Authentication File that will be used to authenticate the source FHIR server endpoint secured through HTTPS.-t [base URL] (or --target [base URL]) – This argument should point to the base URL of the target FHIR server endpoint. In mode COPY or mode SYNTHEA it is possible to specify this argument more than once, in which case requests will be spread round-robin across all supplied targets.--target-basic-auth [username:password] – (optional) This argument may provide a set of credentials to be sent to the target server via HTTP Basic Auth, in the form 'username:password'.--target-bearer-token [token] – (optional) This argument provides a bearer token that will be supplied to the target server in an Authorization header.--target-tls-auth [filename] – (optional) If specified, this parameter supplies a path and filename for a Smile Util TLS JSON Authentication File that will be used to authenticate the target FHIR server endpoint secured through HTTPS.--target-retry-count [count] – (optional) This argument provides the maximum number of times a failed write attempt on the target server should be retried before giving up. Must be a positive integer. If not supplied, failed write attempts will not be retried.-r [resource types] (or --resource-types) – (optional) This argument accepts a comma-separated list of resource types to synchronize. See selecting resources to synchronize below for more information.--resource-ids – (optional) This argument accepts a comma-separated list of resource IDs to synchronize. See selecting resources to synchronize below for more information.--thread-count [count] – (optional) This argument may provide a number of threads to use for processing (default is 10).--prefix-numeric-ids [prefix] – (optional) This argument may provide a prefix to prepend to any purely numeric resource IDs found on the source server when transmitting to the target server. See Prefixing Numeric IDs for more information.-m [mode] (or --mode [mode]) – (optional) This argument specifies the synchronization mode to use (see Synchronization Modes to learn more about available modes).--upload-bundles-as-transactions – (optional) When used with the COPY synchronization mode, any Bundle resources found in the source will be treated as FHIR transaction Bundles and transmitted to the target server using the FHIR Transaction operation. See Uploading Bundles as Transactions below for information on this option. Note this option will modify the value of Bundle.type to be transaction but otherwise make no other changes to Bundle resources found in the source (i.e. request elements will not be automatically added); as such, synchronization will likely fail for other types of Bundle resources found in the source (i.e. of type collection and document).--request-id – (optional) This argument populates the value for the X-Request-Id header, and can be used for matching server logs against a specific smileutil invocation.--semaphore-path [fhirpath] – (optional) If specified, the given FHIRPath expression will be queried for each resource to be uploaded, and the value will be used as a semaphore to avoid multiple threads uploading a resource with the same value at this path concurrently. For example, when uploading Bundle resources containing a transaction, the following path could be used to provide a semaphore on the ID of the Patient resource within each bundle: Bundle.entry.resource.as(Patient).id. Note that this item only applies to mode COPY and must not be used otherwise. For multiple resources: --semaphore-path "Patient.id|Practitioner.id"By default, the synchronize-fhir-servers command will synchronize all non-infrastructure resources on the server. This means that all resources aside from the following set will be synchronized:
The list of resources to synchronize (and the order in which to synchronize them) may be adjusted using the -r or --resource-types parameter. An example is shown below.
--resource-types Patient,Encounter,Observation
Alternately, a list of resources may be specified explicitly using the --resource-ids parameter. An example is shown below.
---resource-ids Patient/123,Patient/456,Encounter/777
Note that resource-types and resource-ids can not be combined.
It is possible to use a FHIR search as the source. This takes the following form:
-s "http://example.org/fhir/Patient?name=smith"
When using this style, all resources are fetched from the search and then uploaded to the target server if necessary (when using COMPARE mode as described below, not all resources will necessarily be uploaded).
It is possible to use a local directory as the source. This takes the following form:
Linux/OSX:
-s "file:///home/test/resources/"
Windows:
-s "file:/C:/Path/to/files/"
When using this style, all files in the given directory are loaded and parsed, and then uploaded to the target server.
When using COMPARE mode as described below, some resources may not be uploaded if they already exist on the target server. When using COPY mode, all resources will be uploaded.
If files are found with a filename ending in .ndjson, the file will be processed as Newline Delimited JSON, where each line is expected to contain a new FHIR resource. This is the format specified in the FHIR Bulk Data Access (Flat FHIR) implementation guide.
This means that a set of NDJSON files produced by a Bulk Export operation may be placed in a directory, and that directory can then be used as the source directory for a synchronize operation. The net result is that all resources in the NDJSON files within the directory will be uploaded to the target server (provided that they match the resource types specified in the selected resource types / -r argument described above).
If your source data consists of FHIR Transaction Bundles you can use the Synchronize FHIR Servers command to upload these Bundles directly to the server as FHIR transactions.
Note that if your Bundles were created using the Synthea tool, it is better to use Synthea Mode.
The following example shows how to upload a directory containing FHIR Transaction Bundles to a server:
bin/smileutil synchronize-fhir-servers --source "file:some/directory" --target "http://target.example.com:8000" --fhir-version R4 --mode COPY --upload-bundles-as-transactions
The synchronize-fhir-servers command supports the use of inline match URLs to auto-create placeholder reference targets where only an identifier is available. See Auto-Create Placeholder Reference Targets with Identifier for more information regarding this feature.
When synchronizing between two Smile CDR (or HAPI FHIR) servers, one challenge is that these servers do not normally allow clients to specify IDs for client-created resources where the ID is purely numeric. For example, on such a server, a client is allowed to create a resource and specify the ID Patient/ABC but the ID Patient/123 is reserved for assignment by the server and may not be client assigned.
In this situation, the --prefix-numeric-ids parameter may be used to specify an alphanumeric prefix. For example, if the command is executed as shown below, a resource on the source server with the ID Patient/123 will be saved on the target server as Patient/PFX123, and any other resources linking to this patient will have their references adjusted during the sync as well.
bin/smileutil synchronize-fhir-servers --prefix-numeric-ids PFX -s "http://source.example.com:8000" -t "http://target.example.com:8000"
There are different mechanisms for synchronizing one server to another, each with its own characteristics.
The copy synchronization mode replicates resources using brute force by copying resources one-by-one.
When using a FHIR server as the source, this mode is fast for small servers with a smaller number of resources but is generally inefficient for servers with a large numbers of resources as it requires many individual HTTP transactions. When using a directory as the source, this mode maybe the fastest option, as it uploads all files without checking whether they already exist on the target server. If many files are expected to already exist, it may be faster to use the COMPARE mode.
This mode works by following the two steps below.
Note the following considerations:
--thread-count parameter may improve performance.--thread-count parameter needs to increased.The compare synchronization mode replicates resources by downloading all resources from both the source and target systems, and then comparing them locally before replicating.
This mode is often faster for larger numbers of resources, although it requires enough free disk space to hold a copy of all resources from both the source and target servers.
This mode works via the following steps:
Note the following considerations:
See Synthea Mode below for information on this mode.
The SYNTHEA mode is used to upload a standard set of synthetic data generated by Synthea to a FHIR server. This command assumes that the generated files are FHIR Transaction Bundles, and uploads them using a standard transaction operation.
The command uploads files in the correct order, uploading Bundles with supporting data (Practitioners, Organizations, etc.) before uploading individual Patient data.
To use this command, first you must execute Synthea to generate a set of Patients. The following example generates 100 patients.
./run_synthea -p 100
Then, execute the Synchronize FHIR Servers command as follows:
bin/smileutil synchronize-fhir-servers -v r4 -m SYNTHEA --source /path/to/synthea/output/fhir --target http://localhost:8000
One common pattern for test systems involves creating a standard set of test resources that a repository can be regularly reset to. Under this scenario, there will be two repositories:
The folowing command may be executed regularly (e.g. using a cron job) in order to replace the contents of the public repository with the contents of the staging repository.
bin/smileutil synchronize-fhir-servers \
-s http://localhost:8000 \
-t http://localhost:18000 \
-v r4 \
--source-basic-auth 'user:password'
You may also want to regularly use the expunge operation to clean up deleted resources on the public server:
curl -X POST "http://localhost:18000/\$expunge" -H "Content-Type: application/json" --data '{"resourceType":"Parameters","parameter":[{"name":"limit","valueInteger":50000},{"name":"expungePreviousVersions","valueBoolean":true},{"name":"expungeDeletedResources","valueBoolean":true}]}'
If you have multiple target servers and wish to spread the load across all of them, you may specify multiple target options. Note that this is not possible in COMPARE mode. It is also important to note that this does not send a full copy of your source data to all target servers. Instead, it spreads the resources among the targets, in round-robin fashion.
If credentials are specified (e.g. using the --target-bearer-token argument), the same credentials will be sent to all targets.
For example:
bin/smileutil synchronize-fhir-servers \
--source http://localhost:8000 \
--target http://example1.com:8000 \
--target http://example2.com:8000 \
--fhir-version R4