Realtime Export Overview
The purpose of the Realtime Export module is to permit realtime extraction of FHIR data from Smile CDR into a remote database. The following diagram shows a quick overview of the architecture of the realtime export process.
Before discussing configuration of the module, it is important to note that there are two main modes in which Realtime Export can operate. These two modes are
POINTCUT mode, the Smile CDR Storage module has an interceptor registered on it which forwards all Creates/Updates/Deletes to an internal channel, which Realtime Export consumes. The advantage of this mode is that it requires no external software, and works out of the box. To accomplish this,
Realtime Export relies on a Storage module as a dependency.
CHANNEL mode, Realtime Export has no knowledge of the Storage module, and instead relies on a third-party piece of software (Debezium) to populate the incoming channel.
More information about setting up Debezium can be found here. The advantage of running Realtime Export in this mode is that it is extremely low latency since Debezium relies on directly listening to database changes as they happen. In
CHANNEL mode, Realtime Export does not rely on a Storage module. In this mode, you must set
channel.name to indicate which channel Debezium is placing messages upon.
The FHIR Storage is the source of the data used by Realtime Export. If the source type is set to
POINTCUT, then this dependency must be filled. If you are instead using
CHANNEL, then Realtime Export module does not require a Storage module dependency. Note that if you do depend on the Storage module, you must enable the Pointcut-based Realtime Export mode so that the module correctly registers all necessary interceptors.
The Cluster Manager module contains the configuration used to connect to the selected message broker.
See the Message Brokers documentation for information on how to select and configure a message broker. By default an embedded Apache ActiveMQ server will be used, and this is acceptable for testing purposes but an external broker should be used in a production scenario.
At a high level, Realtime Export listens for all changes to FHIR resources. This includes creates, updates, and deletes. Upon detecting such a change,
the Realtime Export module will apply any relevant transformations to the resource, in order to convert it into a form that can be easily consumed by an RDBMS. After this,
it will generate and execute the SQL commands necessary to update the remote database. This transformation is done via user-configured
Currently there are several limitations of Realtime Export.
Firstly, the remote database schema must already exist. This means that in order to effectively use realtime export, it will be necessary to spend some time creating the schema manually. During schema creation, there are several needs that must be addressed.
version. This is needed to handle logical updates of remote data.
id. This will be automatically filled with a UUID.
parent_reference, which is constrained by the ID of the parent table it refers to.
source_resource_id.This is needed to handle logical deletes.
Next, it is important to note that each remote table is designed to hold exactly one resource type. This means that in its current implementation, remote tables cannot be partially filled by multiple resources. In general, one resource corresponds to one completed row in one or more remote tables.
Here is an example of an appropriate schema for realtime export. Note that in future examples, some columns may be omitted for brevity.
The Realtime Export Troubleshooting log can be helpful in diagnosing issues relating to Realtime Export. Furthermore, if a message completely fails processing, the cause of the failure is stored along with the failed message.