Smile CDR v2024.05.PRE
On this page:

26.2.1Transaction Log

 

The Smile CDR transaction log is responsible for creating a temporary log of incoming and outgoing transactions as they enter and leave the system via FHIR REST or HL7 v2 endpoints. Note that transactions loaded through other endpoints such as ETL Import will not necessarily be logged.

The transaction log can be setup in many different ways:

  1. The Cluster Manager has a built-in Transaction Log persistence that can be enabled or disabled. If enabled, the transaction log will be written to the database.
  2. The Cluster Manager has a built-in Transaction Log Broker writer that can be enabled.
  3. There is a separate Transaction Log Persistence module. This permits you to separate your cluster manager database schema from your transaction log database schema.
  4. There is a separate Transaction Log Broker module. This module sends transaction logs to a specific topic on the cluster manager broker, with no database persistence. You can also setup multiple modules of this type, with different Allow/Deny lists, in order to route different events to different topics.

Configuration for the Cluster Manager and the Transaction Log modules is stored in the Node Configuration Properties File.

26.2.1.1Transaction Log in the Cluster Manager

This is the original implementation of the transaction log. It can store data in both its local storage, as well as send transaction log data to a broker such as ActiveMQ or Kafka. We no longer recommend using the Cluster Manager transaction log, as it can quickly balloon in size, and reduce portability of the Cluster Manager schema.

26.2.1.2Transaction Log Persistence Module

This module stores Transaction Log events in a dedicated storage layer separate from the Cluster Manager. It supports all the same configuration options as the Cluster Manager, except the configuration for sending data to a broker. Note that you can setup multiple transaction log modules in order to have different data retention periods for events of different types/subtypes. Transaction Logs stored by this module can be viewed in the Web Admin Console's Runtime Transaction Log page.

26.2.1.3Transaction Log Broker Module

This module sends transaction logs off to a topic/queue on a remote broker (ActiveMQ/Kafka). No local persistence of the Transaction Log occurs. Events sent to a module of this type cannot be viewed on the Web Admin Console's Runtime Transaction Log page. When multiple steps of a given Transaction Log are sent separately, (such as is the case with MDM or HL7V2 occasionally), the events are tied together with a matching GUID, allowing you to rebuild a full event downstream, as the sum of its parts.

Note that when Kafka is used as a broker, the Transaction Log's GUID is used as the partitioning key.

26.2.2What Does the Transaction Log Store?

 

The transaction log stores the following fields for each transaction it receives:

  • Initial Timestamp: the time at which the transaction started
  • Transaction GUID: a unique, automatically-assigned identifier for the transaction
  • Request ID: an ID assigned to the request that began the transaction
  • Processsing Time: the duration of the transaction
  • Type: see below
  • Subtype: see below

As relevant, a transaction log entry can contain other information, including facts about the endpoint that facilitated the transaction, as well as the initial payload. On the receiving end, these events are:

  • Timestamp: the times at which the endpoint that received the initial request did so
  • HTTP Request: the HTTP verb and URL of the request
  • Local Address: the URL at which the endpoint is configured to receive requests
  • Body: the body of the request, for example, a FHIR resource or HL7 v2.x message

In terms of replies generated surrounding a transaction, the following fields are recorded:

  • Timestamp: the time at which the reply was issued
  • Outcome: whether the transaction succeeded or failed
  • Local Address: the address from which was issued the reply
  • Body: the body of the reply, for example, a string such as "201 Created" or an HL7 v2.x ACK message
  • HTTP Response Status: the HTTP response status returned with the reply

For more information see the Transaction Log Events API and the Transaction Log Step API.

26.2.3Disabling the Transaction Log on the Cluster Manager

 

The following property may be used to control the transaction log.

  • transactionlog.enabled – Controls whether the transaction log will be written to. The default value is true. If set to false, no new entries will be written (existing entries are not deleted simply by modifying this setting).

The following snippet shows this property being used to disable the transaction log in the configuration properties file.

module.clustermgr.config.transactionlog.enabled=false

26.2.4Transaction Log Configuration

 

26.2.4.1Controlling the Transaction Log Size

  • transactionlog.persist_bodies_in_clustermgr_db – Controls whether payload bodies will be stored in the transaction log. The default value is true. If set to false, items such as raw HTTP requests and responses, HL7 v2.x messages, etc., will not be written to the transaction log. Storing payload bodies is helpful for troubleshooting, but has implications in terms of performance (since this potentially means a lot of extra data is saved in the database) and in terms of privacy (since the transaction log is not aware of the meaning of payloads and therefore can not restrict access by subject or other payload properties). This setting is a master setting, and can be further refined using several other options:

  • retain_transaction_log_days – Controls how many days to keep transaction log entries before deleting them.

26.2.4.2Persisting Transaction Bodies

By default the transaction log will store the request and response bodies for transactions flowing through the system.

26.2.4.3Controlling Transaction Log Events

By default all events will be written to the transaction log. This is useful for troubleshooting, but can quickly make the transaction log very noisy. For this reason, it is often useful to trim certain events out of the log. Note: these transaction bodies may contain PHI, and access should be controlled accordingly.

Events in the Transaction Log can be customized using the Event Whitelist and Event Blacklist properties. If the Event Whitelist property is set, only events matched by the whitelist will be included in the transaction log. If the Event Blacklist is set, any events that are matched by the blacklist will not be processed. Note that the whitelist is processed first, and any events that pass through the whitelist (or all events, if no whitelist is set) are then filtered by the blacklist.

The format of the whitelist/blacklist is a comma separated list of codes, in the format: [Type Code]|[SubType Code]. See Event Types and SubTypes below for a list of valid event codes. Both the Type Code and SubType Code may be a code from the list of available codes, or the value * to match all codes.

The following configuration properties are used to control the lists.

For example, the following settings will cause the transaction log to include ONLY received HL7 v2.x messages, excluding ADT^A28 and ADT^A31 messages.

module.clustermgr.config.transactionlog.event_whitelist=HL7V2_INBOUND|*
module.clustermgr.config.transactionlog.event_blacklist=HL7V2_INBOUND|HL7V2_ADT_A28, HL7V2_INBOUND|HL7V2_ADT_A31

26.2.5Performance Tracing

 

If you are trying to troubleshoot poor performance on a FHIR operation, enabling Performance Tracing with the transaction log as an enabled target can be a helpful way of diagnosing issues.

With this feature enabled, warnings will be added to your transaction log if the system detects certain types of inefficient queries. In addition, if Capture Raw SQL or Equivalent is also enabled, the raw query sent to the database will be included so you can analyze it, look at query plans, etc.

26.2.6Event Types and SubTypes

 

This section lists the event codes for transaction log entries. Every entry in the transaction log will have exactly one type code and one subtype code.

The Type Code can be thought of as the high level category, with the SubType Code being the specific payload type. Note that these codes can have a many-to-many relationship. For example, the HL7V2_ADT_A01 SubType code means that the specific transaction involves an HL7 v2.x ADT^A01 message. This SubType code will be logged with the Type Code of HL7V2_INBOUND when this type of message is received by Smile CDR, and will be logged with a Type Code of HL7V2_OUTBOUND when this message is sent by Smile CDR.

26.2.6.1Event Type Codes

CAMEL_PROCESSOR
Camel Processor Indicates transaction log was generated from a Camel Processor
CDS_HOOKS_REQUEST
CDS Hooks Request Indicates that a REST client request has been handled by a CDS Hooks Endpoint.
ETL_IMPORTER
ETL Importer Indicates that a row has been processed during an ETL import job.
FHIR_REQUEST
FHIR Request Indicates that a FHIR client request has been handled by a Smile CDR FHIR Server Endpoint.
HL7V2_INBOUND
HL7 v2.x Inbound Indicates that an HL7 v2.x message has been received by Smile CDR.
HL7V2_OUTBOUND
HL7 v2.x Outbound Indicates that an HL7 v2.x message has been sent by Smile CDR.
P2P_BATCH_JOB
P2P Batch Job Indicates that a P2P batch job has been handled by Smile CDR.
SOAP_REQUEST
SOAP Request Indicates that a SOAP client request has been handled by Smile CDR.

26.2.6.2Event SubType Codes

CAMEL_PROCESS
Camel Process
CDS_HOOK_SERVICES_LIST
Services List
CDS_HOOK_SERVICE_REQUEST
Service Request
CDS_HOOK_SUBMIT_FEEDBACK_REQUEST
Submit Feedback Request
DOCREF
DOCREF ($docref)
DQM_QPP_BUILD
Qpp Build($qpp-build)
FHIR_APPLY_CODESYSTEM_DELTA_ADD
Apply Codesystem Delta: Add Codes
FHIR_APPLY_CODESYSTEM_DELTA_REMOVE
Apply Codesystem Delta: Remove Codes
FHIR_BINARY_ACCESS_READ
FHIR Binary Access - Read ($binary-access-read)
FHIR_BINARY_ACCESS_WRITE
FHIR Binary Access - Write ($binary-access-write)
FHIR_CODESYSTEM_LOOKUP
Code Lookup ($lookup)
FHIR_CODESYSTEM_SUBSUMES
CodeSystem Subsumes ($subsumes)
FHIR_CODESYSTEM_VALIDATE_CODE
CodeSystem Validate Code ($validate-code)
FHIR_COMPOSITION_DOCUMENT
Generate Document ($document)
FHIR_CONCEPTMAP_TRANSLATE
Translate Code ($translate)
FHIR_CONFORMANCE
Fetch Conformance (/metadata)
FHIR_CREATE
Create Resource
FHIR_DELETE
Delete Resource
FHIR_GET_PAGE
Fetch Page
FHIR_HISTORY_INSTANCE
History (Instance)
FHIR_HISTORY_SYSTEM
History (System)
FHIR_HISTORY_TYPE
History (Type)
FHIR_MARK_ALL_RESOURCES_FOR_REINDEXING
Mark all resources for reindexing
FHIR_META_ADD
Add to Metadata ($meta-add)
FHIR_META_DELETE
Delete from Metadata ($meta-delete)
FHIR_META_GET
Fetch Metadata ($meta)
FHIR_OPERATION_ADD_PARTITION
Add Partition ($fhir-management-add-partition)
FHIR_OPERATION_APPLY
Apply ('$apply')
FHIR_OPERATION_CARE_GAPS
Care Gaps($care-gaps)
FHIR_OPERATION_CQL
CQL($cql)
FHIR_OPERATION_DELETE_EXPUNGE
Delete Expunge Operation ($delete-expunge)
FHIR_OPERATION_DELETE_PARTITION
Delete Partition ($fhir-management-delete-partition)
FHIR_OPERATION_DIFF
Diff Operation ($diff)
FHIR_OPERATION_EVALUATE
Evaluate($evaluate)
FHIR_OPERATION_EVALUATE_MEASURE
Evaluate Measure($evaluate-measure)
FHIR_OPERATION_EVERYTHING_ENCOUNTER_INSTANCE
Search Entire Encounter Record ($everything)
FHIR_OPERATION_EVERYTHING_PATIENT_INSTANCE
Search Entire Patient Chart ($everything)
FHIR_OPERATION_EVERYTHING_PATIENT_TYPE
Search Everything Related to Patients ($everything)
FHIR_OPERATION_EXPUNGE
Expunge Operation ($expunge)
FHIR_OPERATION_EXTRACT
Extract ('$extract')
FHIR_OPERATION_GET_RESOURCE_COUNTS
Get Resource Counts Operation ($get-resource-counts)
FHIR_OPERATION_GRAPHQL
GraphQL Operation ($graphql)
FHIR_OPERATION_IMPORT
FHIR Bulk Import ($import)
FHIR_OPERATION_INITIATE_BULK_EXPORT
Initiate FHIR Bulk Export ($export)
FHIR_OPERATION_LASTN_OBSERVATION_TYPE
Search for most recent or last n=number of Observation records ($lastn)
FHIR_OPERATION_LIST_PARTITIONS
List Partitions ($fhir-management-list-partitions)
FHIR_OPERATION_LIVEBUNDLE
LiveBundle ($livebundle)
FHIR_OPERATION_LIVEBUNDLE_SUBSCRIBER_GROUP_ADD
LiveBundle Subscription Group Add ($livebundle-group-add)
FHIR_OPERATION_LIVEBUNDLE_SUBSCRIBER_GROUP_DELETE
LiveBundle Subscription Group Delete ($livebundle-group-delete)
FHIR_OPERATION_LIVEBUNDLE_WATCHLIST
LiveBundle Watchlist ($livebundle-watchlist)
FHIR_OPERATION_LIVEBUNDLE_WATCHLIST_ADD
LiveBundle Watchlist Add ($livebundle-watchlist-add)
FHIR_OPERATION_LIVEBUNDLE_WATCHLIST_DELETE
LiveBundle Watchlist Delete ($livebundle-watchlist-delete)
FHIR_OPERATION_LIVEBUNDLE_WATCHLIST_SUBSCRIBERS
LiveBundle Watchlist Subscribers ($livebundle-watchlist-subscribers)
FHIR_OPERATION_MATCH_PATIENT
FHIR Patient Match ($match)
FHIR_OPERATION_MATCH_RESOURCE
MDM Match Resource ($mdm-match)
FHIR_OPERATION_MDM_CLEAR
MDM Clear links($mdm-clear)
FHIR_OPERATION_MDM_CREATE_LINK
MDM Create Link ($mdm-create-link)
FHIR_OPERATION_MDM_DUPLICATE_GOLDEN_RESOURCES
MDM Duplicate Golden Resources ($mdm-duplicate-golden-resources)
FHIR_OPERATION_MDM_LINK_HISTORY
MDM Link History ($mdm-link-history)
FHIR_OPERATION_MDM_MERGE_GOLDEN_RESOURCES
MDM Merge Golden Resources ($mdm-merge-golden-resources)
FHIR_OPERATION_MDM_NOT_DUPLICATE
MDM Not Duplicate ($mdm-not-duplicate)
FHIR_OPERATION_MDM_QUERY_LINKS
MDM Query Links ($mdm-query-links)
FHIR_OPERATION_MDM_SUBMIT
MDM Batch Processing($mdm-submit)
FHIR_OPERATION_MDM_UPDATE_LINK
MDM Update Link ($mdm-update-link)
FHIR_OPERATION_MEMBER_MATCH
Member match ($member-match)
FHIR_OPERATION_PACKAGE
Package ('$package')
FHIR_OPERATION_POLL_BULK_EXPORT_JOB_STATUS
Poll For Bulk Export Job Status ($export-poll-status)
FHIR_OPERATION_POLL_BULK_IMPORT_JOB_STATUS
Poll For Bulk Import Job Status ($import-poll-status)
FHIR_OPERATION_POPULATE
Populate ('$populate')
FHIR_OPERATION_PREPOPULATE
Prepopulate ('$prepopulate')
FHIR_OPERATION_READ_PARTITION
Read Partition ($fhir-management-read-partition)
FHIR_OPERATION_REINDEX
Reindex Operation ($reindex)
FHIR_OPERATION_REINDEX_TERMINOLOGY
Reindex terminology ($reindex-terminology)
FHIR_OPERATION_SNAPSHOT
Snapshot ($snapshot)
FHIR_OPERATION_SUBMIT_ATTACHMENT
Submit attachment operation ($submit-attachment)
FHIR_OPERATION_SUBMIT_DATA
Submit Data($submit-data)
FHIR_OPERATION_SUGGEST_KEYWORDS
Suggest Keywords
FHIR_OPERATION_SUMMARY_PATIENT_INSTANCE
Generate IPS Document ($summary)
FHIR_OPERATION_TRIGGER_SUBSCRIPTION
Manually Trigger Subscription ($trigger-subscription)
FHIR_OPERATION_UNCLASSIFIED
FHIR Operation (Unclassified)
FHIR_OPERATION_UPDATE_PARTITION
Update Partition ($fhir-management-update-partition)
FHIR_PATCH
Patch Resource
FHIR_PERFORM_REINDEXING_PASS
Perform reindexing pass
FHIR_PROCESS_MESSAGE
Process Message ($process-message)
FHIR_READ
Read Resource
FHIR_SEARCH_SYSTEM
Search Across All Resource Types
FHIR_SEARCH_TYPE
Search for Resources
FHIR_TRANSACTION
Transaction
FHIR_UPDATE
Update Resource
FHIR_UPLOAD_EXTERNAL_CODE_SYSTEM
Upload External Terminology Code System
FHIR_VALIDATE
Validate Resource ($validate)
FHIR_VALUESET_EXPAND
Expand ValueSet ($expand)
FHIR_VALUESET_INVALIDATE_EXPANSION
Invalidate Expansion ($invalidate-expansion)
FHIR_VALUESET_VALIDATE_CODE
ValueSet Validate Code ($validate-code)
FHIR_VREAD
VRead Resource
HFQL_EXECUTE
Execute HFQL Request
HL7V2_ADT_A01
ADT^A01 (Admit)
HL7V2_ADT_A02
ADT^A02 (Transfer)
HL7V2_ADT_A03
ADT^A03 (Discharge)
HL7V2_ADT_A04
ADT^A04 (Register)
HL7V2_ADT_A05
ADT^A05 (Pre-Admit)
HL7V2_ADT_A06
ADT^A06 (Convert OP to IP)
HL7V2_ADT_A07
ADT^A07 (Convert IP to OP)
HL7V2_ADT_A08
ADT^A08 (Update Visit)
HL7V2_ADT_A11
ADT^A11 (Cancel Admit)
HL7V2_ADT_A12
ADT^A12 (Cancel Transfer)
HL7V2_ADT_A13
ADT^A13 (Cancel Discharge)
HL7V2_ADT_A14
ADT^A14 (Pending Admit)
HL7V2_ADT_A15
ADT^A15 (Pending Transfer)
HL7V2_ADT_A16
ADT^A16 (Pending Discharge)
HL7V2_ADT_A17
ADT^A17 (Swap Patients)
HL7V2_ADT_A21
ADT^A21 (Patient begins LOA)
HL7V2_ADT_A22
ADT^A22 (Patient returns from LOA)
HL7V2_ADT_A24
ADT^A24 (Link Patient Information)
HL7V2_ADT_A25
ADT^A25 (Cancel Pending Discharge)
HL7V2_ADT_A27
ADT^A27 (Cancel Pending Admit)
HL7V2_ADT_A28
ADT^A28 (Add Patient)
HL7V2_ADT_A29
ADT^A29 (Delete Person Information)
HL7V2_ADT_A30
ADT^A30 (Merge Person Information)
HL7V2_ADT_A31
ADT^A31 (Update Patient)
HL7V2_ADT_A34
ADT^A34 (Merge Patient Information)
HL7V2_ADT_A37
ADT^A37 (Unlink Patient)
HL7V2_ADT_A38
ADT^A38 (Cancel Pre-Admit)
HL7V2_ADT_A39
ADT^A39 (Merge Person - Patient ID)
HL7V2_ADT_A40
ADT^A40 (Merge Patient)
HL7V2_ADT_A45
ADT^A45 (Move Visit Information - Visit Number)
HL7V2_ADT_A47
ADT^A47 (Change Patient Identifier List)
HL7V2_ADT_A54
ADT^A54 (Change Attending)
HL7V2_ADT_A60
ADT^A60 (Update Allergies)
HL7V2_ADT_A61
ADT^A61 (Change Consulting)
HL7V2_BAR_P01
BAR^P01 (Establish Billing Account)
HL7V2_BAR_P12
BAR^P12 (Update Diagnosis/Procedures)
HL7V2_DFT_P03
DFT^P03 (Post Detailed Financial Transaction)
HL7V2_OMG_O19
OMG^O19 (General Clinical Order Message)
HL7V2_ORM_O01
ORM^O01 (General Clinical Order Message)
HL7V2_ORU_R01
ORU^R01 (Unsolicited Result)
HL7V2_RAS_O17
RAS^O17 (Medication Administration)
HL7V2_RDE_O01
RDE^O01 (Medication Order - Legacy)
HL7V2_RDE_O11
RDE^O11 (Medication Order)
HL7V2_SIU_S12
SIU^S12 (Appointment Scheduling)
HL7V2_SIU_S13
SIU^S13 (Appointment Rescheduling)
HL7V2_SIU_S14
SIU^S14 (Appointment Modification)
HL7V2_SIU_S15
SIU^S15 (Appointment Cancellation)
HL7V2_SIU_S17
SIU^S17 (Appointment Deletion)
HL7V2_SIU_S26
SIU^S26 (Patient Did Not Show Up for Scheduled Appointment)
HL7V2_UNKNOWN_MESSAGE_TRIGGER
Unknown Message Trigger
HL7V2_VXU_V04
VXU^V04 (Unsolicited Vaccination Record Update)
INVALID_REQUEST
Invalid Request
OPERATION_IMPORT_CDA
CDA Import ($import-cda)
P2P_BATCH_JOB_REQUEST
P2P batch job request
UNSPECIFIED
Unspecified
_FHIR_VALUESET_SUBSUMES
ValueSet Subsumes (Not Used)

26.2.7Realtime Reporting

 

The transaction log can be configured to place transaction events on a broker channel for consumption by other systems using the

module.clustermgr.config.transactionlog.broker.enabled

flag. If set to "true," events will be placed on a queue named smilecdr-transaction.

When submitting data to the broker, the steps are submitted individually, with a GUID as an identifier for the parent event.

26.2.8Hiding Request Body

 

Transaction logs can be configured to display request body contents when viewed in the admin console.

By default, transaction logs will not display body contents, thereby hiding Personal Health Information (PHI) and Personal Identifiable Information (PII).

This can be turned on for debugging purposes by using the following flag:

module_config.clustermgr.transactionlog.show_request_body.enabled

26.2.9Changing Timezone

 

Transaction logs are stored in Coordinated Universal Time (UTC) in the database. When transaction logs are viewed in the list, the time is displayed as in the server local timezone. And when a transaction log is viewed in a detail view, the time is displayed as in the client local timezone.

Note that if the client timezone is switched to be a different timezone than the server, the time displayed in the transaction log list and in the detail transaction event view might contain some inconsistencies. In this case, clear the cache in your browser will resolve the issue.

26.2.10Using multiple Transaction Logs

 

You can configure multiple transaction logs to be used by the system. This is useful if you want to have different transaction logs for different purposes or retention periods. For example, you might want to have a transaction log that only logs FHIR transactions, and another transaction log that only logs HL7 v2.x transactions. This could be accomplished by setting up two transaction logs, and setting their allowlist/denylist events to reflect what you are interested in storing.

To setup multiple transaction logs you will need to modify the cdr-config-Master.properties file in Properties mode since due to security reasons this cannot be setup in the Web Admin Console. In the properties file, duplicate the existing Transaction Log parameters and change the module name and the database. Then you can modify whitelist and blacklist properties accordingly. In the following example the transaction log will record all FHIR Request transactions except the binary read & write transactions. See the list of Event Types and SubTypes.

#################################################################
# External Transaction Log DB Config
#################################################################
module.transactionsFHIR.type=TRANSACTION_LOG_PERSISTENCE
module.transactionsFHIR.config.db.driver=H2_EMBEDDED
module.transactionsFHIR.config.db.url=jdbc:h2:file:./database/h2_transactionsFHIR;DB_CLOSE_ON_EXIT=FALSE
<other module settings>
module.transactionsFHIR.config.transactionlog.event_whitelist=FHIR_REQUEST|*
module.transactionsFHIR.config.transactionlog.event_blacklist=FHIR_REQUEST|FHIR_BINARY_ACCESS_READ,FHIR_REQUEST|FHIR_BINARY_ACCESS_WRITE

In this next example, the transaction logs for HL7V2 events will be recorded to the first broker topic, and all other events will be recorded to the second broker topic.

#################################################################
# External Transaction Log DB Config for HL7V2 message transaction logs, where the bodies of the messages are persisted
#################################################################
module.transaction_broker_hl7.type=TRANSACTION_LOG_BROKER
module.transaction_broker_hl7.config.transactionlog.channel.name=hl7v2-events
module.transaction_broker_hl7.config.transactionlog.persist_bodies_in_clustermgr_db=true
module.transaction_broker_hl7.config.transactionlog.event_whitelist=HL7V2|*

#################################################################
# External Transaction Log DB Config for non-HL7V2 message transaction logs
#################################################################
module.transaction_broker.type=TRANSACTION_LOG_BROKER
module.transaction_broker.config.transactionlog.channel.name=other-events
module.transaction_broker.config.transactionlog.persist_bodies_in_clustermgr_db=false
module.transaction_broker.config.transactionlog.event_blacklist=HL7V2|*