35.2.1Transaction Log

The Smile CDR transaction log is responsible for creating a temporary log of incoming and outgoing transactions as they enter and leave the system via external endpoints such as FHIR REST or HL7 v2 endpoints.

The transaction log is intended to be a troubleshooting tool, which can capture extensive details about transactions as they are being processed. It is not intended to be a permanent audit trail (such as the Audit Log), and is often disabled in production because of its potentially high storage requirements.

The transaction log has several storage mechanisms:

• None: Transaction logs can be completely disabled.
• Cluster Manager Transaction Log Storage: A set of tables within the Cluster Manager database schema can be used to store transaction log entries.
• Transaction Log Persistence Module: A dedicated module with its own database schema and storage can be used to store transaction log events.
• Transaction Log Broker Module: Transaction log entries can be written to a broker channel for processing elsewhere.

Configuration for the Cluster Manager and the Transaction Log modules is stored in the Node Configuration Properties File.

35.2.2Cluster Manager Transaction Log Storage

It is possible to store transaction log events directly in the Smile CDR Cluster Manager database schema. This is a convenient option for low-volume systems, especially non-production systems.

This option is not recommended on systems with large transaction volumes, or where significant log retention is needed since the large amount of data being stored by the Transaction Log can impact the operation of other cluster manager functions, and complicates actions such as backups.

The Cluster Manager Transaction Log Storage can be enabled or disabled using the transactionlog.enabled setting in your configuration file. The following setting enables the Cluster Manager Transaction Log Storage:

module.clustermgr.config.transactionlog.enabled=true

If this setting is enabled, you can also configure settings related to retention and level of detail to store in the log. See Configuration below for more details.

35.2.3Transaction Log Persistence Module

This module stores Transaction Log events in a dedicated database schema separate from the Cluster Manager. Because the Transaction Log tends to be a very high frequency database (i.e. many records are written in quick succession, and old records are purged regularly), it is generally a good idea to use this module and connect it to a dedicated database schema which is not being used for any other purposes.

Any transaction log modules which are active and started will be used to write transaction log events originating on the same node. For example, if you have a FHIR Endpoint module defined as a part of a node configuration, transaction log events will be written by any Transaction Log Persistence modules running on the same node configuration.

This module should be supplied with database connection settings. You can also configure settings related to retention and level of detail to store in the log. See Configuration below for more details.

35.2.4Transaction Log Broker Module

This module sends transaction logs off to a topic/queue on a remote broker (ActiveMQ/Kafka). No local persistence of the Transaction Log occurs. Events sent to a module of this type cannot be viewed on the Web Admin Console's Runtime Transaction Log page. When multiple steps of a given Transaction Log are sent separately, (such as is the case with MDM or HL7V2 occasionally), the events are tied together with a matching GUID, allowing you to rebuild a full event downstream, as the sum of its parts.

Note that when Kafka is used as a broker, the Transaction Log's GUID is used as the partitioning key.

The following is an example of a message as it would be received by the terminal broker:

{
  "payload": {
    "initialTimestamp": "2024-08-20T14:00:30.198-04:00",
    "type": "FHIR_REQUEST",
    "subType": "FHIR_READ",
    "outcome": "SUCCESS",
    "endpointNodeId": "Master",
    "endpointModuleId": "fhir_endpoint",
    "processingTime": 0,
    "requestId": "YBNnt0RgBMLNfJmF",
    "events": [
      {
        "endpointLocalHost": "127.0.0.1",
        "endpointLocalPort": 8000,
        "endpointRemoteHost": "127.0.0.1",
        "endpointRemotePort": 37490,
        "initialTimestamp": "2024-08-20T14:00:30.198-04:00",
        "requestUrl": "http://127.0.0.1:8000/Patient/1403",
        "requestVerb": "GET",
        "type": "ENDPOINT_RECEIVE"
      },
      {
        "body": "{\n  \"resourceType\": \"Patient\",\n  \"id\": \"1403\",\n  \"meta\": {\n    \"versionId\": \"1\",\n    \"lastUpdated\": \"2024-08-20T13:59:08.692-04:00\",\n    \"source\": \"#NX4jAIEtLyvF8Jpf\"\n  }\n}",
        "bodyType": "FHIR_JSON",
        "endpointLocalHost": "127.0.0.1",
        "endpointLocalPort": 8000,
        "endpointRemoteHost": "127.0.0.1",
        "endpointRemotePort": 37490,
        "initialTimestamp": "2024-08-20T14:00:30.223-04:00",
        "outcome": "SUCCESS",
        "type": "ENDPOINT_REPLY",
        "responseStatus": 200
      }
    ],
    "userModuleId": "local_security",
    "userNodeId": "Master",
    "userName": "ADMIN",
    "userFamilyName": "GenericUser",
    "userGivenName": "Admin",
    "transactionGuid": "b5b79664-9bc4-441d-b476-81df6afa6262"
  }
}

35.2.5What Does the Transaction Log Store?

The transaction log stores the following fields for each transaction it receives:

• Initial Timestamp: the time at which the transaction started
• Transaction GUID: a unique, automatically-assigned identifier for the transaction
• Request ID: an ID assigned to the request that began the transaction
• Processing Time: the duration of the transaction
• Type: see below
• Subtype: see below

As relevant, a transaction log entry can contain other information, including facts about the endpoint that facilitated the transaction, as well as the initial payload. On the receiving end, these events are:

• Timestamp: the times at which the endpoint that received the initial request did so
• HTTP Request: the HTTP verb and URL of the request
• Local Address: the URL at which the endpoint is configured to receive requests
• Body: the body of the request, for example, a FHIR resource or HL7 v2.x message

In terms of replies generated surrounding a transaction, the following fields are recorded:

• Timestamp: the time at which the reply was issued
• Outcome: whether the transaction succeeded or failed
• Local Address: the address from which was issued the reply
• Body: the body of the reply, for example, a string such as "201 Created" or an HL7 v2.x ACK message
• HTTP Response Status: the HTTP response status returned with the reply

For more information see the Transaction Log Events API and the Transaction Log Step API.

35.2.6Transaction Log Configuration

35.2.6.1Controlling the Transaction Log Size

• persist_bodies_in_clustermgr_db – Controls whether payload bodies will be included in the transaction log. The default value is true. If set to false, items such as raw HTTP requests and responses, HL7 v2.x messages, etc., will not be written to the transaction log. Storing payload bodies is helpful for troubleshooting, but has implications in terms of performance (since this potentially means a lot of extra data is saved in the database) and in terms of privacy (since the transaction log is not aware of the meaning of payloads and therefore can not restrict access by subject or other payload properties). This setting is a master setting, and can be further refined using several other options:

  • persist_endpoint_receive_bodies – Controls whether the request body for incoming requests such as FHIR create operations will be included in the transaction log entries.

  • persist_endpoint_reply_bodies – Controls whether the response body for incoming requests such as FHIR search operations will be included in the transaction log entries.

• truncate_body.bytes – Limits the size of the transaction log entry by limiting the payload size to a certain number of bytes.

• retain_transaction_log_days – Controls how many days to keep transaction log entries before deleting them.

35.2.6.2Including Transaction Bodies

By default, the transaction log will store the request and response bodies for transactions flowing through the system.

35.2.7Controlling Transaction Log Events

The format of the whitelist/blacklist is a comma separated list of codes, in the format: [Type Code]|[SubType Code]. See Event Types and SubTypes below for a list of valid event codes. Both the Type Code and SubType Code may be a code from the list of available codes, or the value * to match all codes.

The following configuration properties are used to control the lists.

• event_whitelist – The whitelist.

• event_blacklist – The blacklist.

For example, the following settings will cause the transaction log to include ONLY received HL7 v2.x messages, excluding ADT^A28 and ADT^A31 messages.

module.transaction.config.event_whitelist=HL7V2_INBOUND|*
module.transaction.config.event_blacklist=HL7V2_INBOUND|HL7V2_ADT_A28, HL7V2_INBOUND|HL7V2_ADT_A31

An optional Module ID followed by a period (.) character (i.e. fhir_endpoint.) can be prepended to Event Types and SubTypes to further filter transaction logs in the Event Whitelist and Event Blacklist.

Example 1: Transaction logs will include ONLY received HL7 v2.x messages for the endpoint_hl7v2_in_1 module, excluding ADT^02 messages.

module.transaction.config.event_whitelist=endpoint_hl7v2_in_1.HL7V2_INBOUND|*
module.transaction.config.event_blacklist=HL7V2_INBOUND|HL7V2_ADT_A02

Example 2: Transaction logs will include ONLY received HL7 v2.x messages that DO NOT come from the endpoint_hl7v2_in_1 module.

module.transaction.config.event_whitelist=HL7V2_INBOUND|*
module.transaction.config.event_blacklist=endpoint_hl7v2_in_1.HL7V2_INBOUND|*

35.2.8Performance Tracing

If you are trying to troubleshoot poor performance on a FHIR operation, enabling Performance Tracing with the transaction log as an enabled target can be a helpful way of diagnosing issues.

With this feature enabled, warnings will be added to your transaction log if the system detects certain types of inefficient queries. In addition, if Capture Raw SQL or Equivalent is also enabled, the raw query sent to the database will be included so you can analyze it, look at query plans, etc.

35.2.9Event Types and SubTypes

This section lists the event codes for transaction log entries. Every entry in the transaction log will have exactly one type code and one subtype code.

The Type Code can be thought of as the high level category, with the SubType Code being the specific payload type. Note that these codes can have a many-to-many relationship. For example, the HL7V2_ADT_A01 SubType code means that the specific transaction involves an HL7 v2.x ADT^A01 message. This SubType code will be logged with the Type Code of HL7V2_INBOUND when this type of message is received by Smile CDR, and will be logged with a Type Code of HL7V2_OUTBOUND when this message is sent by Smile CDR.

35.2.9.1Event Type Codes

35.2.9.2Event SubType Codes

35.2.10Transaction Storage Outcomes

When ingesting large amounts of data, the transaction log can also be a helpful tool in linking transactions to specific resource storage outcomes.

Note that this feature currently only applies to storage actions performed as a part of FHIR Transaction or Batch operation, since ingestion of large volumes of data is typically performed using these operations.

When Persist Storage Outcome Details is enabled, FHIR Transaction/Batch operations executed through FHIR Endpoint or Camel-based procesing will result in additional details being attached to the corresponsing transaction log entry.

Each write action in the transaction/batch will result in one storage outcome record being attached to the write step within the transaction log entry. For example, if you execute a FHIR Transaction containing 10 POST (create) actions, this will result in 10 Storage Outcome Details being saved.

35.2.10.1Accessing Storage Outcomes

These outcomes can be viewed within the Web Admin Console, but are more likely to be useful as a part of automated analytics. These records can be fetched using the JSON Admin API Transaction Log Endpoint, pushed using the Transaction Log Broker Module, or accessed directly in the transaction log database.

35.2.10.2Data Model

Each record contains the following columns.

35.2.11Hiding Request Body

Transaction logs can be configured to display request body contents when viewed in the admin console.

By default, transaction logs will not display body contents, thereby hiding Personal Health Information (PHI) and Personal Identifiable Information (PII).

This can be turned on for debugging purposes by using:

• transactionlog.show_request_body.enabled

35.2.12Changing Timezone

Transaction logs are stored in Coordinated Universal Time (UTC) in the database. When transaction logs are viewed in the list, the time is displayed as in the server local timezone. And when a transaction log is viewed in a detail view, the time is displayed as in the client local timezone.

Note that if the client timezone is switched to be a different timezone than the server, the time displayed in the transaction log list and in the detail transaction event view might contain some inconsistencies. In this case, clear the cache in your browser will resolve the issue.

35.2.13Using Multiple Transaction Logs

You can configure multiple transaction logs to be used by the system. This is useful if you want to have different transaction logs for different purposes or retention periods. For example, you might want to have a transaction log that only logs FHIR transactions, and another transaction log that only logs HL7 v2.x transactions. This could be accomplished by setting up two transaction logs, and setting their allowlist/denylist events to reflect what you are interested in storing.

To setup multiple transaction logs you will need to modify the cdr-config-Master.properties file in Properties mode since due to security reasons this cannot be setup in the Web Admin Console. In the properties file, duplicate the existing Transaction Log parameters and change the module name and the database. Then you can modify whitelist and blacklist properties accordingly. In the following example the transaction log will record all FHIR Request transactions except the binary read & write transactions. See the list of Event Types and SubTypes.

#################################################################
# External Transaction Log DB Config
#################################################################
module.transactionsFHIR.type=TRANSACTION_LOG_PERSISTENCE
module.transactionsFHIR.config.db.driver=H2_EMBEDDED
module.transactionsFHIR.config.db.url=jdbc:h2:file:./database/h2_transactionsFHIR;DB_CLOSE_ON_EXIT=FALSE
<other module settings>
module.transactionsFHIR.config.event_whitelist=FHIR_REQUEST|*
module.transactionsFHIR.config.event_blacklist=FHIR_REQUEST|FHIR_BINARY_ACCESS_READ,FHIR_REQUEST|FHIR_BINARY_ACCESS_WRITE

In this next example, the transaction logs for HL7V2 events will be recorded to the first broker topic, and all other events will be recorded to the second broker topic.

#################################################################
# External Transaction Log DB Config for HL7V2 message transaction logs
#################################################################
module.transaction_broker_hl7.type=TRANSACTION_LOG_BROKER
module.transaction_broker_hl7.config.broker.channel.name=hl7v2-events
module.transaction_broker_hl7.config.persist_bodies_in_clustermgr_db=true
module.transaction_broker_hl7.config.event_whitelist=HL7V2|*

#################################################################
# External Transaction Log DB Config for non-HL7V2 message transaction logs
#################################################################
module.transaction_broker.type=TRANSACTION_LOG_BROKER
module.transaction_broker.config.broker.channel.name=other-events
module.transaction_broker.config.persist_bodies_in_clustermgr_db=false
module.transaction_broker.config.event_blacklist=HL7V2|*