35.4.1Troubleshooting Logs

Beginning with the 2023.05.R01 release of Smile CDR, the entire logging infrastructure has been re-designed to make it easier to allow customization to occur. With this re-design, there is no longer any reason to change the classes/logback.xml file at all, and a new file has been provided called customerlib/logback-smile-custom.xml which is where we allow you to make any custom enhancements you want to make in order to obtain additional logging from the server. Note that as of this release, the intention of the smile.log file is to remain configured as-is at the INFO level and any new logging you add or enable will appear in a new log file called log/smile-troubleshooting.log. We have also provided several examples in the customerlib/logback-smile-custom.xml file showing you how to enable our pre-configured troubleshooting loggers as well as how to add your own custom loggers. The logging configuration for the customerlib/logback-smile-custom.xml file sets no limits on the Log Level so you may add any Log Level you wish to your custom loggers or our pre-configured loggers and those messages will be stored in this new troubleshooting log file.

There are several pre-configured specialized loggers that can be enabled in order to troubleshoot specific issues within a Smile CDR installation. In the default configuration, only INFO (or higher) level troubleshooting messages will be logged. See below for instructions on enabling DEBUG or TRACE level logging of a troubleshooting logger.

All troubleshooting log messages are directed to the smile-troubleshooting.log log file which is present in the Smile /log directory. Troubleshooting messages logged at INFO or higher are also included in the operational smile.log logfile.

35.4.2Docker Considerations

You can direct any log to the Docker console, and view them via a call to docker logs. To include all troubleshooting logs, uncomment the following snippet in your customer-lib/logback-smile-custom.xml file:

<logger name="ca.cdr.log" level="DEBUG">
    <appender-ref ref="STDOUT_SYNC"/>
</logger>
<logger name="ca.uhn.fhir.log" level="DEBUG">
    <appender-ref ref="STDOUT_SYNC"/>
</logger>

35.4.3Troubleshooting Logs Endpoint

You can change the configuration of the troubleshooting log by updating it via the Troubleshooting Log Endpoint.

To get the actual troubleshooting log configuration, execute Fetch Troubleshooting Log Configuration request:

GET http://localhost:9000/troubleshooting-log/configuration

To change the troubleshooting log level for a single logger, use Update Troubleshooting Logger Level request:

PUT or POST http://localhost:9000/troubleshooting-log/configuration/{logger_name}?loggerLevel=DEBUG

To update troubleshooting log configuration for multiple loggers, execute Update Troubleshooting Log Configuration request:

POST http://localhost:9000/troubleshooting-log/configuration
Content-Type: application/json

[
   {
      "loggerName":"ca.uhn.fhir.log.mdm_troubleshooting",
      "loggerLevel":"DEBUG"
   },
   {
      "loggerName":"ca.cdr.log.hl7v2_troubleshooting",
      "loggerLevel":"DEBUG"
   }
]

35.4.4CDA Exchange+ Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on the operations of the CDA Exchange+ Module in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.cda_troubleshooting" level="DEBUG"/> -->

35.4.5Consent Module Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on the operations of the Consent Module in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.consent_troubleshooting" level="DEBUG"/> -->

35.4.6HL7V2 Troubleshooting Log

The HL7V2 Troubleshooting Log contains details about processing for HL7 v2.x messages. Enable this log to diagnose parsing and processing issues for HL7 v2.x interfaces.

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details about the HL7V2 server processing pipeline in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.hl7v2_troubleshooting" level="DEBUG"/> -->

35.4.7HTTP Troubleshooting Log

The HTTP Troubleshooting Log contains details about any received HTTP requests including request details, HTTP client details, processing time, etc.

The HTTP Troubleshooting Log is an alternative to using HTTP Access Logs that have been defined on individual modules. Note that HTTP Access Logs are more powerful since they can be defined with much more customizable formats and details, but the HTTP Troubleshooting Log can be enabled and disabled on a running server without needing to perform a restart, so each log has advantages.

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details about the HTTP server processing pipeline in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.http_troubleshooting" level="DEBUG"/> -->

35.4.7.1Including Extra Detail

If you are trying to troubleshoot networking issues, you can also enable the HTTP Troubleshooting Log in Trace Mode. This will log additional details about request headers, stream reading, etc. To enable trace mode, change the level from DEBUG to TRACE in both the filter level and the logger level above.

35.4.8Partition Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file to capture details about the partition routing for each FHIR request in the smile-troubleshooting.log file.

<!-- <logger name="ca.uhn.fhir.log.partition_troubleshooting" level="DEBUG"/> -->

35.4.9Security Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details about the security layer in the smile-troubleshooting.log file. This can be handy when trying to determine why a particular operation is or is not being permitted.

<!-- <logger name="ca.cdr.log.security_troubleshooting" level="DEBUG"/> -->

35.4.10Subscription Troubleshooting Log

The Subscription Troubleshooting log contains details about the subscription processing and any resources that pass through it. It shows the various steps along the pipeline including queueing/dequeuing, deliveries, and failures.

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details about the subscription processing pipeline in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.subscription_troubleshooting" level="DEBUG"/> -->

35.4.11LiveBundle Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing LiveBundle details in the smile-troubleshooting.log file. These details include the number of LiveBundle records saved and purged when Subscribers are added to or deleted from Watchlists. Aggregation events have a DEBUG level, and even more detail is available at the TRACE loglevel.

<!-- <logger name="ca.cdr.log.livebundle_troubleshooting" level="DEBUG"/> -->

35.4.12MDM Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on MDM rule evaluation on incoming resources in the smile-troubleshooting.log file. MDM outcome events have a DEBUG level, and even more detail is available at the TRACE loglevel.

<!-- <logger name="ca.uhn.fhir.log.mdm_troubleshooting" level="DEBUG"/> -->

35.4.13Channel Import Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on ingestion of resources via the Channel Import module in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.channel_import_troubleshooting" level="DEBUG"/> -->

35.4.14Realtime Export Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on Realtime Export processing in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.realtime_export_troubleshooting" level="DEBUG"/> -->

Note that this example is set to DEBUG level. Realtime export also supports TRACE level, if you are interested in seeing the actual values populated into the SQL calls for debugging purpose. Note that this should never be enabled in a production environment, and should only be used for testing.

35.4.15FHIR Gateway Troubleshooting Log

If you are having trouble diagnosing routing logic within a FHIR Gateway module, the FHIR Gateway Troubleshooting Log can provide helpful insight.

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on FHIR Gateway activity in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.fhirgateway_troubleshooting" level="DEBUG"/> -->

35.4.16Connection Pool Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on Database Connection Pool activity in the smile-troubleshooting.log file.

DEBUG log level displays stack traces of abandoned connections (if db.connectionpool.log_abandoned and remove_abandoned_on_borrow are both set to true).

<!-- <logger name="ca.cdr.log.connection_pool_troubleshooting" level="DEBUG"/> -->

35.4.17Batch Framework Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on the operations of the Batch Framework in the smile-troubleshooting.log file.

<!-- <logger name="ca.uhn.fhir.log.batch_troubleshooting" level="DEBUG"/> -->

35.4.18Narrative Generation Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on Narrative Generation activity in the smile-troubleshooting.log file.

<!-- <logger name="ca.uhn.fhir.log.narrative_generation_troubleshooting" level="DEBUG"/> -->

35.4.19Terminology Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on Terminology mapping activity in the smile-troubleshooting.log file.

<!-- <logger name="ca.uhn.fhir.log.terminology_troubleshooting" level="DEBUG"/> -->

35.4.20MongoDB Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on MongoDB activity in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.mongodb_troubleshooting" level="DEBUG"/> -->

35.4.21Prior Auth Troubleshooting Log

The following snippet can be uncommented in your customer-lib/logback-smile-custom.xml file in order to start capturing details on Prior Auth activity in the smile-troubleshooting.log file.

<!-- <logger name="ca.cdr.log.pa_troubleshooting" level="DEBUG"/> -->

35.4.22Custom Logging

Custom Logging will allow you to choose what pieces of data to track. Information regarding this can be found at custom troubleshooting log.