OpenTelemetry is a framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs. Starting with the 2024.02 release, support for instrumentation with the OpenTelemetry agent is added to Smile CDR. This makes Smile CDR generate telemetry data that can be used to monitor Smile CDR's performance.
Currently, Smile CDR generates traces and metrics using the OpenTelemetry Java agent. This feature is currently in trial phase and is basically an auto-instrumentation with some minimal Smile CDR related customizations. The auto-instrumentation with the OpenTelemetry Java agent generates telemetry data for the libraries and frameworks that the agent has built-in support for. These include many libraries and frameworks used or supported by Smile CDR as well.
As this integration is still in trial, the details of the data generated are subject to change. These details include the names (such as trace spans and metric names), the trace structure and attributes exposed by trace spans and metrics. Feedback for this feature is welcomed.
To consume OpenTelemetry data generated by Smile CDR, you need observability backends. An observability backend collects, persists the telemetry data and makes it available for monitoring (querying, visualizing and alerting if system is not performing as desired). There are many open source and commercial backends supporting OpenTelemetry. Smile CDR does not recommend any particular observability backend because the choice may depend on your needs, preferences and deployment environment. Since OpenTelemetry is an open standard, you should be able to work with any backend that supports OpenTelemetry.
Some example observability backends and environments that support OpenTelemetry are:
We also have a very basic otel-backend-starter project for learning purposes. This project provides a docker-compose setup to run Jaeger, Prometheus and OpenTelemetry Collector locally.
To enable OpenTelemetry instrumentation in Smile CDR, you need to set an environment variable called CDR_OTEL when running Smile CDR.
For example,
CDR_OTEL=y bin/smilecdr start
The value of the CDR_OTEL variable has no significance, the instrumentation is enabled as long as the variable is set.
When this variable is set, the Smile CDR process is auto-instrumented with the OpenTelemetry Java agent. A version of the OpenTelemetry Java agent is bundled with the Smile CDR release so there is no need to download it separately.
By default, the java agent is configured using the following properties file.
otel.service.name=smilecdr
# Send OTEL logging to slf4j so that javaagent logs gets logged in smile.log
otel.javaagent.logging=application
# capture X-Request-ID as a http span attribute
otel.instrumentation.http.server.capture-response-headers=X-Request-ID
# use stable semantic conventions for http spans
otel.semconv-stability.opt-in=http
This configures the agent so that service name for the telemetry is set to smilecdr, the agent logs are added to the log/smile.log file, the X-Request-Id response header is captured for http requests as a span attribute, and stable http attribute name conventions are used.
If you would like to use your own agent configuration file instead of this default configuration, you need to set the
OTEL_JAVAAGENT_CONFIGURATION_FILE environment variable to specify the path to your agent configuration file. For example,
CDR_OTEL=y OTEL_JAVAAGENT_CONFIGURATION_FILE=<path_to_your_agent_config_file> bin/smilecdr start
Alternatively, you can override or set individual configuration options for the agent using other OpenTelemetry Java environment variables.
Some cloud vendors, such AWS and Azure, provide their own distributions of tools for OpenTelemetry. With such cloud vendors, there are 2 general approaches you can take:
The first approach is to use the OpenTelemetry Java agent bundled with Smile CDR, and use and configure OpenTelemetry Collector to convert the data to vendor specific format. If you follow this approach you need to run Smile CDR with the CDR_OTEL environment variable set as explained in the previous section so that Smile CDR is instrumented with the Java agent.
The second approach is to use the OpenTelemetry Java agent distribution provided by a vendor, if there is one. In this approach when running Smile CDR you do not set the CDR_OTEL environment variable but instead set JAVA_TOOL_OPTIONS environment variable to instrument the Smile CDR process.
Both of these approaches are explained in detail next for AWS and Azure.
AWS provides its own distribution of the OpenTelemetry Java agent and collector.
You can use the AWS Distro for OpenTelemetry Collector to export telemetry data in AWS formats that can be consumed by AWS CloudWatch and AWS X-Ray.
For this to work, you run Smile CDR with the CDR_OTEL environment variable set and configure AWS Distro for OpenTelemetry Collector to export data in AWS formats. You can see some examples for AWS OpenTelemetry Collector Configurations in the AWS Observability repo.
You may also decide to use the AWS Distro for the Java agent instead of the Java agent bundled with SmileCDR. For this,
when running SmileCDR, do not set the CDR_OTEL environment variable, but instead set the JAVA_TOOL_OPTIONS environment variable to instrument Smile CDR.
For example,
JAVA_TOOL_OPTIONS=-javaagent:<path-to-aws-otel-java-agent-jar> OTEL_SERVICE_NAME=smilecdr bin/smilecdr start
You would still need to use AWS Distro for OpenTelemetry Collector to be able to convert telemetry data to AWS specific formats to consume them in AWS Cloudwatch and AWS X-Ray.
Azure provides its own Application Insights OpenTelemetry Java agent, and there is a community provided Azure Monitor Exporter to be used with OpenTelemetry Collector.
If you decide to use the Azure Monitor Exporter for the OpenTelemetry Collector then you need to configure your OpenTelemetry Collector to export to azuremonitor as explained in its readme,
and run Smile CDR with CDR_OTEL environment variable set.
Alternatively, you may decide to use the Application Insights Java agent directly, instead of the Java agent bundled with Smile CDR. In this approach you do not use the OpenTelemetry Collector, and when running Smile CDR, do not set the CDR_OTEL environment variable, but instead set the JAVA_TOOL_OPTIONS environment variable to instrument Smile CDR.
You also need to configure Application Insights Agent according to the instructions provided by Azure.
For example,
JAVA_TOOL_OPTIONS=-javaagent:<path-to-azure-application-insights-agent-jar> bin/smilecdr start
while having a applicationinsights.json configuration file in the same directory as the Application Insights agent jar, with a content similar to following:
{
"connectionString":"<your_connection_string>",
"role": {
"name": "smilecdr"
}
}
By default, the java agent is configured to capture X-Request-ID header from the server responses as shown in the default agent configuration. The header is added to the HTTP spans as an attribute named http.response.header.x-request-id.
For HL7 v2.x inbound messages ingested by Smile CDR through HL7 v2.x endpoint, the following additional telemetry data are available.
For traces, a parent span named smilecdr.hl7v2.inbound_message.process is generated with the following additional span attributes that contain details of the message that is processed:
smilecdr.hl7v2.inbound_message.type (the type of the hl7 v2.x message , e.g. ADT_A01)smilecdr.hl7v2.inbound_message.version (the version of the hl7 v2.x message, e.g. 2.5)smilecdr.hl7v2.inbound_message.control_id (the control id of the hl7 v2.x message)This parent span also captures any conversion issues, that are added to the conversion result during hl7 v2.x to FHIR conversion, as span events with the following attributes:
smilecdr.hl7v2.inbound_message.conversion_issue.level (the severity of the conversion issue)smilecdr.hl7v2.inbound_message.conversion_issue.message (the description of the conversion issue)smilecdr.hl7v2.inbound_message.conversion_issue.path (the location of the conversion issue)As metrics, the following two metrics are generated:
smilecdr.hl7v2.inbound_message.count (a counter that is incremented for each hl7 v2.x message received for processing)smilecdr.hl7v2.inbound_message.error_count (a counter that is incremented for each hl7 v2.x message failed to be processed)Each of these metrics has smilecdr.hl7v2.inbound_message.type and smilecdr.hl7v2.inbound_message.version as attributes so that the counts are available per message type+version pair.
Traces for Camel Routes are generated without requiring any additional configuration. For Smile Component processors, the base URI of the processor is used as the span name. That is, each Camel processor span for the Smile Component is named in the following format smile://[moduleId]/[processorName].
An OpenTelemetry span is generated for any JavaScript callback execution. Such spans are named as
smilecdr.javascript_callback. The name of the JavaScript callback function is available as a span attribute named
smilecdr.javascript_callback.function_name.
An OpenTelemetry span is generated for any interceptor method execution. Such spans are named as
hapifhir.interceptor. The name of the pointcut, the interceptor class name and the interceptor method name are available as the following span attributes: hapifhir.interceptor.pointcut_name, hapifhir.interceptor.class_name, and
hapifhir.interceptor.method_name.