34.0    Monitoring Basics
  • Monitoring
  • 34.0 Monitoring Basics
  • 34.1 Elastic APM Basics
  • 34.2 Externalized Metrics
  • 34.3 OpenTelemetry Integration
    • 34.2    Externalized Metrics   
       Table of Contents

    Elastic APM Basics

    34.1.1Runtime Application Performance Monitoring with Elastic APM
    Deprecated

     

    Smile CDR supports JVM instrumentation, which allows introspection into FHIR query performance as well as JVM performance. Application Performance Monitoring is a set of tools and processes that enable this. This section will cover the basics of setting up APM for Smile CDR.

    34.1.2What is APM?
    Deprecated

     

    APM is a set of tools and processes responsible for observing the performance and availability of a given piece of software. In the context of Smile CDR, APM reports on things such as:

    • How long FHIR queries take to run
    • Database call stacktraces
    • JVM Metrics like heap used, cpu used, etc.

    34.1.3Setting Up an APM Server
    Deprecated

     

    In order to take advantage of Elastic APM, you will need an Elasticsearch cluster, an instance of Kibana (or your Elasticsearch visualization tool of choice), and an instance of the Elastic APM server. There are many ways to deploy this infrastructure.

    For the sake of demonstration, we have set up a docker-compose that supports full functionality out-of-the-box based on the unnofficial docker-elk repository. You will need Docker and Docker Compose available on the same host as Smile CDR.

    In Smile CDR's installation directory there is a folder with all the necessary files for this demo. Navigate to it and, and start the docker-compose contained within.

    cd /path/to/smilecdr/installation/elastic-apm/docker-elk
docker-compose up

    This command will cause Elasticsearch, Elastic APM Server, and Kibana to start. Note this make take a few minutes.

    By default, Kibana listens on port 5601. You can reach it at http://localhost:5601.

    The default credentials are stored in the docker-compose.yml file.

    Note that this is a barebones setup, which is not appropriate for production. Elastic has a Getting Started Guide that details several methods by which you can install and set up a cluster. Once this is up and running, you are now able to instrument Smile CDR using the Elastic agent.

    34.1.4Instrumenting Smile CDR and the JVM
    Deprecated

     

    In order to report transactions and metrics to the APM server, we use Java agent provided by Elastic. When enabled, it automatically records transactions, and then sends them off to the APM server. Smile CDR provides this agent in the form of the elastic-apm-agent-XX.YY.ZZ.jar file, located in elastic-apm directory.

    By default, Smile CDR runs without instrumentation. To enable the instrumentation, navigate the bin/ directory of your installation, and modify the setenv file.

    Uncomment the following lines:

    #JVMARGS="$JVMARGS -javaagent:elastic-apm/elastic-apm-agent-1.13.0.jar"
#JVMARGS="$JVMARGS -Delastic.apm.config_file=elastic-apm/elasticapm.properties"

    This will enable the java agent to instrument Smile CDR. By default, all incoming requests will be instrumented, as well as all JMS messages. Navigate back to Smile CDR's installation directory, and start Smile CDR:

    bin/smilecdr start

    Next up, go ahead and execute a few FHIR searches against your Smile CDR instance. This will populate some APM data in Kibana.

    34.1.5Accessing the User Interface
    Deprecated

     

    Navigate to Kibana at http://localhost:5601, start out by logging in with the credentials provided to you in the docker-compose.yml file. Head over to the left tab, and select APM from the list of options. Next, you will have to select a service. The service we register with APM is called Smile CDR. Once you click it, you will be greeted by the APM interface. From here, you can see the various requests that are submitted to the FHIR module, as well as other administrative modules.

    Kibana

    This interface allows you to inspect average query response times, 95th percentile response times, and a whole bunch of extra metadata related to requests, including the actual SQL that was executed on the database.

    34.1.6Configuring Settings for the Agent
    Deprecated

     

    The configuration for the agent is held in elastic-apm/elasticapm.properties. While the full reference for all configuration options gives details as to what is possible, here we will cover the options that Smile CDR has set as default.

    # Server host
server_urls=http://localhost:8200

    server_urls defines the location that the agent will attempt to send traces to. Supports both HTTP and HTTPS protocols

    #Disable scheduled task transaction recording.
disable_instrumentations=scheduled,quartz

    This property determines which class of instrumentations are disabled. The default setting of scheduled,quartz will prevent your APM server from receiving data on all the various background tasks that occur on Smile CDR. If you need to see these scheduled tasks, simply comment out this line.

    # Disable stack trace collection
stack_trace_limit=0

    When requests come in and cause errors, the agent will attempt to automatically send back the relevant stack traces. We disable this by default. If you want to see stack traces, simply remove this line.

    # Capture all request/response headers in the transaction.
capture_headers=false

    While the APM agent automatically redacts known sensitive headers, we disable capturing request and response headers by default.

    34.1.7Securing the Agent
    Deprecated

     

    This section provides a simple proof of concept that shows how to setup your docker-compose installation with a self-signed certificate. This is in addition to the default security method, which is a secret token.

    Again, this is not for use in a production environment. This is simply to show that TLS configuration is possible between the agent and the APM server. First off, let's generate a self-signed cert from the Elasticsearch cluster using their built-in cert tools.

    34.1.7.1Setting Up a Certificate Authority

    $ docker exec -it docker-elk_elasticsearch_1 /bin/bash

    This will drop us into a shell in our running Elastic container. Now let's generate our certificate authority.

    $ bin/elasticsearch-certutil ca

    This will prompt you for a few bits of information for your new Certificate Authority. Keep the defaults for now. Once that is done, we can generate a certificate for our APM server, and then exit the docker shell.

    34.1.7.2Creating a Certificate

    $ bin/elasticsearch-certutil cert --pem -ca elastic-stack-ca.p12
$ exit

    This will generate a file called certificate-bundle.zip if you kept all the defaults. This file contains both the private key and the certificate, which we will need to install to our APM server. Pull them onto your local host:

    $ docker cp docker-elk_elasticsearch_1:/usr/share/elasticsearch/certificate-bundle.zip ./

    34.1.7.3Enabling TLS on APM Server

    Unzip the certs and add them into the config folder of the APM server

    $ unzip certificate-bundle.zip
$ mv instance.key elastic-apm/demo-apm-server/apm-server/config/
$ mv instance.crt elastic-apm/demo-apm-server/apm-server/config/

    Modify the apm-server.yml file found in elastic-apm/demo-apm-server/extensions/apm-server/config/ and uncomment the following lines:

    #ssl:
#  enabled: true
#  key: "/usr/share/apm-server/instance.key"
#  certificate: "/usr/share/apm-server/instance.crt"
#secret-token: "come_up_with_a_strong_secret_token"

    Next, modify the docker-compose to mount the certificates to the container by uncommenting the following lines in the docker-compose.yml

    #- ./apm-server/config/instance.key:/usr/share/apm-server/instance.key:ro
#- ./apm-server/config/instance.crt:/usr/share/apm-server/instance.crt:ro

    Doing all of the above enables TLS for the APM server, and also defines a secret token that all the agents must pass along to submit data. To apply these changes, you should bring down and rebuild your docker-compose.

    $ docker-compose down && docker-compose up

    34.1.7.4Updating Agent Settings for TLS

    Finally, you have to inform the agent that you are now using HTTPS, and provide it the secret token. Modify elastic-apm/elasticapm.properties, change the server URL to use HTTPS protocol, and add the secret token. You'll have to turn off server cert verification, as this is just a locally self-signed cert running on localhost.

    # Server host
server_urls=https://localhost:8200

# Secret token for communicating to APM server.
secret_token=come_up_with_a_strong_secret_token

# TLS Cert information
verify_server_cert=false

    In a real production situation, you would want to use a properly signed certificate, and have verify_server_cert=true. Restart Smile CDR and your agent is now instrumented and communicating with the APM server using TLS and a secret token.

    $ cd /path/to/smilecdr/installation
$ bin/smilecdr restart

2020-03-13 17:24:53.282 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2019-12-16T20:57:59Z",  "build_sha": "348d8d83c3c823b64fc0692be607b1a5a8fac775",  "version": "7.5.1"}
2020-03-13 17:24:53.371 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.13.0 as Smile CDR on Java 11.0.5 (Eclipse OpenJ9) Linux 5.5.6-1-MANJARO
       34.0    Monitoring Basics
  • Monitoring
  • 34.0 Monitoring Basics
  • 34.1 Elastic APM Basics
  • 34.2 Externalized Metrics
  • 34.3 OpenTelemetry Integration
    • 34.2    Externalized Metrics