Smile CDR v2024.08.PRE
On this page:

3.9.1Upgrading an Existing Installation

 

This page describes the process for upgrading an existing installation of Smile CDR to a newer version.

3.9.2Simple Upgrade Process - Upgrading the Binary on a Single Server

 
Whenever you are upgrading your Smile CDR installation, you should ensure that you have backups of your Smile CDR server folders (at a minimum, the `lib` directory), as well as a full database backup of the Cluster Manager and FHIR Repository databases (possibly multiple FHIR Repository databases, if you have more than one persistence module configured). You may also need to backup any other file system resources that your system uses, depending on how you have configured your installation. For example, if you are storing any FHIR resources in an external directory, or in a Cloud-based storage location, etc. All of your backups will be needed in the case where you cannot successfully upgrade the server and you need to rollback to the current pre-upgrade version.

Within the Smile CDR installation directory, there is a directory called lib that contains all of the binary code for Smile CDR. It is generally possible to simply replace the contents of this directory with the new contents from a release.

The following example shows an in-place upgrade of a single Smile CDR server:

  1. Stop Smile CDR
cd smilecdr
bin/smilecdr stop
  1. Backup "lib/", "bin/" and "otel/" Directory Contents

Change the following commands to use a backup folder that works for your installation:

cp -R lib /path/to/your/backup/folder
cp -R bin /path/to/your/backup/folder
if [ -d otel ]; then cp -R otel /path/to/your/backup/folder; fi
  1. Backup Your Databases

With Smile CDR shut down, no new data will be entering the system and entering/changing the database, so this would be the time to perform a full database backup of all your applicable databases.

However, some installations have very large databases and cannot wait for a full backup to take place, so your situation may require taking a different approach. It may be acceptable in this scenario to use the most recent full backup, along with the latest transaction logs, in case you need to restore the database to this point-in-time, just before the Smile CDR upgrade takes place.

The exact instructions for this step will differ, depending on which database you have chosen to use with your installation. Please consult your DBA Team for assistance with ensuring that you have a full database backup to this point in time, which could be used if you decide to rollback this upgrade.

  1. Delete Previous "lib/" Directory Contents, Selected "bin/" Directory Files, and "otel/" Directory Contents
rm lib/*
rm bin/smilecdr
rm bin/smileutil
mv bin/setenv bin/setenv-2024.08.PRE.backup
rm -rf otel/*
  1. Extract the Smile CDR JAR Files
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the lib directory
tar --strip-components=1 -xvf /path/to/smilecdr-2024.08.PRE.tar.gz smilecdr/lib
  1. Extract the Smile CDR Binary Files
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the bin directory
tar --strip-components=1 -xvf /path/to/smilecdr-2024.08.PRE.tar.gz smilecdr/bin
  1. Extract the Smile CDR OpenTelemetry Files
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the otel directory
tar --strip-components=1 -xvf /path/to/smilecdr-2024.08.PRE.tar.gz smilecdr/otel
  1. Resolve bin/setenv Differences

In a previous step, we extracted a new setenv file and placed it in your bin/ folder, but some customers have made changes to that file to optimize the performance of Smile CDR in their environment. Therefore, you will need to examine the backup version of that file (bin/setenv-2024.08.PRE.backup), compare it to the new one (bin/setenv) and re-apply any manual changes that may have been made in previous versions of Smile CDR.

You can compare these files using the following command:

diff bin/setenv bin/setenv-2024.08.PRE.backup
  1. Check the cdr-config-Master.properties File For New Configuration Settings

Occasionally, we introduce a new configuration setting and it will be added to the default classes/cdr-config-Master.properties file in the new release. Users may choose to manage this file themselves, changing some of the default values to suit their needs, and sometimes using it in Properties Mode to force Smile CDR to use the configured values exclusively from the properties file. If this file is missing some new configuration value, it may not perform properly or it may cause problems during startup.

It is important to compare the old and new properties files to see whether any new configuration settings have been added, and add them to your locally-managed properties file as well.

  1. Upgrading the Database

Starting with version 2020.11.R01, Smile CDR automatically upgrades your database when the new version first starts up.

The automatic database upgrade is convenient and appropriate for smaller systems, development or testing environments. For all other deployments, we strongly recommend running the smileutil: Migrate Database command line tool to perform the migration. This will provide visual feedback of progress, avoid problems with timeouts on long-running tasks, and (optionally) allow a DBA to review the upgrade script to account for any local changes.

As of version 2022.11.R01, Flyway is no longer used to perform database migrations. Instead, Smile CDR now uses the HAPI-FHIR Migration Tool. This tool is backwards compatible with Flyway database migration and shares the same schema migration history tables that Flyway used: FLY_CDR_MIGRATION for the Cluster Manager database and FLY_HFJ_MIGRATION for the FHIR Repository database. The HAPI-FHIR Migration Tool uses a simple lock-record mechanism to ensure that only one process in a cluster is performing a database migration at a time. This lock-record has an installed_rank of -100 and a unique value (UUID) in the description column. The migration operation should automatically delete this lock record upon completion, even if there was a migration error. However in some scenarios, (e.g. loss of database connection in the middle of a migration), it is possible that this lock record will not be properly removed after the migration is complete. If this happens, you can set the CLEAR_LOCK_TABLE_WITH_DESCRIPTION environment variable (or System Property) to the value of the description of this lock record to instruct Smile CDR to delete it before starting a new migration.

For example, if your server fails to start up with an error message like the following:

HAPI-2153: Unable to obtain table lock - another database migration may be running.
If no other database migration is running, 
then the previous migration did not shut down properly 
and the lock record needs to be deleted manually.
The lock record is located in the FLY_CDR_MIGRATION table with 
INSTALLED_RANK = -100 and DESCRIPTION = 2981dc98-6111-4892-b3f9-56b9d559f4d7

Then before starting Smile CDR you could call:

export CLEAR_LOCK_TABLE_WITH_DESCRIPTION=2981dc98-6111-4892-b3f9-56b9d559f4d7

This will cause Smile CDR to delete this stale lock record before starting a new database migration.

3.9.3Upgrade Docker Container as Non-root

 

The docker container no longer runs as root as of release 2023.02.R01. This change is to improve local security. The following actions are required to upgrade to a non-root based docker container structure:

  • When upgrading from releases BEFORE 2023.02.R01 to a release on or AFTER 2023.02.R01
  • Create or use an existing non-root user to run the container per your SOP requirements. EG. 'smilecdr' ($USER)
  • Logon as that $USER
  • Change the ownership of any container volume directories. The original volume mounts were created and owned by root and must be changed to the new $USER so the new user can write to them.
sudo chown -R $USER /path/to/volume-mount

3.9.4Upgrading a Cluster of Servers with Zero Downtime

 

If you are running a cluster of Smile CDR servers and you wish to perform a "zero downtime" upgrade, you can follow the steps outlined in this section.

3.9.4.1Zero Downtime Upgrade Prerequisites

Zero-downtime upgrades require:

  • You must be running Smile CDR version 2021.02.R05 or later on all servers in your Cluster
  • You must have a minimum of two Smile CDR servers or nodes in your cluster
  • You must be using a front-end load balancer or reverse proxy
  • You can only upgrade a system that is out-of-date by at most 2 versions (2 quarterly releases), as any gap in versions larger than 6 months is not supported (see example below)

3.9.4.1.16-month Interval Example

If you are currently running the 2022-08-R05 release version and you wish to upgrade to the most recent 2023.11 release version in 6-month intervals, then you should perform the following upgrades in sequence:

NOTE: These major release versions were accurate at the time this was written, but they could be subject to change in the future.

3.9.4.2Limitations

You must be able to handle your incoming request load on a single server or a defined subset of your servers, since this process involves reducing your entire server cluster to a defined subset of your servers, or just one server if your cluster only has two servers.

During the upgrade, the cluster continues to operate, but does not support changing the configuration on any active server, nor restarting any active server, as these will still be running the older version of Smile CDR and will not be able to be re-initialized against the new database schema.

The following operations will continue to work on all running servers during an upgrade:

  • FHIR Requests
  • OIDC login, logout

Configuration actions will not work during the upgrade. Module restarts on servers running the old software will fail to start, potentially causing a service outage. Once an upgrade has begun, any servers running the old software may not be reconfigured and you must not make changes on these servers via the admin web console, including:

  • changing the configuration of a module
  • restarting a module
  • restarting a server
  • adding or modifying OIDC server definitions
  • viewing the audit and transaction logs

3.9.4.3Steps

Follow these steps to upgrade your server cluster with "zero downtime":

  1. Ensure That You Have Recent Backups

You should ensure that you have recent backups of your Cluster Manager Database as well as all of your FHIR Repository Database(s), before attempting the upgrade. You would typically do this during a maintenance window as part of your normal operational procedures. These will be needed if you run into any problems during the upgrade, you decide not to proceed, and you choose to revert the upgrade by restoring your system from these backups.

  1. Reduce Active Cluster to Servers Running the Older Version of Smile CDR

Using your front-end load balancer or reverse proxy, reduce the number of active servers in your cluster to an acceptable subset of your server cluster, where these remaining active servers are only running the older version of Smile CDR and are still able to serve your incoming requests. If you only have two servers in your cluster, then this will leave just one active server. The active server(s) will remain running the older version of Smile CDR during the Zero Downtime Upgrade process, serving any incoming requests, while a single offline server is upgraded and performs the migration to the newer version of Smile CDR. If you have a high-traffic installation with a large number of servers in your pool, then you may leave several servers on the older version actively serving incoming requests, while you perform this Zero Downtime Upgrade process.

  1. `export CDR_UPGRADE_MODE=true`

Starting with version 2024.02.R0X (any release from 2024.02 onwards), you can now set the environment variable CDR_UPGRADE_MODE to true on the active servers in the cluster and restart these servers. When a server is running in CDR_UPGRADE_MODE, all scheduled tasks are paused, consuming from broker channels (e.g. JMS Queues or Kafka topics) is paused, and changes to the configuration are not permitted in the Web Admin Console. Ignore this step if you are running an earlier release of Smile CDR.

  1. Upgrade One Offline Server

Using only one non-active server, follow the steps above to upgrade that server using the Simple Upgrade Process described above. This involves shutting down the server and then upgrading the binaries, so you should remove this initial upgrade server from your load balancer and drain it of all active sessions, before shutting it down.

  1. Start The Upgraded Server

Unset the CDR_UPGRADE_MODE environment variable on the upgraded server (unset CDR_UPGRADE_MODE) and then start the server up and monitor the smile.log while it starts up and it automatically upgrades the DB. It is very important that you monitor the logs when it is first started up, since it will perform the database migration at this time. Once it completes the startup process and it is fully running, you must verify that all the Modules on that newly-upgraded server are working correctly (i.e. no errors reported and no stopped Modules).

  1. Switch The Server Pool To Use The New Server

Switch your front-end load balancer or reverse proxy to stop using the existing "older version" server(s) and start directing incoming traffic to the newly-upgraded server. This may involve a very short window of perceived downtime by your front-end clients, depending on the front-end technology you are using and the method you use to perform this switchover. If you have a high-volume installation, then you can upgrade several non-active servers and have them ready to start accepting traffic, once you are ready to perform the switchover from the older version to the newer version of Smile CDR.

  1. Shut Down and Upgrade the Older Servers

Shut down the server(s) running the older version of Smile CDR and upgrade them to the newer version of Smile CDR. These servers are no longer receiving requests and can be upgraded. Unset the CDR_UPGRADE_MODE environment variable on each upgraded server before starting it. Verify the upgraded server started successfully and then add it back to the pool.

  1. Upgrade The Remaining Servers

Upgrade the rest of the servers in your cluster one-by-one, using the Simple Upgrade Process described above. Ensure that each upgraded server starts up without problems, before proceeding to the next server. Note that for each server, you must monitor the smile.log file while it starts up to ensure that everything works properly and no ERRORs are encountered during the startup phase of this newly-upgraded server. None of these servers will attempt to perform any database upgrades, because they will discover that it has already been upgraded by the first server. As each server is successfully upgraded to the new version of Smile CDR, it can be added back into your front-end server pool.

3.9.4.4CDR_UPGRADE_MODE Environment Variable

A note about the CDR_UPGRADE_MODE environment variable: Smile CDR batch processing relies on job state being stored in the database and asynchronous broker channels (JMS or Kafka). Smile CDR ensures that the details stored in these job state records and job event messages are forwards compatible. However, they are not guaranteed to be backwards compatible. For this reason, we recommend setting CDR_UPGRADE_MODE on servers running an older version of the software so they do not fail when trying to read job state records or new job event messages. Servers with CDR_UPGRADE_MODE set will continue to submit messages to broker channels (which the new servers will be able to handle), they just won't read from them.

New versions of Smile CDR can also introduce new configuration settings that older versions of the software may not understand. For this reason, CDR_UPGRADE_MODE prevents module restart within the Web Admin Console of old servers. Any required configuration changes should only be made on upgraded servers.

Once a server has been upgraded, it should be restarted after calling unset CDR_UPGRADE_MODE to ensure that normal processing resumes.

3.9.5Subscriptions

 

When upgrading CDR from versions 2019.05 and earlier, if subscriptions are enabled, then you will need to add a new Subscription Matching Module of the same FHIR version as your FHIR Storage module and link the Subscription Matching module to that FHIR Storage module as a dependency.

3.9.6Recommended Upgrade Strategy

 
Note that before attempting to upgrade your server, you must take backups of your database(s) and server components, and you should test your restore procedure as well, since an upgrade is not reversible.

Smile CDR recommends that you implement the following strategy when upgrading your Smile CDR deployments.

  1. Ensure You Are On a Supported Platform

This includes the specific Java version, your chosen database server and other requirements, as described in our documentation:

https://smiledigitalhealth.com/docs/getting_started/platform_requirements.html

  1. Choose Manual Or Automatic Schema Upgrades

Choose whether you wish to manually upgrade your database schema each time you upgrade your Smile CDR Server, or let Smile CDR perform the schema upgrade steps for you on first post-upgrade startup. More details on how to upgrade Smile CDR, as well as how to perform the upgrade manually using our smileutil migrate-database command, are available in our documentation:

https://smiledigitalhealth.com/docs/installation/upgrading.html

https://smiledigitalhealth.com/docs/smileutil/migrate_database.html

We strongly recommend that you choose to upgrade your database by using the "smileutil migrate-database" utility command, if any of the following conditions apply to you:
  • You are upgrading your Production environment or you are upgrading a database in a non-Production environment as preparation to upgrade your Production environment (so that you can test out the process)
  • You are using our MegaScale feature (so that you can control which databases are upgraded and when)
  • Your use a non-privileged user in your database connection or you have implemented Row Level Security inside your database (to ensure that you use a database user account with admin privileges when applying structural schema changes)
  • You have a DBA Team who typically handle the maintenance tasks for your database
  • You are running an older version of our server which does not support automatic upgrades
  • You have made some database changes yourself that you wish to preserve
  • You prefer to have full control over the migration of your database schema
  1. Guiding Principles

When choosing which version(s) to upgrade to, use the following guiding principles:

  • If you are upgrading with Smile CDR shut down or you are less than 6 months behind our Quarterly Releases, you may upgrade to the most recent major release version directly.
  • If you are more than 6 months behind and you are choosing to upgrade with Smile CDR still running as described in our Zero Downtime strategy above, you should upgrade your installation to each major release version in 6-month increments, as this will ensure that your schema changes remain within our 6-month window for backwards-compatibility.
  1. Choose The Most Recent R## Release

When choosing a major release version, always choose the most recent R## version. For example, we have the following major release versions available for the 2023.08 release:

You should upgrade to the 2023-08-R07 release version, and skip over any older versions for that particular Quarterly Release.