Smile CDR v2023.02.PRE
On this page:

3.9Upgrading an Existing Installation

 

This page describes the process for upgrading an existing installation of Smile CDR to a newer version.

3.9.1Simple Upgrade Process - Upgrading the Binary on a Single Server

 
Whenever you are upgrading your Smile CDR installation, you should ensure that you have backups of your Smile CDR server folders (at a minimum, the `lib` directory), as well as a full database backup of the Cluster Manager and FHIR Repository databases (possibly multiple FHIR Repository databases, if you have more than one persistence module configured). You may also need to backup any other file system resources that your system uses, depending on how you have configured your installation. For example, if you are storing any FHIR resources in an external directory, or in a Cloud-based storage location, etc. All of your backups will be needed in the case where you cannot successfully upgrade the server and you need to rollback to the current pre-upgrade version.

Within the Smile CDR installation directory, there is a directory called lib that contains all of the binary code for Smile CDR. It is generally possible to simply replace the contents of this directory with the new contents from a release.

The following example shows an in-place upgrade of a single Smile CDR server:

  1. Stop Smile CDR
cd smilecdr
bin/smilecdr stop
  1. Backup lib/ and bin/ Directory Contents

Change the following commands to use a backup folder that works for your installation:

cp -R lib /path/to/your/backup/folder
cp -R bin /path/to/your/backup/folder
  1. Backup Your Databases

With Smile CDR shut down, no new data will be entering the system and entering/changing the database, so this would be the time to perform a full database backup of all your applicable databases.

However, some installations have very large databases and cannot wait for a full backup to take place, so your situation may require taking a different approach. It may be acceptable in this scenario to use the most recent full backup, along with the latest transaction logs, in case you need to restore the database to this point-in-time, just before the Smile CDR upgrade takes place.

The exact instructions for this step will differ, depending on which database you have chosen to use with your installation. Please consult your DBA Team for assistance with ensuring that you have a full database backup to this point in time, which could be used if you decide to rollback this upgrade.

  1. Delete Previous lib/ Directory Contents and Selected bin/ Directory Files
rm lib/*
rm bin/smilecdr
rm bin/smileutil
mv bin/setenv bin/setenv-2023.02.PRE.backup
  1. Extract the Smile CDR JAR Files
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the lib directory
tar --strip-components=1 -xvf /path/to/smilecdr-2023.02.PRE.tar.gz smilecdr/lib
  1. Extract the Smile CDR Binary Files
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the bin directory
tar --strip-components=1 -xvf /path/to/smilecdr-2023.02.PRE.tar.gz smilecdr/bin
  1. Resolve bin/setenv Differences

In the previous step, we extracted a new setenv file and placed it in your bin/ folder, but some customers have made changes to that file to optimize the performance of Smile CDR in their environment. Therefore, you will need to examine the backup version of that file (bin/setenv-2023.02.PRE.backup), compare it to the new one (bin/setenv) and re-apply any manual changes that may have been made in previous versions of Smile CDR.

You can compare these files using the following command:

diff bin/setenv bin/setenv-2023.02.PRE.backup
  1. Check the cdr-config-Master.properties File For New Configuration Settings

Occasionally, we introduce a new configuration setting and it will be added to the default classes/cdr-config-Master.properties file in the new release. Users may choose to manage this file themselves, changing some of the default values to suit their needs, and sometimes using it in Properties Mode to force Smile CDR to use the configured values exclusively from the properties file. If this file is missing some new configuration value, it may not perform properly or it may cause problems during startup.

It is important to compare the old and new properties files to see whether any new configuration settings have been added, and add them to your locally-managed properties file as well.

  1. Upgrading the Database

Starting with version 2020.11.R01, Smile CDR automatically upgrades your database when the new version first starts up.

You can let Smile CDR perform the upgrade for you (recommended), or you can choose to upgrade it using our Database Migrator Tool as was needed in previous releases. See the smileutil: Migrate Database page for details on how to run the database migrator tool.

Starting with version 2020.11.R01, Smile CDR automatically upgrades your database when the new version first starts up.

You can let Smile CDR perform the upgrade for you (recommended), or you can choose to upgrade it using our Database Migrator Tool as was needed in previous releases. See the smileutil: Migrate Database page for details on how to run the database migrator tool.

As of version 2022.11.R01, Flyway is no longer used to perform database migrations. Instead, Smile CDR now uses the HAPI-FHIR Migration Tool. This tool is backwards compatible with Flyway database migration and shares the same schema migration history tables that Flyway used: FLY_CDR_MIGRATION for the Cluster Manager database and FLY_HFJ_MIGRATION for the FHIR Repository database. The HAPI-FHIR Migration Tool uses a simple lock-record mechanism to ensure that only one process in a cluster is performing a database migration at a time. This lock-record has an installed_rank of -100 and a unique value (UUID) in the description column. The migration operation should automatically delete this lock record upon completion, even if there was a migration error. However in some scenarios, (e.g. loss of database connection in the middle of a migration), it is possible that this lock record will not be properly removed after the migration is complete. If this happens, you can set the CLEAR_LOCK_TABLE_WITH_DESCRIPTION environment variable (or System Property) to the value of the description of this lock record to instruct Smile CDR to delete it before starting a new migration.

For example, if your server fails to start up with an error message like the following:

HAPI-2153: Unable to obtain table lock - another database migration may be running.
If no other database migration is running, 
then the previous migration did not shut down properly 
and the lock record needs to be deleted manually.
The lock record is located in the FLY_CDR_MIGRATION table with 
INSTALLED_RANK = -100 and DESCRIPTION = 2981dc98-6111-4892-b3f9-56b9d559f4d7

Then before starting Smile CDR you could call:

export CLEAR_LOCK_TABLE_WITH_DESCRIPTION=2981dc98-6111-4892-b3f9-56b9d559f4d7

This will cause Smile CDR to delete this stale lock record before starting a new database migration.

3.9.2Upgrading a Cluster of Servers with Zero Downtime

 

If you are running a cluster of Smile CDR servers and you wish to perform a "zero downtime" upgrade, you can follow the steps outlined in this section.

Zero Downtime Upgrade Prerequisites

Zero-downtime upgrades require:

  • You must be running Smile CDR version 2021.02.R05 or later on all servers in your Cluster
  • You must have a minimum of two Smile CDR servers or nodes in your cluster
  • You must be using a front-end load balancer or reverse proxy
  • You can only upgrade a system that is out-of-date by at most 2 versions (typically quarterly releases), as any larger gap in the versions is not officially supported

Limitations

You must be able to handle your incoming request load on a single server or a defined subset of your servers, since this process involves reducing your entire server cluster to a defined subset of your servers, or just one server if your cluster only has two servers.

During the upgrade, the cluster continues to operate, but does not support changing the configuration on any active server, nor restarting any active server, as these will still be running the older version of Smile CDR and will not be able to be re-initialized against the new database schema.

The following operations will continue to work on all running servers during an upgrade:

  • FHIR Requests
  • OIDC login, logout

Configuration actions will not work during the upgrade. Module restarts on servers running the old software will fail to start, potentially causing a service outage. Once an upgrade has begun, any servers running the old software may not be reconfigured and you must not make changes on these servers via the admin web console, including:

  • changing the configuration of a module
  • restarting a module
  • restarting a server
  • adding or modifying OIDC server definitions
  • viewing the audit and transaction logs

Steps

Follow these steps to upgrade your server cluster with "zero downtime":

  1. Ensure That You Have Recent Backups

You should ensure that you have recent backups of your Cluster Manager Database as well as all of your FHIR Repository Database(s), before attempting the upgrade. You would typically do this during a maintenance window as part of your normal operational procedures. These will be needed if you run into any problems during the upgrade, you decide not to proceed, and you choose to revert the upgrade by restoring your system from these backups.

  1. Reduce Active Cluster to Servers Running the Older Version of Smile CDR

Using your front-end load balancer or reverse proxy, reduce the number of active servers in your cluster to an acceptable subset of your server cluster, where these remaining active servers are only running the older version of Smile CDR and are still able to serve your incoming requests. If you only have two servers in your cluster, then this will leave just one active server. The active server(s) will remain running the older version of Smile CDR during the Zero Downtime Upgrade process, serving any incoming requests, while a single offline server is upgraded and performs the migration to the newer version of Smile CDR. If you have a high-traffic installation with a large number of servers in your pool, then you may leave several servers on the older version actively serving incoming requests, while you perform this Zero Downtime Upgrade process.

  1. Upgrade One Offline Server

Using only one non-active server, follow the steps above to upgrade that server using the Simple Upgrade Process described above. This involves shutting down the server and then upgrading the binaries, so you should remove this initial upgrade server from your load balancer and drain it of all active sessions, before shutting it down.

  1. Start The Upgraded Server

Start up the newly-upgraded server and monitor the smile.log while it starts up and it automatically upgrades the DB. It is very important that you monitor the logs when it is first started up, since it will perform the database migration at this time. Once it completes the startup process and it is fully running, you must verify that all the Modules on that newly-upgraded server are working correctly (i.e. no errors reported and no stopped Modules).

  1. Switch The Server Pool To Use The New Server

Switch your front-end load balancer or reverse proxy to stop using the existing "older version" server(s) and start directing incoming traffic to the newly-upgraded server. This may involve a very short window of perceived downtime by your front-end clients, depending on the front-end technology you are using and the method you use to perform this switchover. If you have a high-volume installation, then you can upgrade several non-active servers and have them ready to start accepting traffic, once you are ready to perform the switchover from the older version to the newer version of Smile CDR.

  1. Shut Down and Upgrade the Older Server

Shut down the server(s) running the older version of Smile CDR and upgrade them to the newer version of Smile CDR. These servers are no longer receiving requests and can be upgraded and added back to the pool, once each has been verified as successfully upgraded.

  1. Upgrade The Remaining Servers

Upgrade the rest of the servers in your cluster one-by-one, using the Simple Upgrade Process described above. Ensure that each upgraded server starts up without problems, before proceeding to the next server. Note that for each server, you must monitor the smile.log file while it starts up to ensure that everything works properly and no ERRORs are encountered during the startup phase of this newly-upgraded server. None of these servers will attempt to perform any database upgrades, because they will discover that it has already been upgraded by the first server. As each server is successfully upgraded to the new version of Smile CDR, it can be added back into your front-end server pool.

3.9.3Subscriptions

 

When upgrading CDR from versions 2019.05 and earlier, if subscriptions are enabled, then you will need to add a new Subscription Matching Module of the same FHIR version as your FHIR Storage module and link the Subscription Matching module to that FHIR Storage module as a dependency.

3.9.4Recommended Upgrade Strategy

 
Note that before attempting to upgrade your server, you must take backups of your database(s) and server components, and you should test your restore procedure as well, since an upgrade is not reversible.

Smile CDR recommends that you implement the following strategy when upgrading your Smile CDR deployments.

  1. Ensure You Are On a Supported Platform

This includes the specific Java version, your chosen database server and other requirements, as described in our documentation:

https://smilecdr.com/docs/getting_started/platform_requirements.html

  1. Choose Manual Or Automatic Schema Upgrades

Choose whether you wish to manually upgrade your database schema each time you upgrade your Smile CDR Server, or let Smile CDR perform the schema upgrade steps for you on first post-upgrade startup. More details on how to upgrade Smile CDR, as well as how to perform the upgrade manually using our smileutil migrate-database command, are available in our documentation:

https://smilecdr.com/docs/installation/upgrading.html

https://smilecdr.com/docs/smileutil/migrate_database.html

NOTE: Some database servers are not supported by our automatic schema upgrade feature in Smile CDR. If you are running an older version of a database server that does not allow automatic upgrades, then you will need to perform the upgrade manually using our supplied utility, as described in the second link in Step 2 above.

Automatic upgrades are the easiest and will work for most customers. However, if you have mode some local database changes yourself, or you have a Database Administration Team that manages your databases for you, then you will likely prefer the manual upgrade process.
  1. Guiding Principles

When choosing which version(s) to upgrade to, use the following guiding principles:

  • If you are less than 6 months behind our Quarterly Releases, you may upgrade to the most recent major release version directly.
  • If you are more than 6 months behind, you should choose to upgrade your installation to each major release version in 6-month increments, as this will ensure that your schema changes remain within our 6-month window for backwards-compatibility.
  1. Choose The Most Recent R## Release

When choosing a major release version, always choose the most recent R## version. For example, we have the following major release versions available for the 2021.11 release:

You should upgrade to the 2021-11-R07 release version, and skip over any older versions for that particular Quarterly Release.

  1. Upgrade In 6 Month Intervals

If you are currently running the 2020-11-R02 release version and you wish to upgrade to the most recent 2022.08 release version, then you should perform the following upgrades in sequence:

NOTE: These major release versions were accurate at the time this was written, but they could be subject to change in the future.