This page describes the process for upgrading an existing installation of Smile CDR to a newer version.
Within the Smile CDR installation directory, there is a directory called lib that contains all of the binary code for Smile CDR. It is generally possible to simply replace the contents of this directory with the new contents from a release.
The following example shows an in-place upgrade of a single Smile CDR server:
cd smilecdr
bin/smilecdr stop
Change the following commands to use a backup folder that works for your installation:
cp -R lib /path/to/your/backup/folder
cp -R bin /path/to/your/backup/folder
With Smile CDR shut down, no new data will be entering the system and entering/changing the database, so this would be the time to perform a full database backup of all your applicable databases.
However, some installations have very large databases and cannot wait for a full backup to take place, so your situation may require taking a different approach. It may be acceptable in this scenario to use the most recent full backup, along with the latest transaction logs, in case you need to restore the database to this point-in-time, just before the Smile CDR upgrade takes place.
The exact instructions for this step will differ, depending on which database you have chosen to use with your installation. Please consult your DBA Team for assistance with ensuring that you have a full database backup to this point in time, which could be used if you decide to rollback this upgrade.
rm lib/*
rm bin/smilecdr
rm bin/smileutil
mv bin/setenv bin/setenv-2023.02.PRE.backup
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the lib directory
tar --strip-components=1 -xvf /path/to/smilecdr-2023.02.PRE.tar.gz smilecdr/lib
# The following command assumes you are currently in the root of your smilecdr
# installation, and will extract only the bin directory
tar --strip-components=1 -xvf /path/to/smilecdr-2023.02.PRE.tar.gz smilecdr/bin
In the previous step, we extracted a new setenv file and placed it in your bin/ folder, but some customers have made changes to that file to optimize the performance of Smile CDR in their environment. Therefore, you will need to examine the backup version of that file (bin/setenv-2023.02.PRE.backup), compare it to the new one (bin/setenv) and re-apply any manual changes that may have been made in previous versions of Smile CDR.
You can compare these files using the following command:
diff bin/setenv bin/setenv-2023.02.PRE.backup
Occasionally, we introduce a new configuration setting and it will be added to the default classes/cdr-config-Master.properties file in the new release. Users may choose to manage this file themselves, changing some of the default values to suit their needs, and sometimes using it in Properties Mode to force Smile CDR to use the configured values exclusively from the properties file. If this file is missing some new configuration value, it may not perform properly or it may cause problems during startup.
It is important to compare the old and new properties files to see whether any new configuration settings have been added, and add them to your locally-managed properties file as well.
Starting with version 2020.11.R01, Smile CDR automatically upgrades your database when the new version first starts up.
You can let Smile CDR perform the upgrade for you (recommended), or you can choose to upgrade it using our Database Migrator Tool as was needed in previous releases. See the smileutil: Migrate Database page for details on how to run the database migrator tool.
Starting with version 2020.11.R01, Smile CDR automatically upgrades your database when the new version first starts up.
You can let Smile CDR perform the upgrade for you (recommended), or you can choose to upgrade it using our Database Migrator Tool as was needed in previous releases. See the smileutil: Migrate Database page for details on how to run the database migrator tool.
As of version 2022.11.R01, Flyway is no longer used to perform database migrations. Instead, Smile CDR now uses the HAPI-FHIR Migration Tool. This tool is backwards compatible with Flyway database migration and shares the same schema migration history tables that Flyway used: FLY_CDR_MIGRATION for the Cluster Manager database and FLY_HFJ_MIGRATION for the FHIR Repository database. The HAPI-FHIR Migration Tool uses a simple lock-record mechanism to ensure that only one process in a cluster is performing a database migration at a time. This lock-record has an installed_rank of -100 and a unique value (UUID) in the description column. The migration operation should automatically delete this lock record upon completion, even if there was a migration error. However in some scenarios, (e.g. loss of database connection in the middle of a migration), it is possible that this lock record will not be properly removed after the migration is complete. If this happens, you can set the CLEAR_LOCK_TABLE_WITH_DESCRIPTION environment variable (or System Property) to the value of the description of this lock record to instruct Smile CDR to delete it before starting a new migration.
For example, if your server fails to start up with an error message like the following:
HAPI-2153: Unable to obtain table lock - another database migration may be running.
If no other database migration is running,
then the previous migration did not shut down properly
and the lock record needs to be deleted manually.
The lock record is located in the FLY_CDR_MIGRATION table with
INSTALLED_RANK = -100 and DESCRIPTION = 2981dc98-6111-4892-b3f9-56b9d559f4d7
Then before starting Smile CDR you could call:
export CLEAR_LOCK_TABLE_WITH_DESCRIPTION=2981dc98-6111-4892-b3f9-56b9d559f4d7
This will cause Smile CDR to delete this stale lock record before starting a new database migration.
If you are running a cluster of Smile CDR servers and you wish to perform a "zero downtime" upgrade, you can follow the steps outlined in this section.
Zero-downtime upgrades require:
You must be able to handle your incoming request load on a single server or a defined subset of your servers, since this process involves reducing your entire server cluster to a defined subset of your servers, or just one server if your cluster only has two servers.
During the upgrade, the cluster continues to operate, but does not support changing the configuration on any active server, nor restarting any active server, as these will still be running the older version of Smile CDR and will not be able to be re-initialized against the new database schema.
The following operations will continue to work on all running servers during an upgrade:
Configuration actions will not work during the upgrade. Module restarts on servers running the old software will fail to start, potentially causing a service outage. Once an upgrade has begun, any servers running the old software may not be reconfigured and you must not make changes on these servers via the admin web console, including:
Follow these steps to upgrade your server cluster with "zero downtime":
You should ensure that you have recent backups of your Cluster Manager Database as well as all of your FHIR Repository Database(s), before attempting the upgrade. You would typically do this during a maintenance window as part of your normal operational procedures. These will be needed if you run into any problems during the upgrade, you decide not to proceed, and you choose to revert the upgrade by restoring your system from these backups.
Using your front-end load balancer or reverse proxy, reduce the number of active servers in your cluster to an acceptable subset of your server cluster, where these remaining active servers are only running the older version of Smile CDR and are still able to serve your incoming requests. If you only have two servers in your cluster, then this will leave just one active server. The active server(s) will remain running the older version of Smile CDR during the Zero Downtime Upgrade process, serving any incoming requests, while a single offline server is upgraded and performs the migration to the newer version of Smile CDR. If you have a high-traffic installation with a large number of servers in your pool, then you may leave several servers on the older version actively serving incoming requests, while you perform this Zero Downtime Upgrade process.
Using only one non-active server, follow the steps above to upgrade that server using the Simple Upgrade Process described above. This involves shutting down the server and then upgrading the binaries, so you should remove this initial upgrade server from your load balancer and drain it of all active sessions, before shutting it down.
Start up the newly-upgraded server and monitor the smile.log while it starts up and it automatically upgrades the DB. It is very important that you monitor the logs when it is first started up, since it will perform the database migration at this time. Once it completes the startup process and it is fully running, you must verify that all the Modules on that newly-upgraded server are working correctly (i.e. no errors reported and no stopped Modules).
Switch your front-end load balancer or reverse proxy to stop using the existing "older version" server(s) and start directing incoming traffic to the newly-upgraded server. This may involve a very short window of perceived downtime by your front-end clients, depending on the front-end technology you are using and the method you use to perform this switchover. If you have a high-volume installation, then you can upgrade several non-active servers and have them ready to start accepting traffic, once you are ready to perform the switchover from the older version to the newer version of Smile CDR.
Shut down the server(s) running the older version of Smile CDR and upgrade them to the newer version of Smile CDR. These servers are no longer receiving requests and can be upgraded and added back to the pool, once each has been verified as successfully upgraded.
Upgrade the rest of the servers in your cluster one-by-one, using the Simple Upgrade Process described above. Ensure that each upgraded server starts up without problems, before proceeding to the next server. Note that for each server, you must monitor the smile.log file while it starts up to ensure that everything works properly and no ERRORs are encountered during the startup phase of this newly-upgraded server. None of these servers will attempt to perform any database upgrades, because they will discover that it has already been upgraded by the first server. As each server is successfully upgraded to the new version of Smile CDR, it can be added back into your front-end server pool.
When upgrading CDR from versions 2019.05 and earlier, if subscriptions are enabled, then you will need to add a new Subscription Matching Module of the same FHIR version as your FHIR Storage module and link the Subscription Matching module to that FHIR Storage module as a dependency.
Smile CDR recommends that you implement the following strategy when upgrading your Smile CDR deployments.
This includes the specific Java version, your chosen database server and other requirements, as described in our documentation:
https://smilecdr.com/docs/getting_started/platform_requirements.html
Choose whether you wish to manually upgrade your database schema each time you upgrade your Smile CDR Server, or let Smile CDR perform the schema upgrade steps for you on first post-upgrade startup. More details on how to upgrade Smile CDR, as well as how to perform the upgrade manually using our smileutil migrate-database command, are available in our documentation:
https://smilecdr.com/docs/installation/upgrading.html
https://smilecdr.com/docs/smileutil/migrate_database.html
NOTE: Some database servers are not supported by our automatic schema upgrade feature in Smile CDR. If you are running an older version of a database server that does not allow automatic upgrades, then you will need to perform the upgrade manually using our supplied utility, as described in the second link in Step 2 above.
When choosing which version(s) to upgrade to, use the following guiding principles:
When choosing a major release version, always choose the most recent R## version. For example, we have the following major release versions available for the 2021.11 release:
You should upgrade to the 2021-11-R07 release version, and skip over any older versions for that particular Quarterly Release.
If you are currently running the 2020-11-R02 release version and you wish to upgrade to the most recent 2022.08 release version, then you should perform the following upgrades in sequence:
NOTE: These major release versions were accurate at the time this was written, but they could be subject to change in the future.