5.1.1Upgrade Concepts
Upgrading Smile CDR is a critical part of using the software. The team is constantly working on improving performance, fixing bugs, and adding new features. To take advantage of these developments in Smile CDR, it is critical to have an operational plan on how you will be upgrading your Smile CDR cluster to a newer version. This document discusses the various concepts, modes, and mechanisms that are related to upgrading.
5.1.2What is an upgrade
An upgrade is when you update your installation to a newer version of the software, as well as a newer version of the database schema.
For the purposes of discussion we will use the following terms when we talk about upgrading:
- Upgrading The Binaries; – The binaries are the software that run the application. This includes the contents of the
lib//bin/, and/oteldirectories in an installation of Smile CDR. - Migrating The Database; – The database definitions, comprising table definitions, indexes, generators, and so on.
This page will cover all the information needed to understand which upgrade mode and which database migration mechanism is suitable for your use case.
5.1.2.1Upgrading The Binaries
Depending on how you've installed Smile CDR, the upgrade process will differ.
Upgrading the binaries consists of replacing the contents of the bin/, /lib /otel directories with newer versions. If however you are using the docker container, it is sufficient to upgrade the image you are using to the newer version,
while keeping your volume mounts intact.
5.1.2.2Migrating The Database
Migrating the database consists of applying schema changes to the database. This can include, but isn't limited to:
- Adding new tables
- Altering column definitions
- Adding new indexes
- Removing Indexes
There are three database migration mechanisms:
- Smileutil migrate-database (recommended)
- Smile CDR migrate command (experimental)
- Automatic migrate at boot (deprecated)
The choice of database migration mecahnism does not impact your choice of upgrade mode. These are independent choices, and both online and offline upgrades can be done via any of the defined database migration mechanisms.
5.1.3Upgrade Modes
There are two upgrade modes, which determine whether your Smile CDR cluster remains available during upgrade:
- Offline mode
- Online mode
5.1.3.1Offline Mode
An offline mode upgrade can only occur when all Smile CDR processes in your cluster are shut down. When this can be achieved, Smile CDR may be upgraded to any arbitrary future version. The following table shows some examples of start and end versions for an offline upgrade.
Example Upgrade Paths:
| Start Version | End Version | Notes |
|---|---|---|
| 2023.02.RXX | 2023.05.RXX | Upgrade by one major version |
| 2023.08.RXX | 2024.02.RXX | Upgrade by two major versions |
| 2021.02.RXX | 2024.05.RXX | Upgrade by thirteen major versions |
Each row in this table represents an example migration from a given version to another version. The advantage to an offline upgrade is that Smile can be upgraded an arbitrary number of versions, all at once in a single jump. The tradeoff is that it requires downtime.
| Pros | Cons |
|---|---|
| Allows upgrade to any future release with no intermediate steps | Cluster must be taken fully offline, and cannot service requests |
| Operationally Simple |
5.1.3.2Online Mode
2021.02
An Online Mode upgrade that means that your cluster will be able to continue servicing requests during the upgrade. There will be no downtime for users who connect to your Smile CDR cluster.
In order to perform an Online Mode upgrade, several preconditions must be met:
5.1.3.2.1Prerequisites
- Smile CDR must be at least version
2021.02.R05or later on all servers in your cluster - There must be a minimum of two Smile CDR processes in your cluster
- There must be a front-end load balancer or reverse proxy, directing traffic
- Smile CDR can only be upgraded by two quarterly generally available releases at a time, e.g. 2023.11 2024.05
- If you are using a Microsoft SQL Server or Oracle database, then you must be running an edition that supports adding an index without locking tables (i.e. ONLINE):
5.1.3.2.2Limitations
You must be able to handle incoming request load on a subset of servers during the upgrade process, as the processes being upgraded will be unable to handle requests.
During an online upgrade, FHIR Requests, OIDC requests, and others will continue to be serviced, but some functionality is unavailable:
- Administrative Changes; – Changes to the configuration of the server during an online upgrade is not permitted, including changes to OIDC clients/servers
- Batch Jobs; – Batch Jobs should not be submitted during an online upgrade.
- Restarting non-upgraded nodes; – outdated nodes in the cluster should not be restarted, nor have their modules restarted, during the upgrade. They will be unable to consume new configuration from the database.
5.1.3.2.2.1CDR_UPGRADE_MODE
In order to enforce some of these behaviours, you may set an environment variable on the system running Smile CDR in order to ensure these behaviours:
export CDR_UPGRADE_MODE=true
Usage of the CDR_UPGRADE_MODE system property is available from 2024.02.R01 onwards. It causes Smile CDR to:
- Lock down the administrative console so that changes are not allowed to be made
- Prevent itself from consuming messages from Kafka/ActiveMQ
Once upgrading is complete, those processes can be restarted without this environment variable. This is not mandatory for online upgrades, but ensures that jobs will be processed on the versions of the software on which they were submitted.
5.1.3.2.3Example Upgrade
This below table outlines a "zero downtime" upgrade path, to get from version 2022.11 to 2025.02 (e.g. over 2 years of generally available releases)
Each row in this table represents a single migration event, in which the entire cluster is upgraded.
| Start Version | Target Version |
|---|---|
| 2022.11.RXX | 2023.05.RXX |
| 2023.05.RXX | 2023.11.RXX |
| 2023.11.RXX | 2024.05.RXX |
| 2024.05.RXX | 2024.11.RXX |
| 2024.11.RXX | 2025.02.RXX |
The cluster in this example starts at 2022.11 The migration to 2023.05 is the first step, as that is two releases ahead of the starting point (2022.11 2023.02 2023.05). Once that migration has been completed and all processes are running 2023.05, the next stage can begin. That stage consists of upgrading all processes in the cluster to 2023.11. This process is repeated until your target version is achieved across the cluster, and every process is now running 2025.02.
| Pros | Cons |
|---|---|
| Able to service most user requests during upgrade | Can only upgrade two Generally Available release versions at a time, so can require multiple intermediate steps in cases where you want to upgrade more than two versions at a time |
| Operationally Complex |
5.1.3.3Database Migration Mechanisms
Now that we know about upgrade modes, we can discuss the operational methods by which the database migration can occur. These are the actual commands that need to be run in order to migrate the database. There are currently three options to select from:
5.1.3.3.1Smileutil migrate-database (recommended)
This is the recommended approach for production environments using the command-line migration utility. This method:
- Provides visual feedback of progress
- Avoids timeout issues on long-running tasks
- Allows DBA review of upgrade scripts
- Supports customization through various flags
The documentation for the command covers all the relevant options.
5.1.3.3.2Smile CDR Migrate command (experimental)
As of 2024.11.R01, you may run bin/smilecdr migrate on an existing Smile CDR process. This command will bootstrap its configuration from the properties file or the database, depending on your node.propertysource setting.
To use the experimental migration command, set the cdr.migrate_command.enabled system property to true using one of the methods described here.
The advantage of this method over the [smileutil migrate-database] command is that credentials, database type, and modules are bootstrapped from the cluster, and do not have to be provided at runtime.
- Boots up and upgrades all schemas automatically
- Uses existing configuration for database connection information.
- Exits with code 0 on success
This command currently supports the --dry-run and --enable-heavyweight-migrations flags defined in the smileutil migrate-database options, and will support more flags in the future.
This does not require you to authenticate to the database, nor does it require you to select the driver type or schema type. When using this command, all of that information is determined from your current live configuration.
5.1.3.3.3Automatic Upgrade At Boot (Not recommended)
This mechanism is only supported for backwards compatibility. It is not recommended to use this upgrade mode, on a production installation.
Available since version 2020.11.R01, this mechanism:
- Automatically upgrades database on first startup
- Suitable for smaller systems and development environments
- Requires no additional steps beyond normal startup
- Uses built-in migration mechanisms
This can be enabled by setting the db.schema.update_mode property to UPDATE.
When starting, a Smile CDR process will check if there are any new migrations to be run, and if it finds any, it will run them before continuing its normal boot sequence.
5.1.3.4Demo
[Video content to be added]