13.18.1Working with DuplicatesExperimental
FHIR Repositories can end up with duplicate resources in them. This page details tools that Smile provides to work with duplicated data.
| Cause of Duplicates | Recommended Approach |
|---|---|
| The same entities were imported from different source systems. E.g. Jane Smith's Patient record is imported from the Lab system as Patient/123 and from the Pharmacy system as Patient/456. | Preserve the source data as is so that future data continues to be properly associated to the right patients. Use MDM to establish Golden Resources and MDM LINK records outside of the data and use MDM features like Observation?patient=Patient/123&_mdm=true to link the data at search time. |
| Data was accidentally duplicated on import. E.g. the same data was accidentally loaded twice (as POST new resources) or Conditional Create directives failed to match the intended target correctly. This kind of unintended duplication can also occur when translating HL7v2, CDA, or CSV data into FHIR resources. | In the case where the data was accidentally duplicated, it may make sense to "clean up" the duplicates. See below for details on Smile CDR tools to merge such duplicates. |
13.18.2Deduplication OperationsExperimental
Smile CDR provides several operations to deduplicate data:
$mergewhich is a backport of the FHIR R5 Patient/$merge specification to FHIR R4$hapi.fhir.replace-referencesperforms only the update references part of this$mergeoperation$hapi.fhir.undo-replace-referencesundoes the effects of a$hapi.fhir.replace-referencesoperation$sdh.mdm-bundle-matchprocesses FHIR Bundles to match resources using MDM rules and optionally merge them using survivorship
13.18.3$merge OperationExperimental
See the FHIR R5 Patient/$merge specification page for a description of this operation. See the bottom of this page for details on the current roadmap for enhancing Smile CDR deduplication functionality.
13.18.3.1$merge Input Parameters
| Name | Type | Default | Notes |
|---|---|---|---|
| source-patient-identifier | Identifier | List of source patient identifiers | |
| source-patient | Reference | Source patient | |
| target-patient-identifier | Identifier | List of target patient identifiers | |
| target-patient | Reference | Target patient | |
| result-patient | Patient | Optional merged patient resource | |
| preview | Boolean | false | If true, no changes will be made and response will summarize what would happen were the merge to occur |
| delete-source | Boolean | false | If true, delete the source resource |
| resource-limit | Integer | 512 | If the request is synchrononous and the number of resources to change exceeds this threshold, the operation will fail with 412 Precondition Failed. This parameter has no effect if the Prefer: respond-async header is set |
See the FHIR R5 Patient/$merge specification for a detailed description of these input parameters.
resource-limit is a Smile CDR addition to protect users from accidentally changing too many resources at once. If resource-limit is larger than 10000, the value 10000 will be used.
If you request that the operation be performed asynchronously by providing the Prefer: respond-async HTTP header, then the resource-limit parameter is ignored.
When performed asynchronously, the operation is performed in batches of 1024 resource patches at a time, via PATCH transaction transaction Bundles.
13.18.3.2$merge Output Parameters
| Name | Type | Notes |
|---|---|---|
| input | Parameters | A copy of the input parameters used in the $merge operation |
| outcome | OperationOutcome | Details about the result of the merge |
| result | Patient | The merged Patient resource |
| task | Task | If the merge operation was performed asynchronously, this Task resource provides details about the status of the merge operation |
13.18.3.3Merge Provenance Resource
With the 2025.08 release, the $merge operation creates a Provenance resource upon successful completion.
This Provenance resource contains, in its target element, the versioned references to the target patient,
the source patient (if not deleted during the operation), and all other resources updated as part of the operation.
The Provenance.activity is set to http://terminology.hl7.org/CodeSystem/iso-21089-lifecycle|merge, and the Provenance.agent.who is populated with a logical reference to the request user.
13.18.3.3.1Merge Example 1: Merge Patient/2 into Patient/3 synchronously.
Input:
POST /Patient/$merge
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "source-patient",
"valueReference": {
"reference": "Patient/2"
}
},
{
"name": "target-patient",
"valueReference": {
"reference": "Patient/3"
}
}
]
}
Output:
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "input",
"resource": {
"resourceType": "Parameters",
"parameter": [
{
"name": "source-patient",
"valueReference": {
"reference": "Patient/2"
}
},
{
"name": "target-patient",
"valueReference": {
"reference": "Patient/3"
}
}
]
}
},
{
"name": "outcome",
"resource": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"details": {
"text": "Merge operation completed successfully."
}
}
]
}
},
{
"name": "result",
"resource": {
"resourceType": "Patient",
"id": "3",
"identifier": [
{
"system": "SYS2A",
"value": "VAL2A"
},
{
"system": "SYS2B",
"value": "VAL2B"
},
{
"system": "SYSC",
"value": "VALC"
},
{
"use": "old",
"system": "SYS1A",
"value": "VAL1A"
},
{
"use": "old",
"system": "SYS1B",
"value": "VAL1B"
}
],
"link": [
{
"other": {
"reference": "Patient/2"
},
"type": "replaces"
}
]
}
}
]
}
13.18.3.3.2Merge Example 2: Preview merge Patient/2 into Patient/3 synchronously.
Input:
POST /Patient/$merge
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "source-patient",
"valueReference": {
"reference": "Patient/2"
}
},
{
"name": "target-patient",
"valueReference": {
"reference": "Patient/3"
}
},
{
"name": "preview",
"valueBoolean": "true"
}
]
}
Output:
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "input",
"resource": {
"resourceType": "Parameters",
...copy of input...
}
},
{
"name": "outcome",
"resource": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"details": {
"text": "Preview only merge operation - no issues detected"
},
"diagnostics": "Merge would update 25 resources"
}
]
}
},
{
"name": "result",
"resource": {
"resourceType": "Patient",
...merged patient...
}
}
]
}
13.18.3.3.3Merge Example 3: Merging Patients by identifier asynchronously.
Input:
POST /Patient/$merge
Content-Type: application/fhir+json
Prefer: respond-async
{
"resourceType": "Parameters",
"parameter": [ {
"name": "source-patient-identifier",
"valueIdentifier": {
"system" : "urn:oid:1.2.36.146.595.217.0.1",
"value" : "12345"
}
},
{
"name": "target-patient-identifier",
"valueIdentifier": {
"system" : "urn:oid:1.2.36.146.595.217.0.1",
"value" : "12346"
}
}
]
}
Output:
HTTP/1.1 202 Accepted
{
"resourceType": "Parameters",
"parameter": [
{
"name": "input",
"resource": {
"resourceType": "Parameters",
...copy of input...
}
},
{
"name": "task",
"resource": {
"resourceType": "Task",
"id": "352",
"identifier": [
{
"system": "http://hapifhir.io/batch/jobId",
"value": "26738f4d-266c-4ef6-934f-1d13b1b474b9"
}
],
"status": "in-progress"
}
}
]
}
You can poll the status of the returned Task resource see when it completes. For more detailed status about the background job, you can view the status of the corresponding Smile CDR batch job either through the Web Admin Console, or through the Admin JSON API. The id of the Smile CDR batch job is provided as an identifer on the returned Task.
13.18.4$hapi.fhir.undo-merge OperationExperimental
The $hapi.fhir.undo-merge operation undoes the effects of the most recent $merge operation on the given source and target ids. This operation uses the Provenance resource that was created by the $merge operation, and restores all the resources that were updated as part of the operation back to their versions before the operation. This restore operation is done as an update, so it actually creates a newer version of each restored resource. The operation is performed as a transaction, so it either restores all or none.
13.18.4.1Limitations
The hapi.fhir.undo-merge operation currently has the following limitations:
- It fails if any resources to be restored have been subsequently changed since the
$mergeoperation was performed. - It can only run synchronously.
- It fails if the number of resources to restore exceeds Internal Synchronous Search Size configuration. An external FHIR scripting approach can be used for larger undo operations.
13.18.4.2$hapi.fhir.undo-merge Input Parameters
The $hapi.fhir.undo-merge input parameters are a subset of the $merge operation input parameters. They are used to identify the source and target resources from a merge operation that should be restored to their previous version.
| Name | Type | Default | Notes |
|---|---|---|---|
| source-patient-identifier | Identifier | List of source patient identifiers | |
| source-patient | Reference | Source patient | |
| target-patient-identifier | Identifier | List of target patient identifiers | |
| target-patient | Reference | Target patient |
13.18.4.3$hapi.fhir.undo-merge Output Parameters
| Name | Type | Notes |
|---|---|---|
| outcome | OperationOutcome | Outcome of the operation |
13.18.4.4Undo Merge Example:
Assuming that the $merge operation was previously performed to merge Patient/2 to Patient/3,
you can undo that operation with the following request:
Input:
POST Patient/$hapi.fhir.undo-replace-references
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "source-patient",
"valueString": "Patient/2"
},
{
"name": "target-patient",
"valueString": "Patient/3"
}
]
}
Output:
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "outcome",
"resource": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"details": {
"text": "Successfully restored 8 resources to their previous versions based on the Provenance resource: Provenance/1475/_history/1"
}
}
]
}
}
]
}
13.18.5$hapi.fhir.replace-references OperationExperimental
The $hapi.fhir.replace-references operation searches for all resources in the repository that have a reference to the
source resource, and updates those references to point to the target resource. It is a simplified form of the $merge operation when all you want to do is update references.
This operation creates a Transaction Bundle of Patch operations to update the references and returns the output of performing that transaction.
13.18.5.1$hapi.fhir.replace-references Input Parameters
| Name | Type | Default | Notes |
|---|---|---|---|
| source-reference-id | String | The id of the source resource reference to be replaced | |
| target-reference-id | String | The id of the target resource reference that the references will be replaced with | |
| resource-limit | Integer | 512 | If the request is synchrononous and the number of resources to change exceeds this threshold, the operation will fail with 412 Precondition Failed. This parameter has no effect if the Prefer: respond-async header is set |
The resource-limit parameter is available to control how many resources can be changed by this operation. If resource-limit is larger than 10000, the value 10000 will be used.
If you request that the operation be performed asynchronously by providing the Prefer: respond-async HTTP header, then the resource-limit parameter is ignored.
When performed asynchronously, the operation is performed in batches of 1024 resource patches at a time, via PATCH transaction transaction Bundles.
13.18.5.2$hapi.fhir.replace-references Output Parameters
| Name | Type | Notes |
|---|---|---|
| outcome | Bundle | The result of the Bundle patch transaction |
| task | Task | If the operation was performed asynchronously, this Task resource provides details about the status of the operation |
See the $merge operation above for details about the returned Task resource in the case when the operation is performed asynchronously.
13.18.5.3Replace References Provenance Resource
With the 2025.08 release, the $hapi.fhir.replace-references operation creates a Provenance resource upon successful completion. This Provenance resource contains, in its target element, the versioned references to the target resource,
the source resource, and the resources updated as part of the operation.
The Provenance.activity is set to http://terminology.hl7.org/CodeSystem/iso-21089-lifecycle|link, and the Provenance.agent.who is populated with a logical reference to the request user. Note that a provenance resource for this operation is not created if no resources were actually updated because the source resource is not referenced by any resources.
13.18.5.3.1Replace References Example: Replace all references to Patient/2 with references to Patient/3 synchronously.
Input:
POST /$hapi.fhir.replace-references
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "source-reference-id",
"valueString": "Patient/2"
},
{
"name": "target-reference-id",
"valueString": "Patient/3"
}
]
}
Output:
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "outcome",
"resource": {
"resourceType": "Bundle",
"id": "782add05-549c-4a7e-a687-38c22f2f12d0",
"type": "transaction-response",
"entry": [
{
"response": {
"status": "200 OK",
"location": "CarePlan/62/_history/2",
"etag": "2",
"outcome": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"code": "informational",
"details": {
"coding": [
{
"system": "https://hapifhir.io/fhir/CodeSystem/hapi-fhir-storage-response-code",
"code": "SUCCESSFUL_PATCH",
"display": "Patch succeeded."
}
]
},
"diagnostics": "Successfully patched resource \"CarePlan/62/_history/2\"."
}
]
}
}
},
... etc outcome of the rest of the patch operations ...
]
}
}
}
}
13.18.6$hapi.fhir.undo-replace-references OperationExperimental
The $hapi.fhir.undo-replace-references operation undoes the effects of the most recent $hapi.fhir. replace-references operation on the given source and target ids. This operation uses the Provenance resource that was created by the $hapi.fhir.replace-references operation, and restores all the resources that were updated as part of the operation back to their versions before the operation. This restore operation is done as an update, so it actually creates a newer version of each restored resource. The operation is performed as a transaction,
so it either restores all or none.
13.18.6.1Limitations
The hapi.fhir.undo-replace-references operation currently has the following limitations:
- It fails if any resources to be restored have been subsequently changed since the
$hapi.fhir.replace-referencesoperation was performed. - It can only run synchronously.
- It fails if the number of resources to restore exceeds Internal Synchronous Search Size configuration. An external FHIR scripting approach can be used for larger undo operations.
13.18.6.2$hapi.fhir.undo-replace-references Input Parameters
| Name | Type | Default | Notes |
|---|---|---|---|
| source-reference-id | String | The id of the source resource, this must be the same as the source-reference-id that was used in the $hapi-fhir-replace-references operation being undone | |
| target-reference-id | String | The id of the target resource, this must be the same as the target-reference-id that was used in the $hapi-fhir-replace-references operation being undone |
13.18.6.3$hapi.fhir.replace-references Output Parameters
| Name | Type | Notes |
|---|---|---|
| outcome | OperationOutcome | The outcome of the operation |
13.18.6.4Undo Replace References Example:
Assuming that the $hapi.fhir.replace-references operation was previously performed to replace references from Patient/2 to Patient/3,
you can undo that operation with the following request:
Input:
POST /$hapi.fhir.undo-replace-references
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "source-reference-id",
"valueString": "Patient/2"
},
{
"name": "target-reference-id",
"valueString": "Patient/3"
}
]
}
Output:
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "outcome",
"resource": {
"resourceType": "OperationOutcome",
"issue": [
{
"severity": "information",
"diagnostics": "Successfully restored 8 resources to their previous versions based on the Provenance resource: Provenance/1234/_history/1"
}
]
}
}
]
}
13.18.7$sdh.mdm-bundle-match OperationExperimental
The $sdh.mdm-bundle-match operation processes FHIR Bundles to match resources against existing resources in the repository using Master Data Management (MDM) rules. This operation helps prevent duplicate resources by identifying resources in an input Bundle that match existing resources in the repository.
The operation can operate in two modes:
- Deduplication mode (
merge=false, default): Removes matched resources from the bundle and updates references - Survivorship mode (
merge=true): Applies survivorship rules to merge bundle resources with repository resources
13.18.7.1$sdh.mdm-bundle-match Input Parameters
| Name | Type | Default | Notes |
|---|---|---|---|
| bundle | Bundle | The input Bundle containing resources to process for MDM matching | |
| merge | Boolean | false | When true, matched resources are merged using survivorship rules instead of being removed. The merged resources become UPDATE operations. |
13.18.7.2$sdh.mdm-bundle-match Output Parameters
| Name | Type | Notes |
|---|---|---|
| bundle | Bundle | The processed Bundle where matched resources are either removed (merge=false) or updated (merge=true) |
13.18.7.3Processing Behavior
13.18.7.3.1Deduplication Mode (merge=false, default)
When merge=false:
- For each resource in the input bundle, MDM Rules are applied to find matches in the repository 2. If exactly one match is found, the resource is removed from the bundle 3. All references to the removed resource are updated to point to the matched repository resource 4. If no matches or multiple matches are found, the resource remains unchanged in the bundle 5. The resulting bundle contains only non-duplicate resources and updated references
13.18.7.3.2Merge Mode (merge=true)
When merge=true:
- For each resource in the input bundle, MDM Rules are applied to find matches in the repository 2. If exactly one match is found:
- The bundle resource is merged with the repository resource using Survivorship Rules
- The merged resource retains the ID of the repository resource
- The bundle entry is converted to an UPDATE operation with
If-Matchheader for optimistic locking - All references to the resource are updated to use the repository resource ID 3. If no matches or multiple matches are found, the resource remains unchanged in the bundle 4. The resulting bundle contains merged UPDATE operations and updated references
13.18.7.4Requirements
- An MDM module must be configured and enabled for the partition
- MDM module must be in either
MATCH_AND_LINKorMATCH_ONLYmode - Appropriate MDM rules must be configured for the resource types being processed
- For survivorship mode (
merge=true), survivorship scripts should be configured
13.18.7.5Limitations
- If multiple matches are found for a resource, a
ResourceVersionConflictExceptionis thrown - The operation only removes/merges resources that have MDM Rules configured. Other bundle entries are unchanged except for updating references to resources that were removed/merged by the MDM rules.
- When merge=true, the merge behavior depends on the configured Survivorship Rules
13.18.7.5.1Bundle Match Example 1: Deduplication mode (merge=false, default)
Input Bundle contains a Patient that matches an existing Patient in the repository:
Input:
POST /Bundle/$sdh.mdm-bundle-match
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "bundle",
"resource": {
"resourceType": "Bundle",
"type": "transaction",
"entry": [
{
"request": {
"method": "POST",
"url": "Patient"
},
"resource": {
"resourceType": "Patient",
"identifier": [
{
"system": "http://example.org/mrn",
"value": "12345"
}
],
"name": [
{
"family": "Smith",
"given": ["John"]
}
]
}
},
{
"request": {
"method": "POST",
"url": "Observation"
},
"resource": {
"resourceType": "Observation",
"status": "final",
"code": {
"coding": [
{
"system": "http://loinc.org",
"code": "8302-2"
}
]
},
"subject": {
"reference": "urn:uuid:patient-temp-id"
}
}
}
]
}
}
]
}
Output (Patient removed, Observation reference updated):
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "bundle",
"resource": {
"resourceType": "Bundle",
"type": "transaction",
"entry": [
{
"request": {
"method": "POST",
"url": "Observation"
},
"resource": {
"resourceType": "Observation",
"status": "final",
"code": {
"coding": [
{
"system": "http://loinc.org",
"code": "8302-2"
}
]
},
"subject": {
"reference": "Patient/456"
}
}
}
]
}
}
]
}
13.18.7.5.2Bundle Match Example 2: Survivorship mode (merge=true)
Same input as Example 1, but with merge=true:
Input:
POST /Bundle/$sdh.mdm-bundle-match
Content-Type: application/fhir+json
{
"resourceType": "Parameters",
"parameter": [
{
"name": "bundle",
"resource": {
...same bundle as Example 1...
}
},
{
"name": "merge",
"valueBoolean": true
}
]
}
Output (Patient becomes UPDATE operation, references updated):
HTTP/1.1 200 OK
{
"resourceType": "Parameters",
"parameter": [
{
"name": "bundle",
"resource": {
"resourceType": "Bundle",
"type": "transaction",
"entry": [
{
"request": {
"method": "PUT",
"url": "Patient/456",
"ifMatch": "W/\"2\""
},
"resource": {
"resourceType": "Patient",
"id": "456",
"identifier": [
{
"system": "http://example.org/mrn",
"value": "12345"
}
],
"name": [
{
"family": "Smith",
"given": ["John", "J"]
}
]
}
},
{
"request": {
"method": "POST",
"url": "Observation"
},
"resource": {
"resourceType": "Observation",
"status": "final",
"code": {
"coding": [
{
"system": "http://loinc.org",
"code": "8302-2"
}
]
},
"subject": {
"reference": "Patient/456"
}
}
}
]
}
}
]
}
Note how in survivorship mode, the Patient resource becomes a PUT operation that merges data (e.g., additional given name "J") while maintaining the repository resource ID.
13.18.8Deduplication Roadmap
The Deduplication features of Smile CDR are under active development. Here is a roadmap of new features we are planning to roll out.
13.18.8.1Aug 2025 Release
- ✅ Implement Provenance part of the Patient/$merge specification for both
$mergeand$hapi.fhir.replace-references - ✅ Create new
$hapi.fhir.undo-replace-referencesoperation that uses the Provenance resources to undo the effects of$hapi.fhir.replace-referencesoperations (assuming that none of the affected resources have subsequently changed) - ✅ Enhance
$sdh.mdm-bundle-matchoperation withmergeparameter to support survivorship-based resource merging
13.18.8.2Sometime after the Aug 2025 Release
- Create new
$undo-mergeoperation that uses the Provenance resources to undo the effects of$mergeoperations - Extend
$mergeto all Resource types - Create a third MDM mode:
MATCH_AND_MERGEwhich is similar toMATCH_ONLYin that it uses MDM Rules but does not create any links or Golden Resources. When enabled, aMATCH_AND_MERGEMDM module will automatically perform a$mergeoperation on all inbound resources. Any resources that MATCH a single target will be merged into that resources following the MDM Survivorship rules and all references will be updated to point to the merged resource. - Create a new
$deduplicateoperation that takes a FHIR Bundle resource as input and uses MDM Rules to remove duplicates from the inbound Bundle and update references to point to the matched resource. Think of this as a stronger version of Conditional Create where you now have the full power of MDM matching to find matching resources rather than being limited by the FHIR Conditional Create syntax. - Provide this new
$deduplicateoperation as a Camel Processor - Provide a new
$submit-for-deduplicationoperation that works like$mdm-submitand performs MATCH_AND_MERGE on all resources that match the criteria in the request. E.g. If Organization resources with_source=ABCwere accidentally duplicated in your FHIR Repository, you could call$submit-for-deduplicationwith the criteriaOrganization?_source=ABCto submit all of those Organizations for deduplication. Ones that have existing matches would be deleted and all references updated to point to the remaining copy of that organization.