Deduplication on Ingestion

In the deduplication on ingestion strategy, an incoming resource that matches a resource already in the repository is merged into it or dropped before it is stored. The repository always holds exactly one resource for each real-world entity. This page describes how to deploy the strategy: run the MDM module in MATCH_ONLY mode and reconcile incoming bundles with the $sdh.mdm-bundle-match operation.

The strategy requires the repository to be free of duplicates when it is activated. Duplicates already in the repository can be merged beforehand using the operations described in Working with Duplicates.

MDM MATCH_ONLY mode

By default, MDM runs in MATCH_AND_LINK mode with the complete processing described in Golden Record MDM. If, however, all that is required is support for the Patient/$match operation, MDM Mode can be set to MATCH_ONLY. In MATCH_ONLY mode, no Golden Resources or Links are created and all the infrastructure associated with that (subscription, broker topics, broker consumers etc) are disabled and the MDM Rules definition is used only to define the behavior of the Patient/$match operation.

MATCH_ONLY mode can also provide EID-based query expansion for records linked by an externally assigned enterprise identifier. That configuration supports a different strategy, and is described in Integrating an External Entity Resolution Service.

Deduplicating bundles with $sdh.mdm-bundle-match

The $sdh.mdm-bundle-match operation reconciles an incoming Bundle against the repository before it is persisted. For each resource in the bundle that has MDM matching rules configured, the operation searches for a matching resource in the repository. When a match is found, the bundle resource is either dropped, or merged into the repository resource using survivorship rules (merge=true), and all references within the bundle are rewritten to point at the repository resource. The returned transaction bundle can then be submitted to the repository.

The operation, its parameters and examples are documented in $sdh.mdm-bundle-match Operation. To apply the same reconciliation inside an ingestion pipeline, use the MDM Bundle Match Processor in a Camel route, which can also run it asynchronously as a background step.

If the candidate search for a bundle entry reaches the configured maximum threshold, that entry cannot be reconciled.

Rather than rejecting the whole bundle, the entry is left in place and tagged as omitted from MDM matching.

It is then stored as a new resource instead of being folded into an existing one, and the tag records why.

Other entries in the same bundle are unaffected.