ValueSet Expansion

The FHIR Specification defines two resource types that are used as a part of defining and using codes:

• The CodeSystem resource defines a collection of codes.
• The ValueSet resource creates a collection of codes drawn from one or more CodeSystems, for some specific use.

ValueSets are defined by a collection of rules (the composition). These rules can be simple inclusion rules (e.g. "include codes A, B, and C") or much more complex rules (e.g. "include any codes that are a child of code A", or "include codes with a specific property value", etc).

ValueSet Expansion Pre-Calculation

Smile CDR uses the HAPI FHIR JPA server's pre-expansion mechanism. For details on how it works, including expansion statuses, configuration settings (JpaStorageSettings), the background job lifecycle, and the $expand / $validate-code / $invalidate-expansion operations, see the HAPI FHIR ValueSet Pre-Expansion documentation. The sections below cover Smile CDR-specific behaviour and troubleshooting.

When a ValueSet is uploaded into Smile CDR (e.g. a complete and valid resource of type ValueSet is created in the FHIR repository), Smile CDR starts a background job as soon as that transaction commits, to calculate the expansion and store it in a dedicated set of database tables. Using this pre-calculated expansion has several advantages:

• Requesting the expansion can proceed much more quickly since the expansion doesn't need to be calculated on-demand.
• It is possible to test the ValueSet for membership of a specific code (a common task during resource validation) much more efficiently.
• It is possible to request specific pages (by offset and page size) of ValueSets where the expansion is too large to fit in memory.

Checking Expansion Status

Use the $hapi.fhir.expansion-status operation to list ValueSets by expansion status without database access. The operation returns a Parameters resource with a summary part (counts per status) and one entry per matching ValueSet, each including an errorMessage field on failures.

Invalidating Pre-Calculated Expansion ($invalidate-expansion)

Any time that a ValueSet resource is updated in the database in a way that causes the version number to be incremented, any previously calculated expansion is immediately invalidated and will no longer be used by the system. A new batch job is immediately created that will pre-calculate the expansion in the background. It is also possible to manually request that the existing pre-calculated expansion be invalidated and a new one calculated. This is useful in cases where the underlying CodeSystem has been changed.

To invalidate an existing Pre-Calculated expansion and begin a fresh pre-calculation of the expansion, the $invalidate-expansion operation can be invoked. This operation can either be invoked at the instance level (i.e. against a ValueSet resource ID), or at the type level using a ValueSet canonical URL and version.

Request Parameters

Request Examples

This operation follows the FHIR Asynchronous Interaction Pattern. A Prefer header should be included in the initial request as shown below, but may be omitted for legacy compatibility reasons.

To invalidate any existing pre-calculated expansion and request a new pre-calculated expansion of the ValueSet with ID my-valueset:

POST ValueSet/my-valueset/$invalidate-expansion HTTP/1.1
Prefer: respond-async

To invalidate any existing pre-calculated expansion and request a new pre-calculated expansion of the ValueSet with canonical URL http://foo and version 1.0, use the following form:

POST ValueSet/$invalidate-expansion?url=http%3A%2F%2Ffoo&version=1.0 HTTP/1.1
Prefer: respond-async

The server will respond with a URL which may be polled to request the status of the pre-calculation job:

HTTP/1.1 202 Accepted
Content-Location: http://base/ValueSet/$hapi.fhir.invalidate-expansion.poll-for-status?jobInstanceId=123
Content-Type: application/fhir+json;charset=utf-8

{
  "resourceType": "OperationOutcome",
  "issue": [ {
    "severity": "information",
    "code": "informational",
    "diagnostics": "$invalidate-expansion job has been accepted. Poll for status at the following URL: http://base/ValueSet/$hapi.fhir.invalidate-expansion.poll-for-status?jobInstanceId=123"
  } ]
}

Response

Per the FHIR Asynchronous Interaction Pattern, when the polling URL is invoked with an HTTP GET request, the server will return an HTTP 202 Accepted until the job is complete, at which point it will include an HTTP 200 OK with a Bundle response containing a report. An example of this report follows:

ValueSet Expansion Report
URL: http://example.org/ValueSet/example
Version: 1.0
---------------------------------------------------
Concepts Added               : 24
Concept Designations Added   : 3
---------------------------------------------------
Compose: {"include":[{"system":"http://acme.org"}]}
   Concepts Added               : 24
   Concept Designations Added   : 3

Expanding Hierarchical CodeSystems and ValueSets

Many CodeSystem resources define Concepts in a hierarchy. For example, a fictitious "Animals" CodeSystem might define a code "Pets" with child codes "Dogs" and "Cats".

The hierarchy in a CodeSystem often indicates an "is-a" relationship between the parent code and the child codes. This is not always the case; the hierarchy can imply different kinds of relationships depending on the specific system.

ValueSets can be used to retrieve all of the codes that are a child of a specific code in a CodeSystem. For example, suppose you have the following CodeSystem. Note how there are two codes at the root level ("A" and "B") and each of these codes have children, some of which have further children.

{
  "resourceType": "CodeSystem",
  "url": "http://example.com/my_code_system",
  "content": "complete",
  "concept": [ {
    "code": "A",
    "display": "Code A",
    "concept": [ {
      "code": "AA",
      "display": "Code AA",
      "concept": [ {
        "code": "AAA",
        "display": "Code AAA"
      } ]
    }, {
      "code": "AB",
      "display": "Code AB"
    } ]
  }, {
    "code": "B",
    "display": "Code B",
    "concept": [ {
      "code": "BA",
      "display": "Code BA"
    }, {
      "code": "BB",
      "display": "Code BB"
    } ]
  } ]
}

The hierarchy for the codes above can be visualized as follows:

|-- A
|   |-- AA
|   |   \-- AAA 
|   \-- AB
\-- B 
    |-- BA
    \-- BB

To create a ValueSet containing all of the children of a specific code in this CodeSystem, a ValueSet with a filter can be defined:

{
   "resourceType": "ValueSet",
   "url": "http://example.com/my_value_set",
   "status": "active",
   "compose": {
      "include": [ {
         "system": "http://example.com/my_code_system",
         "filter": [ {
            "property": "concept",
            "op": "is-a",
            "value": "A"
         } ]
      } ]
   }
}

Note that the "is-a" filter will exclude the concept itself. The following example combines an "is-a" filter with a simple explcit code inclusion in order to include the code "A" as well as all of its descendents.

{
   "resourceType": "ValueSet",
   "url": "http://example.com/my_value_set",
   "status": "active",
   "compose": {
      "include": [ {
         "system": "http://example.com/my_code_system",
         "filter": [ {
            "property": "concept",
            "op": "is-a",
            "value": "A"
         } ]
      }, {
         "system": "http://example.com/my_code_system",
         "concept": [ {
           "code": "A"
         } ]
      } ]
   }
}

Requesting A Flat Expansion

Performing a ValueSet expansion is as simple as invoking an HTTP POST on the following URL: http://[server-base-url]/ValueSet/$expand?url=http://example.com/my_value_set

This will produce the following response. Note that the hierarchy is not included in the response.

 {
  "resourceType": "ValueSet",
  "status": "active",
  "compose": {
    "include": [ {
      "system": "http://example.com/my_code_system",
      "filter": [ {
        "property": "concept",
        "op": "is-a",
        "value": "A"
      } ]
    }, {
      "system": "http://example.com/my_code_system",
      "concept": [ {
        "code": "A"
      } ]
    } ]
  },
  "expansion": {
    "identifier": "dcd5e662-bd53-4137-a0af-55771e37b3cb",
    "timestamp": "2021-04-06T14:38:39-04:00",
    "total": 4,
    "offset": 0,
    "parameter": [ {
      "name": "offset",
      "valueInteger": 0
    }, {
      "name": "count",
      "valueInteger": 1000
    } ],
    "contains": [ {
      "system": "http://example.com/my_code_system",
      "code": "AB",
      "display": "Code AB"
    }, {
      "system": "http://example.com/my_code_system",
      "code": "AA",
      "display": "Code AA"
    }, {
      "system": "http://example.com/my_code_system",
      "code": "AAA",
      "display": "Code AAA"
    }, {
      "system": "http://example.com/my_code_system",
      "code": "A",
      "display": "Code A"
    } ]
  }
}

Requesting a Hierarchical Expansion

If you would like the parent-child relationships to be reflected in the response, you can add the includeHierarchy parameter in your request. For example: http://[server-base-url]/ValueSet/$expand?url=http://example.com/my_value_set&includeHierarchy=true

This will produce the following response:

{
  "resourceType": "ValueSet",
  "status": "active",
  "compose": {
    "include": [ {
      "system": "http://example.com/my_code_system",
      "filter": [ {
        "property": "concept",
        "op": "is-a",
        "value": "A"
      } ]
    }, {
      "system": "http://example.com/my_code_system",
      "concept": [ {
        "code": "A"
      } ]
    } ]
  },
  "expansion": {
    "identifier": "442d78a4-1fb5-452e-8df6-94802129a653",
    "timestamp": "2021-04-06T14:38:39-04:00",
    "total": 4,
    "offset": 0,
    "parameter": [ {
      "name": "offset",
      "valueInteger": 0
    }, {
      "name": "count",
      "valueInteger": 1000
    } ],
    "contains": [ {
      "system": "http://example.com/my_code_system",
      "code": "A",
      "display": "Code A",
      "contains": [ {
        "system": "http://example.com/my_code_system",
        "code": "AB",
        "display": "Code AB"
      }, {
        "system": "http://example.com/my_code_system",
        "code": "AA",
        "display": "Code AA",
        "contains": [ {
          "system": "http://example.com/my_code_system",
          "code": "AAA",
          "display": "Code AAA"
        } ]
      } ]
    } ]
  }
}

Troubleshooting Expansion Failures

After installing Implementation Guides or uploading new ValueSets, some ValueSets may fail to expand. This section describes how to identify which ValueSets failed and why.

Identifying ValueSet dependencies

To identify which ValueSets reference a specific CodeSystem, you have the option to use the standard ValueSet-reference SearchParameter:

GET [base]/ValueSet?reference=http://hl7.org/fhir/sid/icd-10-cm

This returns all ValueSets whose compose.include directly references that CodeSystem URL. Note that indirect dependencies (e.g. a ValueSet that includes another ValueSet which depends on the CodeSystem) are not included.

Finding ValueSets that failed to expand

Use $hapi.fhir.expansion-status filtered by status to find failures:

GET [base]/ValueSet/$hapi.fhir.expansion-status?expansionStatus=FAILED_TO_EXPAND

Finding the failure reason

Each FAILED_TO_EXPAND entry in the $hapi.fhir.expansion-status response includes an errorMessage field with the failure cause. The most common causes are:

• A missing CodeSystem means the ValueSet references a CodeSystem that has not been loaded:

  • HAPI-0702: Unable to expand ValueSet because CodeSystem could not be found: <url>. The CodeSystem URL is not registered at all. See Debugging missing content=not-present CodeSystems below for further steps.

• Circular references indicate that the ValueSet or CodeSystem contains a circular parent-child relationship that prevents expansion:

  • HAPI-0849: CodeSystem contains circular reference around code. This means a concept in the CodeSystem hierarchy references itself or an ancestor as a child.

For $validate-code and $lookup, when a content=not-present CodeSystem has no local concepts, the JPA terminology service returns null rather than "code not found", allowing the validation support chain to fall through to the next configured validator (e.g. a remote terminology service). This means $validate-code and $lookup may succeed without uploading the CodeSystem if another validator in the chain can handle the request. See CodeSystems With content=not-present (HAPI FHIR) for details.

See How Uploaded Terminology Interacts with IG Packages for how placeholders work, and Identifying CodeSystems That Need to Be Uploaded for how to find which ones need to be loaded. Once the missing CodeSystem is uploaded, follow the re-expansion steps below.

Re-expanding ValueSets after uploading external CodeSystems

After Uploading CodeSystems, ValueSets that previously failed to expand due to missing CodeSystems need to be re-expanded. How this happens depends on their current expansion status:

There is no periodic sweep that picks up ValueSets needing expansion, so in every case below the re-expansion has to be requested explicitly.

• FAILED_TO_EXPAND or NOT_EXPANDED ValueSets are not retried on their own. A pre-expansion job is only started when a ValueSet resource is created or updated, or when $invalidate-expansion is invoked.
• Previously expanded ValueSets that were already expanded before the CodeSystem was loaded are not re-expanded either. If a ValueSet was expanded while its referenced CodeSystem was missing, the expansion may be incomplete.

In both cases, use the $invalidate-expansion operation, which starts a fresh pre-expansion job immediately and returns a URL for polling its status:

POST ValueSet/[id]/$invalidate-expansion

Note that calling $expand does not start a pre-expansion job. When no usable pre-calculated expansion exists, $expand falls back to expanding in memory to answer that one request, and the stored expansion status is left unchanged.

After re-expansion, verify that the previously failing ValueSets now expand successfully.

Searching for Codes

When building applications that capture data using coded values, a common requirement is to search for codes. This might be done in order to present a user with a dropdown-list, or to provide a type-ahead display with codes matching the characters that a user has entered so far in a text box.

The standard mechanism for performing searches is to use the ValueSet $expand operation. A simple example is shown below, searching for all codes in the http://acme.org CodeSystem (i.e. the CodeSystem in the repository where CodeSystem.url = http://acme.org) having a display name starting with the string "Systolic".

POST /ValueSet/$expand
Content-Type: application/fhir+json

{
  "resourceType": "ValueSet",
  "compose": {
    "include": [ {
      "system": "http://acme.org",
      "filter": [ {
        "property": "display",
        "op": "=",
        "value": "Systolic"
      } ]
    } ]
  }
}