FHIR Bulk Export Operation

LMA

The FHIR Bulk Data Access protocol describes a mechanism for efficiently requesting large amounts of data in a convenient format. This protocol can be used to request all data on a server, or a specific subset that is useful for a given use case.

Enabling Bulk Export

LMA

Bulk export support is configured on the FHIR Storage module via the bulk_export.enabled property.

Note that Bulk Export file generation can involve the creation of many large Binary resources, so it is recommended to enable Externalized Binary Storage when using this feature.

The Maximum Bulk Export File Capacity property may be used to control the size of the generated files.

Types of Bulk Export Requests

LMA

The FHIR Bulk Data Access specification describes several invocation styles. The following table outlines currently supported functionality.

What Resources Are Included in Group Export

When performing a Group bulk export (Group/[id]/$export), the following resources are included:

Group Member Patients:

• All Patient resources that are members of the specified Group.
• With _mdm=true parameter, includes MDM-linked patients and golden resources.

Patient Compartment Resources:

• Resources related to group members.
• Determined by FHIR patient compartment definitions.
• Examples: Observation, Encounter, Condition, AllergyIntolerance, Procedure, etc.

Forward-Referenced Resources:

• Resources in the patient compartment referenced by resources being returned regardless of when the referenced resources were last updated.
• Due to security concerns, Patient resources are excluded from forward-referenced resources.
• Examples: Practitioner or Organization referenced by Observations/Encounters.

The Group Resource:

• The Group resource itself.

Request Parameters

LMA

Requesting A Bulk Extract

LMA

This section describes the user flow when requesting a bulk extract.

First, a user invokes the $export operation to initiate the request. This can be done using either an HTTP GET request with export request parameters included in the URL or as an HTTP POST with the export request parameters included in a FHIR Parameters resource.

Initiating Using HTTP GET

The following example shows a URL that can be used to initiate a Bulk Export using an HTTP GET. For readability, the URL is split over multiple lines but it would not be in a real invocation. https://fhir.example.com:8000/$export
   ?_outputFormat=application%2Ffhir%2Bndjson
   &_type=Patient%2C%20Practitioner
   &_since=2019-10-25T10%3A07%3A58.788-04%3A00
   &_typeFilter=Practitioner%3Fidentifier%3Dbar

The complete HTTP Request is shown below:

GET /$export?[params]
Prefer: respond-async

A similar Group Bulk Export Request can be seen below:

GET /Group/123/$export?_type=Immunization
Prefer: respond-async

Here's an example of fetching all Immunizations within a Group, which have vaccine-code of COVID-19:

GET /Group/123/$export?_type=Immunization&_typeFilter=Immunization%3Fvaccine-code%3DCOVID-19
Prefer: respond-async

Here's an example of exporting a specific patient with MDM expansion enabled:

GET /Patient/123/$export?_mdm=true
Prefer: respond-async

This will export Patient/123 along with all patients linked to the same golden resource via MDM, and all resources in the patient compartment for those patients.

To request multiple resource types, use a comma-separated list for the _type parameter but use multiple instances of the _typeFilter parameter:

GET /$export?_type=Patient,ExplanationOfBenefit&_typeFilter=Patient%3F_id=Patient/123&_typeFilter=ExplanationOfBenefit%3Fpatient=Patient/123
Prefer: respond-async

Initiating Using HTTP POST

The following example shows an example Parameters resource that can be used to initiate a Bulk Export using an HTTP POST.

{
  "resourceType": "Parameters",
  "parameter": [
    {
      "name": "_outputFormat",
      "valueString": "application/fhir+ndjson"
    },
    {
      "name": "_type",
      "valueString": "Patient, Practitioner"
    },
    {
      "name": "_since",
      "valueInstant": "2019-10-25T11:01:45.660-04:00"
    },
    {
      "name": "_typeFilter",
      "valueString": "Patient?identifier=foo"
    }
  ]
}

The complete HTTP Request is shown below:

POST /$export
Prefer: respond-async
Content-Type: application/fhir+json

{ ...payload... }

Initiation Response

Once the server has processed the request, it will respond with a response similar to the following:

HTTP/1.1 202 Accepted
Content-Location: https://fhir.example.com:8000/$export-poll-status?_jobId=0000000-1111111-2222222

The status of 202 means that the server has successfully received the request and has scheduled a background job that will assemble the Bulk Export payload. The Content-Location header indicates a URL that may be used by the client to request an update on the status of the background job. If an identical job has been requested in the last hour, instead of receiving a new job, you will be given the URL of the previously stored job. If you need to override this and force a new job to start, you can do so by setting the cache header as follows: Cache-Control: no-cache. This will force a new job regardless of duplicate jobs in the system.

Authorization for Bulk Export

LMA

A user must have appropriate permissions in order to initiate a bulk export job. For bulk export jobs, often the user is actually a Client that has authenticated using the Client Credentials with JWT Credential system flow, but it can also be a regular user.

A user/client must have appropriate permissions in order to initiate the job. If the user has authorized using SMART on FHIR, the approved scopes must also be appropriate.

The following permissions can be used:

• FHIR_OP_INITIATE_BULK_DATA_EXPORT – This permission allows the user to initiate a bulk export of any kind (system/group/patient).

  • If the permission has no argument, this permission implies that the user can initiate a bulk export for any or all resource types in the system. This is a very powerful permission and should not be used unless you intend to grant the user virtually unlimited access to the data in the repository.
  • The permission may also take an argument, which is a space-separated list of resource types the user is allowed to request a bulk export for. For example, if the user has the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT/QuestionnaireResponse Questionnaire the user will only be allowed to initiate bulk exports for QuestionnaireResponse and Questionnaire resources.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_SYSTEM – This permission allows the user to initiate a system-level bulk export (i.e. a bulk export that is not limited in scope to data about any one patient or group)

  • If the permission has no argument, this permission implies that the user can initiate a bulk export for any or all resource types in the system. This is a very powerful permission and should not be used unless you intend to grant the user virtually unlimited access to the data in the repository.
  • The permission may also take an argument, which is a space-separated list of resource types the user is allowed to request a bulk export for. For example, if the user has the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_SYSTEM/QuestionnaireResponse Questionnaire the user will only be allowed to initiate bulk exports for QuestionnaireResponse and Questionnaire resources.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUP – This permission allows the user to initiate a Group-Level bulk export, meaning an export that contains only files relating to a specific Group resource and its members.

  • This permission must have an argument, containing at a minimum the ID of the Group to export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUP/Group/123 allows the user to initiate a group-level bulk export for the group Group/123.
  • The argument may optionally start with the group ID, but be followed with a space-separated list of resource types the user is allowed to request as a part of the export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUP/Group/123 Patient Observation allows the user to initiate a group-level export, but disallows the user from requesting resource types other than Patient or Observation as a part of that request.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUPS_MATCHING – This permission allows the user to initiate a Group-Level bulk export, meaning an export that contains only files relating to Group resource and its members matching the provided permission FHIR query filter.

  • This permission must have an argument, containing at a minimum a FHIR query matching the Group(s) to export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUPS_MATCHING/?identifier=foo|bar allows the user to initiate a group-level bulk export for the any Group that has the identifier foo|bar. Note that the Group [resource type] part of the URL is left out since the resource type is implied.
  • After the Group query, the argument may optionally contain a space-separated list of resource types the user is allowed to request as a part of the export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_GROUPS_MATCHING/?name=abc Patient Encounter allows the user to initiate a Group-level export on Groups with Group.name=abc, but disallows the user from requesting resource types other than Patient or Observation as a part of that request.
  • Note: having this permission to invoke a Group-level export does not imply that the user has permissions to export Group type resources; unless it is explicitly included in the allowed resource types list or if the user is allowed to export all resource types.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS – This permission allows the user to initiate a Patient-Level bulk export, meaning an export that contains only files relating to one or more specific Patient resource.

  • This permission must have an argument, containing at a minimum the ID(s) of the Patient to export and either a * for all resources or the specific list of permitted resources. Multiple IDs should be comma-separated. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS/Patient/a,Patient/b * allows the user to initiate a Patient-level bulk export for the patients Patient/a and Patient/b for all resources.
  • The argument may optionally start with the patient ID(s), but be followed with a space-separated list of resource types the user is allowed to request as a part of the export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS/Patient/a,Patient/b Patient Encounter allows the user to initiate a Patient-level export, but disallows the user from requesting resource types other than Patient or Encounter as a part of that request.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_ALL_PATIENTS – This permission allows the user to initiate a Patient-Level bulk export with NO restrictions and NO arguments. This permission should take the form of FHIR_OP_INITIATE_BULK_DATA_EXPORT_ALL_PATIENTS

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENT – Deprecated: Users should use FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS or FHIR_OP_INITIATE_BULK_DATA_EXPORT_ALL_PATIENTS instead. This permission allows the user to initiate a Patient-Level bulk export, meaning an export that contains only files relating to a specific Patient resource.

  • This permission must have an argument, containing at a minimum the ID of the Patient to export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENT/Patient/123 allows the user to initiate a Patient-level bulk export for the patient Patient/123.
  • The argument may optionally start with the patient ID, but be followed with a space-separated list of resource types the user is allowed to request as a part of the export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENT/Patient/123 Patient Observation allows the user to initiate a Patient-level export, but disallows the user from requesting resource types other than Patient or Observation as a part of that request.

• FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS_MATCHING – This permission allows the user to initiate a Patient-Level bulk export, meaning an export that contains only files relating to one or more Patient resources that matches the provided permission FHIR query filter.

  • This permission must have an argument, containing at a minimum a FHIR query matching the Patient(s) to export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS_MATCHING/?identifier=foo|bar allows the user to initiate a Patient-level bulk export for the any Patient that has the identifier foo|bar. Note that the Patient [resource type] part of the URL is left out since the resource type is implied.
  • After the Patient query, the argument may optionally contain a space-separated list of resource types the user is allowed to request as a part of the export. For example, the permission FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENTS_MATCHING/?name=Doe Patient Encounter allows the user to initiate a Patient-level export on Patients with the name Doe, but disallows the user from requesting resource types other than Patient or Encounter as a part of that request.
  • Note: having this permission to invoke a Patient-level export does not imply that the user has permissions to export Patient type resources; unless it is explicitly included in the allowed resource types list or if the user is allowed to export all resource types.

Approved SMART Scopes

If the user/client has authenticated using SMART on FHIR, the approved scopes will also be factored into the resource types that are permitted to be requested via bulk export.

For example, suppose a user has the following permission:

FHIR_OP_INITIATE_BULK_DATA_EXPORT_PATIENT/Patient/123 Patient Observation Encounter

And the client has approved the following scopes:

system/Patient.read system/Observation.read

In this case, the user will be allowed to request a bulk export for the Patient and Observation resources, but will be blocked from requesting a bulk export containing Encounter resources.

If the client scopes include filter constraints (see SMART finer-grained resource constraints) such as patient/Observation?category=vital-signs, then the export contents will be restricted to match. For each exported resource type, the filters of the search scopes (e.g. .s, .rs, or .read) covering that type are applied. If no search scope covers the type, the filters of the read-only scopes (e.g. .r) covering it are applied instead. The export filters are not narrowed when the client holds an unfiltered wildcard search scope (e.g. system/*.s), or an unfiltered wildcard read-only scope (e.g. system/*.r) and no search scope.

Polling for Job Status

LMA

Once a job has been initiated, the client may poll the server to request information about the status. This is done by requesting an HTTP GET for the URL specified in the Content-Location header in the previous step.

GET https://fhir.example.com:8000/$export-poll-status?_jobId=0000000-1111111-2222222

The server will respond with a response similar to the following while the job is still being built.

HTTP/1.1 202 Accepted
X-Progress: Build in progress - Status set to BUILDING at 2019-10-25T11:27:16.763-04:00
Retry-After: 120

When the server has completed assembling the export payload, the same polling request will return a response similar to the following. The response payload is a list of files that were assembled by the Binary export job. If no URLs are returned, it means that no resources met the request parameter(s) criteria. Note that the payloads are assembled using Binary resources, so the client will need to have appropriate permission on the server to download Binary resources in order to be able to access the export contents.

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 311 

{
  "transactionTime" : "2019-10-25T11:31:22.487-04:00",
  "output" : [ {
    "type" : "Patient",
    "url" : "https://fhir.example.com:8000/Binary/111"
  }, {
    "type" : "Patient",
    "url" : "https://fhir.example.com:8000/Binary/222"
  }, {
    "type" : "Patient",
    "url" : "https://fhir.example.com:8000/Binary/333"
  } ]
}

Accessing Bulk Export Files

When the polling process has completed, the FHIR endpoint will respond with a list of Binary file URLs that can be used to access the NDJSON data gathered as a part of the export.

If Restrict Download to Initiating User has been enabled on the FHIR Endpoint module (this setting defaults to being enabled) then only the same user on the same node will be able to access the generated Binary files.

Cancelling a Bulk Export Job

The FHIR Bulk Data Access specification describes a mechanism for cancelling a running export by sending a DELETE request to the polling content location (the URL returned in the Content-Location header when the export was initiated).

FHIR Endpoint

On a standard FHIR endpoint, a running bulk export job can be cancelled using the Batch Job Endpoint:

DELETE /admin-json/batch2/job-instance/{instanceId}

The instanceId corresponds to the _jobId returned in the Content-Location header when the export was initiated.

FHIR Gateway

On a FHIR Gateway endpoint, cancelling an export operation using DELETE on the polling content location ($export-poll-status) is not supported. See FHIR Gateway Limitations for details.

As a workaround, if you have access to the Admin JSON API on the target FHIR server(s), you can cancel the underlying job directly using the Admin Batch API endpoint shown above.

Limitations

When using bulk export, it is important to understand the limitations of the system.

• List and Group resources will be omitted from any export requests on Patient compartments. These are omitted for security reasons because these resources typically have references to multiple Patients and could therefore unintentionally expose sensitive information.
• Resources may appear more than once in the generated export files if they are referenced by multiple compartments that are being exported.