Deidentification

All formats that hold medical imaging data derive from MedicalImagingData, which declares a deidentify hook. An implementation strips identifying information from the data and writes the result to a new location. The fileformats-medimage-extras package implements it for DICOM using the deid package, and other packages can implement it for other formats, with recipes in whichever format suits them.

class MedicalImagingData(FileSet):

    @extra
    def deidentify(
        self,
        out_dir: os.PathLike[str],
        recipe: Loaded[FileSet] | None = None,
        **kwargs: Any,
    ) -> Self:
        ...
out_dir

the directory the deidentified data is written to

recipe

the deidentification recipe, already loaded from the recipe format that the implementation for the data’s format expects (see Recipes)

**kwargs

format-specific options, e.g. max_workers for DICOM collections. Implementations ignore options they don’t use.

Implementations only need to strip the identifying data. They don’t report what they changed. Callers that need a record of the changes, e.g. for re-identification, can compare metadata before and after deidentification, which works the same for every format.

Deidentifying DICOM

DICOM files are deidentified with deid recipes, which list actions to apply to the header fields, e.g.

FORMAT dicom

%header

ADD PatientIdentityRemoved YES
REPLACE PatientName ANONYMOUS
REPLACE PatientID var:anon_patient_id
BLANK AccessionNumber
JITTER endswith:Date var:date_jitter
REMOVE startswith:Referring

The recipe is loaded and passed to deidentify, which can be called on a single DicomImage or on a whole DicomSeries/DicomDir. Collections are deidentified file by file, in parallel threads (max_workers sets how many, defaulting to that of concurrent.futures.ThreadPoolExecutor)

>>> from fileformats.medimage import DicomSeries, DeidRecipeX
>>> recipe = DeidRecipeX("/path/to/specs/dicom-series.deid").load()
>>> deidentified = series.deidentify("/path/to/output", recipe=recipe, max_workers=4)

The deidentified files keep their names. A DicomDir is written to a sub-directory of out_dir with the same name as the original directory.

A recipe is required for DICOM. Header fields that the recipe doesn’t mention are left as they are, and so are private tags and sequences (the recipe is applied with deid’s remove_private and strip_sequences turned off), so the recipe needs to account for every field that may hold identifying information, including private tags.

Recipe formats

Two formats are defined for deid recipes:

DeidRecipe (medimage/deid-recipe)

a deid recipe file on its own. Recipes don’t have a standard extension (they are typically named deid.dicom or <name>.deid), so they are identified by the FORMAT <format> line they start with. The recipe can’t use var: or func: values, since nothing defines them.

DeidRecipeX (medimage/deid-recipe-x)

a deid recipe with side-cars that share its stem:

  • <stem>.transforms.yaml (DeidTransforms, required) declares the values of the var: and func: references in the recipe, e.g. a pseudonym hashed from the patient ID, or a date shift read from an environment variable. They are written in a small, declarative language rather than as code, so that recipes can be distributed to and run at sites safely (see Deid transforms specification, version 0.1).

  • <stem>.salt (DeidSalt, optional) holds a secret key that hashes in the transforms are salted with. A warning is logged if other users can read it.

For example, dicom-series.deid, dicom-series.transforms.yaml and dicom-series.salt.

Both are loaded as DeidRecipeWithTransforms, a subclass of deid.config.DeidRecipe that holds the compiled transforms too. Loading fails if the recipe references a var: or func: that isn’t defined, or if the transforms are invalid.

The DICOM implementation accepts either. Its recipe argument is annotated with Loaded[DeidRecipeX] | Loaded[DeidRecipe] | None, in order of preference, so a recipe with side-cars is loaded as DeidRecipeX, and one without them as DeidRecipe (see Finding the recipe format for a datatype).

>>> from fileformats.medimage import DeidRecipe, DeidRecipeX
>>> DeidRecipeX.matches("/path/to/specs/dicom-series.deid")  # has side-cars
True
>>> DeidRecipe("/path/to/specs/deid.dicom").load()  # no side-cars needed

Recipes

The recipe that deidentify takes depends on the format being deidentified. DICOM uses deid recipes, but a recipe for raw scanner data, or for a vendor’s proprietary format, could be completely different. So the hook doesn’t fix the recipe type. Instead, each implementation declares which format(s) its recipe is loaded from, by annotating the recipe argument with Loaded[<recipe format>].

Loaded and loaded_type

Every format has a loaded_type class attribute, the type of the object its load() method returns. Loaded[<format>] is an annotation meaning “data loaded from <format>”. At runtime it evaluates to Annotated[<format>.loaded_type, LoadedMarker(<format>)], so

  • type checkers and most tools see the loaded type itself (static checkers treat it as Any),

  • code that inspects the signature can recover the format the data should be loaded from, from the LoadedMarker,

  • FileSet.load checks that implementations return an instance of the loaded_type, and implementations can check the data they are passed with check_loaded(<format>, data).

The base deidentify hook annotates recipe with Loaded[FileSet], which accepts Loaded[<any format>] in implementations. LoadedMarker is covariant, so data loaded from a subclass of the annotated format is accepted too.

loaded_type can be a class, or a dotted-path string that is only imported when it is needed. Use a string when the class lives in an extras package, so that the format classes don’t depend on it, e.g.

class DeidRecipe(UnicodeFile):
    loaded_type = "fileformats.extras.medimage.deid_recipe.DeidRecipeWithTransforms"

See Loaded data in the FileFormats docs for more details.

Finding the recipe format for a datatype

Callers that deidentify data of many formats, e.g. an ingestion pipeline with a directory of recipes, don’t need to know which recipe format each one uses. They can read it from the recipe annotation of the implementation for the datatype

import typing as ty
from fileformats.core import LoadedMarker, find_extra_implementation
from fileformats.medimage import MedicalImagingData


def load_recipe(datatype: type[MedicalImagingData], recipe_path: Path) -> ty.Any:
    impl = find_extra_implementation(MedicalImagingData.deidentify, datatype)
    hint = ty.get_type_hints(impl, include_extras=True)["recipe"]
    markers = LoadedMarker.all_from_hint(hint)  # in order of preference
    if markers is None:
        raise ValueError(f"The deidentify implementation for {datatype} doesn't take a recipe")
    # Load the recipe from the first of the formats it matches
    recipe_format = next(m.format for m in markers if m.format.matches(recipe_path))
    return recipe_format(recipe_path).load()


recipe = load_recipe(type(series), Path("/path/to/specs/dicom-series.deid"))
deidentified = series.deidentify("/path/to/output", recipe=recipe)

LoadedMarker.all_from_hint returns the formats of a Loaded[X] | Loaded[Y] | None annotation in order, or None if the implementation doesn’t take a recipe (i.e. its recipe argument is annotated with None). The hints must be retrieved with include_extras=True, otherwise the LoadedMarker metadata is stripped. xnat-ingest uses this to pick the recipe for each format from a directory of recipes named after their formats (e.g. medimage/dicom-series).

Custom recipes

There are two ways to use your own recipe format: use it with the DICOM implementation, or use it to deidentify a format of your own.

Extending the DICOM recipe

The DICOM implementation can be passed data loaded from a subclass of DeidRecipe or DeidRecipeX, as long as it loads to a subclass of DeidRecipeWithTransforms. The implementation checks this with check_loaded(DeidRecipe, recipe), and calls the parser method of the loaded recipe.

For example, a recipe that also records the site it was written for. The format goes in your extension package, with its loaded_type given as a dotted path to the class in your extras package

# fileformats/myvendor/deid.py
from fileformats.medimage import DeidRecipeX


class SiteDeidRecipe(DeidRecipeX):
    """A DeidRecipeX that is specific to the site it is in a directory of"""

    loaded_type = "fileformats.extras.myvendor.deid.SiteRecipe"

The loaded type and the load implementation go in your extras package. The implementation constructs the loaded type directly. It doesn’t call DeidRecipeX.load(recipe), which would dispatch back to itself.

# fileformats/extras/myvendor/deid.py
import typing as ty
from fileformats.core import FileSet, Loaded, extra_implementation
from fileformats.extras.medimage.deid_recipe import DeidRecipeWithTransforms
from fileformats.myvendor import SiteDeidRecipe


class SiteRecipe(DeidRecipeWithTransforms):

    def __init__(self, deid: str, site: str, **kwargs: ty.Any) -> None:
        super().__init__(deid, **kwargs)
        self.site = site


@extra_implementation(FileSet.load)
def load_site_recipe(recipe: SiteDeidRecipe, **kwargs: ty.Any) -> Loaded[SiteDeidRecipe]:
    salt = recipe.salt_file.load() if recipe.salt_file is not None else None
    return SiteRecipe(
        str(recipe.fspath),
        site=recipe.fspath.parent.name,
        transforms=recipe.transforms_file.load(salt=salt),
    )

The return annotation of a load implementation has to be Loaded[<format>] or the loaded_type itself, and load raises a TypeError if it returns anything else. The recipe can then be loaded and passed to the DICOM implementation

>>> recipe = SiteDeidRecipe("/path/to/site-a/dicom-series.deid").load()
>>> recipe.site
'site-a'
>>> deidentified = series.deidentify("/path/to/output", recipe=recipe)

Callers that find the recipe format from the annotation (see Finding the recipe format for a datatype) will still load DeidRecipeX/DeidRecipe. To have them use your format, implement deidentify for the datatype, with recipe: Loaded[SiteDeidRecipe], as below.

Recipes for other formats

To deidentify a format of your own, define a recipe format for it with a loaded_type, implement load for the recipe format, and implement deidentify for the data format, annotating recipe with Loaded[<recipe format>]. For example, for a raw data format whose header is key=value lines, with a YAML recipe listing the fields to blank

# fileformats/myvendor/__init__.py (no dependencies)
from fileformats.generic import BinaryFile, UnicodeFile
from fileformats.medimage import MedicalImagingData


class RawDeidRecipe(UnicodeFile):
    """Lists the header fields of MyRawData files to blank"""

    ext = ".raw-deid.yaml"
    loaded_type = "fileformats.extras.myvendor.deid.RawDeidRules"


class MyRawData(MedicalImagingData, BinaryFile):
    ext = ".myraw"

# fileformats/extras/myvendor/deid.py
import os
import typing as ty
from dataclasses import dataclass, field
from pathlib import Path

import yaml
from fileformats.core import FileSet, Loaded, check_loaded, extra_implementation
from fileformats.medimage import MedicalImagingData
from fileformats.myvendor import MyRawData, RawDeidRecipe


@dataclass
class RawDeidRules:
    blank: list[str] = field(default_factory=list)


@extra_implementation(FileSet.load)
def load_raw_deid_recipe(recipe: RawDeidRecipe, **kwargs: ty.Any) -> Loaded[RawDeidRecipe]:
    with recipe.open() as f:
        spec = yaml.safe_load(f) or {}
    return RawDeidRules(blank=list(spec.get("blank", [])))


@extra_implementation(MedicalImagingData.deidentify)
def my_raw_deidentify(
    data: MyRawData,
    out_dir: os.PathLike[str],
    recipe: Loaded[RawDeidRecipe] | None = None,
    **kwargs: ty.Any,
) -> MyRawData:
    if recipe is None:
        raise ValueError("A recipe is required to deidentify MyRawData")
    check_loaded(RawDeidRecipe, recipe)  # e.g. not a recipe for another format
    header = []
    for line in data.fspath.read_text().splitlines():
        key = line.split("=", 1)[0]
        header.append(f"{key}=" if key in recipe.blank else line)
    out = Path(out_dir) / data.fspath.name
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text("\n".join(header))
    return MyRawData(out)

Notes on the implementation’s signature:

  • the first argument is the format it is implemented for, and the return type has to match it,

  • recipe can be annotated with a union of formats in order of preference, e.g. Loaded[RawDeidRecipeV2] | Loaded[RawDeidRecipe] | None, and with None alone (recipe: None = None) if the implementation doesn’t take a recipe,

  • other options are keyword arguments with defaults, and **kwargs takes the options meant for other implementations.

Callers can then use your recipe without knowing about it, e.g. with the load_recipe function above

>>> recipe = load_recipe(MyRawData, Path("/path/to/rules.raw-deid.yaml"))
>>> recipe
RawDeidRules(blank=['PatientName'])
>>> MyRawData("/path/to/scan.myraw").deidentify("/path/to/output", recipe=recipe)