De-identifying Clinical Text with SpaCy and NLP

Introduction

Before clinical notes, discharge summaries, or pathology reports can be used for secondary research or training Large Language Models (LLMs), they must be rigorously de-identified to comply with regulations like HIPAA. Protected Health Information (PHI) such as names, dates, phone numbers, and locations must be redacted. In this tutorial, we will use the open-source NLP library SpaCy to build a custom pipeline that identifies and redacts PHI from unstructured clinical text.

Architecture Overview

graph TD
    Text(Raw Clinical Text) --> Tokenizer[SpaCy Tokenizer]
    Tokenizer --> Rules[Rule-Based Matcher: EntityRuler]
    
    Rules -.-> |Custom Lists: Hospitals, Doctors| Tagged1[Pre-tagged Entities]
    Tagged1 --> NER[Transformer NER Model: en_core_web_trf]
    
    NER --> |Statistical Predictions: PERSON, DATE| Tagged2[Combined Entities]
    
    Tagged2 --> RegEx[RegEx Matcher]
    RegEx -.-> |SSN, Phone, MRN| FinalList[Final Entity List]
    
    FinalList --> Redact[Redaction Engine: Replace with Tags]
    Redact --> Clean(De-identified Text)
        

Prerequisites

• Python 3.8+
• SpaCy
• Basic understanding of Named Entity Recognition (NER)

Step 1: Install Dependencies and Language Models

We need to install SpaCy and download a pre-trained English transformer model which is highly accurate at recognizing standard entities (like PERSON, ORG, DATE).

pip install spacy
python -m spacy download en_core_web_trf

Step 2: Basic Named Entity Recognition

Let's see how the out-of-the-box model performs on a mock clinical note.

import spacy

# Load the transformer model
nlp = spacy.load("en_core_web_trf")

clinical_note = """
Patient Name: Johnathan Doe
DOB: 05/14/1965
Date of Admission: October 12, 2023
Attending Physician: Dr. Sarah Jenkins
Hospital: Massachusetts General Hospital

History of Present Illness:
Mr. Doe, a 58-year-old male from Boston, presented to the ED on 10/12/2023 complaining of severe chest pain.
His daughter, Emily, brought him in. He works at Acme Corp and his emergency contact number is 555-0198.
"""

doc = nlp(clinical_note)

print("Entities found:")
for ent in doc.ents:
    print(f"{ent.text:<25} {ent.label_}")

Step 3: Building the Redaction Pipeline

The base model will find most names and dates, but we need to actively replace them in the text. Furthermore, standard NER models might miss highly formatted PHI like phone numbers or SSNs, so we should augment the pipeline with regular expressions (RegEx).

import re

# Define PHI entity labels we want to redact from the NER model
PHI_LABELS = {"PERSON", "ORG", "GPE", "LOC", "DATE", "TIME"}

def redact_phi(text):
    doc = nlp(text)
    redacted_text = text
    
    # 1. Redact NER Entities (Sort reverse so indices don't shift during replacement)
    entities = [(ent.start_char, ent.end_char, ent.label_) for ent in doc.ents if ent.label_ in PHI_LABELS]
    entities.sort(key=lambda x: x[0], reverse=True)
    
    for start, end, label in entities:
        # Replace the entity with its label, e.g. [PERSON]
        replacement = f"[{label}]"
        redacted_text = redacted_text[:start] + replacement + redacted_text[end:]
        
    # 2. Redact Phone Numbers using RegEx
    # Simple regex for matching xxx-xxxx or (xxx) xxx-xxxx
    phone_pattern = re.compile(r'\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}')
    redacted_text = phone_pattern.sub("[PHONE]", redacted_text)
    
    # 3. Add more RegEx for SSNs, MRNs, Zip Codes as needed...
    
    return redacted_text

clean_note = redact_phi(clinical_note)
print("\n--- Redacted Note ---\n")
print(clean_note)

Step 4: Handling Edge Cases (Custom Rules and Fuzzy Matching)

Sometimes the model makes mistakes (e.g., misclassifying "Acme Corp" or missing a doctor's name if the capitalization is weird). We can add an EntityRuler to the SpaCy pipeline to force specific matches (like known hospital names or specific provider lists) before the statistical model runs. For patient names that might be misspelled, we can use fuzzy matching.

# Add a custom rule-based matcher BEFORE the NER model
ruler = nlp.add_pipe("entity_ruler", before="ner")
patterns = [
    {"label": "HOSPITAL", "pattern": "Massachusetts General Hospital"},
    {"label": "HOSPITAL", "pattern": "MGH"},
    # We can also add patterns for known local doctors
    {"label": "PERSON", "pattern": [{"LOWER": "dr"}, {"IS_PUNCT": True, "OP": "?"}, {"LOWER": "jenkins"}]}
]
ruler.add_patterns(patterns)

# Now, "Massachusetts General Hospital" will definitively be tagged as HOSPITAL 
# rather than relying on the statistical model's ORG prediction.

# --- Handling Fuzzy Matching for Known Patient Names ---
import regex # Note: the 'regex' module supports fuzzy matching, standard 're' does not
# pip install regex

known_patient_name = "Johnathan Doe"
# Search for the name allowing up to 2 substitutions, insertions, or deletions (e.g., "Jonothan Do")
fuzzy_pattern = regex.compile(f"({known_patient_name})", regex.IGNORECASE)

def apply_fuzzy_redaction(text, pattern):
    return pattern.sub("[PATIENT_NAME]", text)

# clean_note = apply_fuzzy_redaction(clean_note, fuzzy_pattern)

Conclusion

De-identifying clinical text is a critical first step in medical AI research. While pure regular expressions are brittle, and pure machine learning models can be unpredictable, combining SpaCy's state-of-the-art transformer NER with custom regular expressions, deterministic rule-based matching, and fuzzy logic creates a robust, highly accurate de-identification pipeline capable of scaling to millions of documents.