De-identifying Clinical Text with Microsoft Presidio and NLP

Introduction

Before clinical notes can be shared for research or used to train open-source models, they must be stripped of Protected Health Information (PHI) to comply with HIPAA. In this tutorial, we will use Microsoft Presidio, an open-source data protection library, to automatically detect and anonymize PII/PHI in unstructured medical text.

Prerequisites

• Python 3.9+
• Microsoft Presidio (Analyzer and Anonymizer)
• Spacy (for underlying NLP models)

Step 1: Install Dependencies

pip install presidio-analyzer presidio-anonymizer spacy
python -m spacy download en_core_web_lg

Step 2: Detecting PHI

We configure the Presidio Analyzer to use a large Spacy model to detect standard entities like names, dates, phone numbers, and locations in clinical narratives.

from presidio_analyzer import AnalyzerEngine

analyzer = AnalyzerEngine()

clinical_note = "Patient John Doe, DOB 05/12/1975, presented to Mass General on 10/24/2023 complaining of chest pain. Call his wife Jane at 555-0198."

# Analyze text for PII/PHI
results = analyzer.analyze(text=clinical_note, entities=["PERSON", "DATE_TIME", "PHONE_NUMBER", "LOCATION"], language='en')

for result in results:
    print(f"Entity: {result.entity_type}, Score: {result.score:.2f}, Text: {clinical_note[result.start:result.end]}")

Step 3: Anonymizing the Text

Once entities are detected, we use the Anonymizer engine to replace the PHI with placeholder tags or synthetic data.

from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

anonymizer = AnonymizerEngine()

# Replace with entity type (e.g., )
anonymized_result = anonymizer.anonymize(
    text=clinical_note,
    analyzer_results=results,
    operators={
        "DEFAULT": OperatorConfig("replace", {"new_value": "<REDACTED>"}),
        "PERSON": OperatorConfig("replace", {"new_value": "<PATIENT_NAME>"}),
        "PHONE_NUMBER": OperatorConfig("replace", {"new_value": "<PHONE>"})
    }
)

print(anonymized_result.text)
# Output: Patient , DOB , presented to  on  complaining of chest pain. Call his wife  at .
</code></pre>

        
Conclusion
        
By integrating Microsoft Presidio into your data pipelines, you can safely automate the de-identification of massive clinical text datasets, unlocking them for safe downstream ML research and LLM fine-tuning.
    </section>
</article>