Supporting structured / semi-structured data with Presidio

[omri374]

Handling structured / semi-structured data with Presidio

• Context / Problem Statement
  • Use cases we would like to support initially
  • Use cases we might want to consider in the future
• Considered Options
  • Code examples
• Consequences
• Next steps

Context / Problem Statement

Presidio's main strength lies in the ability to detect PII entities in unstructured text. However, many use cases involve either fully structured or semi-structured data, where Presidio cannot be used out of the box. The scenarios for structured/semi structured data de-identification are mainly around tabular data which might or might not contain free text columns, and json/xml files with specific fields containing PII.

Use cases we would like to support initially

1. Read tabular data iteratively (via cursor or iterator) and return an iterable with anonymized rows.
2. Read json file + mapping of fields and return the same json with the anonymized values.

Use cases we might want to consider in the future

1. Automatically detect which columns need anonymization and which can be ignored by either:
   1. Run a function on column names/keys which returns which columns/keys require de-identification.
   2. Allow sampling of data to decide whether to de-identify the entire column or leave it as is.
2. Allow users to define which keys can be useful for detection as additional context, i.e. use an entity in one column (e.g. customer name) as a hint for detection in other columns (e.g. free text entered by the same customer).
3. Allow users to configure which entity types should be searched for in every key/column.
4. Allow users to use non-string objects for identification and anonymization.
5. Allow customization of the anonymization mechanism per column / entity.
6. Entire redaction of column vs. anonymization of entities in values ("" vs "My name is ").
7. Let users run the analyzer on their structured data to say which column/key contains which PII entities.

Considered Options

1. Key/value pairs:
   a. Provide a new analyzer API for reading a flat dict/json document, and iterate over keys and values.
   b. Provide a new anonymizer API for recreating the anonymized json/dict.

   This will support both structured data in tables (assuming each row is treated as key:value) and flat json documents. This option will allow users to scan and de-identify tabular data (sql, pandas) as well as shallow dictionaries (i.e. Dict[str, str]).

2. Key/list of values: In addition to (1), provide the ability to run on dicts/json containing lists, such as this one:

   {
    "a": "Scottie",
    "b": ["Michael", "Dennis"],
    "c": 5  
   }

   This will extend (1) with the ability to run on dictionaries of type Dict[str, List[str]] as well as on entire columns on tabular data (assuming the tabular data is represented as a dictionary, where every column has a key and list of values).

3. Dictionaries/json of arbitrary structure: In addition to (1) and (2), provide the ability to parse deeper json/dict structures. A naive approach would be to flatten the dict, run the logic in (1) or (2) and unflatten it back (potentially with something like this). Note that this library does not support the flattening of list of objects, so it would not support all use cases (note: a thorough analysis of options is required).

Code examples

High level code illustration of option 1

• Analyzer:

@dataclass
class DictAnalyzerResult:
    key: str
    value: str
    recognizer_results: List[RecognizerResult]

def analyze_dict(self, input_dict: Dict[str, str], **kwargs) -> Iterator[DictAnalyzerResult]:
    for key, value in input_dict.items():
        results = self.analyze(text=value, **kwargs) if isinstance(value, str) else []
        yield DictAnalyzerResult(key=key, value=value, recognizer_results=results) 

• Anonymizer:

def anonymize_dict(self, analyzer_results:Iterator[DictAnalyzerResult],**kwargs) -> Dict[str, str]:

    return_dict = {}
    for result in analyzer_results:
        if isinstance(result.value, str):
            resp = self.anonymize(text=result.value, analyzer_results=result.recognizer_results, **kwargs)
            return_dict[result.key] = resp.text              
        else:
            return_dict[result.key] = result.value             

    return return_dict

High level code illustration of option 2

1. Generalize the DictAnalyzerResult to support lists as well

@dataclass
class DictAnalyzerResult:
    key: str
    value: Union[str, List[str]]
    recognizer_results: Union[List[RecognizerResult], List[List[RecognizerResult]]]

2. Analyzer:
   • Add an analyze_list method
   • Same logic as 1 + checking whether the input is a string or an iterable

def analyze_list(self, list_of_texts: Iterable[str], **kwargs) -> List[List[RecognizerResult]]:
   list_results = []
   for text in list_of_texts:
      results = self.analyze(text=text, **kwargs) if isinstance(text, str) else []
      list_results.append(results)

   return list_results

def analyze_dict(
 self, input_dict: Dict[str, Union[str, Iterable[str]]], **kwargs) -> Iterator[DictAnalyzerResult]:

    for key, value in input_dict.items():
        if not value:
            results = []
        else:
            if isinstance(value, str):
                results: List[RecognizerResult] = self.analyze(text=value, **kwargs)
            elif isinstance(value, collections.Iterable):
                results: List[List[RecognizerResult]] = self.analyze_list(
                            list_of_texts=value, 
                            key=key, 
                            **kwargs)
            else:
                results = []

    yield DictAnalyzerResult(key=key, value=value, recognizer_results=results)

3. Anonymizer:

def anonymize_list(self, texts:List[str], recognizer_results_list: List[List[RecognizerResult]], **kwargs) -> List[EngineResult]:
    return_list = []
    for text, recognizer_results in zip(texts, recognizer_results_list):
        if isinstance(text,str):
            res = self.anonymize(text=text,analyzer_results=recognizer_results,**kwargs)
            return_list.append(res.text)
        else:
            return_list.append(text)
        
 return return_list


def anonymize_dict(self, analyzer_results: Iterator[DictAnalyzerResult],**kwargs) -> Dict[str, str]:

    return_dict = {}
    for result in analyzer_results:
        if isinstance(result.value, str):
            resp = self.anonymize(text=result.value, analyzer_results=result.recognizer_results, **kwargs)
            return_dict[result.key] = resp.text
        elif isinstance(result.value, collections.Iterable):
            anonymize_respones = self.anonymize_list(texts=result.value,
                                                     recognizer_results_list=result.recognizer_results, 
                                                     **kwargs)
            return_dict[result.key] = anonymize_respones 
        else:
            return_dict[result.key] = result.value
	
    return return_dict

Note: Handling of collections of texts is not done via iterators but as lists in memory.

Consequences

• Allow users to de-identify tabular data
• Allow users to de-identify 1-level json documents
• Allow users to extend the capability with additional filtering and analysis capabilities
• Users would have to handle multi-level json objects themselves
• Non-string columns are completely ignored at this point

Next steps

Dear community members, we would like to hear your thoughts on this. Is this approach supporting your use case? Please share your feedback in the comments. Contributions from the community into Presidio are also welcome!
For specific questions, feel free to reach out to the team at presidio@microsoft.com.

[willsthompson]

All of the constraints related to input and structure seem reasonable, but I think it's crucial that keys be included in the analysis. Keys and values are part of the document structure, and IMHO much of the value from being able to analyze structured data comes from being able to analyze those relationships, not only the discrete values.

In our case, we're trying to recognize psudoidentifiers, and many of them have values that are inherently hard to detect with a practically useful level of confidence, like age or US zipcode. But if we can also apply patterns to the key and use those results in combination to validate, invalidate, enhance, etc. the value results, we can perform detection with a much higher level of confidence. We would need this capability to support the majority of our use case.

I like the idea of having keys and values being used together similar to text and recognizer_context_words in enhance_using_context(), but such that keys and values are are more equivalent, i.e. a key may enhance a value, or a value may enhance a key, or there may be no need to consider one of the two. However, I could also see the advantage to having key/value analysis occur at a higher level, like in the analyzer engine, where maybe you could more easily take advantage of overlapping/nested matching relationships, i.e. the value matches a date pattern and key could specify (or not) the type of date.

[omri374]

Thanks @willsthompson! These are great suggestions and feedback. Could you give a few examples of such keys? I'm wondering if the current setup of context words and lookup would be useful, or we would need a different module.

To make sure I understand, does your suggestion fall under point 1 of "Use cases we might want to consider in the future" (1. Automatically detect which columns...) or do you see it as something else? What would you add to point 1 above to support the keys-> values and values -> keys relationship? Maybe adding something about increasing confidence if a key is likely to contain PII based on its name?

[willsthompson]

Hi @omri374! I wasn't sure whether 1-i in the list of future use cases covered my use case, but it's possible it does. What you describe here is what I had in mind:

increasing confidence if a key is likely to contain PII based on its name

Reviewing the code samples, I was thinking that instead of calling analyze() like this:

    for key, value in input_dict.items():
        results = self.analyze(text=value, **kwargs)  ...

I would need something more like:

    for key, value in input_dict.items():
        results = self.analyze(text=value, key=key, **kwargs)  ...

Then we could more accurately handle, e.g.:

{ 'userAge' : 35,
  'userOrders': 35,
  'userZip' : 90210,
  'userID': 90210,
  'plaintextPassword': '2u4LxuMiUd7c'
}

Without the key context available for these values, I don't think it would be possible to detect zipcode, age, or password with enough confidence to be practically useful. Do you see this as something that could be included in your initial release, or would it have to be a followup?

[omri374]

Thanks @willsthompson, this is great. I'm not sure yet at what stage this will be added, but we would make sure we allow proper customization and not to block future improvements in case this gets added in a future iteration.

[willsthompson]

If you can share any thoughts/preferences on how you would want to design this feature after you ship the first iteration of this, it would be super helpful. If we make plans to refactor our current fork to work on a newer Presidio build, we will need to implement this feature anyway, so it would be ideal if we could write something close to PR-ready.

[omri374]

First iteration is likely going to contain the code from option 2 above + tests + sample usage.

[omri374]

Please see an initial sample here: https://github.com/microsoft/presidio/blob/main/docs/samples/python/batch_processing.ipynb

[Onenguyen]

Hi @omri374! Said this on another thread, but I'm pretty new to all this so apologies if this is just a gap in knowledge. I didn't see any mention of how to do this in the sample:

1. Automatically detect which columns need anonymization and which can be ignored by either:

• Run a function on column names/keys which returns which columns/keys require de-identification.
• Allow sampling of data to decide whether to de-identify the entire column or leave it as is.

This is important for my use case because I'm often presented with large tables and I need to quickly identify which columns require attention. It would be nice if there could be an output of column name and identified entities.

[omri374]

Hi @Onenguyen, The points you mentioned are not yet implemented. This is in our roadmap.

[ashishmgofficial]

@omri374 Iam pretty new to Presidio, but have similar use case, where we are presented with files of format csv , json etc and we need to anonymize sensitive data by identifying the PII columns. Is this still in progress, or in roadmap or already supported ?

[omri374]

Hi @ashishmgofficial, this is still on our roadmap, but no ETA. Plan is to add the existing example into the package + support some logic for finding out which keys we should look at using:

1. Key name (e.g. by comparing with the context values in each recognizer)
2. Sample values from each table / list to see if Presidio detects PII in them.

Would that answer your need?

So essentially we're thinking of turning this into a two step process:

1. Find out which columns/keys to focus on
2. Perform analysis / anonymization on these columns only.

The reason for doing it in two phases is that the user would likely want to see which columns are suggested, and update the list if necessary. Second is that it doesn't require any modification to existing Presidio code, it's a wrapper on top of the existing functionality.

[omri374]

Supporting context from external sources

With the latest release of Presidio (2.2.25+), users are able to pass context words from external sources. This could be especially useful in structured/semi-structured settings. For example, the column name (in a data frame / sql table) or the key (in a json/xml file) could be passed as context, and would be compared with each recognizer's context words. For example:

# Imports
from presidio_analyzer import AnalyzerEngine, PatternRecognizer, RecognizerRegistry, Pattern

# Define a simple ZIP regex
regex = r"(\b\d{5}(?:\-\d{4})?\b)" # very weak regex pattern
zipcode_pattern = Pattern(name="zip code (weak)", regex=regex, score=0.01)

# Define the recognizer with the defined pattern and context words that are relevant for zip codes
zipcode_recognizer = PatternRecognizer(supported_entity="US_ZIP_CODE", patterns = [zipcode_pattern], context= ["zip","zipcode"])

# Create recognizer registry and add new recognizer
registry = RecognizerRegistry()
registry.add_recognizer(zipcode_recognizer)

# Create analyzer engine
analyzer = AnalyzerEngine(registry=registry)

# Run with external context
external_context = ["zip"] # e.g. the column name
result = analyzer.analyze(text="My code is 90210", language="en", context=external_context)
print(result)

Result:

[type: US_ZIP_CODE, start: 11, end: 16, score: 0.4]

[RaviThej0803]

Hi @omri374 , We are also trying to solve the same use case w.r.t to excel and csv , can you please point us to an existing example if there was anything published? I tried both BatchAnonymizer and BatchAnalyzerEngine, it wasn't working as expected

[omri374]

Hi @RaviThej0803, could you provide some more information on your use case? And some details on what didn't work as expected with BatchAnonymizer and BatchAnalyzerEngine?

[omri374]

@willsthompson @RaviThej0803 @ashishmgofficial @Onenguyen not sure if this is still relevant, but there's a draft PR (#878) on structured/semi-structured processing. If you'd like to influence how this is implemented, please review it and provide your feedback. Thanks!

[willsthompson]

@omri374 yes, exactly! Thank you!

[omri374]

@willsthompson we have a relatively new method for each recognizer called `enhance_using_context'.
While it's originally there to enhance the score in custom ways, it could also be used to create new results based on context.

For example, one could use it to match the input context with the recognizer's context, and return a result which spans the entire text.
Something like:

from typing import List, Optional
from presidio_analyzer import AnalyzerEngine, EntityRecognizer, RecognizerResult, AnalysisExplanation
from presidio_analyzer.nlp_engine import NlpArtifacts

class ContextBasedRecognizer(EntityRecognizer):
    """Return a result if the input context matches the recognizer's context."""
    
    def enhance_using_context(
        self,
        text: str,
        raw_recognizer_results: List[RecognizerResult],
        other_raw_recognizer_results: List[RecognizerResult],
        nlp_artifacts: NlpArtifacts,
        context: Optional[List[str]] = None,
    ) -> List[RecognizerResult]:
            if not context or not self.context:
                return []

            for context_term in context:
                if context_term in self.context:
                    return [RecognizerResult(entity_type=self.supported_entities[0],
                                            start=0, 
                                            end=len(text), score=1.0,
                                            analysis_explanation=self._build_explanation(score=1.0)
                                          )]
            return []
    
    
    def _build_explanation(
        self,
        score: float
    ) -> AnalysisExplanation:
        explanation = AnalysisExplanation(
            recognizer=self.name,
            original_score=score,
            pattern_name="Context",
            pattern=None,
            validation_result=True,
        )
        return explanation

Running it through the AnalyzerEngine:

zip_context_recognizer = ContextBasedRecognizer(supported_entities = ["ZIP"],
                                                context=["zipcode", "zip", "zip code", "zip_code"])

analyzer = AnalyzerEngine()
analyzer.registry.add_recognizer(zip_context_recognizer)

analyzer.analyze(text="90820",language="en",context=["zip"])

Would return these results:
[type: ZIP, start: 0, end: 5, score: 1.0]

Running it with a different context term:

analyzer.analyze(text="90820",language="en",context=["postal"])

Would not return a result.

Could this be aligned with your requirements? Happy to get your feedback. We could create a generic recognizer containing this logic.

[willsthompson]

I think so, yes. As long as it doesn't require a value match, it should work for us. We're still on a fork of Presidio v1, so I wasn't familiar with this newer method. Thank you for the detailed example!

[omri374]

Worth considering a move to V2 :) The analyzer API hasn't changed much, so even if you keep the V1 infra (genproto etc.) you should be able to update the presidio-analyzer code to V2. Feel free to reach out if you have any questions.

[willsthompson]

We're only using the analyzer, but my fork added several things to support tabular data, so the transition might be a little rocky. But now that it's clear v2 supports all the features we need, I should be able to jettison my customization and use the public v2 package. It will definitely be an improvement for long term maintenance, so I'm hoping we can set aside some time later this year to get it done.

[JGillette71]

Hello,
Want to start by saying Presidio has been a game changer. Stacking PII detection models from scratch would have over extended my small team. It's also exciting to see data protection measures this accessible.

I wanted to check in to see if any developments have been made on future use case #2 "Allow users to define which keys can be useful for detection as additional context, i.e. use an entity in one column (e.g. customer name) as a hint for detection in other columns (e.g. free text entered by the same customer)." I refer to this as "entity integrity" or "entity preservation" throughout anonymization.

I was able to easily write a new recognizer and feed it a customer name to distinguish the customer from other persons within semi-structured data. I'm now struggling to scale. I've tried experimenting with batch analyzer, but I can't seem get anything going. I'm currently working with row-wise apply of analyzer, but again, not a scalable solution.

IMO Presidio's batch processing with a entity preservation feature would be an invaluable tool for ML engineers and data augmentation - providing the ability preserve the semantic context of unstructured data for NLP tasks.

[omri374]

Hi,
Have you looked into Presidio's ad-hoc recognizers? Perhaps it would be useful for entity preservation. There is a sample and a tutorial on creating those, and the extension to batch processing should be feasible.

Please let us know if this is helpful, and if anything specific is missing.

[millefalcon]

Hi, a little late to the party, but I was having a similar use case and thought I'd share my 2 cents.

My use case was mainly around point 3:

3. Dictionaries/json of arbitrary structure: In addition to (1) and (2), provide the ability to parse deeper json/dict structures. A naive approach would be to flatten the dict, run the logic in (1) or (2) and unflatten it back (potentially with something like this). Note that this library does not support the flattening of list of objects, so it would not support all use cases (note: a thorough analysis of options is required).

I was mainly concerned with text in nested json, thanks to theodore's answer, I found one could hook into the stdlib json decoder while it parses the string. One can hook into other type parsing as well similarly. If one were only concerned with object paris, we could probably just use the object_hook and run the batch anonymization already implemented while loading the json. So now we just had to run the anonymization on the parsed text, and we're more or less good to go ':). This may have performance issues since it is using the py_make_scanner instead of the c_make_scanner.

Code:

import json
import sys
from json.decoder import py_scanstring
from json.scanner import py_make_scanner

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

try:
    from _json import scanstring as c_scanstring
except ImportError:
    c_scanstring = None

scanstring = c_scanstring or py_scanstring


class AnonDecoder(json.JSONDecoder):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self._analyzer = AnalyzerEngine()
        self._anonymizer = AnonymizerEngine()

        def parse_str(*args, **kwargs):
            text, end = scanstring(*args, **kwargs)
            anon_text = self._anonimize(text)
            return anon_text, end

        self.parse_string = parse_str
        self.scan_once = py_make_scanner(self)

    def _anonimize(self, text):
        analyzer_results = self._analyzer.analyze(
            text=text,
            language="en",
        )
        anonymized_results = self._anonymizer.anonymize(
            text=text,
            analyzer_results=analyzer_results,
        )
        return anonymized_results.text


data = """{
    "key_a": {"key_a1": "My phone number is 212-121-1424"},
    "key_b": ["www.abc.com"],
    "key_c": 3,
    "names": ["James Bond", "Clark Kent", "Hakeem Olajuwon", "No name here!"],
    "key_d": [
        {
            "key2": "Peter Parker"
        },
        {
            "key3": "My phone number is 414-121-1424"
        }
    ]
}"""


json.dump(json.loads(data, cls=AnonDecoder), fp=sys.stdout, indent=4)

Result:

{
    "key_a": {
        "key_a1": "My phone number is <PHONE_NUMBER>"
    },
    "key_b": [
        "<URL>"
    ],
    "key_c": 3,
    "names": [
        "<PERSON>",
        "<PERSON>",
        "<PERSON>",
        "No name here!"
    ],
    "key_d": [
        {
            "key2": "<PERSON>"
        },
        {
            "key3": "My phone number is <PHONE_NUMBER>"
        }
    ]
}

@omri374 Love to hear you folks thoughts. Thank you for the awesome library ❤️

[omri374]

Thanks @millefalcon! This is a great suggestion. I will try it out. Any ideas on how to incorporate it? could it complement the existing functionality or would it replace it?

[omri374]

This is definitely cleaner than what we have currently. One of the reasons for the existing design, is to leverage spaCy's batch processing, and not handle inputs one by one, but aside from this, I really like your approach!
How would it handle non-string PII? for example a 5 digit zip code? can we tweak it somehow, or will casting to string be necessary?

[millefalcon]

@omri374 I'm not sure this is fit to incorporate into presidio(to be fair, I'm not sure how to incorporate this, since this is part of the data serialization/deserialization part), on the other hand this maybe an Okay fit for some other lib making use of presidio to make it easier to anonymize json/csv.

[millefalcon]

We don't need to cast to str. We could just hook into whichever type parser we need. Please see JSONDecoder src. One can write a similar function like parse_str from the example for parse_int to anonymize integer data.

[omri374]

Presidio now has more in this space: https://microsoft.github.io/presidio/structured/

Note that there are currently two ways to use Presidio with structured data:
Option 1: presidio-structured
Option 2: Batch analysis and anonymization

Main differences are:

• Presidio structured (option 1) scans a structured dataset to identify which columns / keys have PII in them, and then redacts the entire column/values for this key. If 90% of the cells have a PERSON in them, then 100% of the values would be redacted. It redacts the cell completely, so if a cell contains "My name is David" then the entire cell would be removed.
• Batch analysis / anonymization (option 2) scans every cell for PII. If a cell has "My name is David", it would replace it with "My name is ". If PII was detected on 90% of the cells, only those cells with identified PII would be redacted.

Note that in the future we plan to have a more seamless integration of the batch analysis / anonymization into presidio-structured.