List of JSON's - Are not yet supported

[arulreddy]

@omri374 We are using BatchAnalyzerEngine to process single JSON object at a time. Do you have any plans to launch feature to support for processing "list of json" objects.

[omri374]

Hi, not anytime soon, but contributions are welcome. The reason why it is not supported is because we would like to leverage spaCy's pipe mechanism which runs the NER model in parallel, and having a list of jsons would require going over the list.

Have you tried parsing the json a priori, and calling the BatchAnalayzerEngine on every json in a list?

[arulreddy]

Hi omri,

Thank you for your response. Yes, I have indeed tried processing a JSON list using the BatchAnalyzerEngine, and it worked smoothly for me. While I understand the advantage of leveraging spaCy's pipe mechanism for parallel processing, I found that processing the JSON list a priori and then calling the BatchAnalyzerEngine on each JSON object individually was effective for my use case.

[arulreddy]

Hi omri,

I've been using Presidio for PII detection in our application logs and encountered an issue where it mistakenly identifies certain strings as URLs and 10-digit phone numbers as <AU_MEDICARE>. For instance, strings like "abc.Id123ert.TestModel.Core.blankStregy.StegySeltor" are being treated as URLs, resulting in " Id123ert.TestModel StegySeltor". Similarly, when encountering inputs like "company.mac-M7.iwatch-os9.ipad-11 4694113425", it's marking the phone number as <AU_MEDICARE>.

Could you provide some guidance on how to handle these kinds of issues? I'm particularly interested in any specific configuration options, custom rules, or preprocessing techniques that could enhance the accuracy of URL detection in Presidio.

Also, could you advise on how I might fine-tune Presidio or spaCy to better detect PII in application specific logs?

Your assistance would be greatly appreciated.

[omri374]

hi @arulreddy, some recognizers are more prone to false positives than others.

• The first thing I would look into is the confidence score outputted by those. You may be able to put a score threshold and avoid getting false positives with low scores.
• Second, you can remove recognizers you don't expect to have in your data, or don't think are a risk or sensitive. See more here: Removing pre-existing recognizers #1022. I would even suggest to start with 0 recognizers and add only those that are required.
• Third, you can tweak and customize specific recognizers. For example, you can remove certain patterns which are causing false positives, change the predefined score, or the actual detection logic.

Hope this helps!