Measurements

What detection finds, measured and published.

We measure the real engine, network off, on fictional files built like real law-firm case files. An item counts as found if it no longer appears in the copy. Here are the results, the method and the limits.

What are the results?

By type, on law-firm documents: people 166/166, organizations 58/58, addresses 72/72, e-mails 21/21, phones 21/21, IBANs 10/10, dates 14/14, identifiers 12/12. The same person never gets two different labels.

What about a public dataset?

So that a third party can rerun the measurement, we ran the engine on the first 2,000 French texts of Ai4Privacy's public OpenPII 1.5M dataset (validation split, texts from France, Switzerland and Canada), licensed CC BY 4.0. With full detection, 97.8% of the values the product claims to mask disappear: people 98.6%, addresses 93.6%, e-mails 100%, phones 99.7%, identifiers 98.7%, and 97.2% of dates.

With light detection, the result drops to 51.9%: its rules recognize French formats, while this synthetic dataset mostly uses Swiss, Canadian or made-up formats (only 26 of 770 phone numbers are in French format, and all 26 are found). Names (91%) and e-mails (100%) hold. The dataset contains no IBANs or organizations, and its short texts do not look like law-firm documents. Method, label mapping, sample and raw results: on request at contact@anonymfiles.com.

Data: OpenPII 1.5M, Ai4Privacy / Ai Suisse SA, CC BY 4.0. Ai4Privacy is not associated with this measurement.

How does it compare with Microsoft Presidio?

On the same corpus with the same counting rule, Presidio (French configuration, medium spaCy model) finds 85% of items in its best configuration, and recognizes neither French dates nor French identifiers. Method, versions and results by type: Presidio on French documents, measured.

How is it measured?

• The app's engine, with its local models and rules, network off.
• A fully fictional corpus: judgments, briefs, deeds, minutes, payroll exports, e-mails and letters, written like real documents.
• An item is found if it no longer appears in the redacted copy. Amounts, docket numbers and land references (45 items) are masked by hand and not counted.
• The corpus also contains 57 terms that must stay readable (courts, public bodies, legal terms): a masked term counts as an error.

What are the limits?

• It is our corpus, built by us: it replaces neither an independent audit nor a test on your own files.
• No automatic detection is perfect: each document is reviewed side by side before export, and nothing leaves without approval.
• The corpus and measurement scripts are available on request at contact@anonymfiles.com.

Further reading: the detection benchmark and offline test · re-identification risk.