AnonymFiles detection benchmark and offline test: method and results

The results at a glance

• 364 out of 366 items masked, that is 99% recall, on 40 fictitious firm documents.
• On real formats: 100% in Word, 97% in email, 98% in text PDFs, 100% in scanned PDFs (OCR) and spreadsheets.
• In light detection (switched on by default on computers with 8 GB of memory or less, about 1 GB less memory): 360 out of 366, that is 98%.
• 1 term to keep masked by mistake out of the 55 in the text documents (“URSSAF Pays de la Loire”), and two in the formats: a public body and a court, masked out of caution.
• Measured on October 14, 2026, with the application's real engine (two local models and rules), network disabled.

The benchmark method

The set has forty fictitious documents in French, written like those of the seven professions AnonymFiles serves: written submissions, writ of summons, judgment, sale agreement, deed of gift, engagement letter, minutes of a general meeting, payroll export, bailiff's report, service of process, insolvency report, list of creditors, draft patent, contract, non-disclosure agreement, emails. Every person, company, address, contact detail and number is invented. The set includes deliberate traps: namesakes, names in tables, names in capitals, companies named by their legal form, dates of birth next to dates of statutes, SIREN and social security numbers.

For each document, a list sets two things:

• the items that must disappear from the redacted copy (366 in total), classified by type: person, organization, address, email, phone, IBAN, private date, identifier;
• the terms that must stay readable (55 in total): references to statutes, courts, legal concepts.

Forty-five other elements (amounts, docket numbers, land registry references, vehicle registrations) are masked by hand during review: they are not counted. Part of the documents is then produced as Word, text PDF, scanned PDF, email and spreadsheet, and prepared by the application's full pipeline, extraction and OCR included. An item counts as found if its value no longer appears, as a whole word, in the redacted copy; a term to keep counts as an error if it has disappeared. The set, the expected answers and the scripts are published with the code: the benchmark can be re-run identically.

Results by type of information

Results by file format

The same documents, prepared in their real format by the application. For scans, an item that OCR does not read exactly (lost accent, confused letter) is counted as “not read” and is left to check during review.

On a PDF with tight line spacing, a redacted area no longer erases words on the neighbouring line: the defect, found with this set, is fixed (on a scanned page, pixel erasure still covers the full height of the words).

What this benchmark does not tell you

• The set is fictitious. Forty documents written for the purpose do not cover the full variety of a real case file: rare names, typos, abbreviations, documents in a foreign language, poor-quality scans.
• Some misses remain. The six missed items are organizations without a legal form (“Clinique du Parc”, “Nord Habitat Expertise”) and surnames alone in a file heading or an email subject. These are exactly what to watch for during review.
• It is not an audit. The benchmark is run by the publisher, on its own set. Benchmarks published by other publishers use other test sets: the figures cannot be compared.
• Amounts and sensitive passages are not counted. They are masked by hand, in one click, during review.

Hence the application's rule: every document is shown side by side with its original, and nothing is exported without approval. The benchmark is repeated at every model or detection-rule change.

What the application does on the network

Here, according to the application's code, is everything that touches the network:

The 127.0.0.1 local bridge is not the internet

The 127.0.0.1 address refers to the computer itself: a connection to it never leaves the machine. AnonymFiles opens a small service there, on a randomly chosen port and protected by a secret token, that only AI agents installed on the same computer (Claude Code, Codex…) can call to read the redacted and approved copies, never the originals. A firewall or monitoring tool may display this address: that is normal, nothing goes outside.

The details are on the security and network scope page.

Checking offline operation yourself

On a Mac

1. Turn off Wi-Fi (and unplug the network cable) right after installation, before the first launch.
2. Process a test case file: detection, review, export. Everything works, models included: they ship with the application.
3. For finer proof, install an outbound firewall: LuLu, free and open source, or Little Snitch. It shows every attempt to connect to the internet, application by application. Only connections to 127.0.0.1 can appear, those of the local bridge.
4. Without installing anything, Activity Monitor (“Network” tab) shows the bytes sent and received by each process.

On Windows, once the Windows version is released

1. Let the first processing run download the public models: it is the only planned connection, with no document text.
2. Then block the application in Windows Defender Firewall, or turn off the network: processing continues.
3. To observe, Resource Monitor (“Network” tab) lists the processes that communicate, and the netstat -b command, in a command prompt run as administrator, shows their connections.

And on the exported file

Open an exported PDF, select all the text and paste it into an editor: the masked text is no longer there, it has been deleted, not covered. The six checks to run on any redacted PDF are in our guide to checking that a PDF is really anonymized.