Denegra Derbent · Measurement
How much personal data
does Derbent catch?
We gave Derbent and a widely used open-source tool the same 500 Turkish documents. Derbent masked 98.0% of the personal values in them; the open-source tool, as it comes, masked 72.6%.
Lab measurement, not live traffic. Measured on 15 October 2026 on made-up test documents, with the Derbent version of that day. The rules have changed since, and this set has not been measured with the current rules yet. We have no measurement on real customer traffic yet; a pilot measures that on your own prompts.
The result
Derbent next to Presidio, the open-source masking tool that LiteLLM’s guardrail uses, with its default settings.
-
Values masked
98.0%
2476 of 2526 personal values. Presidio: 72.6%.
-
Documents fully masked
364 of 400
Every value in the document masked. Presidio: 71 of 400.
-
False alarms
11 of 100
Clean documents where something was masked anyway. Presidio: 100 of 100.
-
Precision
95.2%
Of the pieces Derbent masked, the share that hit personal data or a company name. Presidio: 45.0%.
By type of data
Values masked out of the values in the set. Presidio is shown with no score threshold (its default in LiteLLM) and with a threshold of 0.5, as an administrator might set it to cut noise.
| 500 documents | Derbent | Presidio, default | Presidio, threshold 0.5 |
|---|---|---|---|
| All values masked | 2476/2526 (98.0%) | 1835/2526 (72.6%) | 1252/2526 (49.6%) |
| Documents with every value masked | 364/400 | 71/400 | 19/400 |
| Clean documents with a false alarm | 11/100 | 100/100 | 100/100 |
| Precision | 95.2% | 45.0% | 38.3% |
| Names | 1166/1183 | 800/1183 | 800/1183 |
| Addresses | 149/151 | 0/151 | 0/151 |
| Phone numbers | 368/369 | 338/369 | 25/369 |
| National ID (TCKN) | 164/164 | 164/164 | 13/164 |
| Tax ID (VKN) | 46/49 | 49/49 | 3/49 |
| Company register (MERSİS) | 30/30 | 30/30 | 2/30 |
| Social security (SGK) | 22/23 | 1/23 | 1/23 |
| IBAN | 68/68 | 68/68 | 68/68 |
| Card numbers | 46/46 | 46/46 | 46/46 |
| 204/204 | 204/204 | 204/204 | |
| Licence plates | 68/78 | 1/78 | 1/78 |
| Passports | 39/42 | 33/42 | 6/42 |
| Driving licences | 23/32 | 18/32 | 0/32 |
| Dates of birth | 83/87 | 83/87 | 83/87 |
What we tested on
500 Turkish documents of 16 kinds: online-shop complaints, car rental contracts, dental patient records, bank account-opening forms, customs letters, job applications and more. 400 contain personal data and 100 are clean, so we can count false alarms too.
Every value is made up. Identity, tax and IBAN numbers have valid check digits but belong to nobody.
How we counted
Both tools were scored with the same rules:
- A value counts as masked only if every character of every place it appears is masked.
- A clean document counts as a false alarm if anything in it is masked. The model would then get a damaged question.
- Company names are labelled, but neither counted nor held against a tool.
- Precision counts the masked pieces that touch a labelled value. Company names are labelled, so masking one counts as correct here, for both tools.
Why Presidio
Presidio is the open-source tool that LiteLLM’s guardrail turns on, so “mask personal data before the model” is one setting away for anyone. We ran it as it comes: its built-in recognizers and spaCy’s English model, with no Turkish model and no custom rules. It can be extended; these numbers show it untuned.
Limits
- Lab, not live. All documents were written for testing. Real traffic is messier; we measure it only in a pilot, on the customer’s own prompts.
- Our own set. We wrote it for this project. It was hidden from Derbent’s rules when it was written and they were never tuned on it, but we measured it again after several rounds of rule changes from 12 October 2026 on. The set is not published yet. Read it as our own measurement, not an independent audit.
- Labels not hand-checked. The labels were written with the documents and no person has reviewed every one. A wrong label counts against both tools alike.
- One language. This set is Turkish. English results are below.
Our other test sets
We keep four more sets private and tune the rules on other data (one exception is noted below). Derbent alone, measured on 15 October 2026, the English and chat-prompt rows on 16 October 2026, before the latest rule changes.
| Set | Values masked | Clean items with a false alarm |
|---|---|---|
| 500 Turkish documents of 16 other kinds | 4450/4561 (97.6%) | 36/100 |
| 80 handwritten Turkish documents | 574/591 | 1/16 |
| 500 English documents | 4174/4522 (92.3%) | 16/100 |
| 300 chat prompts, half Turkish | 1829/1917 | 22/100 |