Build1 publisher3 min readPublished
A one-line prompt edit replaces retraining in AWS's model-agnostic PII detector
AWS keeps the PII entity list in the instructions and reaches any model through a messages-in, text-out interface, and it claims a span-level benchmark against eight other detectors whose scores the write-up does not show.
The Engineer · Build desk

What happened
- AWS has described a configurable, instruction-driven PII detector that runs on any large language model managed on Amazon Bedrock, with the code shipping as the pii-detector package.
- The entity set that defines what counts as PII lives in the instructions, so extending it to a domain identifier is a one-line edit with no retraining and no redeployment.
- AWS says it scored the detector span-for-span against eight other LLM-based detectors, including OpenAI's PrivacyFilter, on a common ground truth drawn from five public PII corpora.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability Teams that cannot call a managed API get the same detector by writing an adapter, which puts scanning inside a VPC or an air-gapped network without a second implementation.
- constraint Portability is bought by asking almost nothing of the model, so any vendor feature for guaranteeing well-formed output stays outside the interface and the burden lands on your parser.
- decision With no scores in the published material, the part a buyer can evaluate is the architecture, which means anyone adopting this still has to fund an evaluation on their own entity set.
The portability claim rests on one interface. The model is reached through the Inferencer, and any object that takes a list of messages and returns the assistant's text satisfies it [9]. That is a deliberately thin contract. It asks for nothing model-specific, which is why the package can ship an Amazon Bedrock adapter and also take a custom adapter for OSS-GPT 20B served on your own GPU, including in environments that cannot reach Bedrock at all [7][8]. The price of that thinness is that whatever a given vendor offers for making structured output reliable sits outside the contract, in your parsing layer.
That parsing layer is where the interesting failure lives. The model reads the text, labels each span with an entity type from the schema, and returns the spans as JSON; a separate post-processing step then computes exact character offsets and removes duplicates [10]. The model is not asked for indices, only for substrings. AWS does not say how the offset step distinguishes a substring that genuinely occurs twice in a transcript from one the model reported twice, and in a redaction pipeline that is the difference between masking both phone numbers and masking one.
The entity-set argument is the part that holds up on its own. Token-classification taggers fix their PII types at training time, so an employee ID or a crypto-wallet address that a custom fine-tuning corpus introduces falls outside the frozen schema, and adding it means relabeling and retraining [13]. Here the schema is a sentence in the prompt, and changing it takes no retraining and no redeployment [12]. Retraining a tagger to recognise a wallet address is a project; editing a sentence is a Tuesday. The second dial is the model itself, which AWS says sets accuracy, latency and cost together [11], so the quality you want has a per-call price rather than a one-off training bill.
The comparison is where the material thins out. AWS says the detector was evaluated on five public PII corpora across nine LLM-based detectors including OpenAI's PrivacyFilter [2], while the evaluation section describes scoring it span-for-span alongside eight other detectors on a common ground truth [3]. Those reconcile if AWS's own detector is one of the nine, leaving eight external systems [4]. The published material states the method and does not carry the scores [17], so there is nothing yet to check.
Span-for-span scoring on a shared ground truth is the harder test to pass, because boundary disagreement costs you where a document-level flag would not. For a number produced that way to predict anything about your corpus, three things have to hold: your entity set is close to theirs, your span-match rule is the same one, and your text looks like the five public corpora. AWS's own framing argues against the third. The motivating data is customer-support transcripts, HR records, chat logs and long free-text columns arriving in messy multilingual formats that no fixed schema anticipated [14], and the detector is offered as reasoning across eight languages without a translation step [15]. Public benchmark corpora are the tidy end of that.
The stated reason to care is memorisation: a model fine-tuned on uncleaned text can reproduce a real person's details through a prompt that was never meant to surface them [16]. Against that risk, what this release actually hands you is an architecture where the entity list is configuration and the backend is swappable [6], plus sample code you can point at your own columns [5]. The accuracy number is still yours to measure.
What to watch
- Whether AWS publishes per-corpus and per-entity precision and recall against PrivacyFilter, and states the span-match rule used.
- Whether the sample repository grows Inferencer adapters beyond Bedrock, since the air-gapped case depends entirely on custom adapters.
- Whether a domain identifier the base model has never seen named stays a one-line instruction edit or needs examples and a parsing change.