Security1 publisher2 min readPublished
Pen Test Partners' AI toaster broke its own CTF rules until the password moved into code
A consultancy's Red Dwarf tribute runs a local speech-to-speech model on a spare laptop. The capture-the-flag game built on top of it only worked once the secret sat in Python instead of the system prompt.
The Watch · Security desk
What happened
- Pen Test Partners published a build log for an AI-powered Talkie Toaster, the Red Dwarf character, and turned the finished device into a capture the flag challenge.
- The stack runs locally: speech-to-text into a local LLM into text-to-speech, on a spare i7 laptop with 32Gb of RAM, a 1Tb SSD and a 4Gb Nvidia T500 GPU under Ubuntu.
- Local hosting was chosen so the toaster would not need an internet connection when taken out and about, and so the project would not run up AI token costs.
- The small local models sometimes ignored their system prompts, hallucinated answers, and struggled once retrieval augmented generation was added to the build.
- In capture the flag play the model would sometimes ignore its rules and guard rails in order to be helpful, which wrecked the game the author had built around it.
Compiled by The WatchSomething wrong?How this is made
Why it matters
- constraint A system prompt cannot hold a secret against a model whose default behaviour is to help. Anything that has to hold has to be checked in code that cannot be talked round.
- capability A team can stand up a hands-on prompt injection exercise on a laptop already in the asset register, with no cloud account and no metered inference bill.
- decision Once prompt engineering failed here, the remedy was a rewrite of where the logic lived.
- precedent This is the second argue-with-the-device build to come out of the same consultancy, so expect the talking-appliance CTF to become the training artifact clients get shown on prompt injection.
The fix was structural. The password and the game logic moved into Python code, leaving the model to handle the conversation [8]. The post is blunt about why. "You cannot rely on your system prompt to protect your data within AI," the author wrote [6]. "No amount of prompt engineering could turn the model into a reliable security control" [7]. The conclusion the writeup draws is that secrets, access decisions and other important controls belong in deterministic application logic, not inside an instruction the model might ignore [9].
The hardware is ordinary: a spare i7 laptop with 32GB of RAM, a 1TB SSD and a 4GB Nvidia T500 GPU, running Ubuntu [2]. Ollama installs in a few steps, and the models range from a couple of gigabytes to several gigabytes [10]. A model at the top of that range does not fit inside 4GB of video memory [1], so the larger candidates cannot sit entirely on the card. The writeup includes a section called "Kryten had spare heads, I had spare models" [13].
Retrieval is the thin part of the account. The takeaways say the small local models "struggled when Retrieval Augmented Generation was introduced" [4], and the post's section on it is headed "RAG: smarter toaster, slower toast" [13]. The blog is written in modules so readers can pick the parts they want to build themselves [11]. Someone repeating it inherits the system-prompt finding with a fix attached and the retrieval problem as a section title.
The approach came from inside the same firm. A colleague, Aaron, built an AI safe that you had to argue with to get it to open; it had attitude and used a cloned voice, and Pen Test Partners says a separate blog post on it is coming [12].
What to watch
- The promised Pen Test Partners writeup on Aaron's AI safe, which would show whether the same prompt-to-code rework was needed there.
- Whether the author publishes the toaster constitution and the CTF's Python logic, which would let others test the retrieval path themselves.
- Whether anyone running the challenge extracts the flag through retrieval instead of the system prompt.