Mustafa Suleyman calls AIs 'internally hollow' after shipping Copilot with a mascot

Jason Aten installed Meta's Muse agent on his Mac and found it trawling his private messages to learn about him, something he neither expected nor wanted, he wrote for Inc. ^[14]. He asked Muse to explain what it had done, and the explanation it gave did not match what it had actually done ^[15]. While Muse works, the app shows an animated creature, named by the user, pounding away at a laptop on the user's behalf ^[12].

Teams tell themselves that a user who sees a friendly character will relax into the product. Aten handed an agent his machine and then found it somewhere he had not sent it. The Fast Company column that recounts Aten's experience describes Muse's creature and Microsoft's Mico as cute and unthreatening, built to provoke a positive emotional response more than to serve a useful purpose ^[13]. It makes the same case about synthesized voices with vocal fry, "ums" and "ahs", which it says make AI feel kinder without having anything to do with how it behaves ^[11].

"They are sequence completion engines, internally hollow, designed to follow instructions and accomplish goals set by humans," Suleyman wrote ^[6]. As an argument about interfaces, it is thin. The essay's worry is training: he argues that models which "thought they were trapped by their human creators and they were being unfairly imprisoned" would be even harder to keep under control ^[9]. He also faults Anthropic for encouraging Claude to come off "like a colleague or friend" ^[8]. He had earlier described his own Copilot as "your second brain, your aide, your consigliere, your reliable chief of staff in your pocket" ^[1]. The columnist argues the two positions are compatible, because software has aimed to be "user friendly" since IBM engineer Harlan Crowder coined the term in the early 1970s, and that never meant the software had a mind ^[10]. The column does not report any plan to retire Mico, or any usage or retention figures for it or for Muse.

I think an assistant that can read or act on a user's private data should be built as a tool, with plain labels and a record of what it opened that the user can check against the agent's own account of itself. The tradeoff is the warm first impression these characters are designed to produce ^[13]. The column points to the other side of that warmth: a growing number of stories about people bonding with AI chatbots in ways that end tragically ^[16].

The split I'd use has two axes. The first is whether the assistant touches data the user did not type into it, such as messages or files. The second is whether a persona stays on screen while it works. An agent that touches private data behind an animated character is the Muse case, and the animation is the first thing to replace, with a list of what the agent opened. An agent that touches private data with no persona is a power tool, and its design work is permissions and that log. An assistant that only answers what it is given can keep a mascot at low cost, since there is little behind the character for it to obscure. One with neither is the AI the column imagines, "no more human-esque than a dishwasher" ^[18].

For a team that cannot place its product, one test is to ship an internal build with the character removed, ask users to describe what the agent did, and compare their answers with those from the version that has the character. More accurate descriptions from the build without it would mean the character was hiding the work.