When your AI assistant outputs sensitive-looking data, is it a leak or a fabrication? Not knowing which is the new enterprise security nightmare.
The worst security call I ever sat through wasn’t about a breach. It was about whether we’d had one.
Our internal assistant, a retrieval bot bolted onto wikis, tickets, and a few too many Slack exports, had returned something to a support engineer that looked like a customer’s partial card number and a billing address. She did the right thing and escalated. Then the fun started. Was it real? Did the model retrieve a record it should never have touched? Or did it assemble something plausible out of training data and the shape of the prompt? We spent the next six hours unable to answer that question with confidence, which is a polite way of saying we spent six hours treating a maybe-nothing like a definite everything.
I thought about that call this month reading Ars Technica’s report on Muse, Meta’s internal AI assistant. According to Ars, which obtained internal employee chats, staffers were alarmed by outputs that appeared to show sensitive financial information, reportedly including bank and bitcoin balances. Meta’s position, as reported, is that these were hallucinations: the model fabricating realistic-looking numbers, not surfacing real records. I have no independent view of Muse’s retrieval logs, so I’m taking both the employee concern and Meta’s explanation as reported, not settled.
Here’s the thing the headline-writers are getting backwards. If Meta’s explanation is accurate, that is not the reassuring version of this story. It might be the worse one.
Why “it’s just a hallucination” is not a relief
Walk the two branches.
If the outputs were genuine data, if Muse reached into a system it had no business touching, that’s bad, but it’s a bounded bad. It’s an authorization failure. You find the connector, you find the over-scoped token or the index that ingested a dataset it shouldn’t have, you revoke, you rotate, you notify. Grim, finite, fixable. The industry has run this play a hundred times. We know the runbook.
If the outputs were fabricated, convincingly enough that Meta’s own engineers reportedly couldn’t dismiss them on sight, you’ve lost something harder to get back. You’ve destroyed the signal. Every security triage process I’ve ever built assumes that “this looks like sensitive data” is evidence that sensitive data was involved. A model that manufactures realistic account balances on demand turns that assumption into noise. Now every escalation is Schrödinger’s leak, and you have to open the box by hand, every time.
That’s the trust failure. A clean exfiltration you can detect by its signature. A plausible fabrication wears the signature of a leak while being something else entirely, and it does it at machine scale. You can’t triage your way out of that with human reviewers. You’ve seen the pattern before even if you haven’t seen Muse: the shape is familiar from early RAG systems confidently producing data-shaped text that nobody could quickly tie back to a source.
The three buckets, and why you can’t skip any of them
When your assistant surfaces something that looks sensitive, there are at least three explanations, and the uncomfortable part is that they can look identical from the outside:
- Authorization failure, the model retrieved real data from a source it shouldn’t reach. A boundary problem.
- Retrieval from an unexpected-but-permitted source, the data is real, the access was technically allowed, and nobody realized that connector was in scope. A governance problem wearing an authorization costume.
- Fabrication, the model generated data-shaped text with no underlying record. A trust problem.
Meta’s explanation, as reported, is the third. The employee concern, as reported, leans toward the first. The second, the hybrid, is the one nobody wants to talk about because it’s the most embarrassing: access you granted, through a path you forgot about, pulling data that’s real. I don’t know which applies to Muse. Neither do you. That’s precisely the problem worth fixing, regardless of which branch is true here.
What I changed after that six-hour call
The lesson I took away, the one I’d hand to anyone wiring an LLM into corporate systems, is that you cannot answer “leak or hallucination” at incident time. You have to make the system answer it for you at generation time. That means provenance isn’t a nice-to-have feature for citations. It’s your security control.
Concretely: every factual-looking claim an internal assistant emits should carry a verifiable pointer back to a retrieved chunk, and anything that can’t be grounded gets withheld or flagged, not shown to a user who will then escalate it. The cheap version looks like this:
It’s not elegant and it won’t win a benchmark. But it changes the economics of your next incident. When a sensitive-looking output does escape, you already know whether it had a source behind it. The triage question, the one that cost me six hours, becomes a database lookup: show me the provenance record for this response. No record, no retrieval, no real data touched. A record, and you know exactly which store to go lock down.
Two warnings, because I got both wrong at first.
First, content filters are the wrong tool here, and reaching for them makes things worse. A filter that scrubs anything resembling a bank balance will happily scrub real retrieved balances too, which means you’ve now hidden your actual leaks behind the same curtain as your fabrications. You wanted to raise signal. You lowered it. Provenance separates the two cases; filtering smears them together.
Second, “it looks real” is now worthless as a severity input. Modern models produce flawless-looking IBANs and plausible crypto wallet amounts without breaking a sweat, that’s table stakes, not a tell. If your escalation playbook still treats realism as evidence, rewrite it. Grounding is evidence. Realism is theater.
I don’t know what Muse actually did, and I’d distrust anyone outside Meta who claims they do. But the architecture lesson doesn’t depend on the verdict. Build the system so that the question “was this real?” is already answered before a human ever asks it. Because the old world, where a leak announced itself by looking like one, is gone. Your model can now forge that announcement on demand. Plan for the forgery, not just the theft.
