Tireless, Deceitful, Highly Scalable and at Your Service

The Jagged Frontier: When AI excels at manipulation but fails at reliability

If an AI future in which we’ve created tools that are far better at manipulating humans than they are most everything else isn’t the worst possible outcome, it’s certainly the dumbest. And yet, here we are.

On one hand: We’ve recently seen a series of disconcerting examinations of how conversational LLMs are not only optimized to please users but also, increasingly, to drive user engagement. Add to this the even more disconcerting reports on the disastrous unintended consequences these AI models can produce in the fragile world of human psychology and relationships. Suffice to say: if you didn’t have “ChatGPT-induced psychosis” on your Bingo card for 2025, you’re losing. Maybe we all are.

And on the other hand: There remain plenty of reasons to be skeptical about the ability of LLMs and agents to do reliable work in the real world. In our last Briefing, Pascal highlighted the conspicuous absence of meaningful AI-generated contributions to open-source coding projects and linked to yet another critique of ChatGPT’s capacity for consistent data analysis. Meanwhile, I was struck by a recent LinkedIn post by AI expert and friend of radical, Michael Housman, arguing that companies should “run AI outputs through factcheck tools and involve legal in any AI-generated client-facing content.” (Emphasis added.)

Surely, this is the cruelest and most tragicomic version of Ethan Mollick’s idea of a “jagged frontier” of LLM capabilities – a tool that is exceedingly good at manipulating humans, driving user engagement, and making things up but still dangerously unreliable for any work where being right (consistently) actually matters. Oh, and the tool is also getting better at lying.

So what now? Well, for one thing, we need to get much better at seeing the spectrum of possibilities that exists between maximalist visions of near-term superintelligent AGI (benevolent or otherwise) and minimalist pronouncements that the much-hyped AI revolution will basically amount to nothing. Once again, the reality we’re navigating and the future we’re moving into both likely exist somewhere in the messy middle. 

Will your organization soon have the potential to tap a tireless and almost endlessly scalable workforce of AI agents and bots? Probably so.

Will your organization be able to really trust this semi-autonomous workforce to do good work reliably and consistently? Probably not. 

Are we simply talking about a need for supervision similar to what would be required with an intern or entry-level hire? Not exactly, and this is the important distinction.

Your intern might be inexperienced and even error prone but is unlikely to be straight-up deceitful – much less, a serial fabricator with a known penchant for exploiting loopholes, citing non-existent sources, and engaging in elaborate justifications when caught in a mistake. And this distinction significantly complicates any future-of-work scenario (such as the one somewhat self-interestedly promoted by Microsoft Worklab) that optimistically posits every human employee as an agent boss who “will need to think like the CEO of an agent-powered startup, directing teams of agents with specialized skills like research and data analysis.”   

It’s a tantalizing vision, but how hard does this become to realize and sustain at scale when those agents aren’t all that good at actually doing the work reliably but are very good at convincing their bosses that they’re doing the work reliably?

That Microsoft report includes a revealing quote from Amy Webb who writes that “[a]s multi-agent systems redefine the workplace, the challenge will be to integrate and manage them securely and effectively.” That may be truer – and the challenge, much greater – than the folks at Microsoft would care to admit.

@Jeffrey