Discussion about this post

User's avatar
Swag Valance's avatar

The returns on AI opener is a great reminder that undisciplined product velocity is often worse than no velocity at all. Or, as Zalando's Marcel Toben put it recently, "velocity without judgment is just chaos with a PR merged".

Check out his two-loop model of the AI agent delivery loop versus the product release control loop:

https://experimentation-club.beehiiv.com/p/experimentation-club-letter-3#confidence-is-the-work

Greater velocity backfires if you cannot enable confidence at a similar scale and speed.

When I co-lead a global center of excellence for experimentation, we would present our experimentation program learnings at QBRs (Quarterly Business Reviews) with our executive sponsors. One of the biggest shifts in perspective came when we went beyond our projected revenue improvements as a result of our collective experimentation: we also included projections of avoided losses. I.e., the garbage we prevented from going in before "garbage in, garbage out" took over.

And when the typical product change is a conclusive winner only 10-25% of the time, that's a lot of added code complexity for either no conclusive improvement or, worse, a net harm to the product overall. Scale the product velocity without also scaling the confidence in what you release, guess what inevitably happens?

No posts

Ready for more?