101 posts tagged with "Test Automation"

Google I/O 2026 made one thing unmistakably clear: the era of AI-assisted coding is over. The era of AI-agentic coding has begun.

The keynote opened with a line I have been thinking about since:

"We've transitioned from AI that simply assists you, to agents that can independently navigate complex tasks across your entire workflow."

What followed was a two-hour parade of agent-first announcements: Antigravity 2.0, the Antigravity CLI and SDK, Managed Agents in the Gemini API, Gemini 3.5 Flash, WebMCP, Chrome DevTools for agents, an Android migration agent that converts entire React Native apps to Kotlin, and vibe-coding Android apps directly in Google AI Studio.

Most commentary since has focused on productivity. How fast can you ship? How many agents can you orchestrate in parallel? How much code can Gemini 3.5 Flash produce per minute?

I want to focus on a different question — one Google did not spend much time on during the keynote:

Who tests what the agents produce?

Last week I watched a developer build an entire payment integration in 35 minutes using Cursor.

User authentication. Stripe checkout. Webhook handling. Invoice generation. All wired up and functional.

In 2023, that is a week-long sprint. In 2026, it is a Tuesday morning before standup.

Then we ran DevAssure's O2 Agent on the PR.

Focus on the merge gate? Read the companion: Why your vibe-coded PR keeps breaking production — the handoff from coding agent to CI, not the quality-gap theory.

For a decade, the testing industry rallied behind a simple mantra: shift left.

Find bugs earlier. Test sooner. Put quality in the hands of developers.

The theory was sound. A bug caught in development costs roughly 10× less than one found in production. Move testing to the left of the timeline, and you save money, ship faster, and improve quality.

But here is what actually happened:

Playwright is one of the best browser automation frameworks ever built. It is fast, reliable, developer-friendly, and powerful. If your team knows exactly what to test, how the UI is structured, which locators are stable, and who will maintain the tests forever, Playwright is an excellent choice.

But that is also the problem.

Most product teams do not struggle because Playwright is weak. They struggle because writing and maintaining browser tests is still manual engineering work:

• Every test needs scripts.
• Every script needs locators.
• Every locator can break.
• Every UI change needs test maintenance.
• Every failure needs someone to investigate whether the app is broken or the test is broken.

At some point, the test suite becomes another codebase to maintain.

Agent-based testing changes this model. Instead of writing scripts that tell the browser exactly what to click, type, and assert, you give the agent an objective. The agent understands the application, interacts with the UI, adapts to changes, validates behavior, and reports issues.

That is why developers should start moving away from writing Playwright tests manually.

Every team that adopted vibe coding eventually hits the same wall: code ships faster, but validation does not keep up. This post is the implementation manual for closing that gap on every pull request.

Cursor changed the default for how features get built. What has not changed is that untested code still breaks production — it just gets there faster.

For twenty years, automated testing meant writing more code. That era is ending — and most teams haven't noticed yet.

The first automated test I ever wrote was in Playwright. It was 2012. It launched a browser, filled in a login form, and checked that the dashboard loaded. It passed. I felt like a wizard.

More than a decade later, the fundamental contract hasn't changed. To test software, you write more software. You describe, in code, what your code is supposed to do. Then you maintain that second codebase forever.

We've built entire careers, conferences, certifications, and consultancies on this premise. Selenium. Cypress. Playwright. Test pyramids. BDD. Page object models. The whole apparatus rests on a single assumption: humans must specify, in writing, what to test.

That assumption is quietly dying.

A CTO told me last month, very pleased with himself:

"We're already paying $200/month per dev for Claude. Testing is basically free now — we just ask Claude to also write the tests."

I asked him to pull up his Anthropic bill. The number was 14x what he'd budgeted at the start of the quarter. And his team still hadn't shipped the regression suite.

This is the most expensive trap in the AI tooling stack right now, and it's expensive precisely because it looks free. If you've already bought a coding agent, asking it to do double duty as a testing agent feels like the obvious move. One subscription, one workflow, one bill.

Except there is no "one bill." There are six.

Last week, on a customer call, a CTO asked me the question I now get every single week:

"I'm already using Claude to write my code. Why can't I just point the same agent at the code and have it test itself?"

It's a fair question. If one AI can write a React component, surely it can write the test for that component too. The economics look seductive — one tool, one workflow, one bill.

But here's an insight:

Testing your own PR is like proofreading your own essay. You'll read it 10 times. You'll miss the same typo 10 times. Because your brain autocorrects what it wrote.

That insight is what this blog is about. And it explains why a coding agent, no matter how capable, is structurally the wrong tool to verify its own work.