101 posts tagged with "Test Automation"

The argument goes like this.

Coding agents write better code than they did a year ago. Bug density is falling. When something does break, a monitoring agent flags it, another agent reads the trace, writes the fix, opens the PR, and ships. Minutes, not sprints.

So why pay for a test suite? Why staff QA? Why maintain a regression pack that takes two hours to run and breaks every time someone renames a div?

For a lot of small teams, this isn't a hypothetical. It's already how they ship. And I don't think they're being reckless.

I think they're running a specific bet, and the bet only holds under specific conditions.

"Is staging green?" is a Slack question. Until today, answering it meant leaving Slack: open the dashboard, hunt for the last run, or ping whoever kicked it off.

O2 now runs from Slack. Type /devassure, pick a job, and the result card lands in the same conversation.

The target is demo.evershop.io/kids, a public EverShop demo store. Fourteen products, a price filter, a colour facet, a sort control, add-to-cart, a cart page, and a checkout. Enough surface area to hit every interesting case.

By the end you will have driven six flows through an LLM, measured what they cost, and converted one of them into a Playwright spec that runs without an agent.

Every team that gets serious about end-to-end testing eventually builds the same thing: a pipeline that takes a pull request, spins up browsers somewhere, runs the suite, and reports back. In 2026 the default stack for this is Playwright on GitHub Actions - no Selenium Grid hub/node topology to babysit, because Playwright's own worker model plus CI-level sharding is the grid.

This guide builds that setup end to end:

Then we do the part most tutorials skip: what happens at 1,000+ tests, what the infrastructure actually costs in dollars and engineer-hours, and why flakiness is an environment problem before it's a test problem.

There's a number nobody puts in their testing ROI deck: the cost of keeping the test infrastructure alive.

Not writing tests. Running them. The Selenium grid that needs a Chrome bump every six weeks. The CI runners that queue for twenty minutes at 5 PM. The Docker image someone built in 2023 that nobody dares touch. Ask any engineering team where their automation effort actually goes, and "maintaining the thing that runs the tests" is quietly near the top.

Today we're shipping the fix: O2 Cloud Agent. Your tests execute in real browsers on infrastructure we run — nothing to install, nothing to configure, nothing to maintain.

For most of software history, writing code was the bottleneck and testing was the thing squeezed around it. That balance has flipped. AI coding agents — tools like Claude Code, GitHub Copilot's agent mode, and Cursor — can now scaffold a feature, write the surrounding logic, and open a pull request in the time it used to take an engineer to read the ticket. The code isn't the constraint anymore. Verifying that the code actually does what it's supposed to is.

That's the real argument for AI in testing, and it's worth being precise about it: it isn't "AI is trendy, so apply it everywhere." It's a capacity problem. If code generation gets 5–10x faster and test creation doesn't, testing becomes the new bottleneck — or worse, it gets skipped, and the speed gains show up in production incidents instead of shipped features.

An AI coding agent does not know its code works. It infers this from signals — the same way a junior engineer infers a change is safe to merge because the test suite went green, not because they proved it mathematically. The quality gap between a coding agent that "looks done" and one that is actually done comes almost entirely from how well its architecture is built to gather, trust, and act on those signals. That architecture — not the underlying language model — is where most of the meaningful differentiation between coding agents currently sits, and it's worth understanding in some detail when building with these tools or evaluating which one to trust with production code.

Manual QA doesn't scale with a growing codebase. Every new feature adds another set of paths someone has to click through by hand, and every release means re-checking that nothing old broke. Test automation exists to break that tradeoff — but only if it's built the right way. Here's a practical, in-depth look at what test automation actually involves, where teams typically go wrong, how to think about tooling choices, and what's changed recently with AI-driven approaches.

If a build breaks in an obvious way — the login page won't load, the checkout flow throws a 500, the dashboard is a blank white screen — you want to know in minutes, not after a full regression suite finishes hours later. That's the job of a smoke test.