38 posts tagged with "CI/CD"

Regression testing exists to answer one question cheaply and repeatedly: did this change break something that used to work? That's the whole job.

Somewhere along the way, most teams ended up spending more energy keeping the regression suite alive than they spend getting the answer it was supposed to give them. This post looks at why that happened, and at a different approach — DevAssure's O2 agent, which runs regression tests written in plain English instead of Selenium or Playwright code — and what changes when you let an agent decide what to test and how, rather than encoding every decision into a script.

Legacy code, in Michael Feathers' well-known definition from Working Effectively with Legacy Code, is simply code without tests. Most healthcare software fits that description more than teams like to admit — years of accumulated business logic, several generations of frontend stacked on top of each other, and a regression suite that either doesn't exist or stopped being trustworthy sometime around the last major redesign.

That's a hard enough problem for a typical B2B app. For healthcare software specifically, it gets multiplied by a variable most testing strategies don't account for well: the sheer number of distinct people using the same system in completely different ways.

Most of what gets published about AI coding agents and test quality falls into one of two buckets: vendor marketing that reports whatever numbers make the product look good, or developer opinion pieces built on anecdote. A paper accepted to the 23rd International Conference on Mining Software Repositories (MSR '26), posted to arXiv in March 2026 by researchers at the Nara Institute of Science and Technology and the University of Groningen, is neither. It's an empirical study of how AI coding agents actually generate tests in real open-source repositories — and the findings complicate the tidy narrative in both directions.

If you've watched an AI coding agent work through a GitHub issue, you've probably seen it stop mid-task to scaffold a quick test file, run it, print some values, and adjust its patch accordingly. This has become such a normalized part of the agentic coding loop that "the agent writes its own tests" is often treated as a proxy for rigor — a sign the fix is being validated, not just guessed at.

A study posted to arXiv in February 2026 and still circulating in QA and agent-research discussions this summer pushes back on that assumption with actual measurement. The results are worth sitting with if your team is building agent-driven testing workflows.

A theme has been building in AI-coding-agent research over the past couple of weeks, and it's one that should matter a great deal to anyone doing software testing or QA: coding agents are getting good enough to satisfy the check you wrote rather than the request you meant. That distinction — between passing a test and actually doing the job — is turning into one of the more consequential debates in agentic software engineering right now, and it has direct implications for how testing and QA teams should be evaluating AI-generated code and AI-generated tests alike.

If you've shipped code in the last year, you've probably noticed the shift: AI coding agents aren't just autocompleting functions anymore — they're running in execution loops, writing tests, reading failures, and fixing their own bugs before a human ever opens the pull request. Anthropic's 2026 Agentic Coding Trends Report frames the change plainly: software development is shifting from writing code to orchestrating agents that write code.

For testing teams, that changes the job description as much as the tooling.

Here's what's actually happening in agentic testing and AI-assisted QA right now, and what it means if you're building (or buying) test automation.

The test still passes. The alert never renders.Direct link to The test still passes. The alert never renders.

Your team migrated a patient medication dashboard to Next.js App Router. You refactored MedicationList as a Server Component to cut Time to First Byte. You wrapped DrugInteractionAlert in a <Suspense> boundary so the medication list appears immediately while the interaction check runs in parallel. Perceived performance improved.

CI stayed green. Your Playwright suite passed every run.

Two weeks later, a nurse filed a bug report: the drug interaction banner wasn't showing on slow hospital wifi connections. It would flash briefly and disappear, or sometimes never appear at all. The medications rendered. The alert — the one that warns about a contraindicated combination — didn't.

The test wasn't catching it because the test was using waitForLoadState('networkidle'). By the time Playwright declared the page idle, the medication list had streamed in and the interaction check was still in-flight. The test asserted the page, saw no alert, logged a pass.

This is not a hypothetical failure mode. It is a specific class of bug introduced by migrating to App Router without updating your test assumptions — and it is consequential in proportion to how much your UI surfaces clinically relevant information.

This post covers what App Router actually changed in the rendering model, why it breaks existing test suites in ways that are especially dangerous for health-adjacent applications, and how O2 — DevAssure's PR-native testing agent — handles these rendering patterns natively without requiring you to write or maintain a single test script.

Last month I had a conversation with a CTO that stuck with me.

Their team has 3,400 tests. 94% coverage. A CI pipeline that runs on every PR. Tests pass reliably — less than 2% flaky rate. By every industry metric, this is a well-tested codebase.

They also had 6 production bugs in the past 30 days. All reported by users. All missed by the test suite.

I asked him to send me the bugs. Here's what they were:

1. A modal didn't close when clicking outside it. Users had to refresh the page to dismiss a confirmation dialog.
2. A price displayed as $1,299 in the cart but charged $12.99. Decimal formatting inconsistency between the display component and the payment API.
3. The "Export to CSV" button worked on Chrome, broke on Safari. Downloaded an empty file.
4. A newly added field was editable for admins but displayed as read-only for regular users — the opposite of what it should have been. Permission logic was inverted.
5. A search that returned 0 results showed the previous results instead of an empty state. Stale state from a React component not resetting.
6. The onboarding flow skipped step 3 entirely when the user's timezone was UTC+0. A conditional that checked for a truthy timezone value — and 0 is falsy in JavaScript.

None of these are exotic edge cases. Every one of them is something a human using the app would hit within 5 minutes.

And none of them were caught by 3,400 tests at 94% coverage.

Why?

At Microsoft Build 2026, Microsoft announced something that quietly confirms the core thesis behind DevAssure: as AI agents take over more of the software development lifecycle, the agent that writes the code cannot be the agent that grades the code.

The announcement was a pair of open-source projects — ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing) and the Agent Control Specification (ACS) — designed to give developers a portable, framework-agnostic way to evaluate and govern AI agents before their behavior ships to production. Coming from the company now positioning itself as the "agent-first" platform for enterprise development, this is a meaningful signal about where the industry is heading.

I want to walk through what Microsoft actually shipped, why it matters beyond agent safety, and what it means for teams where 30–40% of code is already AI-generated — because the validation gap Microsoft just named at the agent layer is the same gap most engineering teams still have at the application layer.