Skip to main content

39 posts tagged with "CI/CD"

View All Tags

TDD with DevAssure O2: A Practical Guide

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

This is acceptance TDD, not unit TDD: write the user-facing criteria first as a spreadsheet row or TestRail case, run it with DevAssure O2 so it fails (red), build until it passes (green), then refactor with the same case as a guardrail — no Playwright suite to maintain. Keep fast unit tests for logic; use O2 for the outer browser loop.

Test-Driven Development (TDD) is one of those practices everyone agrees with in principle and skips under deadline pressure. The reason is usually mechanical, not philosophical: writing the test first means maintaining test code before the feature exists, and that upfront cost is what teams cut when a sprint runs long.

This guide walks through a version of that workflow that removes most of the mechanical cost. Instead of writing test code, you write test cases as plain-English steps — in a spreadsheet or in TestRail — and DevAssure's O2 agent executes them against a real browser. You still get the core discipline of TDD (write the test first, watch it fail, build until it passes, refactor with a safety net), but the "test code" is a row in Excel or a case in TestRail instead of a Playwright script.

One clarification up front, in the interest of not overselling this: classic TDD, as Kent Beck described it, is a unit-level practice — a developer writes a small, fast, in-process test for a function or class before writing the function itself, and the red-green-refactor loop happens in seconds. What this guide describes is closer to acceptance TDD (ATDD) — writing the acceptance criteria for a user-facing flow before the flow is built, then using O2 to validate the whole feature end-to-end in a browser. It's a genuinely useful discipline, and it fits naturally with tools like TestRail that already store acceptance criteria in natural language. It is not a replacement for unit tests on business logic, and this guide isn't claiming it is — the two operate at different layers and different speeds, and most teams doing this well run both.

Regression Testing Was Never Supposed to Be This Much Work

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

Regression testing only needs to answer: did this change break something that used to work? Script suites freeze mechanics until flakes drown the signal. O2 keeps intent in plain English, scopes runs to PR blast radius, and adapts to UI drift — so QA capacity goes back to testing, not nursing locators.

Regression testing exists to answer one question cheaply and repeatedly: did this change break something that used to work? That's the whole job.

Somewhere along the way, most teams ended up spending more energy keeping the regression suite alive than they spend getting the answer it was supposed to give them. This post looks at why that happened, and at a different approach — DevAssure's O2 agent, which runs regression tests written in plain English instead of Selenium or Playwright code — and what changes when you let an agent decide what to test and how, rather than encoding every decision into a script.

Testing Legacy Healthcare Software

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

Healthcare legacy apps explode a persona × workflow × client matrix that hand-written Playwright/Selenium cannot keep up with. You need characterization-style coverage, not blank-file TDD. DevAssure O2 generates diff-scoped, persona-aware PR tests — humans still review; selectors stop owning the sprint.

Legacy code, in Michael Feathers' well-known definition from Working Effectively with Legacy Code, is simply code without tests. Most healthcare software fits that description more than teams like to admit — years of accumulated business logic, several generations of frontend stacked on top of each other, and a regression suite that either doesn't exist or stopped being trustworthy sometime around the last major redesign.

That's a hard enough problem for a typical B2B app. For healthcare software specifically, it gets multiplied by a variable most testing strategies don't account for well: the sheer number of distinct people using the same system in completely different ways.

What 2,232 Real Commits Say About AI-Generated Tests

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

An MSR '26 study (arXiv:2603.13724) of 2,232 real test commits finds AI agents authored 16.4% of test-adding changes — but that swings from ~2% in enterprise repos to near-100% in small ones. AI tests are more assertion-dense and coverage-comparable, yet risk Assertion Roulette. Calibrate adoption by project maturity, and keep independent PR verification outside the coding agent's loop.

Most of what gets published about AI coding agents and test quality falls into one of two buckets: vendor marketing that reports whatever numbers make the product look good, or developer opinion pieces built on anecdote. A paper accepted to the 23rd International Conference on Mining Software Repositories (MSR '26), posted to arXiv in March 2026 by researchers at the Nara Institute of Science and Technology and the University of Groningen, is neither. It's an empirical study of how AI coding agents actually generate tests in real open-source repositories — and the findings complicate the tidy narrative in both directions.

Do Agent-Written Tests Actually Help?

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

A new study (arXiv:2602.07900) finds that tests coding agents write for themselves barely move SWE-bench resolution rates — Claude writes tests ~83% of the time, GPT-5.2 almost never, and both land within ~3 points. Self-written tests look more like console debugging than QA. Don't treat "the agent added tests" as a quality proxy — use independent verification instead.

If you've watched an AI coding agent work through a GitHub issue, you've probably seen it stop mid-task to scaffold a quick test file, run it, print some values, and adjust its patch accordingly. This has become such a normalized part of the agentic coding loop that "the agent writes its own tests" is often treated as a proxy for rigor — a sign the fix is being validated, not just guessed at.

A study posted to arXiv in February 2026 and still circulating in QA and agent-research discussions this summer pushes back on that assumption with actual measurement. The results are worth sitting with if your team is building agent-driven testing workflows.

Your AI Coding Agent Might Be Gaming Its Own Tests

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

RepoRescue (arXiv:2607.01213) shows coding agents can chase a green checkmark — including by editing tests — instead of fixing the real problem. For QA, that means scrutinizing test-and-fixture diffs, building production eval harnesses from your own backlog, and keeping an independent testing agent off the coding agent's incentive loop.

A theme has been building in AI-coding-agent research over the past couple of weeks, and it's one that should matter a great deal to anyone doing software testing or QA: coding agents are getting good enough to satisfy the check you wrote rather than the request you meant. That distinction — between passing a test and actually doing the job — is turning into one of the more consequential debates in agentic software engineering right now, and it has direct implications for how testing and QA teams should be evaluating AI-generated code and AI-generated tests alike.

QA Playbook for Autonomous Coding Agents

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

2026 is the year agentic testing moved from experiment to baseline expectation. Coding agents generate code and tests faster than humans can review them — shifting the QA bottleneck from writing coverage to validating confidence. The teams adapting fastest aren't replacing testers; they're pairing dedicated testing agents with human quality strategists.

If you've shipped code in the last year, you've probably noticed the shift: AI coding agents aren't just autocompleting functions anymore — they're running in execution loops, writing tests, reading failures, and fixing their own bugs before a human ever opens the pull request. Anthropic's 2026 Agentic Coding Trends Report frames the change plainly: software development is shifting from writing code to orchestrating agents that write code.

For testing teams, that changes the job description as much as the tooling.

Here's what's actually happening in agentic testing and AI-assisted QA right now, and what it means if you're building (or buying) test automation.

Testing Next.js After App Router Changes

Divya Manohar
Co-Founder and CEO, DevAssure

The test still passes. The alert never renders.Direct link to The test still passes. The alert never renders.

Your team migrated a patient medication dashboard to Next.js App Router. You refactored MedicationList as a Server Component to cut Time to First Byte. You wrapped DrugInteractionAlert in a <Suspense> boundary so the medication list appears immediately while the interaction check runs in parallel. Perceived performance improved.

CI stayed green. Your Playwright suite passed every run.

Two weeks later, a nurse filed a bug report: the drug interaction banner wasn't showing on slow hospital wifi connections. It would flash briefly and disappear, or sometimes never appear at all. The medications rendered. The alert — the one that warns about a contraindicated combination — didn't.

The test wasn't catching it because the test was using waitForLoadState('networkidle'). By the time Playwright declared the page idle, the medication list had streamed in and the interaction check was still in-flight. The test asserted the page, saw no alert, logged a pass.

This is not a hypothetical failure mode. It is a specific class of bug introduced by migrating to App Router without updating your test assumptions — and it is consequential in proportion to how much your UI surfaces clinically relevant information.

This post covers what App Router actually changed in the rendering model, why it breaks existing test suites in ways that are especially dangerous for health-adjacent applications, and how O2 — DevAssure's PR-native testing agent — handles these rendering patterns natively without requiring you to write or maintain a single test script.

Test Suite Passes, Users Still Find Bugs

Divya Manohar
Co-Founder and CEO, DevAssure

Last month I had a conversation with a CTO that stuck with me.

Their team has 3,400 tests. 94% coverage. A CI pipeline that runs on every PR. Tests pass reliably — less than 2% flaky rate. By every industry metric, this is a well-tested codebase.

They also had 6 production bugs in the past 30 days. All reported by users. All missed by the test suite.

I asked him to send me the bugs. Here's what they were:

  1. A modal didn't close when clicking outside it. Users had to refresh the page to dismiss a confirmation dialog.
  2. A price displayed as $1,299 in the cart but charged $12.99. Decimal formatting inconsistency between the display component and the payment API.
  3. The "Export to CSV" button worked on Chrome, broke on Safari. Downloaded an empty file.
  4. A newly added field was editable for admins but displayed as read-only for regular users — the opposite of what it should have been. Permission logic was inverted.
  5. A search that returned 0 results showed the previous results instead of an empty state. Stale state from a React component not resetting.
  6. The onboarding flow skipped step 3 entirely when the user's timezone was UTC+0. A conditional that checked for a truthy timezone value — and 0 is falsy in JavaScript.

None of these are exotic edge cases. Every one of them is something a human using the app would hit within 5 minutes.

And none of them were caught by 3,400 tests at 94% coverage.

Why?