Skip to main content

Self-Healing Test Automation Treats the Wrong Disease

Divya Manohar
Co-Founder and CEO, DevAssure

TL;DR

Self-healing test automation patches a broken locator and keeps the run green. We built it, then removed it. A heal cannot tell a moved button from a broken feature, and at the churn rate of AI-written UI it becomes a tax. Write the test as user intent. There is nothing left to heal.

We shipped self-healing test automation. Then we took it out.

Not because it didn't work. It worked often enough to look good in a demo. It failed in the one place that matters: telling you whether the red test was a moved button or a broken feature.

This post is the developer-side view of self-healing test automation, from a team that shipped it, paid for it, and replaced it.

A stitched locator chip marked HEALED, beside a dead cracked checkout control with its wires pulled

What self-healing test automation actually doesDirect link to What self-healing test automation actually does

A scripted UI test finds elements by locator: an ID, XPath, CSS selector, name, or class. When the UI changes, the locator stops matching and the test fails with ElementNotFound.

Self-healing intercepts that failure. It looks at nearby text, DOM position, visual cues, and past runs, guesses the closest matching element, swaps the locator, and keeps the test running.

That's the whole idea: keep a locator-based suite alive through UI churn without someone rewriting selectors by hand.

It's a reasonable fix for a real problem. But it's a fix for a symptom.

Why we stopped healing locatorsDirect link to Why we stopped healing locators

When DevAssure started in 2024, we were a no-code automation platform for QA teams. Customers' UIs started changing faster as they adopted coding agents, and locator maintenance became the top complaint. So we built healing.

Four things went wrong.

1. Every heal is an inference call. Healing that uses an LLM or vision model to re-match elements costs tokens on every broken step, every run. The bill scales with UI churn, which is exactly the thing that's increasing. We couldn't optimize it down to something predictable.

2. Healing introduced its own flakiness. A heal is a guess with a confidence score. Low-confidence guesses that pass on one run and pick a different element on the next are, by definition, flaky tests. We had added a non-deterministic layer on top of a deterministic one.

3. It created a new triage job. Every ElementNotFound now had two possible meanings: the locator drifted, or the feature is broken. Someone had to decide which, for every heal. That's the maintenance work healing was supposed to remove, in a different shape.

4. It didn't change the underlying model. The test was still coupled to markup. Healing moved the breakage from "fails loudly" to "gets patched quietly." The script was still the thing you had to keep alive.

The over-healing problem: green tests on broken featuresDirect link to The over-healing problem: green tests on broken features

This is the failure mode that should worry engineering leads most.

Say a PR breaks the checkout Submit button's handler, and in the same change an AI agent restructures the form. The old locator no longer matches. Healing finds the closest candidate: another button in roughly the same spot with similar text. It clicks it. The page responds. The assertion that follows is loose enough to pass.

You now have a green test on a broken checkout.

A 0.71-confidence heal locks onto a lookalike Place order control, the run is marked PASS, and the order confirmation stays empty

Vendors know this risk exists, which is why serious self-healing tools add confidence thresholds, human review queues, audit logs, and separate agents to classify failures. All of that is sensible. It's also an admission: healing can't tell drift from defects on its own, so you need more machinery and more humans to supervise it.

If your testing layer needs a second layer to check whether the first one lied, it's worth asking whether the first layer is the right abstraction. A suite that passes while users find the bug is the outcome this produces.

AI coding agents made the locator problem unwinnableDirect link to AI coding agents made the locator problem unwinnable

Locator maintenance used to be a release-cadence problem. A redesign every quarter, some broken selectors, a sprint of cleanup.

With Cursor, Claude Code, Copilot, and similar agents writing a growing share of frontend code, markup changes on far more PRs. Agents don't preserve your data-testid conventions unless told to. They rename classes, re-nest components, and swap libraries as a side effect of an unrelated change.

At that churn rate, healing isn't an occasional safety net. It's running on most PRs, burning inference on every one, and asking a human to approve its guesses. That doesn't scale with the speed agents ship code.

The deeper problem is position, not just volume. A coding agent testing its own output shares the assumptions that produced the change. Healing sits on the same side of that line: it tries to keep the old script attached to the new markup, instead of asking whether the user flow still does what it should.

What to do instead: remove the locatorDirect link to What to do instead: remove the locator

The alternative isn't better healing. It's tests that never referenced the DOM structure in the first place.

Write the test as user intent, in plain English, and let an agent resolve each step against the live page at run time. This is the schema O2 actually runs. Each step is one action or one assertion. There is no selector, no XPath, and no CSS in the file.

# .devassure/tests/guest-checkout.yaml
summary: Guest checkout with saved address
steps:
- Open the product page for "Trail Runner 2"
- Select "9" as the size
- Click Add to cart
- Navigate to checkout
- Continue as a guest
- Select the saved address "Home"
- Click Place order
- Verify that an order confirmation with an order number is shown
- Verify that the cart is empty
priority: P0
tags:
- checkout

O2 does not parse a keyword DSL. Those lines are the forms from the YAML step reference: one intent per line, credentials and the base URL kept out of the test, and at least one Verify so the case has a pass/fail signal.

There's no locator to break, so there's nothing to heal. When the markup changes, the test doesn't care. When the behavior changes, the test fails, which is what you want.

Broken selectors fall apart while plain-English steps resolve onto the live checkout page

The triage question flips. A failure no longer means "maybe a selector moved." It means the user flow didn't do what the spec says.

This is how we built O2: a PR-native testing agent that reads the diff, writes or updates plain-English YAML tests, and runs them in a real browser from one line in GitHub Actions. It sits outside the coding agent that wrote the change, as an independent verification layer for AI-generated code.

Self-healing vs. locator-free testingDirect link to Self-healing vs. locator-free testing

Self-healing automationLocator-free (intent-based) testing
What the test is coupled toDOM locators (ID, XPath, CSS)User intent and expected outcome
When the UI changesLocator breaks; AI guesses a replacementNothing to break
When the feature breaksMay heal onto the wrong element and passFails on the outcome
Cost driverInference per heal, scales with UI churnInference per run, independent of churn
Human reviewApprove or reject healsReview real failures only
Who maintains itQA / automation engineersNobody, in the steady state
Fit for AI-written codeStrained at high churnDesigned for it

Where self-healing still makes senseDirect link to Where self-healing still makes sense

We don't think healing is useless. It's a reasonable choice when:

  • You have a large existing scripted suite and a rewrite isn't on the table this year. Healing buys time.
  • Your UI is stable and changes are infrequent, so heals are rare and easy to review.
  • You need tightly controlled, deterministic scripts and can't tolerate an agent interpreting steps. Some of our own customers prefer scripted tests for exactly this reason.

If that's you, healing with strict confidence thresholds and mandatory review is the sane setup.

If your frontend is changing on most PRs because agents are writing it, healing is a tax on a model that no longer fits.

How to evaluate your optionsDirect link to How to evaluate your options

Questions worth asking before you buy into any self-healing or AI testing tool:

  1. What happens on a heal with low confidence? Does the test pass, fail, or queue for review?
  2. Can it distinguish a moved element from a broken one? Ask for a demo of the second case, not the first.
  3. What does a heal cost? Per step, per run, at your PR volume.
  4. Who triages? If it's your QA team, you've moved maintenance, not removed it.
  5. Where does it run? On the PR, before merge, or on a nightly build after the damage is merged?
  6. Is it independent of the code author? A coding agent testing its own output shares its own blind spots.

A technique where a test tool detects a broken element locator during a run, uses AI or heuristics to find the most likely replacement element, and updates the locator so the test can continue.