AI-generated E2E tests become useful when they check the business outcome you intended and fail when that outcome is broken. A generated script that clicks through a page and reports success is only a draft. Review its assumptions, control its data and challenge its assertions before adding it to CI.
An AI-generated E2E test is browser-test code or a test plan produced with model assistance. Its reliability comes from the test contract and execution environment, not the fact that a model authored it. A repeatable green result is still weak evidence if the test never checks whether the right order was saved.
ego (lite) can help your coding agent inspect the real flow in a visible Space while you review what it sees. Turn those observations into a fixed test plan and reviewable code. Keep the agent's narrative separate from the assertions that decide whether a build passes.
Why can generated E2E tests pass while the feature is broken?
A generated test can encode the wrong expectation, observe a superficial signal or accidentally depend on state left by another run. Those problems have different remedies. Calling all of them “flaky” can hide a test that consistently passes for the wrong reason.
The official Playwright Assertions guide explains how web assertions wait for a specified condition. Choose that condition from the requirement: a visible success message alone does not establish that a record survives a fresh read.
Use this table to classify the problem before editing the test. The examples are possible failure modes, not measured failure frequencies.
| Symptom | Possible cause | First check |
|---|---|---|
| Test always passes despite a failed save | Assertion checks a toast, not stored state | Suppress the write and inspect the exact record |
| Local pass, CI timeout | Environment, authentication or timing mismatch | Compare trace, fixtures and readiness conditions |
| Test passes only after another test | Shared account or backend data | Run it alone with fresh seeded state |
| Click hits the wrong item | Unscoped locator or ambiguous requirement | Match the business record before acting |
| A repair removes the failing assertion | Changed test contract | Review the original requirement and diff |
A timing failure needs synchronization. A weak assertion needs a stronger oracle: the evidence that decides correctness. A missing requirement needs a human decision. Increasing retries cannot resolve those last two problems.
What should you give the generator before it writes code?
Give the assistant a small acceptance contract with a controlled start and an observable finish. Specify the record, account role, input data, permitted actions and cleanup. Include at least one behavior that must remain unchanged.
For example, an owned order fixture could require: create an order for one seeded product at the fixture's known price; after submission, its ID, quantity and total match the input; a fresh application read returns that same order; no second order is created. These are example requirements, not observations of a live store.
A focused generation request can look like this:
Draft a Playwright Test for our seeded order fixture.
Start with a new cart and a permitted test account.
Scope the product by its unique test identifier.
Verify quantity and total against the seeded values.
Submit once, then verify the order through a fresh UI read
and the authorized read API.
Do not change expected values to match the application.
Stop and ask for the missing contract if a field or API is unclear.Playwright's test-agent documentation separates planning, test generation and healing roles. That division is useful when organizing the work, but the application owner still needs to approve what constitutes a correct result.
Do not give a generator production credentials just to make a test convenient. Use an authorized test environment and define what a failed setup should report. A test that cannot establish its starting state should fail setup rather than inventing another journey.
How do you review the first generated test?
Review the generated diff as application code. Inspect every assumption the script adds: guessed selectors, expected prices, hidden setup, permissive fallbacks and assertions that merely check an element exists. Ask whether each assertion could pass if the feature were broken.
The following changes are review priorities. They improve the test contract, but they still need execution against your own fixture.
| Generated shortcut | Review change | Reason |
|---|---|---|
waitForTimeout(3000) | Wait for a specific visible or stored state | Elapsed time does not establish readiness |
| First “Submit” button on the page | Scope the form and control | Prevent an unintended action |
| Success banner is visible | Verify exact ID and persisted fields | Detect a write that never happened |
| Assert the amount read from the same result | Compare with seeded expected amount | Avoid a circular oracle |
| Reuse a shared cart | Seed per-test state and clean it up | Reduce cross-test dependencies |
| Catch errors and continue | Fail with an explicit diagnostic | Preserve a useful failure signal |
Playwright's best practices emphasize user-facing behavior, resilient locators and isolated tests. Apply those ideas to the actual page, rather than adding a role locator that does not match the application's accessible markup.
A weak illustrative test might end here:
await page.getByRole("button", { name: "Place order" }).click();
await expect(page.getByText("Order placed")).toBeVisible();That checks a message. To check the order, use a stable identity, known expected inputs and a fresh read. Keep display assertions too when they are part of the requirement, but do not let them replace persistence checks.
What did the generated TodoMVC draft actually do?
The October 12, 2026 recording uses the public Playwright TodoMVC demo. Its task is explicit: create one uniquely named todo, mark that exact todo completed, reload, then check that both the exact text and completed state remain. This exercises browser-local persistence rather than an order API or backend database. Select any image to inspect it at full resolution.
The controller reports that the unchanged first draft ran once with zero retries and verified exact text, the checked checkbox and completed styling after reload. The retained browser shows a completed todo. The Playwright report below provides native test-run evidence: one test passed. The test source and trace are referenced in the recording but have not been supplied for independent assertion review.
What does a reviewed persistence check look like?
A stronger check compares the final record with values known before the action. The example below assumes an owned fixture with test-only seed and cleanup endpoints, a cart URL tied to the seed, and an order ID exposed through a test identifier. Those endpoints and UI contracts must be implemented or adapted before this test can run.
The seed creates one product, a private cart and an authorized fixture session. baseURL and authentication must be configured so browser and request fixtures have access to the same environment. The snippet is illustrative; it is not a report of an executed test.
import { randomUUID } from "node:crypto";
import { test, expect } from "@playwright/test";
test("the submitted order is stored with the expected amount", async ({
page, request,
}) => {
const runId = randomUUID();
let failed = false;
try {
const seed = await request.post("/api/test/seed-order", {
data: { runId, quantity: 1, totalCents: 7900 },
});
expect(seed.ok()).toBeTruthy();
await page.goto(`/test/carts/${runId}`);
await expect(page.getByTestId("cart-total")).toHaveText("$79.00");
await page.getByRole("button", {
name: "Place order", exact: true,
}).click();
const idField = page.getByTestId("order-id");
await expect(idField).toHaveText(/\S+/);
const orderId = (await idField.innerText()).trim();
await page.reload();
await expect(page.getByTestId("order-id")).toHaveText(orderId);
await expect.poll(async () => {
const response = await request.get(
`/api/orders/${encodeURIComponent(orderId)}`,
);
if (!response.ok()) return null;
const order = await response.json();
return {
id: order.id, runId: order.runId,
quantity: order.quantity, totalCents: order.totalCents,
};
}, { timeout: 10000 }).toEqual({
id: orderId, runId, quantity: 1, totalCents: 7900,
});
} catch (error) {
failed = true;
throw error;
} finally {
try {
const cleanup = await request.delete(
`/api/test/seed-order/${encodeURIComponent(runId)}`,
);
expect(cleanup.ok()).toBeTruthy();
} catch (cleanupError) {
if (!failed) throw cleanupError;
console.error("Cleanup also failed", cleanupError);
}
}
});The Playwright assertions reference documents retrying assertions and expect.poll. Bounded polling lets a known asynchronous write settle without silently accepting the wrong record. Adapt the timeout to your system's contract and retain the final failed observation.
This test checks identity and amount; it does not yet establish the absence of a duplicate order. If that is part of your contract, add a read scoped to runId and assert that exactly one order exists. Keep fixture cleanup failures visible without losing the original failure evidence.
How do you prove the test catches a regression?
Run a passing control and a deliberately broken case against a controlled environment. A test that fails during login in both cases has not demonstrated that its order assertion catches the injected defect.
The table below is an acceptance protocol, not test results. For each mutation, record the exact failure and show that restoring the behavior restores the expected outcome.
| Controlled case | Expected test outcome | Evidence to retain |
|---|---|---|
| Correct seeded order persists | Pass | Test revision, seed and stored order |
| Success message appears but write is suppressed | Fail on persistence | Trace plus fresh read showing no stored order |
| Stored amount differs from seeded price | Fail on amount | Expected and actual values |
| Wrong record is returned | Fail on identity | Requested and observed IDs |
| Duplicate order is created | Fail on count, once that assertion exists | Records scoped to the test run |
| Starting state cannot be created | Fail setup explicitly | Seed request status and diagnostic |
Use only failures you can inject safely in your own test environment. Name the exact requirement each mutation exercises. Count an inconclusive run separately from a meaningful pass or detected regression.
Keep the generator prompt, reviewed code and mutation evidence together. A before/after comparison should say what changed and what was actually observed. Do not claim improved reliability from a code diff alone.
What must stay stable when the test moves into CI?
Pin dependencies, install the matching browser binaries and use the same configuration you qualified locally. Define how the environment becomes ready, how test accounts are provisioned and who owns its seeded data. A visible homepage is not proof that all backend dependencies are ready.
Playwright browser contexts isolate browser state between tests. They do not reset a shared backend database. Use unique run identities or isolated environments for server-side data, and make cleanup part of the fixture contract.
The official CI guide covers running Playwright in CI. Start qualification with retries disabled so failures remain visible. If your production suite later uses retries, report first-attempt failures and recovered attempts separately; a final green build should not erase the instability you are investigating.
Retain traces and failure artifacts with access appropriate to their contents. The Trace Viewer guide explains how to inspect actions, DOM snapshots and network activity. Compare those observations with the intended contract before blaming the model, locator or application.
How should an agent help without weakening the release gate?
Let the agent investigate failures and propose edits. Require review when an edit changes expected values, drops an assertion, expands permitted actions or substitutes another business record. A locator repair and a changed acceptance criterion are different decisions.
ego (lite)'s visible Space workflow can help a coding agent reproduce the relevant flow with a person checking the target and outcome. Use that inspection to clarify the test plan and repair the code. Run the reviewed test again through the controlled cases before promoting it into CI.
This article covers the initial path from generation to qualification. Long-term suite maintenance, changing requirements and locator conventions deserve their own processes. Start with one important flow whose initial state and final outcome you can independently inspect.
How did the agent check the same task in ego (lite)?
The coding agent opened TodoMVC in a named ego (lite) Space and was asked to create and complete one unique todo without modifying unrelated existing items. The overview shows the named task running alongside another Space. It also records setup discovery for an older installed command; that issue is not hidden by the final result.
The final capture shows the result table beside the live task, including Agent is in control, Take over and Stop. It reports that the same unique row was created, completed and still matched after reload. Other existing todos remain visible. This is a separate browser inspection of the business requirement, not execution of the generated Playwright test.
Which checks are reported in the result?
The ego (lite) result identifies the todo as Ego persistence check 2026-10-12T09:18:45.895Z. Its four checks are transcribed below so the essential evidence does not depend on reading small screenshot text.
| Check | Reported result | Observed evidence reported by the controller |
|---|---|---|
| Created | Pass | Exactly one matching row, initially unchecked |
| Completed | Pass | Same row checked and struck through, with completed class |
| Text persisted after reload | Pass | Exact text remained in one matching row |
| Completed state persisted after reload | Pass | Checkbox remained checked and completed styling remained |
These normal-run captures do not show a reviewed test diff or a suppressed-storage-write experiment. They therefore do not establish that the generated draft misses a fault, that a revision detects one, or that ego (lite) improved its reliability. The next qualification step is to challenge the preserved draft and reviewed version under the same deliberate fault, retain actual failures, and accept the result if both already detect it.
What does qualification leave untested?
Qualification establishes only what the chosen scenarios exercise. The example order contract does not cover performance, accessibility, authorization boundaries or every supported browser. Select those checks from the application's requirements and risk, rather than treating one qualified flow as complete coverage.
The code fragments require an application fixture and have not been used here to measure a generator's reliability. Keep your executed controls, mutations and environment details with the reviewed test. Only those observations can support claims about the behavior of your suite.
FAQ
Are AI-generated E2E tests inherently flaky?
The authoring method alone does not determine flakiness. Examine synchronization, isolation, environment assumptions and assertions before assigning a cause.
Is a test that passes ten times reliable?
Repeated passes show behavior under those runs' conditions. They do not demonstrate that the test can detect a regression it never encounters.
Can AI generate the expected values too?
Use values from the approved requirement or controlled seed. An expectation inferred from the same application output can make an incorrect result pass.
Should I replace every fixed sleep?
Identify the readiness condition each sleep was intended to approximate. Prefer a specific bounded assertion; replacing a sleep with an unrelated wait does not fix the contract.
Does a fresh browser context reset test data?
A browser context isolates browser state. Shared server-side records need their own seed, namespace or reset mechanism.
Should a healed test commit itself?
Review the proposed diff and rerun its qualification cases. Give special scrutiny to changes in assertions and expected outcomes.
Can I run these tests on a third-party checkout?
Use owned or explicitly authorized environments for writes and deliberate faults. A third-party live flow is unsuitable for injecting a failed purchase or duplicate order.
What if there is no read API for verification?
Check the exact record through a fresh UI navigation or reload and document what that evidence can establish. Do not claim a backend check you cannot perform.
How many retries should qualification use?
Begin without retries so first-attempt failures are visible. Choose any later retry policy based on diagnosed causes and retain the original failed attempts.
What is the smallest useful starting point?
Qualify one controlled flow with a known passing state and one deliberate business failure. Expand the suite once its assertions and setup are understood.
Related reading
Read Playwright locators for target selection and Claude Code browser testing for an interactive testing workflow. For browser execution and review, see computer use agents and their controls. Pick one generated test and write the defect it must catch before adding more tests.
