Why this checklist exists
When a test goes red, the slowest part is usually not fixing it — it is working out who should. A failure can come from the application under test, from the test itself, from the test data, from a transient condition, or from the Functionize platform. Until you know which, the ticket sits in the wrong queue and the work gets repeated.
This checklist is six checks, in order. Work them from the top and stop at the first one that explains the failure. Most failures are explained in the first four, without a ticket ever being raised.
The goal is not to decide whether Functionize is at fault. It is to reach the right owner quickly, with enough evidence that nobody has to repeat your work.
The five owners
Every check below ends by assigning the failure to one of these:
| Intermittent | Passed on a re-run. Worth noting, not worth a ticket unless it comes back. |
| Application | The application under test, or the environment it runs in, behaved differently. |
| Test | The test needs updating — usually a selector that no longer matches. |
| Test data | The data the test depends on is stale, missing, or already used. |
| Functionize | None of the above explains it. Raise a ticket. |
Step 1 — Re-run the failed test once
What to do: re-run the failed test a single time, without changing anything.
Why: a single re-run is the cheapest way to separate a genuine, repeatable failure from a transient one. Network blips, a slow environment, a service still warming up, and timing races all produce failures that do not reproduce.
If it passes: record it as intermittent and move on. Note the test ID and the date somewhere you can look back at. Raise a ticket only if the same test fails intermittently again — at that point the pattern itself is the evidence, and it is far more useful to us than a single run.
If it fails again: continue to step 2.
Re-running once is the rule. Re-running repeatedly until a test passes hides real problems and is how flaky suites are created.
Step 2 — Check the failed step's screenshot
What to do: open the failed step and look at the screenshot captured at the moment of failure. Ask one question: did the application actually load?
What you are looking for:
- An application error page — for example "Oops! Something went wrong"
- A blank or white frame
- A partly rendered screen, where some elements are present and others never arrived
- A login or session-expired screen you did not expect
If you see any of those: this is an application or environment issue, not a test issue. The test failed because the thing it was testing was not there. Re-run when the environment is healthy.
This step matters more than its position suggests. A warning or failure at a step is often correct behaviour — the test is accurately reporting that the expected elements never rendered. Treating that as a platform defect sends the ticket to the wrong place.
If the application loaded correctly: continue to step 3.
Step 3 — Compare the element with the recorded selector
What to do: look at the element the step is trying to act on, and compare it with what the application is serving now. Has the identifier, the visible text, or the surrounding structure changed?
What you are looking for:
-
Identifier changes. These are often small enough to miss. A change from
select-item-0-holidaygrouptoselect-item-0-holidayGroupis a change in capitalisation only, and it is enough to invalidate the stored selector. - Label or text changes. A button renamed from "Save" to "Save and close" will not be matched by a selector looking for the old text.
- Structural changes. A field moved into a new container, a table gaining a column, a modal becoming a separate window.
- Position-based selectors. If a step targets "the tenth row" rather than "the row labelled X", it will action whatever happens to sit in that position. When the ordering changes, the test silently does the wrong thing rather than failing cleanly.
If the element has changed: update the selector. Where you have the choice, prefer text or label based selectors over position based ones, and prefer a stable unique identifier over a display name. Identifiers tend to survive cosmetic changes; positions and labels do not.
If you find one position-based selector, it is worth checking whether the same pattern was used elsewhere in the suite. These tend to arrive in groups, and fixing them as a group is far cheaper than meeting them one failure at a time.
If the element is unchanged: continue to step 4.
Step 4 — Check the test data
What to do: look at the data the test depends on, and at what the application said when the step failed.
What you are looking for:
- A record the test tries to create that already exists from an earlier run. The usual tell is a message on save such as "a similar record already exists", instead of the success confirmation the test expects.
- Reference data that has changed underneath the test — a dropdown option renamed or removed, an account closed, a date now in the past.
- Data consumed by a previous run in the same cycle.
If the data is the problem: reset it, or make it unique per run. Generating a unique value per execution is the more durable fix, because it removes the dependency on run order and on clean-up completing successfully.
Tests that pass the first time after a data refresh and fail on every subsequent run are almost always this.
If the data is correct: continue to step 5.
Step 5 — Try the same step manually
What to do: perform the same action by hand, in the application, outside the platform. Use the same environment, the same user, and the same data.
If it fails manually too: it is application behaviour, and no change to the test will fix it. This is a strong, unambiguous finding — worth capturing with a short recording or a screenshot, because it settles the question of ownership immediately.
If it works manually but not in the test: that difference is itself the most useful thing you can tell us. Note exactly what you did by hand, then continue to step 6.
Step 6 — If none of the above explains it, raise a ticket
By this point you have ruled out a transient failure, an application error, a changed element, stale data, and application behaviour. That is genuinely useful work, and the ticket should say so.
Include all of the following:
- Test ID
- Step number where the failure occurred
- Run link to the failed execution
- What you already checked — the steps above you worked through, and what you found
- Screenshot of the failed step
- Whether it reproduces manually
The fourth item is the one most often left out and the one that most affects resolution time. A ticket that says "fails at step 6.15, re-ran twice, application loads correctly, selector unchanged, data is unique per run, works manually" can be picked up immediately. A ticket that says "step 6.15 fails" starts with somebody repeating everything you already did.
If a step fails intermittently rather than every time, say so and give more than one run link. Intermittent issues and consistent ones are investigated differently.
A note on error messages
If you encounter an error message in the platform whose meaning is not clear, please raise it. An error string that a skilled engineer cannot interpret is a problem worth fixing on our side, and we would rather hear about it than have teams lose time guessing.
Download the one-page version
A printable one-page summary of this checklist, including the triage flow diagram, is attached to this article: Functionize — Failure Triage Checklist (PDF)
It is designed to be pinned up or kept open next to a test run, and it covers the same six steps in short form.