Search

Debugging Failed Tests

One of the biggest challenges in test automation is test case maintenance—especially when the application under test (AUT) changes. This is where Functionize is built to help.

Our platform combines machine learning, self-healing, and robust debugging tools to minimize maintenance overhead while giving you deep visibility when issues occur.

Categories of Failures

Before fixing a failed test, it’s important to identify what type of problem occurred. We break these down into two categories:

Category 1: Automated Test Issues

The test itself didn’t execute as expected, often due to changes in element properties or timing:

  • The wrong element was selected for a click, input, or checkbox action.
  • A click action executed, but nothing in the AUT responded.
  • An input field did not accept the entered value.
  • A drop-down did not select the intended item.
  • The test did not wait long enough for an element to appear.
  • The test was unable to find, identify, or interact with a page element.

Category 2: AUT Issues

The application itself behaved differently than expected:

  • An unexpected popup or modal appeared.
  • An error was displayed in the AUT.
  • The application took significantly longer to load.
  • A workflow or UI element changed.

Self-Healing Tests

At execution time, Functionize automatically attempts to self-heal failed selectors using our ML models.

  • How it works: Unlike Selenium or other tools that rely on a single selector (XPath, CSS), Functionize captures thousands of data points per element. During execution, the ML model analyzes these attributes and adapts when it detects a change.
  • Visibility: Self-heals are marked with yellow tags in the Test Details page and Slider View.
  • When human input is required:
    • The site has changed significantly.
    • Workflow logic is different but the test has not been updated.
    • A genuine defect exists in the AUT.

Debugging Tools

Functionize provides multiple layers of debugging so you can quickly diagnose the root cause:

1. Screenshots

For every action, four screenshots are captured:

  • Pre-Action: State before the action.
  • Mid-Action: During execution of the step.
  • Post-Action: After the action completes.
  • Full-Screen: Expanded view of the entire application.

These allow step-by-step visual verification, helping pinpoint exactly where behavior diverged.

2. Live Debug

  • Set breakpoints within a test.
  • Execute step-by-step on a VM hosted in the Functionize Cloud.
  • Watch the test interact with the AUT in real time, re-execute specific steps, and manually adjust actions as needed.

3. Anomalous Attribute Tags

These highlight which element attributes changed compared to previous runs, helping you understand why a selector may have failed or healed.

4. Comparisons

  • Previous Successful Run vs Current Run: Side-by-side screenshots of past passes versus the current failure. This is the most powerful diagnostic tool for spotting changes.
  • Current Run vs Architect Run: Compare the latest run against the original modeled version.
  • Common Attributes: Review data attributes across Architect, previous runs, and the current run.

5. Computer Vision (CV) Failures

When verifications rely on screenshots rather than selectors, CV compares full-page screenshots against baselines. If the expected content isn’t present, the CV verification fails.

6. Video Recording

If screenshots and comparisons aren’t enough, you can enable video capture. Replay the execution like a session recording to see the precise moment of failure.

How to Debug Failed Tests

  1. Open a project and navigate to the Test Listing page.
    • Review the Browsers column for quick pass/fail status.
  2. Open the failed test and go to the Browser tab.
  3. Each action banner shows pass/fail status:
    • Green = passed
    • Red = failed
    • Yellow = healed
    • Purple = incomplete
  4. Expand a failed action to view Execution Error details.
  5. If a Verify Action fails, note that execution continues by design. This ensures you capture maximum results from each run.
  6. From the failed step, click View to open Slider View.
  7. Use arrows to step through screenshots before and after the failure, looking for clues such as:
    • Page not loading
    • Error message displayed
    • Wrong account or data state
  8. Review the left-side panel for detailed action metadata.
  9. Use the bottom panel to check historical data from previous runs of this step.

Fixing Failed Tests

Once you’ve diagnosed the issue, you have several ways to fix it:

  • SmartFix: One-click ML-powered fix suggestions (e.g., update element selection or verifications).
  • Action Settings: Add flags for optional actions, popups, or skip logic when the AUT varies by environment.
  • Executor Updates: Switch execution strategies to ensure inputs and clicks are performed correctly.
  • Selector Override: Manually choose the element if ML selection missed.
  • Live Debug: Modify and update tests interactively in real time.
  • Local Edit (Architect): Re-model flows or elements from your local environment.
  • Support Ticket: When advanced troubleshooting is needed, escalate to our support team.

You’ll want to ensure that the test flow is the same as the manual flow. Any changes in the AUT need to be reflected in the test.

Handling False Positives (Force Failing)

Self-healing can occasionally result in a false positive—a test passes even though the AUT contains a defect.

  • When this happens: Typically due to missing or incomplete verifications.
  • Solution: Use the Force Fail feature to manually mark the step (and test) as failed. This retrains the ML model and ensures accuracy in future runs.

How to Force Fail

  • From the Test Detail page:
    • Expand the action → Settings tab → Fail → Confirm → Save.
  • From the Slider View:
    • Open results → Gear icon → Fail → Confirm.

When a Force Fail is applied, SmartFix may also suggest alternatives (e.g., different elements to select), giving you another way to improve the test.

Summary

Functionize’s approach to test maintenance is built on three pillars:

  1. Self-Healing: Automated recovery from minor UI changes.
  2. Debugging Tools: Multi-layered visibility—screenshots, comparisons, tags, Live Debug, CV, and video.
  3. Repair Actions: SmartFix, selector overrides, local edits, and Force Fail for long-term resilience.

Together, these capabilities ensure you can keep your automated tests robust, reliable, and accurate—even as your AUT evolves.