Search

Debugging Failed Test Cases

  • Updated

The biggest pain point in test automation is test case maintenance, when changes occur in the application under test. This is one of the main challenges that Functionize aims to help you solve. In most cases, we will self-heal the test for you using a robust set of machine learning algorithms. If the test fails for other reasons, we provide several methods for you to view and debug the test or update it as needed.

The goal with debugging is to diagnose why test(s) failed and then once the problem(s) are identified, fix the problem(s).  Problems can be categorized as either an automated test issue or an application under test (AUT) issue.

What Types of Problems Can Be Encountered?

It is important to first determine what the problem is in order to then figure out how to fix the problem. We have identified two different categories of problems:

CATEGORY 1 - The automated test did not do what you expected.

  • The wrong element was selected for clicking/text/input/checkbox/etc
  • A click did not work, nothing changed after the click action was executed
  • An input did not work, nothing enter in the AUT
  • A drop-down field did not select the correct item from the list
  • The test did not wait long enough for an element to appear in the AUT

CATEGORY 2 - The AUT did not do what you expected, a defect was found.

  • The AUT had an unexpected popup
  • The AUT had an unexpected error
  • The AUT took significantly longer time to load than expected
  • The AUT workflow changed

Self-Healing Tests

Self-Healing is completed in the cloud at the time of test execution and can be viewed on the Test Details page and Slider View.  There are small, yellow tags that indicate a Self-Heal has taken place for a given action.

If a test case fails in Functionize, it means the test case has not Self-Healed for one reason or another, and human input is required to resolve the failed action(s). Human input is typically needed if the site has changed significantly, the workflow has changed, but the test hasn't been updated or there is a valid defect in the application under test.

Note: Selenium and many other test automation tools capture only one data point (XPath or some kind of Selector) for every element. Functionize identifies not just a single Selector but thousands of data elements, forming the basis of our machine learning. Functionize collects all of the data and analyzes it to find discrepancies and changing patterns from one test execution to the next for the purpose of rating, ranking, and categorizing the elements, ultimately resulting in the elimination of Selector maintenance altogether, the basis of our Self-Heal feature.

Debugging and Diagnosing Tips

We have many tools available to figure out what the problem is by allowing the user to see what happened during the test's execution.

  1. Screenshots are typically the easiest way to track down the cause of failure. We capture 4 screenshots per action. The Pre, Mid, and Post Action screenshots are found in the Screenshot Interval menu. The fourth screenshot, Full Screen, is available by clicking the Expand button. The Pre, Mid, and Post-Action screenshots of a test case involves capturing screenshots of an application at various stages of its use to ensure correct functionality of the AUT.Pre-Action Screenshot: The test captures a screenshot of the application before performing any actions. This establishes a baseline and ensures that the application is in the expected state before any actions are taken.Mid-Action Screenshot: The test captures a screenshot during the performance of an action. This ensures the AUT functions correctly during the action and captures any errors or unexpected behavior.Post-Action Screenshot: The test captures a screenshot after the action has been completed. This ensures that the AUT is in the expected state and verifies that any changes made during the action are saved correctly.By comparing these three screenshots, the tester can identify discrepancies or issues that may have occurred during the action execution and can verify the correct function of the AUT. The screenshots are particularly useful for verifying the functionality of the AUT user interfaces, as it allows the tester to see exactly what the user sees at each stage of the interaction with the AUT.
  2. Live Debug gives the user the ability to set Breakpoint(s) and have access to the Virtual Machine on the Functionize Cloud to watch and interact with the test during execution.
  3. Anomalous Attribute Tags will be displayed on the Test Detail page and Slider View for actions with attributes that have changed from the previous execution and have Self-Healed.
  4. Previous Successful Run vs Current Run Functionize also offers, most importantly, the ability to view the Previous Successful Run along with multiple screenshots per step, allowing the user to visually compare what passed previously with the current failed execution. This can be accomplished across multiple steps and is accessible at the top left of any test action in the Slider View.Note: The Previous Successful Run is possibly the most important and useful diagnostic tool offered because it allows the comparison of a failed execution to a passed execution and makes it easy to see the visual difference between the two screenshots, allowing for a side-by-side comparison .In the screenshot below, Prev stands for Previous, Cur stands for Current, and Arc stands for Architect. Toggling between these will show the screenshots for the corresponding execution. Look at multiple steps leading up to the failed step to see when screenshots start to go wrong.The comparison of screenshots shows a side-by-side view of the Previous Successful Run versus the Current Run.
  5. Current Successful Run vs Architect This comparison of screenshots shows a side-by-side view of the current successful run versus the architect run.
  6. Common Attributes Compare Data Attributes from when the test was Modeled, Previous Successful Run and Current run.
  7. CV Failures - Full Page Screenshots Computer Vision (CV) is based on the Full Page screenshots. If these screenshots don't show the expected value, CV will likely not work as expected. To locate the Full Page screenshot, look in the header of the Slider View and click the Cur, side-by-side  or Base tabs to see the Current and/or Baseline screenshots in a side-by-side view. See below example:
  8. Video Enabled If all other methods don't provide enough information, you can turn on video recording for the test, rerun, and then determine the root of the issue.

How to Debug Failed Test Cases

  1. Open any Project
  2. The Test Listing page includes a column called Browsers where you can quickly see the Pass/Fail status of your test and the last run information details are on this page as well
  3. Open a test and then open the Browser tab for the failed execution
  4. Each action displays its Pass/Fail status in the banner. Green indicates passed, red indicates failed, and yellow indicates
  5. Expand  the Failed action in order to see the Execution Error details
  6. If a Verify Action fails, the test will continue executing, so there may be a series of passed steps, a failed step, and then additional passed steps - this is expected and allows maximum value to be extracted from test results and is controlled under Test Settings > Advanced tab
  7. From the failed step, click the View button to the right of the failed test step, which will open the Slider View to offer a more inclusive context, with screenshots, to the reason for the failure
  8. Using the Arrows, click through before and after the failed action, looking for visual clues in the screenshots of the failure, these can include error messages, page not loading, or already registered accounts
  9. Stop at the failed test action screenshot, usse the information shown on and around the screenshot to diagnose the test case and in the example, Functionize is highlighting that the Found and Expected text does not match
    • On the left side slider panel, information about the failed step is displayed
    • On the bottom, highlights what data was found on Previous executions of this step, which in this case matched the Expected text 

Fixing Failed Test Cases

When a test case fails, it's essential to diagnose and fix the issues promptly. Functionize offers various methods and tools to help you update and fix failed tests, ensuring your automated tests are robust and reliable. Here is an overview of the available methods:

SmartFix

Functionize's machine learning (ML) will suggest ways to fix the test with a one-click solution. This feature can help update the test when the wrong element is selected, an incorrect password is entered, or verifications need updating.

Update Action Setting Flags

Each action in a test has certain settings, flags, and information associated with it. These can be accessed via the Test Details page or Slider View. Action settings are useful for handling unexpected pop-ups, optional actions, or skipping actions when a website has changed in one environment.

Update the Executor

If a click didn't actually click or an input didn't input text as expected, you can change the Executor to improve the action's performance.

Update the Selection Method

Functionize's default ML selection is highly accurate, but sometimes it may select the wrong element. The Selectors Override feature allows you to overrule the ML selection.

Force Fail a Passing Test

There are times when self-healing attempts do not perform as expected, resulting in false positives. You can manually mark the test as failed to ensure the ML model does not mistakenly learn incorrect behavior.

Live Debug

Live Debug is the most robust method for updating a test case. It allows you to interact with a test while it runs on Virtual Machines in the Functionize Cloud, enabling quick diagnosis and modification of tests in a clean execution environment.

Local Edit

Local Edit recreates all the ML data used to execute the test. This feature allows you to use Architect on your local machine to make changes in test flows, element selections, and verifications.

Submit a Support Ticket

If you are unable to resolve your test failure with the above options, please reach out to the Functionize Customer Support team.

By utilizing these methods and tools, you can efficiently diagnose and fix failed test cases, ensuring the reliability and accuracy of your automated tests in Functionize.
 

Force Failing a Passing Test Case

Self-healing helps the user save countless hours fixing broken test cases. However, in certain circumstances, this may result in a false positive or a test case that passed when it should have been considered a failed test case. In the context of testing, a false positive occurs when a test indicates success (pass) even though there is an actual defect or issue present. False positives should be corrected so that the machine learning model can learn and ensure that tests are passed or failed correctly when executed again in the future. This corrective action is called Force Fail, which gives the user the ability to manually fail a test that has passed.

The most common reason for false positives is incorrect and/or missing verifications in a test case.

How to Force Fail a Test Case

Whether a missing verification is the culprit for your false positive or some other reason, a user has the ability to Force Fail a test case to intentionally mark a test as failed and ultimately correct the false positive.Functionize provides two different options to Force Fail a test case:

Force Fail on the Test Detail Page

  1. Open the Test Details page
  2. Expand the action to be Force Failed in order to view the Settings tab
  3. Click the Fail button
  4. Click the Fail button in the Confirm model
  5. The test action will change to Failed status with the Execution Error message "Failed manually by User on Date"
  6. Click the Save button.

Force Fail on the Slider Page

  1. Open the Test Details page
  2. Click on the View Results button or click the View button to the right of any Action
  3. Click on the gear icon for the specific step to fail
  4. Click the Fail button
  5. Click Fail in the Confirm model
  6. Once the failure has been confirmed, the overall test is marked failed with an indication that the test was manually failed

Note: When Force Fail is used, in certain situations, a SmartFix suggestion may be provided to recommend how the test can be optimized for future executions. As an example, alternate suggestions or elements may be offered to select on the page. Should that SmartFix suggestion be accepted (e.g. selecting a different element on the page), then in future executions, the test will check for that element instead. These SmartFix suggestions offer another peek into our machine learning capabilities that are continually looking for ways to optimize tests. Using the Force Fail feature and modifying the test case with proper verifications will allow the user to better run tests in the future.