Search

Using Visual Verifications

  • Updated

There are two ways to perform Visual Verifications using Functionize's Computer Vision feature:

  • Full Page Verification
  • Element-Based Verification

We employ a machine learning approach to compare images, making the process faster and taking into consideration trivial changes that can occur within the browser. Just imagine the browser rendering an image off by a few pixels in each execution. Our baselines are established per browser to ensure that browser-specific false negatives do not occur.

Both of these visual verifications operate similarly:

  • Match Requirement: (Default % set to 92%) Your project has a default match requirement % for both types of verification. We keep this tolerance in place for common variations, but, of course, you can adjust this value higher or lower. The default value will be passed into Architect and can be overwritten on an action-by-action basis when creating the test or from the UI after the test has been executed.
  • Baselines: Baselines for the image are established during the first successful run, per browser. However, Element baselines are set to Chrome's default when creating a test with Architect.

Full Page Verification

A full-page visual verification, referred to as a Visual Page Check, allows you to verify that an entire screenshot, even when not in the current viewport, always matches your baseline. There are two types of Full-Page verifications:

  • Baseline Verifications: In every execution, we compare against the baseline, which is set for Chrome via Architect during creation or during the first successful execution of the test. The baselines may be updated after any execution to match the current screenshot.
  • Previous Step Verification: This should be used when you want to verify a screenshot against a previous step's verification. For example, if you would like to compare a screenshot with the previous screenshot, you might use the Previous Step Verification method.

Element-Based Verification

Element-based verifications are a feature of any Verify Action. The element could be an image, but also a div, span, or any kind of element that you would typically baseline. On the Slider View screenshot, the area highlighted in green for the selected element will be used as the image for comparison. There are two types of Element-based verifications:

  • Baseline Verifications: In every execution, we compare against the baseline, which is set for Chrome via Architect during creation or during the first successful execution of the test. The baselines may be updated after any execution to match the current run's element.
  • Previous Step Verification: This should be used when you want to verify an element against a previous step's verification. For example, if you would like to compare a company logo that always appears just like the first step's logo, you might use the Previous Step Verification method.

Creating a Baseline Element Verification

This verification must be created using Architect. It may be created on an image element, or a div, or any other kind of element. The entire element must fit into the viewport, the visible screen, in order to properly perform a visual validation.

  1. Launch Architect to create the test case
  2. Select eye icon to perform a visual verification, this works identically to a regular checkpoint verification
  3. Once the eye icon is selected, choose the element from the webpage that you wish to verify
  4. Once the element has been selected, you will be brought to the Visual Check modal to verify that your element appears as expected - this is where you set the Visual Check Type to Baseline Comparison, by default and set the default Visual Match Requirement %, the default can be updated under the Advanced TAB in Project Settings
  5. Click Save on the Visual Check model
  6. The new Verify Action with an eye icon will appear in the action list of the Architect
  7. Alternatively, if you wish to add the visual verification to an existing verify, click the white dot to the left of any Verify Action, then select Visual Check button in the left side panel to access the same feature
  8. If navigating via the white dot or another method, remember to click Save on the left side panel to add the Visual Check to the test

Creating a Step Comparison Element Verification

  1. Follow the same steps as Creating a Baseline Verification until the Visual Check modal appears
  2. Set the Visual Check Type to Step Comparison
     
  3. Select the Step # for the previous step to which you'd like to compare the current image to, only previous verifications will appear in the dropdown when selecting the step
  4. Update or keep the default Visual Match Requirement %
  5. Click the Save button in the modal
  6. If navigating via the white dot, or another method, remember to then click Save in the left side panel to add the Visual Check to the test

Viewing Visual Comparison Results

  1. Go to the Architected test case where the visual verification was created
  2. Notice the Visual Check on the steps and element verification
  3. Hovering over the Visual Check tag will show you the Current run's element on the right, and the Baseline image on the left, the Actual and Expected Matches are the values used during the execution being viewed, Match Requirement is the value used across all browsers of the test, and Baseline is the baseline image set for the particular browser for which you are reviewing results
  4. Clicking on Edit Match Requirement allows you to quickly update the setting used on future executions of any browser
  5. Each browser renders web pages slightly differently, so they must have their own baseline image, clicking the View chrome baseline will allow you to view what the Chrome baseline is currently set to
  6. Clicking Update Baseline allows you to update the baseline image for the given Browser

Viewing Baseline

  1. If you would like to view the baseline images across all browsers, view the Actions Log
  2. Element Baselines will have a Baseline tag
  3. Hovering over the Baseline tag will show you the current settings used for the test and the Baseline tag is clickable and will show you the full image used for that Browser baseline

Setting the Default Match Requirement %

  1. The default value may be set at the project level
  2. On the Test Listing page, click the Project Actions …  icon, on the top right side, to view Project Settings information
  3. Navigate to Project setting > Advanced Tab > Element Match Requirement Default % in order to update the default value for Tests under this Project

Full Page Visual Verification Overview

Visual Page Check is an action type used to visually verify a full page of a website. This action type is abbreviated as VisPageChk. This verification type utilizes Machine Learning (ML) for intelligent matching of the current screenshot against the baselined screenshot for the site. The baseline will be automatically set during the first successful test run, or you can manually set or update the baseline image when needed.

Note: Currently, this is only supported in desktop web browsers.

Here are two types of full page visual verifications:

  • Baseline Verifications: In every execution, we compare against the baseline, which is set per browser during the first successful execution of the test on that browser and the baselines may be updated after any execution to match the current screenshot
  • Previous Step Verification: This should be used to visually verify a screenshot against a previous step's visual verification of the same screenshot

Adding a Visual Page Check Action

  1. Select the +  in Architect to open the Architect Advanced Actions sidebar panel
  2. Select the Visual Page Chk action
  3. Verify or set the Visual Match Requirement % - the default value can be updated at the project level
  4. Click the Add button in the Architect Advanced Actions sidebar panel
  5. The resulting VisPageChk Action should appear in the Architect Action list

Viewing Full Page Visual Check Results

  1. Once the test has been executed, view the full page visual check results in the Slider View page
  2. Similar to the Prev (previous), Cur (current), and Arc (architect) screenshots, there is an additional section for Visual Page Check Actions that displays the Cur (current) and Base (baseline) images side-by-side
  3. When viewing Visual Page Check Actions, there is a control widget, and when in side-by-side comparison, a yellow draggable bar
  4. In the CV Controls modal, the following options are available:
    • Select Show Ignore Regions checkbox to display regions of the full page to ignore
    • Select to view Modified areas within the screenshot that are different from the baseline
    • These overlays may appear on any set of images but will always reflect the current versus baseline changes
    • In the Settings area, you can make changes to the full page visual verification settings, Edit Match %, Update Curr to Base, Ignore Elements and Compare Regions - similar to other settings changes, these adjustments will be reflected across the entire test, and any enabled browser will adopt these settings for the next execution
    • Click on Compare Regions Popup window will open and select the Region and Save
  5. Click the gear icon on Vispagechk Action
  6. The Settings pop-up window will display, click on Advanced tab.
  7. Select the Check box Basic Pixel comparison and click on save

Setting the Default Match Requirement %

The Full page visual check default value may be set at the project level.

  1. On any project page, select the … icon to view project level settings
  2. Navigate to Project Settings >> Advanced tab >> Full page visual check default value % in order to update the default value

Insert Full Page Check action

Full Page Visual Check Actions can also be inserted via the Test Detail page and Slider View.

Test detail page:

  1. Open your Test
  2. Hover where you would like to insert the action
  3. Open the Insert menu and select Full Page Check Action
  4. Enter a value in the Visual Match Requirement (%) and select Visual Check Type (Baseline Comparison or Previous Action)
  5. Click Insert

Slider Page:

  1. Open your Test
  2. Open Slider View via the View Results button or the View button next to any action
  3. Click the Quick Add button at the step where you would like to insert the action.
  4. Open the Insert menu and select Full Page Check Action
  5. Enter a value in the Visual Match Requirement (%) and select Visual Check Type (Baseline Comparison or Previous Action)
  6. Click Insert

Limitations

Full page visual verification screenshots always start at the currently scrolled position. To ensure it starts at the top of a particular page, follow these steps:

  1. Set the scroll location using the ADV Actions Architect button
  2. Select Scroll Action
  3. Then scroll to the desired location on the website for setting the top of the screenshot, click the Close button on the Architect sidebar panel

Note: The full page visual verification may not always capture the entire screenshot. Certain sites behave differently or are particularly large, causing us to capture only partial screenshots.