Search

Understanding Test Case Settings

  • Updated

Functionize's Test Settings feature allows users to add, edit, and review various settings related to a test case. These settings include General & Advanced Settings, Webhooks, Timeouts, Cookies, Headers, Auth, and Alerts. Users can customize these settings to configure the behavior and execution of their test cases.

Accessing Test Settings

Method 1: Through Project Test Listing

  1. Open a project to the Test Listing page.
  2. Locate the test case to update Test Settings.
  3. Under the Actions column, click the Test Actions button and select the Settings menu option.

Method 2: Through Test Details Page

  1. Click on the Test Actions button and select Details or click on the test case Name.
  2. On the Test Details page, click the Settings button and select Test Settings.

Regardless of the path chosen, the Test Settings screen will display, starting with the General tab.

Test Settings Tabs

General Tab

The General settings for a test case are found in the General Tab. The fields are as follows:

  • Test Name: Name of the test.
  • Project: Name of the project.
  • Site Environment: The project environment in which the test is created.
  • Folder: A categorization assigned to this test to help filter tests on the Test Listing page.
  • Schedule: Shows the schedule of the test when running.
  • Browser Run Order: Shows run order as parallel or sequential.
  • Notes: Any notes for this test.
  • Tags: A categorization assigned to this test to help filter tests on the Test Listing page.
  • Not Approved for Learning: Approve the test for AI learning and self-healing.

Webhooks Tab

The Webhooks tab is used to set the state of a test case, either before or after a test case executes. Currently, only GET and POST requests are supported for both Pre- and Post-test hooks. The fields are as follows:

  • Pre-Test Hook: The Pre Test Hook allows you to define an API call that will be executed before the test starts. This can be useful for preparing the test environment or initializing certain conditions required for the test.
    1. API Call:
      • Enter the URL for the API call in the API Call field.
    2. API Method:
      • Select the HTTP method for the API call from the dropdown (e.g., POST, GET).
    3. Parameters:
      • Click the Add Pre Hook Parameter button to add any parameters required for the API call.
      • Enter the Parameter Name and Value in the respective fields.
  • Post-Test Hook: The Post-Test Hook allows you to define an API call that will be executed after the test completes. This can be useful for cleaning up the test environment or logging results to an external system.
    1. API Call:
      • Enter the URL for the API call in the API Call field.
    2. API Method:
      • Select the HTTP method for the API call from the dropdown (e.g., POST, GET).
    3. Parameters:
      • Click the Add Post Hook Parameter button to add any parameters required for the API call.
      • Enter the Parameter Name and Value in the respective fields.
  • Extensions Post-Hook: Test Clean Up - executes after the test runs, allowing for custom integrations with Test Management solutions.

Timeouts Tab

The Timeouts tab shows performance level timeouts and thresholds for the test case. If any of these set thresholds are exceeded during test case execution, the test case status will display a Warning.

  • Page Load Timeout: Maximum time the ML timing model will wait for a whole page to load before attempting selection.
  • Missing Element Timeout: Maximum time the ML timing model will wait for an individual element to load before attempting selection.
  • DOM Loaded Threshold: The point at which the DOM loads on the screen and provides a warning message if the threshold is exceeded.
  • First Paint Threshold: The point at which the first pixel renders on the screen and provides a warning message if the threshold is exceeded.

Note: These timeouts will not fail a test case but will alert that one of these thresholds has been met/exceeded.

Advanced Tab

The Advanced tab showcases an extensive list of optional test case settings, including options such as enabling video execution in the cloud, setting a proxy at the test case level, selecting the region for test case execution, Self-Heal validation, setting viewport width and height for responsive sites, and more. The most commonly used settings include:

  • Language: You can select the Language on which you want to execute the test case at Runtime
  • Enable Video (on-demand runs): Enables video recording of a test executed in the Functionize Cloud.
  • Ignore Certificate Errors: Ignores any Browser raised certificate errors (default ON for new and modified test cases created via Architect).
  • Continue on Verification Failure: Continues execution if a test fails on a Verify Action (may slow execution significantly, with only the first 3 failures displayed on the orchestration report page).
  • Maximum Verification Failures: Sets the maximum number of verification failures allowed on a test.
  • Continue On Error: Continues execution if a test Action (other than a Verify Action) fails (may slow execution significantly, with only the first 3 failures displayed on the orchestration report page).
  • Capture Network logs: Captures network logs (use for troubleshooting, as it can collect large amounts of data).
  • Capture Console logs: Captures console logs (use for troubleshooting, as it can collect large amounts of data).
  • Display PDF in Browser: Opens PDF files in the browser window if the test interacts with them.
  • Disable Self heal validation: Disables AI self-healing for test failures.
  • Disable Web Security: Turns off certain browser security mechanisms, allowing access to resources from any domain.
  • Disable iOS CORS: Disables Cross-Origin Resource Sharing restrictions on iOS devices.
  • Hide Address Bar: Allows the application to open in app mode, hiding the browser’s address bar for a more immersive testing experience.
  • Memory Saver Mode On: Reduces browser/application memory footprint to improve system performance.
  • Block 3rd Party Cookies: Blocks third-party cookies in all browsing modes.
  • Region: Select the Google Datacenter region in which all test cases are executed and after selecting the Region Functionize will display a message to a user “Functionize will try to run the test in the selected data center. If it’s unavailable, another data center in the same region will be used.”
  • Timezone: Select the Timezone for test case execution.
  • Window Width/Height: Set the window width and height for responsive sites (resets after the first step of the test).
  • Orientation: Select Portrait or Landscape page orientation.
  • Enable Screenshots: Captures screenshots (default ON, but can be turned off for sensitive data).
  • Environment Management: Replaces the environment's URL for the test execution.https://www.existing-site.com::https://www.replace-with-site.com
  • Proxy Settings: Configures a proxy server for accessing the application under test behind a firewall.
  • Enable Timing Model: Determines how long the AI Timing Model waits for an element to appear (Aggressive = faster, Conservative = slower).
  • Override Default Self-Heal: Overrides the default self-heal settings.
  • Self Heal: Uses machine learning to detect and update tests to reflect changes in the application or website.

Headers Tab

The Headers tab allows simulated data parameters, such as IP addresses, to create and deliver a specific experience at execution.

Cookies Tab

Inject cookies into a test case in the Cookies tab. For example, logging in a Tester without having to go through the login steps. Enter the Parameters for each cookie by typing in the cookie name and associated value.

Alerts Tab

The Alerts tab sends alerts via email if this option is selected. The Alerts Delivery drop-down menu allows a Tester to select parameters for when alerts are sent. Enter the email address(es) in the field, separating each with a comma.

Auth Tab

If the site under test uses HTTP authentication, the Tester can enter the following details under the Auth tab:

  • Initial Pageinit HTTP Authentication: Enter username:password.
  • MTLS client key: Enter client key.
  • MTLS client certificate (pem format): Enter client certificate.Note: Please see additional details on mTLS certificates here
  • Additional HTTP Authentication: Enter the URL without protocol, domain, username, and password.

When finished reviewing and editing your Test Case Settings, click Save.

Using the URL Override Setting

In certain testing scenarios, you might need to test multiple sites without creating separate tests for each one. Functionize provides a URL Override feature that allows you to replace the originally recorded URL with a different one, making it convenient to test in multiple environments. This feature also supports comma-separated lists to override multiple URLs per test.Note: URL Override will not function correctly without the leading https:// for each URL.

URL Override Steps

  1. Open your project and select the test you wish to modify to view the Test Details page.
  2. In the Settings menu, select Test Settings.
  3. Open the Advanced tab.
  4. In the Environment Management field, enter the new site(s) to test using the specified format.
  5. Click Save
  6. A success message will be displayed to confirm the change.
  7. The new URL will be tested during the next execution of the test. The Current URL will reflect the new site, compared to the original site listed in the Modeled and Previous Successful Run columns.

Field Input Format

Single URL Override:

Use the format string::replacement

Example: https://www.yoursite.com::https://qa.yoursite.com

Multiple URL Override:

Use the format string::replacement,string::replacement,string::replacement

Example: https://www.site.com::https://qa.site.com,https://www.site.com::https://dev.site.com

By utilizing the URL Override feature, you can efficiently test multiple environments without the need to create redundant tests, thereby saving time and ensuring consistency across your testing scenarios.

Recording Video of Test Executions

Functionize provides users with the ability to record videos of test executions. This feature can be particularly helpful for debugging test cases. Video recording must be enabled on a per-test basis within the Test Settings. Once enabled, any on-demand executions of the test on supported browsers will have videos recorded for visual debugging efforts.

Enabling Video Recording

From the Test Detail Page

  1. Open a Test:
    • Navigate to the test you want to enable video recording for.
  2. Access Test Settings:
    • Open the Settings menu and select Test Settings.
  3. Enable Video Recording:
    • Go to the Advanced tab in the Test Settings window.
    • Check the box next to Enable Video (on demand runs).
    • Click Save.

From the Test Listing Page

  1. Open a Project:
    • Navigate to the project containing the test you want to modify.
  2. Access Project Settings:
    • Open the Actions menu and select Settings.
  3. Enable Video Recording:
    • Go to the Advanced tab in the Settings window.
    • Check the box next to Enable Video (on demand runs).
    • Click Save.

Running an On-Demand Execution

Once video recording is enabled, run an on-demand execution of the test on one of your enabled browsers. The Video Playback button will appear after the first execution finishes and video is available. Return to the Test Detail page to access the video.

Note:

  • The playback button will not be visible on browser results tabs where the test has not yet been executed after enabling video.
  • The Video Playback button will remain visible once a video has been recorded, even if video recording is later disabled for the test.

Accessing Video Playback

From the Browser Results Tab

  1. Open the Browser Results Tab:
    • Navigate to the Browser Results tab for the browser where you ran the test.
  2. Click the Video Playback Button:
    • Click the Video Playback button to open the Watch Test Video window.
  3. Video Playback Options:
    • The Watch Test Video window allows you to:
      • View fullscreen
      • Download the video
      • Adjust playback speed
      • View the video as picture-in-picture

From the History Tab

  1. Open the History Tab:
    • Navigate to the History tab for the test.
  2. View Execution:
    • Click on the View icon of an execution to open the read-only view.
  3. Click the Video Playback Button:
    • Click the Video Playback button to open the Watch Test Video window.
    • The Watch Test Video window will open for viewing as on the browser tabs

By following these steps, you can enable video recording for your test executions, making it easier to debug and review test cases visually.

Configuring Learning Mode

Functionize's Learning Mode is designed to optimize the Machine Learning (ML) models that power our test automation by selectively learning from high-quality test executions. By enabling Learning Mode, you ensure that Functionize gathers data only from accurate and complete workflows, reducing the risk of false positives and improving the system's ability to adapt to changes on your site.

Why Learning Mode?

Previously, Functionize modeled and learned from every test, regardless of its pass or fail status. This approach could lead to longer learning times and inaccurate modeling if the system captured data from broken or failing tests. Learning Mode allows you to fine-tune your tests before the system begins to learn from them, ensuring that the ML models are built from high-quality data.

How It Works

  1. Test Refinement: Create and refine your test until it runs successfully and meets your requirements.
  2. First Successful Execution: After the first successful execution of a new test, you will be prompted to approve the test for learning.
  3. Approval: Approve the test for learning via the banner or the Test Settings.

Note: This feature is applicable only to tests created after the Functionize v5.2.2.1 release. Tests created prior to this release will not be affected and do not require Learning Mode approval.

Learning Mode Banner

After the first successful execution of a test, an approval banner will appear on the Test Detail page.

  1. Click Approve: If the test is ready for ML data gathering, click the Approve button.
  2. Ignore: To temporarily dismiss the message, click Ignore. The banner will reappear the next time you log in to Functionize.
  3. Don't Show Again: To permanently dismiss the message for this test, check the box for Don't Show notification again.

Enabling Learning Mode in Test Settings

  1. Open Settings Menu: Navigate to the Test Settings via the Settings Menu.
  2. General Tab: On the General tab, the setting Not Approved for Learning will be enabled by default, indicating that Learning Mode is not yet activated.
  3. Enable Learning Mode: Uncheck the setting Not Approved for Learning to enable Learning Mode and click Save.

Learning Mode Status

The Test Listing page will display a reminder under each test that has not been approved for learning.

Summary

By using Learning Mode, you ensure that Functionize's ML models are built from accurate and complete data, improving the system's ability to adapt to changes and reducing the risk of false positives. Enable Learning Mode once your test is refined to your satisfaction to start gathering high-quality data for your ML models.