1. Functionize
  2. User Guide
  3. Test Executions & Orchestrations
  4. Orchestrations

View Orchestration Results

Follow

View Orchestration Results Overview

Orchestrations in Functionize allow you to execute test cases across multiple projects, making it particularly useful for end-to-end scenarios and system testing. An Orchestration Result is provided after a Single or Group Orchestration is executed. The results break down which individual tests failed or passed, and also show warnings or incomplete executions.

Orchestration Listing Page

Types of Actions in an Orchestration

  1. Result

    • Clicking on "Result" will display the results of that Orchestration.
    • Total Execution Time

      This is the overall time taken by the queue to complete execution. It represents the sum of all stages in the process, from the first test case being picked up by an agent to the completion of the last test case.


    • Agent Availability Time

      This refers to the time during which one or more agents were actively available and executing test cases. This time begins when free agents are assigned to a test case and continues until all the cases in the queue are executed.


    • Agent Queue Time

      This is the time when the queue was blocked because no agents were available to pick up new test cases. It reflects the portion of time where all agents were occupied, causing a delay in the execution process as the queue awaited agent availability.


  2. Edit

    • Clicking on "Edit" opens the edit window, allowing you to modify all the details of the orchestration, such as name, environment, folder, test case scheduling, and more.

  3. Audit Log

    • Clicking on "Audit Log" allows you to see every type of change, including the name of the person who made the changes, under the Editor tab in that Orchestration.

  4. Option Tab

    • Clicking on the "Option" tab opens a dropdown with multiple options: Reorder, Clone, API calls, and Delete.

      • Reorder: Allows to Reorder the test case for execution if the run order is selected as sequential
      • Clone: Opens a popup to create a clone of the Orchestration.
      • API Calls: Opens a popup window from which you can use API calls to remotely access this orchestration.
      • Delete: Opens a popup for deleting the Orchestration.


How to View Orchestration Results

  1. Log into the Functionize app.
  2. In the top navigation menu, select the Orchestrations tab.

  3. The Orchestrations List page will display with the most recent Orchestration listed at the top by default. You can also filter by "Created by" or specific Project from the dropdown fields. The "Execution Details" column displays a quick-view results summary of each Orchestration.
  4. To view more options or more detailed orchestration results, under the Actions column in the far-right column click the Results button for the specific Orchestration and select Orchestration Results

Orchestration Results Report

The Orchestration Results report opens in a new browser tab, displaying execution details such as:

  • All execution runs via a dropdown menu.
  • Basic results at a glance.
  • Results per browser.
  • Summary of each test in the orchestration.
  • Performance trends and action durations.


Export CSV for Downloading Results

The Export CSV feature in Functionize allows you to download a detailed summary of the execution results for your orchestrations. This feature is especially useful for reviewing test case outcomes, understanding failure points, and sharing results with stakeholders.

  • On the Orchestration Results page, you will see an Export CSV button on the right side.
  • Click the Export CSV button to download the results file

Details in the Downloaded CSV File

The downloaded CSV file contains comprehensive details about the orchestration execution. Here’s what you can expect to find in the CSV file:

  1. Test Case: The name of the test case.
  2. Project: The project to which the test case belongs.
  3. Environment: The environment where the test was executed (e.g., live, staging).
  4. CreatedBy: The "Created By" field represents the name of the user who created an Orchestration.
  5. Browser: The browser used for the test execution.
  6. Dataset: The dataset used during the test run.
  7. Dataset Data: The Dataset Date for the test executed.
  8. Status: The outcome of the test (e.g., Passed, Failed).
  9. Actions: The number of actions executed in the test case.
  10. No. of actions Passed: This total Number of test actions that have successfully executed without any errors or failures.
  11. No. of actions Failed: This total Number of test actions that encountered failures during execution.
  12. Execution Time: The total time taken to execute the test case.
  13. Number of POs in Test: The number of Page Objects (POs) included in the test case.
  14. Failing/Warning Step: The specific step where the test failed or a warning was generated.
  15. PO Failing: The specific Page Object that caused the failure.
  16. Failure/Warning Comments: Detailed comments about the failure or warning, including the expected and actual results.
  17. Details: Links to detailed reports or logs for further investigation.

Notes

  • Status of Orchestration: The CSV file includes the status of each test case within the orchestration, clearly indicating whether it passed or failed.
  • Page Objects: If Page Objects are added, the number of Page Objects will be mentioned.
  • Failure Details: The specific step number where the test failed and the failure/warning comments provide detailed insights into what went wrong.

By utilizing the Export CSV feature, you can effectively analyze your test results, identify failure points, and ensure your test cases are functioning as expected. This feature enhances your ability to maintain high-quality standards in your testing processes.

 

Viewing More Details

  1. Failed tests will include details at the point of failure which can be viewed by clicking the arrow button on the left side of the test in the summary section


  2. Click the Details button for any test to open that test's execution Test Details page


  3. A new tab will open with a read-only Test Detail page for this execution


  4. The Information button displays basic statistics of the test


  5. The Action menu contains options to download a PDF of the test or view performance reports


  6. Click View Results or View button next to any action for a step-by-step screenshot, read-only view in Slider view


  7. Actions may be submitted to our Train the AI feature in the Slider View
    See User Guide: Report to AI

Rerun Individual Tests

Tests in an Orchestration can be rerun without executing the entire orchestration. Each test can be rerun irrespective of its Passed, Failed, or Warning status. However, currently, individual tests must be launched for reruns. Hover over the browser icon to select the option to rerun your test.

When a test case is rerun as part of an orchestration, the latest result will supersede older results for that browser and dataset (when applicable). Although both results will appear, integrations or notifications to test management tools, and results accessed via API, will only provide the most recent result for the browser and dataset combination.

 

Rerun test cases through Bulk Action 

Tests in an Orchestration can be rerun without executing the entire orchestration. Multiple tests can be rerun irrespective of its Passed, Failed, or Warning status. 

  • Select the test case(s) to be Rerun via the checkbox



  • Click on the the Bulk Actions dropdown and Click Rerun Test



  • Select the checkbox for the Browser in which to Rerun the Orchestration on and click Confirm

 

Orchestration Results History

The top of the Orchestration Results page contains several menus to filter your view. From this page you can view a different Orchestration, view a different Date/Time, filter by Tags, and filter by Browser. Viewing the history of your orchestration is as simple as selecting a previous run from the Date/Time drop-down on the left. The page will refresh to show the newly selected Orchestration Results report.

 

Orchestration Performance Trends and Action Duration Reports

The Orchestration Results page provides some high-level performance trends and action duration graphs to provide insight into site under test performance metrics.  The following detailed graphs are provided as the user scrolls down through the Orchestration Results page:

 

Performance Trend Line

Displays the different performance metrics (i.e. Page Loading Time, DOM Interactive, Visual Completion, etc.) in a line graph for the test cases used in the Orchestration.


Page Performance

Displays the different performance metrics (i.e. Page Loading Time, DOM Interactive, Visual Completion, etc.) in a bar graph for the test cases used in the Orchestration.


Performance Details

Select the specific Performance metric (i.e. Page Load, DNS, Visual Completion, etc.) to view in the graph.


Action Duration

Select the specific test case from the Orchestration to view each action's, within the test case, duration to execute.

Related articles