View Orchestration Results

November 28, 2019 03:48

View Orchestration Results Overview

Orchestrations in Functionize allow you to execute test cases across multiple projects, making it particularly useful for end-to-end scenarios and system testing. An Orchestration Result is provided after a Single or Group Orchestration is executed. The results break down which individual tests failed or passed, and also show warnings or incomplete executions.

Orchestration Listing Page

Types of Actions in an Orchestration

Result
- Clicking on "Result" will display the results of that Orchestration.
- Total Execution Time
  This is the total time taken for the entire execution process, if there were no other limitations like concurrency
- Agent Execution Time
  This is when the VMs are actually executing test cases
- Agent Queue Time
  This is when the tests are blocked from execution because of a lack of concurrency
Edit
- Clicking on "Edit" opens the edit window, allowing you to modify all the details of the orchestration, such as name, environment, folder, test case scheduling, and more.
Audit Log
- Clicking on "Audit Log" allows you to see every type of change, including the name of the person who made the changes, under the Editor tab in that Orchestration.
Option Tab
- Clicking on the "Option" tab opens a dropdown with multiple options: Schedule, Clone, API calls, and Delete.
  - Schedule: Allows scheduling the execution of an Orchestration by selecting from the dropdown.
  - Clone: Opens a popup to create a clone of the Orchestration.
  - API Calls: Opens a popup window from which you can use API calls to remotely access this orchestration.
  - Delete: Opens a popup for deleting the Orchestration.

How to View Orchestration Results

Log into the Functionize app.
In the top navigation menu, select the Orchestrations tab.
The Orchestrations List page will display with the most recent Orchestration listed at the top by default. You can also filter by "Created by" or specific Project from the dropdown fields. The "Execution Details" column displays a quick-view results summary of each Orchestration.
To view more options or more detailed orchestration results, under the Actions column in the far-right column click the Results button for the specific Orchestration and select Orchestration Results

Orchestration Results Report

The Orchestration Results report opens in a new browser tab, displaying execution details such as:

All execution runs via a dropdown menu.
Basic results at a glance.
Results per browser.
Summary of each test in the orchestration.
Performance trends and action durations.

Export CSV for Downloading Results

The Export CSV feature in Functionize allows you to download a detailed summary of the execution results for your orchestrations. This feature is especially useful for reviewing test case outcomes, understanding failure points, and sharing results with stakeholders.

On the Orchestration Results page, you will see an Export CSV button on the right side.
Click the Export CSV button to download the results file

Details in the Downloaded CSV File

The downloaded CSV file contains comprehensive details about the orchestration execution. Here’s what you can expect to find in the CSV file:

Test Case: The name of the test case.
Project: The project to which the test case belongs.
Environment: The environment where the test was executed (e.g., live, staging).
Browser: The browser used for the test execution.
Dataset: The dataset used during the test run.
Dataset Data: The Dataset Date for the test executed.
Status: The outcome of the test (e.g., Passed, Failed).
Actions: The number of actions executed in the test case.
Execution Time: The total time taken to execute the test case.
Number of POs in Test: The number of Page Objects (POs) included in the test case.
Failing/Warning Step: The specific step where the test failed or a warning was generated.
PO Failing: The specific Page Object that caused the failure.
Failure/Warning Comments: Detailed comments about the failure or warning, including the expected and actual results.
Details: Links to detailed reports or logs for further investigation.

Notes

Status of Orchestration: The CSV file includes the status of each test case within the orchestration, clearly indicating whether it passed or failed.
Page Objects: If Page Objects are added, the number of Page Objects will be mentioned.
Failure Details: The specific step number where the test failed and the failure/warning comments provide detailed insights into what went wrong.

By utilizing the Export CSV feature, you can effectively analyze your test results, identify failure points, and ensure your test cases are functioning as expected. This feature enhances your ability to maintain high-quality standards in your testing processes.

Viewing More Details

Failed tests will include details at the point of failure which can be viewed by clicking the arrow button on the left side of the test in the summary section
Click the Details button for any test to open that test's execution Test Details page
A new tab will open with a read-only Test Detail page for this execution
The Information button displays basic statistics of the test
The Test Action menu contains options to download a PDF of the test or view performance reports
Click View Results or View button next to any action for a step-by-step screenshot, read-only view in Slider view
Actions may be submitted to our Train the AI feature in the Slider View
See User Guide: Report to AI

Rerun Individual Tests

Tests in an Orchestration can be rerun without executing the entire orchestration. Each test can be rerun irrespective of its Passed, Failed, or Warning status. However, currently, individual tests must be launched for reruns. Hover over the browser icon to select the option to rerun your test.

When a test case is rerun as part of an orchestration, the latest result will supersede older results for that browser and dataset (when applicable). Although both results will appear, integrations or notifications to test management tools, and results accessed via API, will only provide the most recent result for the browser and dataset combination.

Rerun test cases through Bulk Action

Tests in an Orchestration can be rerun without executing the entire orchestration. Multiple tests can be rerun irrespective of its Passed, Failed, or Warning status.

Select the test case(s) to be Rerun via the checkbox
Click on the the Bulk Actions dropdown and Click Rerun Test
Select the checkbox for the Browser in which to Rerun the Orchestration on and click Confirm

Orchestration Results History

The top of the Orchestration Results page contains several menus to filter your view. From this page you can view a different Orchestration, view a different Date/Time, filter by Tags, and filter by Browser. Viewing the history of your orchestration is as simple as selecting a previous run from the Date/Time drop-down on the left. The page will refresh to show the newly selected Orchestration Results report.

Orchestration Performance Trends and Action Duration Reports

The Orchestration Results page provides some high-level performance trends and action duration graphs to provide insight into site under test performance metrics. The following detailed graphs are provided as the user scrolls down through the Orchestration Results page:

Performance Trend Line

Displays the different performance metrics (i.e. Page Loading Time, DOM Interactive, Visual Completion, etc.) in a line graph for the test cases used in the Orchestration.

Page Performance

Displays the different performance metrics (i.e. Page Loading Time, DOM Interactive, Visual Completion, etc.) in a bar graph for the test cases used in the Orchestration.

Performance Details

Select the specific Performance metric (i.e. Page Load, DNS, Visual Completion, etc.) to view in the graph.

Action Duration

Select the specific test case from the Orchestration to view each action's, within the test case, duration to execute.