Overview
A test case may execute successfully when run locally but fail when executed as part of an orchestration. This discrepancy often points to differences in the execution environment between the two run types.
How It Works
A primary cause for a test failing in an orchestration while passing locally is the use of an outdated runtime version. Local runs might default to the latest runtime, which includes the most recent features and bug fixes. However, an orchestration might be configured to use an older version. If a test's functionality relies on an update or fix present only in a newer runtime, it will fail when executed on the older version.
For example, a step that fails to save a value during an orchestrated run might be due to the test case executing on an older runtime. Updating the test case to the latest runtime version can resolve the issue, allowing the step to execute successfully.
Limitations
This solution specifically addresses failures caused by runtime version discrepancies. Other factors can also cause a test to fail in an orchestration but not locally, such as differences in test data, application environment configurations, or timing issues that only appear during a full orchestration.
Related Info
It is best practice to ensure that test cases are updated to the latest runtime version before being added to an orchestration. If a test continues to fail in an orchestration after a runtime update, further investigation into environmental or data-related differences between the local and orchestration execution environments is recommended.