Test results concepts
Here are all terms you may encounter when reading test run results.
The concepts are arranged in the order you may encounter them in the Runs and results tab.
- Test results
- Test results for functional testing is a comparison of the expected and the actual value. It can be either Passed or Failed. For comparative testing, it is a list of differences, Diffs, found for the comparisons you set in test cases within test definitions for which the test was run. See also: How to read test run results.
- Passed test
- In functional testing result, the Passed result means that the Expected result of the test run was the same as the Actual result.
- Failed test
- In functional testing result, the Fail result means that the Expected result of the test run was different from the Actual result and the implemented changes need to be further investigated.
- Diffs are results of comparative testing. They are differences detected by
Test Center when comparing your current website configuration with the one you want to
implement. They are neither good nor bad, some of them might indicate expected outcomes of
the configuration change you are testing. It’s up to you to decide whether the diff was
expected or unexpected. The expected diff is what you want to achieve by implementing
tested configuration changes. An unexpected diff might indicate a configuration error
that, if activated on production, could lead to a denial of service or other production
issues. Such unexpected diffs should be marked as not accepted and investigated.
For example, if you updated the time-to-live for a file name, then you would expect to find a caching option diff in the results for that file name. In that case, you accept this diff to signify that it indicates expected behavior and needs no further investigation.
- Diff accepted
- An accepted diff is a detected difference that was expected by configuration changes.
- Diff unaccepted
- An unaccepted diff is a detected difference that was unexpected by configuration changes and needs to be further investigated.
- Error occurred
- Test results, the Error occurred message means that the test could not be executed because of an error. To learn more about the error, you need to hover over the message on the interface.