Nested Compositions and the Test Suite Dashboard

Nested Compositions and the Test Suite Dashboard are distinct features that can be used in tandem for debugging functional tests and to quick identify regression errors or issues. Both are relevant to functional testers using CloudTest and TouchTest.

Nested Compositions: The Nested Compositions feature introduced a new (CloudTest 53.05) clip element type, Launch Composition. This clip element can be placed into the test clip the same as any other clip element. Using this feature CloudTest and TouchTest functional testers can launch compositions from other compositions, and view Test Suite results in the new Test Suite Dashboard. The ability to leverage Jenkins CI and other CI tools remains undiminished.

Test Suite Dashboard: This System Dashboard (also introduced in CloudTest 53.05) is purpose-built for web and mobile Functional Testing and allows the tester to easily browse through the history of a test suite and the test compositions in it, right down to the action level where validations pass/fail, and where regressions can be detected and fixed.

Using the Launch Composition Clip Element: The Launch Composition clip element is a call to a Composition from another Composition (via a Test Clip in the calling Composition). A Composition is added into a Test Clip via the Clip Editor lower panel, Compositions tab, and the resulting clip element is a Launch Composition clip element.

Adding a Launch Compositions clip element into the Clip works precisely the same as adding any other clip element—double click the item to place it at the insertion point, or, drag it into place in the Clip Editor workspace.

More than one Launch Composition clip element can be placed into a Test Clip(s) in order to create a Test Suite.

Additionally, the Set Launch Composition context command can be used to set another Composition as a Launch Composition.

When this command is selected, the Choose a Composition box appears. Select the Composition to set as a Launch Composition.

Launch Compositions have all of the following capabilities:

  • They can Repeat, be placed in Transactions, have properties, and all of the things that Clip elements do.
  • They have properties including "whether or not to wait for the comp to complete", "whether failure should fail the parent", "optional name for the result", etc.
  • When the parent Composition that contains the Clip is played its result can be viewed in the Test Suite Dashboard (which loads by default); an individual result is also created for each Launch Composition.
  • Results of parent Compositions show the Launch Composition elements in the tree, but not its children.
Note: In this release, Device Selection is done in the nested composition itself and cannot be overridden in the parent composition.

Launch Composition Options

Launch Compositions, like Nested Clips, have their own in situ properties that are applicable only to the nested instance (these options are distinct from the stand-alone properties that apply to the stand-alone Composition). After adding a Launch Composition to your Clip, expand the new clip element in Clip Editor, List View to display its Options. Alternately, select the Launch Composition clip element and view it in the Selected: <""> panel.

  • Location – For non-functional tests specify the Maestro Location to use. Refer to Specifying Maestro Locations for more information. Locations doesn't apply to functional testing so functional testers can accept the default.
  • Errors Should – This drop-down specifies which failure action to take if an error occurs in the Launch Composition. Select one of the standard Failure Actions that apply to all clip elements
    • If the failure action is set to record only, the validation outcome will be recorded in the result, and the launch composition will continue whether the validation passes or fails.
    • If the failure action is set to fail the parent, the parent clip will stop if the validation fails.
    • Wait For Completion – Checked by default.
      • If checked, this Clip Element will not complete execution until the Composition completes.
      • If unchecked, this Clip Element will complete once the Composition has been started, and will not wait for its completion.
    • Verify Successful Play – Unchecked by default.
      • If checked, this Clip Element will fail if the launched Composition does not complete with status Completed. What then happens is determined by the value of the Failure Action attribute.
      • This attribute only has meaning if the Wait For Completion attribute is checked. If that attribute is unchecked, this attribute is not relevant.
    • Allow Partial Load – Unchecked by default.
      • Checked indicates that Composition play should proceed even if all of the necessary servers were not found. Locations doesn't apply to functional testing so functional testers can leave this unchecked.
    • Result Name – Optionally, specify an alternate name for the Result name (e.g. other than the default name).
    • Result Description – Optionally, specify a description for the result.

Building a Test Suite

Test Suites can be built in a number of ways, including adding all of the relevant Compositions as Launch Compositions into a test clip that resides in a parent test composition with no other test clips.

Playing this parent Composition creates a result for the Test Suite (although individual test composition results are also created).

More sophisticated Test Suites can be built that contain many such Clips, each of which contains one or more Launch Compositions. For example, a Test Suite for mobile apps and devices such as the example below can be arranged by Track, each of which corresponds to a device type, OS version, or to a network type. The Clips that have Launch Compositions are organized into the appropriate Track.

When you play a parent Composition (e.g. your Test Suite), the Composition Editor opens to the Play tab with the Test Suite Dashboard in display.

The Test Suite Dashboard presents functional data as a summary of the output from a "test suite." The Test Suite is used to analyze the results of a functional test across many results, and in the TouchTest use case, mobile devices. The Test Suite Dashboard incorporates key metrics, and the results of a functional test, including Success Summary [the X and number of successful Compositions, Clips, Validations]; Details about the Compositions and its test results [pass/fail, time/date of last test run], and the ability to drill down into details.

Test Suite Dashboard

As part of the Test Suite Dashboard, a new Widget Type category—Functional—has been added to the Widget Type list.

The Test Suite Dashboard includes the following widgets:

Functional Summary – This widget displays Summary Statistics, including the Success percentage of successful out of the total for Compositions, Validations, and Clips. The Functional Summary can also be added to custom dashboards.

Compositions Overview – This widget shows the components of the parent Composition and their history by component.

The Composition Overview widget includes the following columns:

  • A Composition column that can be expanded to drill into child components
  • A Last Result column that shows the most recent result (pass/fail) of the specific item for that line (e.g. in the result that was opened)
  • A Prior 10 Results column that shows the pass/fail result for each row before the Last Result (most recent from left to right).
  • A Start Time column that shows the last start time for the item in the row
  • A Time to Complete column that shows the total time that the item (a Composition, Clip, or Validation) last took to execute

The Compositions Overview tree starts from the Clips inside the parent Composition. Only Compositions, Clips, and Validations are shown here, while Tracks, Bands, Messages, App, and all grouping Clip Elements are omitted.

The selected dot in Last 10 Results drives the Result Details selection below.