Understanding Metrics in CloudTest Analytics

The following table defines CloudTest Analytic metrics as they appear in the Results Details and other dashboard widgets in test results. Definitions are categorized by the order of hierarchy in the test composition itself, first by action, then by Containers inside of a Clip, and then by the Clip itself).

Metric Level User Interface Name Definition
Actions

Metrics for an individual action

Start Time The time that play started as an offset from the start of the test, in milliseconds (ms).
Delta From Scheduled Time If in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms).
Response Time The complete amount of time spent establishing the connection, sending the request, and receiving and processing the response. Excludes any time spent waiting for a connection to be available from the internal connection pool. If no response is received from the server, then this will be 0.

For WebSocket Send, the response time is very small or zero because sending is handled asynchronously.

For WebSocket Message (received), the response time will depend on whether a server-side timestamp (set by the target server) is present in the response.

  • If a timestamp is present in the response, the response time measures from the timestamp to CloudTest receiving the message.
  • If there is no timestamp in the response, the response time is very small or zero.
Bytes Sent The total number of bytes sent to the remote server, including the request line, the headers, the body, and all separating whitespace.
Bytes Received The total number of bytes received from the remote server, including the status line, the headers, the body, and all separating whitespace. If the response requires any post-processing (e.g. decompression or binary-to-text-conversion), this will be the number of bytes *before* the post-processing took place.
Retries due to unavailable local ports The total number of times that the request needed to be re-tried due to insufficient local ports on the CloudTest server. Typically, 0.
Retry time due to unavailable local ports The total amount of time that the request spent on requests that failed due to insufficient local ports on the CloudTest server. Does not include the time spent on the request that succeeded. Typically, 0.
Connection Establish Time The amount of time spent establishing the HTTP connection, including the SSL handshake (if any) and DNS lookup. If a previously-opened connection is re-used, then this will be 0.
Send Time The total amount of time spent sending the request content to the server (including request line, headers, and body). Does not include any of the following: establishing the HTTP connection, waiting for the server to respond, or processing the server response.
Receive Time The total amount of time spent receiving and processing the response content from the server. The term "processing" includes de-compressing (if the response was compressed), performing any binary-to-text conversions (e.g. AMF), and/or parsing the bytes according to the selected character set.
Total Time The total amount of time spent sending a Message and receiving the response. Includes "Send Time", "Receive Time" and any necessary re-tries. Messages that did not receive a response due to connection timeouts or other errors, will still have a total time. For Non-Messages, the Total Time is the time to complete the action. Delays, Browser Actions, App Actions, and Database Actions will have a total time.
TTFB The amount of time from the start of the request to the first byte being received from the server. Includes the following: establishing the HTTP connection, sending the request to the remote server, and waiting for the first response byte.
TTLB The amount of time from the start of the request to the last byte being received from the server. Includes the following: establishing the HTTP connection, sending the request to the remote server, and receiving the complete response. Does not include response "processing" (e.g. de-compressing, etc.).
DNS Lookup Time The amount of time spent resolving the host name using DNS. If a cached DNS result is re-used, or if the request is being sent directly to an IP address, then this will be 0. This is a subset of "Connection Establish Time".
Wait for Connection Pool Time The amount of time spent waiting for a free connection from the HTTP connection pool. If there is no need to wait (i.e. new connection is opened, or free connection is available immediately), then this will be 0.
Status Completed, Stopped or Failed.
Containers inside of a Clip

Metrics for an individual Container inside of a Clip

Start The time that play started as an offset from the start of the test, in milliseconds (ms).
Delta From Scheduled Time If in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms).
Full Duration (Shown by default as Effective Duration) The length of time for the container to start until it completed, in milliseconds (ms). This is not the time for the elements within the container. The duration can be greater than or less than the sum of the elements within it. For example, a Page container will contain Messages which run in parallel so the sum will often be greater than the duration.
Message Play Count The number of Messages that were completed in any way (success or fail) in this container and any containers it contains
Browser Action Play Count The number of Browser Actions that were completed in any way (success or fail) in this container and any containers it contains
App Action Play Count The number of App Actions that were completed in any way (success or fail) in this container and any containers it contains
Database Action Play Count The number of Database Actions that were completed in any way (success or fail) in this container and any containers it contains
Script Play Count The number of Scripts that were completed in any way (success or fail) in this container and any containers it contains
Response Count The number of any type of operation (Message, Browser Action, App Actions, Database Action) that completed a "request" (success or fail). This number could be different than the number of Actions completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Messages being played going up, but not in the Response Count going up, since no request could be sent. For actions in this container and containers it contains.
Min Response Time The smallest value of the clip element’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Max Response Time The largest value of the clip element’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
Total Response Time The sum of all of the clip element’s duration values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count). Note that actions can execute in parallel, especially in Pages, so this sum can be longer than the duration of the container itself.
Base Page Average Response Time The first clip element in a Page container is the Base Page (e.g. index.html). The Base Page Avg Response Time is the mean time to get that base page not including its assets (i.e. .png, .js, .xml, etc.).
Total Bytes Sent The sum of all of the Total Bytes Sent values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
Total Bytes Received The sum of all of the clip element's Total Bytes Received values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Effective Message Throughput The number of messages per second, and bytes per second sent and bytes per second received, for all messages in this container and containers it contains that did complete a request.
Min Port Retry Count The smallest value of the clip element's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Max Port Retry Count The largest value of the clip element's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Total Port Retry Count The sum of all of the clip element's Total Port Retry Count values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Min Port Retry Time The smallest value of the clip element's Total Port Retry Time for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Max Port Retry Time The largest value of the clip element's Total Port Retry Timet for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Total Port Retry Time The sum of all of the clip element's Total Port Retry Time values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Delay Play Count The number of Delay Clip Elements that were completed in any way (success or fail) in this container and any containers it contains
Main Message Response Count The number of Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). This number could be different than the number of Pages completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Pages being played going up, but not in the Response Count going up, since no request could be sent even though the Page did start. For Pages in this container and containers it contains.
Main Message Response Time The sum of all of the clip element’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
Effective Duration The total amount of time, in milliseconds (ms), that this container was "effective" (doing work). Doesn't include the time between items (such as inter-item delays or inter-repeat delays). Takes into account overlapping executions -- the effective time is when at least one or more Clip Elements are executing anywhere within the container or its child containers.
Clip

Metrics for an individual Clip

Start The time that play started as an offset from the start of the test, in milliseconds (ms).
Response Count The number of any type of action (Message, Browser Action, App Actions, Database Action) that completed a "request" (success or fail). This number could be different than the number of Actions completed. For example, a Message might have failed to create the request due to some error, such as an improper ISSE in the request. This would result in the count of Messages being played going up, but not in the Response Count going up, since no request could be sent. For actions in this clip and containers it contains.
Min Response Time The smallest value of the clip’s duration for all actions in this container and containers it contains that did complete a request (see the collection's Response Count).
Max Response Time The largest value of the clip’s duration for all actions in the clip and containers it contains that did complete a request (see the collection's Response Count).
Total Response Time The sum of all of the clip’s duration values for all actions in this container and containers it contains that did complete a request (see the collection's Response Count). Note that actions can execute in parallel, especially in Pages, so this sum can be longer than the duration of the container itself.
Duration The length of time for the container to start until it completed, in ms. This is not the time for the elements within the container. The duration can be greater than or less than the sum of the elements within it. For example, a Page container will contain Messages which run in parallel so the sum will often be greater than the duration.
Delta From Scheduled Time If in a timed container, the offset of the actual start time from the scheduled time, in milliseconds (ms). Generally, used only for timed clips.
Delta Scheduled Duration The difference between the scheduled duration and the actual one. Generally, used only for timed clips.
Total Bytes Sent The sum of all of the Total Bytes Sent values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count).
Total Bytes Received The sum of all of the clip's Total Bytes Received values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count).
Min Port Retry Count The smallest value of the clip's Total Port Retry Time for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
Max Port Retry Count The largest value of the clip's Total Port Retry Count for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Total Port Retry Count The sum of all of the clip's Total Port Retry Count values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
Min Port Retry Time The smallest value of the clip's Total Port Retry Time for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
Max Port Retry Time The largest value of the clip's Total Port Retry Timet for all actions in this container and containers it contains that did complete a request (see the collection's Response Count)
Total Port Retry Time The sum of all of the clip's Total Port Retry Time values for all actions in this clip and containers it contains that did complete a request (see the collection's Response Count)
Start Offset from Container The offset of this clip's start time from its parent's start time, in milliseconds (ms).
Total Main Message Response Count The sum of all of the clip’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
Total Main Message Response Time The sum of all of the clip’s duration values for all Messages in a Main (Document) section of a Page that completed a "request" (success or fail) (see the collection's Response Count). For all Pages in this container and containers it contains.
Effective Duration The total amount of time, in milliseconds (ms), that this clip was "effective" (doing work). Doesn't include the time between items (such as inter-item delays or inter-repeat delays). Takes into account overlapping executions—the effective time is when at least one or more Clip Elements are executing anywhere within the container or its child containers.
Errors in Descendants