CloudTest produces several types of test result objects in order to give both detailed and aggregated actionable information about tests. The following are the types of result objects which are available:
General – Basic information such as a result name, a description, start and end times for the test, and the total number of HTTP messages, browser actions, scripts and other objects which were executed during the test.
Message Details – This contains metrics for each HTTP/SOAP Message, Browser Action, Script, Checkpoint, or Delay that were run during the test. This includes information such as time sent, response time, bytes sent and received as well as the container object information for Bands/Tracks/Test Clips/and Chains.
Log – This object contains detailed even information for the test including when the Test Composition started, when Messages are sent, when responses are received, and whether the Test Clip failed or not, as well as other statistical information about the test.
General – The general result object is contained in the Soasta_repository database in a table named PlaybackResult. It contains the following columns:
- Id – Unique primary key value for the result.
- Result Name – this can be changed in the Repository View but by default a result name is composed of Result from followed by a date. For example: Result from Wed Mar 12 09:27:29 PDT 2008.
- Short Description – Not currently used.
- Long Description - Full text description of the result. This can be changed in the Repository View.
- Start Time – The time the test started.
- End Time – The time the test completed.
- Total Time – the total test time expressed in milliseconds.
- Status – The current status of the test such as Completed, Failed, Stopped, or Playing.
- Summary - For a test that had errors this will contain a text description of the error which caused the test to fail.
- Total Messages – the number of messages sent in the test.
- Total Checkpoints – the number of checkpoints executed in the test
- Total Transitions – the number of scripts executed in the test
- Total Browser Actions – the number of browser actions executed in the test
- Total Events – the number of logging events. Events are detailed actions which are recorded during the test such as sending a message, completion of a track, or starting of a monitor.
- Total Errors – the number of errors in the test.
- Total Delays – the number of timing delays executed in the test
- Indexed – not used.
- OwnerID – the id of the user who ran the test
- Created – the date and time the test result was created.
- Last Modified – the date and time the test result was last modified.
- Compostion VersionId - The internal version number for the test composition associated with this result.
- Content – The Play Settings for the Test Composition.
- Results Service Id – the foreign key that links the data in the Soasta_resultsservice database to the result.
Message Details - The message details are contained in a star-schema format. There is a set of tables with names of the form xxxx9999 where xxxx is the fact or dimension table name and 9999 is a result id number.
The MessageFact table contains the important metrics for each http message, browser action or script run in a test. The columns are defined below:
- Message_key – unique primary key
- Band_key - key to Band Dimension table
- Track_key – key to Track Dimension table
- Clip_key – key to Clip Dimension table
- Chain_key – key to Chain Dimension table
- Clip_element_key – key to Clip Element Dimension table
- Target_key – key to Target Dimension table
- Server_key – key to Server Dimension table
- Band, Track, Test Clip, Chain, ClipElement Index columns – when an element has a repeat property such as the number of Virtual Users for a Track, the index value reflects that instance of the repetition. Indexes are 0 based.
- ClipElementType – A code that indicates the type of test clip element. Values are:
6 = Message 7 = Script 8 = Checkpoint 9 = Delay 10 = Browser Action
- Start Time – the time that the message was sent, in milliseconds, measured from the start of the Test Composition.
- DeltaFromScheduledTime – the difference between the time the message was scheduled to be sent and when it was actually sent. In milliseconds.
- Response Time – The difference between when the message was sent and when the response was received, in milliseconds. The Response Time can be broken down into three components – Send Time, Receive Time, and Connection Establishment Time.
- BytesSent – The number of bytes of the message.
- BytesReceived – The number of bytes in the response
- Send Time – the time it takes to send the message.
- Receive Time – the time from when the message completed sending to when the response is received.
- ConnectionEstablishTime – the time it takes to create the socket
- ValidationError – Set to 1 if there was a validation error, 0 for a validation passing, and blank for no validation
There are seven Dimension tables which are linked to the Fact tables. These are:
- Band Dimension – The band associated with any given message
- Track Dimension – The track associated with any given message
- Clip Dimension – The Test Clip associated with any given message
- Chain Dimension – The chain associated with any given message
- Clip Element Dimension – The Test Clip element associated with any given message
- Target Dimension – The target associated with any given message
- Server Dimension – The Maestro server that sent any given message
Most of these contain the same types of data which are listed below along with some dimension specific additions:
- Key – Primary key to the table
- Name – The external name such as the Track Name or Test Clip Name
- Definition – The definition for the component at the time the test was run stored in an XML format.
- In addition, the Test Clip, Clip Element, and Target Dimensions contain the directory path for the Test Clip as well as the Local Name which is unique in the Test Composition when the same clip/clip element/target is used more than once.
- The ClipElement dimension also contains the name of the Test Clip it is contained in; a local name, and the HTTP operation that corresponds to the message.
- The Server Dimension also contains the Location Name and Server Name and URL that correspond to the Locations and Servers for the Maestro servers that ran the test.
- The Target Dimension also contains the MimeType and an associated image for the target. The image is stored as a BLOB.
Log – The Event Log table contains event information for the test such as Starting Chain, and it also contains the actual messages sent and received. The amount and type of event data that is captured is governed by the Test Composition Logging level settings. The log is stored as an additional fact table named EventFact using the same Dimensions described above for the Message Fact table.
- Event_key – Primary key for the event
- Band_key, Track_key, Clip_key, Chain_key, Clip_element_key, Target_key, Server_key – See Message Fact table above.
- BandIndex, Trackindex, Chainindex, ClipElementIndex – see Message Fact table above.
- EventNumber – A sequential number assigned by the Maestro server for the event. These numbers are unique but differ from the Event_key because events can reach the Results Service in a different order than they occurred in.
- Level -
- EventTime – The time in milliseconds, from the start of the Test Composition.
- Code – The type of information contained in this row of data. Values are:
- start – composition/band/track/clip/chain starting;
- end – composition/band/track/clip/chain ended;
- fail – when a component fails for any reason;
- send – message is being sent;
- sent – message has been sent;
- open – a target is being opened;
- close – a target was closed;
- gc – java garbage collection was forced to occur;
- machinef – information about machines associated with the test;
- mnstart – a monitor was started;
- mnstop – a monitor was stopped;
- stats – statistics associated with the component;
- exstats – additional statistics;
- Source – the source of the event. Valid values are:
- Transport – sending or receiving a message
- Text – A description of the event as specified by the Maestro server.
- Details1, Details2, and Details3 – Can contain the actual message sent or received; the URL for a target; or other relevant information.
- Data – Can contain the contents of the message posted in an XML format. Stored in a Blob field.
- ResponseCount – The number of responses received for the given component.
- MinResponseTime, MaxResponseTime, TotalResponseTime – The minimum, maximum, and total response times for the given component.
- MinPortRetryCount, MaxPortRetryCount, TotalPortRetryCount – The min, max, and total number of retries for sending messages because of sending errors, for the given component.
- MinPortRetryTime, MaxPortRetryTime, TotalPortRetryTime – The min, max, and total time it took to retry sending messages because of sending errors, for a given component.
- TotalBytesReceived, TotalBytesSent – The total number of bytes sent/received for the given component.
- Duration – The difference in time between when the component started and when it completed. For a test clip or chain this might represent the duration of a user scenerio.
- StartTime – The offset into the Test Composition that the event occurred.
- MessagePlayCount , BrowserActionPlayCount, ScriptPlayCount, CheckpointPlayCount, DelayPlayCount – Counts for the number of each type of Clip Elements in the test.
- MaxThreads – The maximum number of threads used by Maestro in the test. This is blank for components other than the Test Composition component.
- EffectiveDuration – The difference in time between when the Test Composition effectively started and when it effectively ended. The difference between the EffectiveDuration and the Duration represent the overhead of running the Test Composition and saving the results.
- DeltaScheduledDuration – Some message test clips have an expected duration – for example the length of an evenly spaced test clip. This field shows any difference between that expected duration and the actual duration.
- DeltaScheduledStart – The difference in time between the time the test clip element was scheduled to be sent and when it was sent. This difference can occur for many different reasons such as the response time of earlier messages or the load on the Maestro server.