CMAF qualification checklist

Akamai uses the following test cases to qualify CMAF ingest.

CMAF qualification tests
Test Description Notes
Basic streams test (DASH) Check that the MPD file is downloadable.

Verify that the MPD@type attribute is present and it’s value is dynamic (not static).
Basic streams test (HLS)

Verify if the first line of the bitrate playlist files is #EXTM3U. Check if bitrate playlist files have the tags #EXT-X-TARGETDURATION, #EXT-X-MEDIA-SEQUENCE.

Verify that the number of streams received is the same as the number of streams expected.

 See Bitrate and playlist.
Bit rate playlist alignment Verify that the tag EXT_X_MEDIA_SEQUENCE is consistent across playlists.
CMAF supported segments or bitrate URL file

Check that the segment numbers are sequentially increasing.

Verify if response received is 200 by requesting random ts segments (first one, last one, middle one) from the playlist.

Check that the EXT-X-MEDIA-SEQUENCE corresponds to first segment in the bitrate playlist.

Verify whether any #EXTINF has negative values in the bitrate m3u8 files.

 See Unique segment numbering
DASH spec conformance.

Verify that the @publishTime and @availabilityStartTime attributes are present. According to the spec, for @type=dynamic, these attributes must be present.

Check if the @minimumUpdatePeriod attribute is present.
Error recovery test Verify the ts segment IDs after reconnect.

The encoder must use the Testnet DNS. If it uses only a host file for resolution, it will not work and will fail.
Error recovery test (failover to a different entry point – DNS lookup test) The transition to the secondary entry point should be seamless.
Error recovery test (failover to a different EP – DNS lookup test) The transition to the secondary EP should be seamless.
Error recovery test (streams reconnect to same EP) Verify if the streams reconnect successfully to the same EP.
HTTP request type Check that the HTTP request type for the ts segments is either PUT or POST.
Long duration publish Verify playback and analyze the stream publish after 48 hours of publishing.
Master playlist upload interval Verify that the master playlist file does not get uploaded too frequently. See Upload order (HLS) and

Upload order (DASH).
MPD@availabilityStartTime is constant Verify that the value of the attribute MPD@availabilityStartTime does not change when you publish. The availability start time should not change during the course of publishing, unless restarted on purpose.
MPD file update interval Check that the MPD file is not uploaded too frequently or too late. An update interval of about 30 seconds is ideal. Failure in the test will usually trigger the team to evaluate more deeply and consider why the vendor is sending frequent MPD updates.
Playback Verify playback on CMAF supported players. See Playback workflow (HLS).
Playback Verify playback on HLS and DASH players. Playback workflow (HLS) and Playback workflow (DASH).
Segment names Check that the segment names in the URL are according to the ones advertised in the MPD file. If segments uploaded are not the same as advertised by the derived MPD calculation, this test will not pass.
Segment retry on 5xx error and verify retry interval Check retry behaviour of encoder after recieving 5xx response code.

Expect the encoder to retry the segment within segment duration. This is required to support usage with IAS.
Segment retry on 5xx error, and verify retry interval Verify that the encoder re-tries a segment at least once when any other 5xx response is sent back.
Stream alignment Check that the timestamps across representations / bitrates are aligned.
Unique path on republish When an encoder stops the publish and starts another one, the path should be unique (not mandatory for MSL 4).
Upload interval check for bitrate playlists Verify that the bitrate playlist upload interval is almost equal to the segment upload interval.
Upload interval check for segments Verify that the segment upload interval is almost equal to the advertised values.
Upload interval check for segments Verify the segment upload interval is almost equal to the advertised values.
User-Agent string Verify that the encoder is sending a valid User-Agent string. The User-Agent string format should be similar to Vendor Model version #. See User-Agent string.
User-Agent string Verify that the Encoder is sending a valid User-Agent string.