CMAF qualification checklist

Akamai uses the following test cases to qualify CMAF ingest.

CMAF qualification tests
Test Description
Basic streams test (DASH) Check that the MPD file is downloadable.

Verify that the MPD@type attribute is present and it’s value is dynamic (not static).
Basic streams test (HLS)

Verify if the first line of the bitrate playlist files is #EXTM3U. Check if bitrate playlist files have the tags #EXT-X-TARGETDURATION, #EXT-X-MEDIA-SEQUENCE.

Verify that the number of streams received is the same as the number of streams expected.
Bit rate playlist alignment Verify that the tag EXT_X_MEDIA_SEQUENCE is consistent across playlists.
CMAF supported segments or bitrate URL file

Check that the segment numbers are sequentially increasing.

Verify if response received is 200 by requesting random ts segments (first one, last one, middle one) from the playlist.

Check that the EXT-X-MEDIA-SEQUENCE corresponds to first segment in the bitrate playlist.

Verify whether any #EXTINF has negative values in the bitrate m3u8 files.
DASH spec conformance.

Verify that the @publishTime and @availabilityStartTime attributes are present. According to the spec, for @type=dynamic, these attributes must be present.

Check if the @minimumUpdatePeriod attribute is present.
Error recovery test Verify the ts segment IDs after reconnect.

The encoder must use the Testnet DNS. If it uses only a host file for resolution, it will not work and will fail.
Error recovery test (failover to a different entry point – DNS lookup test) The transition to the secondary entry point should be seamless.
Error recovery test (failover to a different EP – DNS lookup test) The transition to the secondary EP should be seamless.
Error recovery test (streams reconnect to same EP) Verify if the streams reconnect successfully to the same EP.
HTTP request type Check that the HTTP request type for the ts segments is either PUT or POST.
Long duration publish Verify playback and analyze the stream publish after 48 hours of publishing.
Master playlist upload interval Verify that the master playlist file does not get uploaded too frequently. See Upload order for HLS and Upload order (DASH).
MPD@availabilityStartTime is constant Verify that the value of the attribute MPD@availabilityStartTime does not change when you publish. The availability start time should not change during the course of publishing, unless restarted on purpose.
MPD file update interval Check that the MPD file is not uploaded too frequently or too late. An update interval of about 30 seconds is ideal. Failure in the test will usually trigger the team to evaluate more deeply and consider why the vendor is sending frequent MPD updates.
Playback Verify playback on CMAF supported players. Refer to Playback URLs (CMAF)
Playback Verify playback on Primary and backup HLS stream URLs and Primary and backup DASH stream URLs players.
Segment names Check that the segment names in the URL are according to the ones advertised in the MPD file. If segments uploaded are not the same as advertised by the derived MPD calculation, this test will not pass.
Segment retry on 5xx error and verify retry interval Check the retry behavior of the encoder after receiving a 5xx response code. Expect the encoder to retry the segment within the segment duration.
Segment retry on 5xx error, and verify retry interval Verify that the encoder re-tries a segment at least once when any other 5xx response is sent back.
Stream alignment Check that the timestamps across representations / bitrates are aligned.
Unique path on republish When an encoder stops the publish and starts another one, the path should be unique (not mandatory for MSL 4).
Upload interval check for bitrate playlists Verify that the bit rate playlist upload interval is almost equal to the segment upload interval.
Upload interval check for segments Verify that the segment upload interval is almost equal to the advertised values.
Upload interval check for segments Verify the segment upload interval is almost equal to the advertised values.
User-Agent string Verify that the encoder is sending a valid User-Agent string. The User-Agent string format should be similar to Vendor Model version #.
User-Agent string Verify that the Encoder is sending a valid User-Agent string.
Note: For details, refer to HTTP ingest requirements