Serve from Zip considerations

There are some issues to bear in mind when using Serve From Zip.

Recommendations and limitations for indexed zip files

Consider storing your zip files in different upload directories if performance is impacted by serving indexed content. These limitations apply to indexed zip files:

  • Don't use NetStorage as your master copy. Indexed zip files stored on NetStorage has a different MD5 than what was uploaded. You won't be able to verify going back.
  • Don't check the MD5 or files size of indexed files on NetStorage. Use the return response from most protocol clients to confirm a successful transfer.
  • Don't use Rsync (or Aspera sync) with indexed zip files. Using rsync to upload new zip files is okay, but all other rsync options require that files sizes match. Because the index added to the NetStorage zip files will always make the file different, rsync will continuously upload files resulting in lots of extra bandwidth.

Review this table of frequently asked questions and considerations:

Consideration Description
NetStorage translates backslashes when indexing When indexing archive files, NetStorage translates backward slashes ( \ ) to forward slashes ( / ).

This can be problematic if the archive contains any UNIX system-based files with that character in their name. You can overcome this problem by uploading the file external to the archive (i.e., do not include it in the archive, and upload it separately).

Archive file size or object limitations Archive files are limited as follows:
  • ObjectStore: 100 gigabytes in size and/or may contain no more than 1,000,000 objects.
Note: An object constitutes a file, symlink or directory.

As a workaround, you can upload multiple archive files that fit within these restrictions.

Multiple disk archive files are not supported You cannot use multiple disk archive files with Serve from Zip.
No Support with “FTP Download” If you are using NetStorage as your origin for the FTP Download product, Serve from Zip is not supported for use.
Delivery issues with HTTP byte-range requests Using HTTP byte-range requests to a file within a zip archive could impede performance. This is because these requests need to decompress the bytes for that file up to the starting offset. A bad use case example is asking for byte 299,999,999 of a 300 megabyte file.

Observe the potential drawbacks:

  • Range-requests are not supported with Content Management protocols to objects inside a zip file. (Content management protocols are the HTTP-API and any session-based protocol.)
  • A 404 error results in a larger performance penalty.
  • Performance of range-request functionality to objects within a NetStorage indexed zip file will be penalized when the delivery product cache doesn't have the range requested, the full object will be requested from the NetStorage origin.
Note: Putting objects larger than 100 MB into an indexed zip archive isn't recommended. Large File Optimization (LFO) delivery options won't be usable for those objects, and high midgress traffic is the likely result.
The indexing operation adds information to the archive file If the CMShell command, md5sum is run after indexing, the hash differs from that of your local archive version. For the same reason, running FTP’s site az2z command in tandem with its site chkhash <MD5Digest> command produces an error.
Can be problematic for NetStorage’s “MD5 Sum” feature

This setting is set via the Upload Directory Attribute interface in Control Center. See the NetStorage - Configuration Guide for more details on this component.

When enabled, the feature sends an MD5 hash in the response header of each file served from NetStorage. However, in the case of Serve from Zip content, the MD5 hash is from the archive file, not of the content file itself. So, if the end user’s client (including Download Manager) checks the response header, the hash will not match that of the file received, and the file will be rejected.

This behavior also prevents content caching at Edge servers, if you have asked your account representative to enable the Edge’s hash-checking feature. In this case, it does not interfere with the actual serving of content to end users, but it does require Edge Servers to retrieve content from NetStorage each time content is requested. In the future, Serve from Zip will override the MD5 Sum feature and NetStorage will not send an MD5 hash with Serve from Zip content.

Only basic zip format features are supported For example, encryption and bzip compression are not supported.
Emergency directories or stale content protection are not supported Zip logic does not support these features.
Aspera does not support zip indexing as part of the upload After a zip archive is uploaded via Aspera, the indexing must be done separately using the CMShell az2z command.
Double indexing an archive is not supported Once a file has been indexed, any attempt to re-index the same file will result in failure.
You can use these archive utilities
  • Mac Finder
  • Command-line (Mac)
  • Windows Explorer
  • WinRar
  • 7-Zip
Serve from Zip uses relative paths Uploaded zip files should contain relative paths that do not start with a slash ( / ). Absolute paths starting with a slash are not supported.
What does auto-indexing do to a zip file pre-indexed? Error 451 will occur after uploading pre-indexed zip files with auto-index enabled, and the file will not transfer to NetStorage.
What does the CMShell “az2z” cmd do to a zip file that was auto-indexed? Once a file has been indexed, any attempt to re-index the same file will result in failure.
How do I find out which zip files are, or are not, indexed? Indexed zip file MD5s and file sizes stored on NetStorage will not match your local copy:
  1. Get the list of all zip files by scanning a local copy of the content tree. For example, using Linux: find . -name '*.zip' -printf '%s %p\n'
  2. Use the HTTP-API "stat" action (or equivalent on another protocol) for each path and compare the size of the zip file. If the uploaded file size is the same, the uploaded zip file is not indexed.

Otherwise you'll need to know the name of a sub-object within each zip file (e.g., a video manifest name.) and be able to download via a delivery config using serve-from-zip.

  1. Use the HTTP-API list action to scan all objects in NetStorage.
  2. Test if a sub-object can be retrieved using the HTTP Delivery with the HEAD method.
How do I remove the Akamai index from a zip file on NetStorage? Removing an index isn't supported. Re-upload the original zip file without auto-indexing to replace an indexed zip file.
How do I upload zip files without auto-indexing? You can choose any of these options:
  • Create a new, or use another upload account that does not auto-index.
  • Use the HTTP-API with the zip-index=0 parameter, if using an upload account with auto-indexing enabled.
  • Temporarily disable the auto-index option on the upload account.