Stream logs to Amazon S3

DataSream supports sending logs to Amazon Simple Storage Service (Amazon S3). Amazon S3 is a static file storage that lets you organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements.

DataStream sends GZIP-compressed logs to Amazon S3.

For security reasons, DataStream sends logs over TLS even if Amazon S3 policies allow insecure requests.

Before you begin

How to

  1. In Destination name, enter a human-readable description for the destination.
  2. In Destination type, select S3.
  3. In Bucket, enter the name of the Amazon S3 bucket where you want to send and store logs.
  4. In Folder path, enter the path to the folder within the bucket where you want to store the logs. The path ends with a trailing slash. If this field is left empty, the files will be saved in the container's root path.
    Amazon S3 dynamically creates folders based on folder paths specified in your configuration. In a folder path, you can use the following variables: year, month, day, and hour.

    For example, for these folder paths: raw-logs/{year}/{month}/{day}/{hour} and raw-logs/{year}/make/model, Amazon S3 could store data in such folders: raw-logs/2019/03/15/12 and raw-logs/2019/make/model

    Important: You can't start a folder path with "/". If you add "/" at the beginning of a path, Amazon S3 treats your bucket as a folder and your data will not reach the intended destination
  5. In Region, select the AWS region where the bucket resides.
  6. In Access key ID, enter the access key associated with the Amazon S3 bucket.
  7. In Secret access key, enter the secret key associated with the Amazon S3 bucket.
  8. Click Save to save the details you provided.