Stream logs to Amazon S3
DataStream supports sending logs to Amazon Simple Storage Service (Amazon S3). Amazon S3 is a static file storage that lets you organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements.
DataStream sends GZIP-compressed logs to Amazon S3.
For security reasons, DataStream sends logs over TLS even if Amazon S3 policies allow insecure requests.
Before you begin
- Create an Identity and Access Management (IAM) user in Amazon S3. See Overview of access management: permissions and policies.
- Create a dedicated storage bucket in an AWS region. See Create storage buckets.
- Grant the user or role that can
access the bucket the appropriate permissions to the bucket contents, including
- Make note of the access keys associated with your account. See Understanding and getting your security credentials.
- Set up and manage server side encryption (SSE) in the container's settings. See Server side encryption for Amazon S3.
- In Destination, select S3.
- In Name, enter a human-readable description for the destination.
- In Bucket, enter the name of the Amazon S3 bucket where you want to send and store logs.
In Folder path,
provide the path to the folder within the bucket where you want to store the
logs. The path ends with a trailing slash.
Amazon S3 dynamically creates folders based on folder paths specified in your configuration. In a folder path, you can use the following variables: year, month, day, and hour.Note: You can't start a folder path with "/". If you add "/" at the beginning of a path, Amazon S3 treats your bucket as a folder and your data will not reach the intended destination.
- In Access key ID, enter the access key associated with the Amazon S3 bucket.
In Secret access
key, enter the secret key associated with the Amazon S3
Tip: You can check your authentication details in the .csv file that you saved when creating your access key. If you didn’t download the .csv file, or if you lost it, you may need to delete the existing access key and add a new one. See Managing access keys (console) in AWS.
- In Region, select the AWS region name where the bucket resides.
to validate the connection to the destination and save the details you
As part of this validation process, the system uses the provided access key identifier and secret and to save an akamai_write_test_2147483647.txt file in your S3 folder. You can only see this file if the validation process is successful, and you have write access to the Amazon S3 bucket and folder that you are trying to send logs to.