The following are frequently asked questions (FAQs) about DataStream.

DataStream FAQs
Question Answer
What is DataStream’s output data format? The output format for DataStream is JSON. For sample schemas and data sets, see DataStream API.
What are the usage limits? For DataStream, the rate limit on the free tier is 380 requests per second, which can add up to one billion request or 5000GB per month. If the actual traffic is higher, the system samples the data to ensure the limit of 380 requests per second.

DataStream offers a paid tier for customers who need data delivery in excess of 5000GB or 1 billion requests per month.

How many endpoints does DataStream send logs from? DataStream sends logs to customer destinations from multiple source endpoints. Typically, the number of source endpoints range from 10s to 100s and can change without prior notice.
When would I use raw logs as opposed to aggregated metrics? You can start with aggregated metrics for a snapshot of CDN health. If you see any anomalies, you can start streaming raw logs for root cause analysis or diagnostics. For example, if you’re experiencing a large number of 4xx errors over a period of time, raw logs can help you find the cause and potential fixes.

You can turn raw log streams on before and after a new deployment and turn them off once the deployment is stable. For example, before and after offload monitoring. This helps avoid billing when you don’t need the logs.

I don’t use a log analytics tool. Can I still use DataStream’s raw logs or aggregated metrics? You need a tool to parse and visualize the DataStream output. You could choose lower cost, open source stacks for data parsing and visualization in human readable formats.
How does DataStream aggregation work? DataStream aggregation is based on a tumbling window over the selected aggregation time frame. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals.

DataStream aggregation operators start collecting live data based on the aggregation time frame chosen at the time of defining a stream. For example, if you create a stream at 10.00 with a five-minute time frame, DataStream will collect and aggregate data from 10.00 to 10.05. DataStream aggregation operators will repeat the aggregation every five minutes.

What are the minimum and maximum aggregation time frames available? These are the available time frames:
  • 1 minute
  • 5 minutes
  • 15 minutes
  • 30 minutes
  • 1 hour
Note: The aggregated data is retained for 12 hours on a rolling window basis.
How is DataStream different from the CloudMonitor or LDS products that I already use? There are new features that make DataStream the next generation log delivery product compared to LDS and CloudMonitor.
  • Data retention
  • Data aggregation
  • Pull APIs in addition to the traditional Push mechanism
  • Lower latency than CloudMonitor or LDS
  • Ability to define a stream with only the chosen data sets
  • Ability to turn a stream on and off as needed
Does DataStream support security event logs? DataStream is a log delivery product for all transactional events and associated metrics. You can use the SIEM Integration product to deliver security logs. See SIEM Integration.
Why does one or more custom fields return ^ in log files? DataStream requires enabling custom fields, such as User-Agent, Accept-Language, Cookie, Referrer or X-Forwarded-For in the Log Delivery Service behavior in your property configuration. If log lines return ^, first enable the fields in Property Manager. See Enable logging custom parameters.