Event-time Aggregation and Watermarking in Apache Spark’s Structured Streaming

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. This is the fourth post in a multi-part series about how you can perform complex streaming analytics using Apache Spark.
   
2. 
   
3. Continuous applications often require near real-time decisions on real-time aggregated statistics—such as health of and readings from IoT devices or detecting anomalous behavior. In this blog, we will explore how easily streaming aggregations can be expressed in Structured Streaming, and how naturally late, and out-of-order data is handled.

复制代码

本帖隐藏的内容

Event-time Aggregation and Watermarking in Apache Spark’s Structured Streaming.pdf (673.32 KB)

2017-5-28 02:32:56 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Apache Spark Aggregation structured streaming Structure

1. Streaming Aggregations
   
2. Structured Streaming allows users to express the same streaming query as a batch query, and the Spark SQL engine incrementalizes the query and executes on streaming data. For example, suppose you have a streaming DataFrame having events with signal strength from IoT devices, and you want to calculate the running average signal strength for each device, then you would write the following Python code:
   
3. 
   
4. # DataFrame w/ schema [eventTime: timestamp, deviceId: string, signal: bigint]
   
5. eventsDF = ...
   
6. 
   
7. avgSignalDF = eventsDF.groupBy("deviceId").avg("signal")
   
8. This code is no different if eventsDF was a DataFrame on static data. However, in this case, the average will be continuously updated as new events arrive. You choose different output modes for writing the updated averages to external systems like file systems and databases. Furthermore, you can also implement custom aggregations using Spark’s user-defined aggregation function (UDAFs).

复制代码