Apply These Techniques to Learn Complex Event Processing with Apache Flink.

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. 1. Objective
   
2. 
   
3. This tutorial on Complex Event Processing with Apache Flink will help you in understanding Flink CEP library, how CEP programs are written using Pattern API, various CEP pattern operations with syntax, Pattern detection in CEP and advantages of CEP operations in Flink. You will also learn Flink Complex event processing use cases and examples to get in depth knowledge of complex event processing for Flink.
   
4. 
   
5. 2. Introduction to Complex Event Processing with Apache Flink
   
6. 
   
7. With the increasing size of data and smart devices continuously collecting more and more data, there is a challenge to analyze this growing stream of data in near real-time for reacting quickly to changing trends or for delivering up to date business intelligence which can decide company’s success or failure. Detection of event patterns in data streams is a key problem in real time processing.
   
8. 
   
9. Flink handles this problem through Complex event processing (CEP) library that addresses this problem of matching the incoming events against a pattern to produce complex events which are derived from the input events. CEP executes relevant data on a stored query unlike traditional RDBMSs and discards irrelevant data. This enables CEP queries to be applied on a potentially infinite stream of data and also enables inputs to be processed immediately. This aspect effectively leads to CEP’s real time analytics capability. This gives the opportunity to quickly get hold of what’s really important in data. In this manner, Flink CEP is 1 of the key component of Apache Flink ecosystem.
   
10. 
    
11. Apache Flink is a natural fit for CEP workloads due to its true streaming nature and its capabilities for low latency as well as high throughput stream processing. Consequently, CEP has found application in a wide variety of use cases as described below and has provided several features of Apache Flink that has created huge difference between Apache Flink, hadoop and Apache Spark.
    
12. 
    
13. 3. Pattern API
    
14. 
    
15. Flink CEP program can be written using the pattern API that allows defining complex event patterns. Each pattern consists of multiple stages or states. The pattern needs to start with initial state and then to go from one state to the next, the user can specify conditions. Each state must have a unique name to identify the matched events later on. We can append states to detect complex patterns.
    
16. 
    
17. Do you know different pattern operations? Let us see some of the most commonly used CEP pattern operations:
    
18. 
    
19. a. Begin
    
20. 
    
21. It defines pattern starting state and is written as below:
    
22. 
    
23. Pattern<Event, ?> start = Pattern.<Event>begin("start");
    
24. b. Next
    
25. 
    
26. It appends a new pattern state and matching event need to succeed the previous matching pattern as below:
    
27. 
    
28. Pattern<Event, ?> next = start.next("next");
    
29. c. FollowedBy
    
30. 
    
31. It appends a new pattern state but here other events can occur between 2 matching events as below:
    
32. 
    
33. Pattern<Event, ?> followedBy = start.followedBy("next");
    
34. d. Where
    
35. 
    
36. It defines a filter condition for current pattern state and if the event passes the filter, it can match the state as below:
    
37. 
    
38. patternState.where(new FilterFunction <Event>() {
    
39. @Override
    
40. public boolean filter(Event value) throws Exception {
    
41. return ... // some condition
    
42. }
    
43. });
    
44. e. Or
    
45. 
    
46. It adds a new filter condition which is ORed with an existing filter condition. Only if an event passes the filter condition, it can match the state.
    
47. 
    
48. f. Within
    
49. 
    
50. It defines the maximum time interval for an event sequence to match the pattern post which it is discarded. It’s written as below:
    
51. 
    
52. patternState.within(Time.seconds(10));
    
53. 4. Pattern Detection
    
54. 
    
55. We need to create pattern stream to run stream of events as below using input stream and a pattern:
    
56. 
    
57. DataStream <Event> input = ...
    
58. Pattern <Event, ?> pattern = ...
    
59. PatternStream <Event> patternStream = CEP.pattern(input, pattern);
    
60. 5. Use cases for Flink CEP
    
61. 
    
62. Apache Flink CEP is used for large number of applications like for financial applications such as stock market trend, credit card fraud detection and RFID-based tracking and monitoring. It also find its usage in detecting network intrusion by specifying patterns of suspicious user behaviour.
    
63. 
    
64. Refer Flink use case tutorial to get real time use cases of Apache Flink and how industries are using Flink for their various purposes.
    
65. 
    
66. 6. Conclusion
    
67. 
    
68. Flink CEP includes various challenges like:
    
69. 
    
70. Ability to achieve high throughput and low latency processing
    
71. Ability to produce the results as soon as the input event stream is available
    
72. Ability to provide aggregation over time, timeout between two events of interest and other computations
    
73. Ability to provide real time alerts & notifications on detection of complex event patterns
    
74. Follow DataFlair on LinkedIn for More Updates on #ApacheFlink #BigData

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Techniques Processing Technique processI Complex Complex