【Jerry Lead】Spark Internals

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. Introduction
   
2. 
   
3. This series discuss the design and implementation of Apache Spark, with focuses on its design principles, execution mechanisms, system architecture and performance optimization. In addition, there's some comparisons with Hadoop MapReduce in terms of design and implementation. I'm reluctant to call this document a "code walkthrough", because the goal is not to analyze each piece of code in the project, but to understand the whole system in a systematic way (through analyzing the execution procedure of a Spark job, from its creation to completion).
   
4. 
   
5. There're many ways to discuss a computer system. Here, We've chosen a problem-driven approach. Firstly one concret problem is introduced, then it gets analyzed step by step. We'll start from a typical Spark example job and then discuss all the related important system modules. I believe that this approach is better than diving into each module right from the beginning.
   
6. 
   
7. The target audiences of this series are geeks who want to have a deeper understanding of Apache Spark as well as other distributed computing frameworks.
   
8. 
   
9. I'll try my best to keep this documentation up to date with Spark since it's a fast evolving project with an active community. The documentation's main version is in sync with Spark's version. The additional number at the end represents the documentation's update version.
   
10. 
    
11. For more academic oriented discussion, please check out Matei's PHD thesis and other related papers. You can also have a look at my blog (in Chinese) blog.
    
12. 
    
13. I haven't been writing such complete documentation for a while. Last time it was about three years ago when I was studying Andrew Ng's ML course. I was really motivated at that time! This time I've spent 20+ days on this document, from the summer break till now (August 2014). Most of the time is spent on debugging, drawing diagrams and thinking how to put my ideas in the right way. I hope you find this series helpful.

复制代码

1. Contents
   
2. 
   
3. We start from the creation of a Spark job, and then discuss its execution. Finally, we dive into some related system modules and features.
   
4. 
   
5. Overview Overview of Apache Spark
   
6. Job logical plan Logical plan of a job (data dependency graph)
   
7. Job physical plan Physical plan
   
8. Shuffle details Shuffle process
   
9. Architecture Coordination of system modules in job execution
   
10. Cache and Checkpoint Cache and Checkpoint
    
11. Broadcast Broadcast feature
    
12. Job Scheduling TODO
    
13. Fault-tolerance TODO

复制代码

本帖隐藏的内容

Spark Internals.pdf (23.91 MB, 需要: 5 个论坛币)

2017-2-15 10:45:55 上传

需要: 5 个论坛币  [购买]

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Internal Intern inter jerry Spark understand principles execution addition document

谢谢分享

kankan