Using sparklyr in Databricks

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. In September 2016, RStudio announced sparklyr, a new R interface to Apache Spark. sparklyr’s interface to Spark follows the popular dplyr syntax. At Databricks, we provide the best place to run Apache Spark and all applications and packages powered by it, from all the languages that Spark supports. sparklyr’s addition to the Spark ecosystem not only complements SparkR but also extends Spark’s reach to new users and communities.
   
2. 
   
3. Today, we are happy to announce that sparklyr can be seamlessly used in Databricks clusters running Apache Spark 2.2 or higher. In this blog post, we show how you can install and configure sparklyr in Databricks. We also introduce some of the latest improvements in Databricks R Notebooks.

复制代码

本帖隐藏的内容

Using sparklyr in Databricks.pdf (201.34 KB)

2017-5-28 02:03:27 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：BRICKS Using abric Spark BRIC

1. Installing sparklyr
   
2. sparklyr is under active development and new versions are regularly released with new API and bug fixes. We do not pre-install sparklyr, allowing our users can install and enjoy the latest version of the package. You can install the latest development version from GitHub:
   
3. 
   
4. devtools::install_github("rstudio/sparklyr")
   
5. After sparklyr 0.6 is released to CRAN, the installation process can be done much simpler.
   
6. 
   
7. install_packages(sparklyr)

复制代码

1. Using sparklyr API
   
2. After setting up the sparklyr connection, you can use all sparklyr APIs. You can import and combine sparklyr with dplyr or MLlib. You can also use sparklyr extensions. Note that if the extension packages include third-party JARs, you may need to install those JARs as libraries in your workspace.
   
3. 
   
4. library(dplyr)
   
5. iris_tbl <- copy_to(sc, iris)
   
6. iris_summary <- iris_tbl %>%
   
7.     mutate(Sepal_Width = ROUND(Sepal_Width * 2) / 2) %>%
   
8.     group_by(Species, Sepal_Width) %>%
   
9.     summarize(count = n(),
   
10. Sepal_Length = mean(Sepal_Length),
    
11. stdev = sd(Sepal_Length)) %>% collect
    
12. 
    
13. library(ggplot2)
    
14. ggplot(iris_summary,
    
15.    aes(Sepal_Width, Sepal_Length, color = Species)) +
    
16.     geom_line(size = 1.2) +
    
17.     geom_errorbar(aes(
    
18. ymin = Sepal_Length - stdev,
    
19. ymax = Sepal_Length + stdev),
    
20.    width = 0.05) +
    
21.     geom_text(aes(label = count),
    
22. vjust = -0.2, hjust = 1.2, color = "black") +
    
23.     theme(legend.position="top")

复制代码

1. Using SparkR and sparklyr Together
   
2. We find SparkR and sparklyr complementary. You can use the packages next to each other in a single notebook or job. To do so you can import SparkR along with sparklyr in Databricks notebooks. The SparkR connection is pre-configured in the notebook, and after importing the package, you can start using SparkR API. Also, remember that some of the functions in SparkR mask a number of functions in dplyr.
   
3. 
   
4. library(SparkR)
   
5. The following objects are masked from ‘package:dplyr’:
   
6. 
   
7.     arrange, between, coalesce, collect, contains, count, cume_dist,
   
8.     dense_rank, desc, distinct, explain, filter, first, group_by,
   
9.     intersect, lag, last, lead, mutate, n, n_distinct, ntile,
   
10.     percent_rank, rename, row_number, sample_frac, select, sql,
    
11.     summarize, union

复制代码