【Use Case】Approximate quantiles in Apache Spark

相似文件 换一批

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. Introduction
   
2. Apache Spark is fast, but applications such as preliminary data exploration need to be even faster and are willing to sacrifice some accuracy for a faster result. Since version 1.6, Spark implements approximate algorithms for some common tasks: counting the number of distinct elements in a set, finding if an element belongs to a set, computing some basic statistical information for a large set of numbers. Eugene Zhulenev, from Collective, has already blogged in these pages about the use of approximate counting in the advertising business.
   
3. 
   
4. The following algorithms have been implemented against DataFrames and Datasets and committed into Apache Spark’s branch-2.0, so they will be available in Apache Spark 2.0 for Python, R, and Scala:
   
5. 
   
6. approxCountDistinct: returns an estimate of the number of distinct elements
   
7. approxQuantile: returns approximate percentiles of numerical data
   
8. Researchers have looked at such algorithms for a long time. Spark strives at implementing approximate algorithms that are deterministic (they do not depend on random numbers to work) and that have proven theoretical error bounds: for each algorithm, the user can specify a target error bound, and the result is guaranteed to be within this bound, either exactly (deterministic error bounds) or with very high confidence (probabilistic error bounds). Also, it is important that this algorithm works well for the wealth of use cases seen in the Spark community.
   
9. 
   
10. In this blog, we are going to present details on the implementation of approxCountDistinct and approxQuantile algorithms and showcase its implementation in a Databricks notebook.

复制代码

本帖隐藏的内容

Approximate Algorithms in Apache Spark.pdf (247.22 KB)

2017-5-28 03:08:37 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Apache Spark Approximate quantiles quantile Approx

1. dataset = sqlContext.read.load("/databricks-datasets/amazon/data20K")
   
2. sqlContext.registerDataFrameAsTable(dataset, "amazon_partitioned")
   
3. 
   
4. # A few imports
   
5. import numpy as np
   
6. import time
   
7. import pandas as pd
   
8. from pyspark.sql.functions import expr

复制代码

1. Approximate distinct counts
   
2. import numpy as np
   
3. from pyspark.sql.functions import approxCountDistinct
   
4. import time
   
5. user_dataset = sqlContext.read.format("csv").load("/databricks-datasets/amazon/users").toDF("user")
   
6. sqlContext.registerDataFrameAsTable(user_dataset, "transaction_users")
   
7. users = sqlContext.sql("select user from transaction_users").cache()
   
8. users.count()

复制代码