Learning PySpark by Tomasz Drabas

一直在用pyspark ，谢谢分享

1. The .filter(...) transformation
   
2. Another most often used transformation is the .filter(...) method,
   
3. which allows you to select elements from your dataset that fit specified
   
4. criteria. As an example, from the data_from_file_conv dataset, let's
   
5. count how many people died in an accident in 2014:
   
6. data_filtered = data_from_file_conv.filter(
   
7. lambda row: row[16] == '2014' and row[21] == '0')
   
8. data_filtered.count()

复制代码

1. The .flatMap(...) transformation
   
2. The .flatMap(...) method works similarly to .map(...), but it returns
   
3. a flattened result instead of a list. If we execute the following code:
   
4. data_2014_flat = data_from_file_conv.flatMap(lambda row:
   
5. (row[16], int(row[16]) + 1))
   
6. data_2014_flat.take(10)

复制代码

1. The .distinct(...) transformation
   
2. This method returns a list of distinct values in a specified column. It is
   
3. extremely useful if you want to get to know your dataset or validate it.
   
4. Let's check if the gender column contains only males and females; that
   
5. would verify that we parsed the dataset properly. Let's run the following
   
6. code:
   
7. distinct_gender = data_from_file_conv.map(
   
8. lambda row: row[5]).distinct()
   
9. distinct_gender.collect()

复制代码

1. The .sample(...) transformation
   
2. The .sample(...) method returns a randomized sample from the
   
3. dataset. The first parameter specifies whether the sampling should be
   
4. with a replacement, the second parameter defines the fraction of the
   
5. data to return, and the third is seed to the pseudo-random numbers
   
6. generator:
   
7. fraction = 0.1
   
8. data_sample = data_from_file_conv.sample(False, fraction, 666)

复制代码

Learning PySpark by Tomasz Drabas

谢谢分享啊！

Learning PySpark by Tomasz Drabas

好书谢谢

康看看看