[Case Study]WordCount using Java

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

1. 1 package com.hq;
   
2. 2
   
3. 3 /**
   
4. 4  * User: hadoop
   
5. 5  * Date: 2014/10/10 0010
   
6. 6  * Time: 19:26
   
7. 7  */
   
8. 8
   
9. 9 import org.apache.spark.SparkConf;
   
10. 10 import org.apache.spark.api.java.JavaPairRDD;
    
11. 11 import org.apache.spark.api.java.JavaRDD;
    
12. 12 import org.apache.spark.api.java.JavaSparkContext;
    
13. 13 import org.apache.spark.api.java.function.FlatMapFunction;
    
14. 14 import org.apache.spark.api.java.function.Function2;
    
15. 15 import org.apache.spark.api.java.function.PairFunction;
    
16. 16 import scala.Tuple2;
    
17. 17
    
18. 18 import java.util.Arrays;
    
19. 19 import java.util.List;
    
20. 20 import java.util.regex.Pattern;
    
21. 21
    
22. 22 public final class JavaWordCount {
    
23. 23   private static final Pattern SPACE = Pattern.compile(" ");
    
24. 24
    
25. 25   public static void main(String[] args) throws Exception {
    
26. 26
    
27. 27     if (args.length < 1) {
    
28. 28       System.err.println("Usage: JavaWordCount <file>");
    
29. 29       System.exit(1);
    
30. 30     }
    
31. 31
    
32. 32     SparkConf sparkConf = new SparkConf().setAppName("JavaWordCount");
    
33. 33     JavaSparkContext ctx = new JavaSparkContext(sparkConf);
    
34. 34     JavaRDD<String> lines = ctx.textFile(args[0], 1);
    
35. 35
    
36. 36     JavaRDD<String> words = lines.flatMap(new FlatMapFunction<String, String>() {
    
37. 37       @Override
    
38. 38       public Iterable<String> call(String s) {
    
39. 39         return Arrays.asList(SPACE.split(s));
    
40. 40       }
    
41. 41     });
    
42. 42
    
43. 43     JavaPairRDD<String, Integer> ones = words.mapToPair(new PairFunction<String, String, Integer>() {
    
44. 44       @Override
    
45. 45       public Tuple2<String, Integer> call(String s) {
    
46. 46         return new Tuple2<String, Integer>(s, 1);
    
47. 47       }
    
48. 48     });
    
49. 49
    
50. 50     JavaPairRDD<String, Integer> counts = ones.reduceByKey(new Function2<Integer, Integer, Integer>() {
    
51. 51       @Override
    
52. 52       public Integer call(Integer i1, Integer i2) {
    
53. 53         return i1 + i2;
    
54. 54       }
    
55. 55     });
    
56. 56
    
57. 57     List<Tuple2<String, Integer>> output = counts.collect();
    
58. 58     for (Tuple2<?, ?> tuple : output) {
    
59. 59       System.out.println(tuple._1() + ": " + tuple._2());
    
60. 60     }
    
61. 61     ctx.stop();
    
62. 62   }
    
63. 63 }

复制代码

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏0 回帖

关键词：Case study Using Count study Case package import