[GitHub]Mastering Java for Data Science

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Mastering Java for Data Science

本帖隐藏的内容

Mastering-Java-for-Data-Science-master.zip (1.48 MB)

2017-7-7 03:43:53 上传

This is the code repository for Mastering Java for Data Science, published by Packt. It contains all the supporting project files necessary to work through the book from start to finish.

This is the official code repository for the book. You can find the unofficial one at https://github.com/alexeygrigorev/mastering-java-data-science. The unofficial repository contains the latest code fixes and is maintained by the author.

About the Book

Java is the most popular programming language, according to the TIOBE index, and it is a typical choice for running production systems in many companies, both in the startup world and among large enterprises.

Not surprisingly, it is also a common choice for creating data science applications: it is fast and has a great set of data processing tools, both built-in and external. What is more, choosing Java for data science allows you to easily integrate solutions with existing software, and bring data science into production with less effort.

This book will teach you how to create data science applications with Java. First, we will revise the most important things when starting a data science application, and then brush up the basics of Java and machine learning before diving into more advanced topics. We start by going over the existing libraries for data processing and libraries with machine learning algorithms. After that, we cover topics such as classification and regression, dimensionality reduction and clustering, information retrieval and natural language processing, and deep learning and big data.

Finally, we finish the book by talking about the ways to deploy the model and evaluate it in production settings.

Instructions and Navigation

All of the code is organized into folders. Each folder starts with a number followed by the application name. For example, Chapter02.

The code will look like the following:

List<String> list = new ArrayList<>();list.add("alpha");list.add("beta");list.add("beta");list.add("gamma");System.out.println(list);

You need to have any latest system with at least 2GB RAM and a Windows 7 /Ubuntu 14.04/Mac OS X operating system. Further, you will need to have Java 1.8.0 or above and Maven 3.0.0 or above installed.

Related Products

• Java for Data Science

• Mastering R for Data Science

• Mastering Spark for Data Science

  
  

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏1 回帖

关键词：Data Science Mastering Science Master GitHub

1. package chapter03;
   
2. 
   
3. import java.beans.BeanInfo;
   
4. import java.beans.Introspector;
   
5. import java.beans.PropertyDescriptor;
   
6. import java.lang.reflect.Method;
   
7. import java.util.List;
   
8. import java.util.Map;
   
9. import java.util.stream.Collectors;
   
10. import java.util.stream.IntStream;
    
11. 
    
12. import com.google.common.collect.Lists;
    
13. import com.google.common.collect.Maps;
    
14. 
    
15. import joinery.DataFrame;
    
16. 
    
17. public class BeanToJoinery {
    
18. 
    
19.     public static <E> DataFrame<Object> convert(List<E> beans, Class<E> beanClass) {
    
20.         try {
    
21.             return doConvert(beans, beanClass);
    
22.         } catch (Exception e) {
    
23.             throw new RuntimeException(e);
    
24.         }
    
25.     }
    
26. 
    
27.     private static <E> DataFrame<Object> doConvert(List<E> beans, Class<E> beanClass) throws Exception {
    
28.         int nrow = beans.size();
    
29.         BeanInfo info = Introspector.getBeanInfo(beanClass);
    
30.         PropertyDescriptor[] properties = info.getPropertyDescriptors();
    
31. 
    
32.         Map<String, List<Object>> columns = Maps.newLinkedHashMap();
    
33. 
    
34.         for (PropertyDescriptor pd : properties) {
    
35.             String name = pd.getName();
    
36.             if ("class".equals(name)) {
    
37.                 continue;
    
38.             }
    
39. 
    
40.             Method getter = pd.getReadMethod();
    
41.             if (getter == null) {
    
42.                 continue;
    
43.             }
    
44. 
    
45.             List<Object> column = Lists.newArrayListWithCapacity(nrow);
    
46.             for (E e : beans) {
    
47.                 Object value = getter.invoke(e);
    
48.                 column.add(value);
    
49.             }
    
50. 
    
51.             columns.put(name, column);
    
52.         }
    
53. 
    
54.         List<Integer> index = IntStream.range(0, nrow)
    
55.                 .mapToObj(Integer::valueOf)
    
56.                 .collect(Collectors.toList());
    
57. 
    
58.         List<String> columnNames = Lists.newArrayList(columns.keySet());
    
59.         List<List<Object>> data = Lists.newArrayList(columns.values());
    
60.         return new DataFrame<>(index, columnNames, data);
    
61.     }
    
62. 
    
63. }

复制代码

1. package chapter03;
   
2. 
   
3. import java.io.IOException;
   
4. import java.util.List;
   
5. import java.util.Map;
   
6. import java.util.Map.Entry;
   
7. import java.util.function.Function;
   
8. import java.util.function.ToDoubleFunction;
   
9. import java.util.stream.Collectors;
   
10. 
    
11. import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
    
12. import org.apache.commons.math3.stat.descriptive.SummaryStatistics;
    
13. 
    
14. import com.google.common.collect.Maps;
    
15. 
    
16. public class CommonsMathExample {
    
17. 
    
18.     public static void main(String[] args) throws IOException {
    
19.         List<RankedPage> data = Data.readRankedPages();
    
20.         summaryStats(data);
    
21.         descStats(data);
    
22.         proportion(data);
    
23.         groupByDescStats(data);
    
24.     }
    
25. 
    
26.     private static void summaryStats(List<RankedPage> data) {
    
27.         SummaryStatistics summary = new SummaryStatistics();
    
28.         data.stream().mapToDouble(RankedPage::getBodyContentLength).forEach(summary::addValue);
    
29.         System.out.println(summary.getSummary());
    
30.     }
    
31. 
    
32.     private static void descStats(List<RankedPage> data) {
    
33.         double[] dataArray = data.stream().mapToDouble(RankedPage::getBodyContentLength).toArray();
    
34.         DescriptiveStatistics desc = new DescriptiveStatistics(dataArray);
    
35.         System.out.printf("min: %9.1f%n", desc.getMin());
    
36.         System.out.printf("p05: %9.1f%n", desc.getPercentile(5));
    
37.         System.out.printf("p25: %9.1f%n", desc.getPercentile(25));
    
38.         System.out.printf("p50: %9.1f%n", desc.getPercentile(50));
    
39.         System.out.printf("p75: %9.1f%n", desc.getPercentile(75));
    
40.         System.out.printf("p95: %9.1f%n", desc.getPercentile(95));
    
41.         System.out.printf("max: %9.1f%n", desc.getMax());
    
42.     }
    
43. 
    
44.     private static void proportion(List<RankedPage> data) {
    
45.         double proportion = data.stream()
    
46.                 .mapToInt(p -> p.getBodyContentLength() == 0 ? 1 : 0)
    
47.                 .average().getAsDouble();
    
48.         System.out.printf("proportion of zero content length: %.5f%n", proportion);
    
49.     }
    
50. 
    
51.     private static void groupByDescStats(List<RankedPage> data) {
    
52.         System.out.println();
    
53. 
    
54.         Map<Integer, List<RankedPage>> byPage = data.stream()
    
55.                 .filter(p -> p.getBodyContentLength() != 0)
    
56.                 .collect(Collectors.groupingBy(RankedPage::getPage));
    
57. 
    
58.         List<DescriptiveStatistics> stats = byPage.entrySet().stream()
    
59.                 .sorted(Map.Entry.comparingByKey())
    
60.                 .map(e -> calculate(e.getValue(), RankedPage::getBodyContentLength))
    
61.                 .collect(Collectors.toList());
    
62. 
    
63.         Map<String, Function<DescriptiveStatistics, Double>> functions = Maps.newLinkedHashMap();
    
64.         functions.put("min", d -> d.getMin());
    
65.         functions.put("p05", d -> d.getPercentile(5));
    
66.         functions.put("p25", d -> d.getPercentile(25));
    
67.         functions.put("p50", d -> d.getPercentile(50));
    
68.         functions.put("p75", d -> d.getPercentile(75));
    
69.         functions.put("p95", d -> d.getPercentile(95));
    
70.         functions.put("max", d -> d.getMax());
    
71. 
    
72.         System.out.print("page");
    
73.         for (Integer page : byPage.keySet()) {
    
74.             System.out.printf("%9d ", page);
    
75.         }
    
76.         System.out.println();
    
77. 
    
78.         for (Entry<String, Function<DescriptiveStatistics, Double>> pair : functions.entrySet()) {
    
79.             System.out.print(pair.getKey());
    
80.             Function<DescriptiveStatistics, Double> function = pair.getValue();
    
81.             System.out.print(" ");
    
82.             for (DescriptiveStatistics ds : stats) {
    
83.                 System.out.printf("%9.1f ", function.apply(ds));
    
84.             }
    
85.             System.out.println();
    
86.         }
    
87.     }
    
88. 
    
89.     private static DescriptiveStatistics calculate(List<RankedPage> data, ToDoubleFunction<RankedPage> getter) {
    
90.         double[] dataArray = data.stream().mapToDouble(getter).toArray();
    
91.         return new DescriptiveStatistics(dataArray);
    
92.     }
    
93. 
    
94. }

复制代码

1. package chapter04.classification;
   
2. 
   
3. import java.io.PrintStream;
   
4. import java.util.List;
   
5. import java.util.function.Function;
   
6. 
   
7. import org.apache.commons.io.output.NullOutputStream;
   
8. import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
   
9. 
   
10. import chapter04.cv.Dataset;
    
11. import chapter04.cv.Split;
    
12. import de.bwaldvogel.liblinear.Feature;
    
13. import de.bwaldvogel.liblinear.FeatureNode;
    
14. import de.bwaldvogel.liblinear.Linear;
    
15. import de.bwaldvogel.liblinear.Model;
    
16. import de.bwaldvogel.liblinear.Parameter;
    
17. import de.bwaldvogel.liblinear.Problem;
    
18. 
    
19. public class LibLinear {
    
20. 
    
21.     public static void mute() {
    
22.         PrintStream devNull = new PrintStream(new NullOutputStream());
    
23.         Linear.setDebugOutput(devNull);
    
24.     }
    
25. 
    
26.     public static DescriptiveStatistics crossValidate(List<Split> folds, Function<Dataset, Model> trainer) {
    
27.         double[] aucs = folds.parallelStream().mapToDouble(fold -> {
    
28.             Dataset foldTrain = fold.getTrain();
    
29.             Dataset foldValidation = fold.getTest();
    
30.             Model model = trainer.apply(foldTrain);
    
31.             return auc(model, foldValidation);
    
32.         }).toArray();
    
33. 
    
34.         return new DescriptiveStatistics(aucs);
    
35.     }
    
36. 
    
37.     public static Model train(Dataset dataset, Parameter param) {
    
38.         Problem problem = wrapDataset(dataset);
    
39.         return Linear.train(problem, param);
    
40.     }
    
41. 
    
42.     private static Problem wrapDataset(Dataset dataset) {
    
43.         double[][] X = dataset.getX();
    
44.         double[] y = dataset.getY();
    
45. 
    
46.         Problem problem = new Problem();
    
47.         problem.x = wrapMatrix(X);
    
48.         problem.y = y;
    
49.         problem.n = X[0].length + 1;
    
50.         problem.l = X.length;
    
51. 
    
52.         return problem;
    
53.     }
    
54. 
    
55.     public static double auc(Model model, Dataset dataset) {
    
56.         double[] scores;
    
57.         if (model.isProbabilityModel()) {
    
58.             scores = predictProba(model, dataset);
    
59.         } else {
    
60.             scores = predictValues(model, dataset);
    
61.             scores = sigmoid(scores);
    
62.         }
    
63. 
    
64.         return Metrics.auc(dataset.getY(), scores);
    
65.     }
    
66. 
    
67.     public static double[] predictProba(Model model, Dataset dataset) {
    
68.         int n = dataset.length();
    
69. 
    
70.         double[][] X = dataset.getX();
    
71.         double[] results = new double[n];
    
72.         double[] probs = new double[2];
    
73. 
    
74.         for (int i = 0; i < n; i++) {
    
75.             Feature[] row = wrapRow(X[i]);
    
76.             Linear.predictProbability(model, row, probs);
    
77.             results[i] = probs[1];
    
78.         }
    
79. 
    
80.         return results;
    
81.     }
    
82. 
    
83.     public static double[] predictValues(Model model, Dataset dataset) {
    
84.         int n = dataset.length();
    
85. 
    
86.         double[][] X = dataset.getX();
    
87.         double[] results = new double[n];
    
88.         double[] values = new double[1];
    
89. 
    
90.         for (int i = 0; i < n; i++) {
    
91.             Feature[] row = wrapRow(X[i]);
    
92.             Linear.predictValues(model, row, values);
    
93.             results[i] = values[0];
    
94.         }
    
95. 
    
96.         return results;
    
97.     }
    
98. 
    
99.     public static double[] sigmoid(double[] scores) {
    
100.         double[] result = new double[scores.length];
     
101. 
     
102.         for (int i = 0; i < result.length; i++) {
     
103.             result[i] = 1 / (1 + Math.exp(-scores[i]));
     
104.         }
     
105. 
     
106.         return result;
     
107.     }
     
108. 
     
109.     private static Feature[][] wrapMatrix(double[][] X) {
     
110.         int n = X.length;
     
111.         Feature[][] matrix = new Feature[n][];
     
112.         for (int i = 0; i < n; i++) {
     
113.             matrix[i] = wrapRow(X[i]);
     
114.         }
     
115.         return matrix;
     
116.     }
     
117. 
     
118.     private static Feature[] wrapRow(double[] row) {
     
119.         int m = row.length;
     
120.         Feature[] result = new Feature[m];
     
121. 
     
122.         for (int i = 0; i < m; i++) {
     
123.             result[i] = new FeatureNode(i + 1, row[i]);
     
124.         }
     
125. 
     
126.         return result;
     
127.     }
     
128. }

复制代码

THANKS FOR SHARING

谢谢分享                  

看看，似乎不错

thanks for sharing