Top 20 Python Machine Learning Open Source Projects

Fig. 1: Python Machine learning projects on GitHub, with color corresponding to commits/contributors. Bob, Iepy, Nilearn, and NuPIC have the highest such value.

• scikit-learn, 18845 commits, 404 contributors,
  www.github.com/scikit-learn/scikit-learn
  scikit-learn is a Python module for machine learning built on top of SciPy.It features various classification, regression and clustering algorithms including support vector machines, logistic regression, naive Bayes, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.
• Pylearn2, 7027 commits, 117 contributors,
  www.github.com/lisa-lab/pylearn2
  Pylearn2 is a library designed to make machine learning research easy. Its a library based on Theano
• NuPIC, 4392 commits, 60 contributors,
  www.github.com/numenta/nupic
  The Numenta Platform for Intelligent Computing (NuPIC) is a machine intelligence platform that implements the HTM learning algorithms. HTM is a detailed computational theory of the neocortex. At the core of HTM are time-based continuous learning algorithms that store and recall spatial and temporal patterns. NuPIC is suited to a variety of problems, particularly anomaly detection and prediction of streaming data sources.
• Nilearn, 2742 commits, 28 contributors,
  www.github.com/nilearn/nilearn
  Nilearn is a Python module for fast and easy statistical learning on NeuroImaging data. It leverages the scikit-learn Python toolbox for multivariate statistics with applications such as predictive modeling, classification, decoding, or connectivity analysis.
• PyBrain, 969 commits, 27 contributors,
  www.github.com/pybrain/pybrain
  PyBrain is short for Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library. Its goal is to offer flexible, easy-to-use yet still powerful algorithms for Machine Learning Tasks and a variety of predefined environments to test and compare your algorithms.
• Pattern, 943 commits, 20 contributors,
  www.github.com/clips/pattern
  Pattern is a web mining module for Python. It has tools for Data Mining, Natural Language Processing, Network Analysis and Machine Learning. It supports vector space model, clustering, classification using KNN, SVM, Perceptron
• Fuel, 497 commits, 12 contributors,
  www.github.com/mila-udem/fuel
  Fuel provides your machine learning models with the data they need to learn. it has interfaces to common datasets such as MNIST, CIFAR-10 (image datasets), Google's One Billion Words (text). It gives you the ability to iterate over your data in a variety of ways, such as in minibatches with shuffled/sequential examples
• Bob, 5080 commits, 11 contributors,
  www.github.com/idiap/bob
  Bob is a free signal-processing and machine learning toolbox The toolbox is written in a mix of Python and C++ and is designed to be both efficient and reduce development time. It is composed of a reasonably large number of packages that implement tools for image, audio & video processing, machine learning and pattern recognition
• skdata, 441 commits, 10 contributors,
  www.github.com/jaberg/skdata
  Skdata is a library of data sets for machine learning and statistics. This module provides standardized Python access to toy problems as well as popular computer vision and natural language processing data sets.
• MILK, 687 commits, 9 contributors,
  www.github.com/luispedro/milk
  Milk is a machine learning toolkit in Python. Its focus is on supervised classification with several classifiers available: SVMs, k-NN, random forests, decision trees. It also performs feature selection. These classifiers can be combined in many ways to form different classification systems.For unsupervised learning, milk supports k-means clustering and affinity propagation.
• IEPY, 1758 commits, 9 contributors,
  www.github.com/machinalis/iepy
  IEPY is an open source tool for Information Extraction focused on Relation Extraction
  It's aimed at users needing to perform Information Extraction on a large dataset. scientists wanting to experiment with new IE algorithms.
• Quepy, 131 commits, 9 contributors,
  www.github.com/machinalis/quepy
  Quepy is a python framework to transform natural language questions to queries in a database query language. It can be easily customized to different kinds of questions in natural language and database queries. So, with little coding you can build your own system for natural language access to your database.
  Currently Quepy provides support for Sparql and MQL query languages, with plans to extended it to other database query languages.
• Hebel, 244 commits, 5 contributors,
  www.github.com/hannes-brt/hebel
  Hebel is a library for deep learning with neural networks in Python using GPU acceleration with CUDA through PyCUDA. It implements the most important types of neural network models and offers a variety of different activation functions and training methods such as momentum, Nesterov momentum, dropout, and early stopping.
• mlxtend, 135 commits, 5 contributors,
  www.github.com/rasbt/mlxtend
  Its a library consisting of useful tools and extensions for the day-to-day data science tasks.
• nolearn, 192 commits, 4 contributors,
  www.github.com/dnouri/nolearn
  This package contains a number of utility modules that are helpful with machine learning tasks. Most of the modules work together with scikit-learn, others are more generally useful.
• Ramp, 179 commits, 4 contributors,
  www.github.com/kvh/ramp
  Ramp is a python library for rapid prototyping of machine learning solutions. It's a light-weight pandas-based machine learning framework pluggable with existing python machine learning and statistics tools (scikit-learn, rpy2, etc.). Ramp provides a simple, declarative syntax for exploring features, algorithms and transformations quickly and efficiently.
• Feature Forge, 219 commits, 3 contributors,
  www.github.com/machinalis/featureforge
  A set of tools for creating and testing machine learning features, with a scikit-learn compatible API.
  This library provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, etc.), and particularly helpful if you use scikit-learn (although this can work if you have a different algorithm).
• REP, 50 commits, 3 contributors,
  www.github.com/yandex/rep
  REP is environment for conducting data-driven research in a consistent and reproducible way. It has a unified classifiers wrapper for variety of implementations like TMVA, Sklearn, XGBoost, uBoost. It can train classifiers in parallel on a cluster. It supports interactive plots
• Python Machine Learning Samples, 15 commits, 3 contributors,
  www.github.com/awslabs/machine-learning-samples
  A collection of sample applications built using Amazon Machine Learning.
• Python-ELM, 17 commits, 1 contributor,
  www.github.com/dclambert/Python-ELM
  This is an implementation of the Extreme Learning Machine in Python, based on scikit-learn.
  

This post used some content from www.pansop.com/1039/

Related:

• Interesting Open-Source Projects in Machine Learning, Data Mining, Data Science
• Open Source Tools for Machine Learning
• Will the Real Data Scientists Please Stand Up?
  

thanks for shairng

1. """
   
2. The Haxby dataset: face vs house in object recognition
   
3. =======================================================
   
4. This example does a simple but efficient decoding on the Haxby dataset:
   
5. using a feature selection, followed by an SVM.
   
6. """
   
7. 
   
8. #############################################################################
   
9. # Retrieve the files of the Haxby dataset
   
10. from nilearn import datasets
    
11. 
    
12. haxby_dataset = datasets.fetch_haxby_simple()
    
13. 
    
14. # print basic information on the dataset
    
15. print('Mask nifti image (3D) is located at: %s' % haxby_dataset.mask)
    
16. print('Functional nifti image (4D) is located at: %s' %
    
17.       haxby_dataset.func[0])
    
18. 
    
19. #############################################################################
    
20. # Load the behavioral data
    
21. import numpy as np
    
22. y, session = np.loadtxt(haxby_dataset.session_target[0]).astype("int").T
    
23. conditions = np.recfromtxt(haxby_dataset.conditions_target[0])['f0']
    
24. 
    
25. # Restrict to faces and houses
    
26. condition_mask = np.logical_or(conditions == b'face', conditions == b'house')
    
27. y = y[condition_mask]
    
28. conditions = conditions[condition_mask]
    
29. 
    
30. # We have 2 conditions
    
31. n_conditions = np.size(np.unique(y))
    
32. 
    
33. #############################################################################
    
34. # Prepare the fMRI data
    
35. from nilearn.input_data import NiftiMasker
    
36. 
    
37. mask_filename = haxby_dataset.mask
    
38. # For decoding, standardizing is often very important
    
39. nifti_masker = NiftiMasker(mask_img=mask_filename, sessions=session,
    
40.                            smoothing_fwhm=4, standardize=True,
    
41.                            memory="nilearn_cache", memory_level=1)
    
42. func_filename = haxby_dataset.func[0]
    
43. X = nifti_masker.fit_transform(func_filename)
    
44. # Apply our condition_mask
    
45. X = X[condition_mask]
    
46. session = session[condition_mask]
    
47. 
    
48. #############################################################################
    
49. # Build the decoder
    
50. 
    
51. # Define the prediction function to be used.
    
52. # Here we use a Support Vector Classification, with a linear kernel
    
53. from sklearn.svm import SVC
    
54. svc = SVC(kernel='linear')
    
55. 
    
56. # Define the dimension reduction to be used.
    
57. # Here we use a classical univariate feature selection based on F-test,
    
58. # namely Anova. We set the number of features to be selected to 500
    
59. from sklearn.feature_selection import SelectKBest, f_classif
    
60. feature_selection = SelectKBest(f_classif, k=500)
    
61. 
    
62. # We have our classifier (SVC), our feature selection (SelectKBest), and now,
    
63. # we can plug them together in a *pipeline* that performs the two operations
    
64. # successively:
    
65. from sklearn.pipeline import Pipeline
    
66. anova_svc = Pipeline([('anova', feature_selection), ('svc', svc)])
    
67. 
    
68. #############################################################################
    
69. # Fit the decoder and predict
    
70. 
    
71. anova_svc.fit(X, y)
    
72. y_pred = anova_svc.predict(X)
    
73. 
    
74. #############################################################################
    
75. # Visualize the results
    
76. 
    
77. # Look at the SVC's discriminating weights
    
78. coef = svc.coef_
    
79. # reverse feature selection
    
80. coef = feature_selection.inverse_transform(coef)
    
81. # reverse masking
    
82. weight_img = nifti_masker.inverse_transform(coef)
    
83. 
    
84. 
    
85. # Create the figure
    
86. from nilearn import image
    
87. from nilearn.plotting import plot_stat_map, show
    
88. 
    
89. # Plot the mean image because we have no anatomic data
    
90. mean_img = image.mean_img(func_filename)
    
91. 
    
92. plot_stat_map(weight_img, mean_img, title='SVM weights')
    
93. 
    
94. # Saving the results as a Nifti file may also be important
    
95. weight_img.to_filename('haxby_face_vs_house.nii')
    
96. 
    
97. #############################################################################
    
98. # Obtain prediction scores via cross validation
    
99. 
    
100. from sklearn.cross_validation import LeaveOneLabelOut
     
101. 
     
102. # Define the cross-validation scheme used for validation.
     
103. # Here we use a LeaveOneLabelOut cross-validation on the session label
     
104. # divided by 2, which corresponds to a leave-two-session-out
     
105. cv = LeaveOneLabelOut(session // 2)
     
106. 
     
107. # Compute the prediction accuracy for the different folds (i.e. session)
     
108. cv_scores = []
     
109. for train, test in cv:
     
110.     anova_svc.fit(X[train], y[train])
     
111.     y_pred = anova_svc.predict(X[test])
     
112.     cv_scores.append(np.sum(y_pred == y[test]) / float(np.size(y[test])))
     
113. 
     
114. # Return the corresponding mean prediction accuracy
     
115. classification_accuracy = np.mean(cv_scores)
     
116. 
     
117. # Print the results
     
118. print("Classification accuracy: %.4f / Chance level: %f" %
     
119.       (classification_accuracy, 1. / n_conditions))
     
120. # Classification accuracy: 0.9861 / Chance level: 0.5000
     
121. 
     
122. show()

复制代码

1. """
   
2. Different classifiers in decoding the Haxby dataset
   
3. =====================================================
   
4. Here we compare different classifiers on a visual object recognition
   
5. decoding task.
   
6. """
   
7. 
   
8. #############################################################################
   
9. # We start by loading the data and applying simple transformations to it
   
10. 
    
11. # Fetch data using nilearn dataset fetcher
    
12. from nilearn import datasets
    
13. haxby_dataset = datasets.fetch_haxby(n_subjects=1)
    
14. 
    
15. # print basic information on the dataset
    
16. print('First subject anatomical nifti image (3D) located is at: %s' %
    
17.       haxby_dataset.anat[0])
    
18. print('First subject functional nifti image (4D) is located at: %s' %
    
19.       haxby_dataset.func[0])
    
20. 
    
21. # load labels
    
22. import numpy as np
    
23. labels = np.recfromcsv(haxby_dataset.session_target[0], delimiter=" ")
    
24. stimuli = labels['labels']
    
25. # identify resting state labels in order to be able to remove them
    
26. resting_state = stimuli == b'rest'
    
27. 
    
28. # find names of remaining active labels
    
29. categories = np.unique(stimuli[np.logical_not(resting_state)])
    
30. 
    
31. # extract tags indicating to which acquisition run a tag belongs
    
32. session_labels = labels["chunks"][np.logical_not(resting_state)]
    
33. 
    
34. # Load the fMRI data
    
35. from nilearn.input_data import NiftiMasker
    
36. 
    
37. # For decoding, standardizing is often very important
    
38. mask_filename = haxby_dataset.mask_vt[0]
    
39. masker = NiftiMasker(mask_img=mask_filename, standardize=True)
    
40. func_filename = haxby_dataset.func[0]
    
41. masked_timecourses = masker.fit_transform(
    
42.     func_filename)[np.logical_not(resting_state)]
    
43. 
    
44. 
    
45. #############################################################################
    
46. # Then we define the various classifiers that we use
    
47. 
    
48. # A support vector classifier
    
49. from sklearn.svm import SVC
    
50. svm = SVC(C=1., kernel="linear")
    
51. 
    
52. # The logistic regression
    
53. from sklearn.linear_model import LogisticRegression, RidgeClassifier, \
    
54.     RidgeClassifierCV
    
55. logistic = LogisticRegression(C=1., penalty="l1")
    
56. logistic_50 = LogisticRegression(C=50., penalty="l1")
    
57. logistic_l2 = LogisticRegression(C=1., penalty="l2")
    
58. 
    
59. # Cross-validated versions of these classifiers
    
60. from sklearn.grid_search import GridSearchCV
    
61. # GridSearchCV is slow, but note that it takes an 'n_jobs' parameter that
    
62. # can significantly speed up the fitting process on computers with
    
63. # multiple cores
    
64. svm_cv = GridSearchCV(SVC(C=1., kernel="linear"),
    
65.                       param_grid={'C': [.1, .5, 1., 5., 10., 50., 100.]},
    
66.                       scoring='f1', n_jobs=1)
    
67. 
    
68. logistic_cv = GridSearchCV(LogisticRegression(C=1., penalty="l1"),
    
69.                            param_grid={'C': [.1, .5, 1., 5., 10., 50., 100.]},
    
70.                            scoring='f1')
    
71. logistic_l2_cv = GridSearchCV(LogisticRegression(C=1., penalty="l2"),
    
72.                               param_grid={
    
73.                                   'C': [.1, .5, 1., 5., 10., 50., 100.]},
    
74.                               scoring='f1')
    
75. 
    
76. # The ridge classifier has a specific 'CV' object that can set it's
    
77. # parameters faster than using a GridSearchCV
    
78. ridge = RidgeClassifier()
    
79. ridge_cv = RidgeClassifierCV()
    
80. 
    
81. # A dictionary, to hold all our classifiers
    
82. classifiers = {'SVC': svm,
    
83.                'SVC cv': svm_cv,
    
84.                'log l1': logistic,
    
85.                'log l1 50': logistic_50,
    
86.                'log l1 cv': logistic_cv,
    
87.                'log l2': logistic_l2,
    
88.                'log l2 cv': logistic_l2_cv,
    
89.                'ridge': ridge,
    
90.                'ridge cv': ridge_cv}
    
91. 
    
92. 
    
93. #############################################################################
    
94. # Here we compute prediction scores and run time for all these
    
95. # classifiers
    
96. 
    
97. # Make a data splitting object for cross validation
    
98. from sklearn.cross_validation import LeaveOneLabelOut, cross_val_score
    
99. cv = LeaveOneLabelOut(session_labels)
    
100. 
     
101. import time
     
102. 
     
103. classifiers_scores = {}
     
104. 
     
105. for classifier_name, classifier in sorted(classifiers.items()):
     
106.     classifiers_scores[classifier_name] = {}
     
107.     print(70 * '_')
     
108. 
     
109.     for category in categories:
     
110.         task_mask = np.logical_not(resting_state)
     
111.         classification_target = (stimuli[task_mask] == category)
     
112.         t0 = time.time()
     
113.         classifiers_scores[classifier_name][category] = cross_val_score(
     
114.             classifier,
     
115.             masked_timecourses,
     
116.             classification_target,
     
117.             cv=cv, scoring="f1")
     
118. 
     
119.         print("%10s: %14s -- scores: %1.2f +- %1.2f, time %.2fs" % (
     
120.             classifier_name, category,
     
121.             classifiers_scores[classifier_name][category].mean(),
     
122.             classifiers_scores[classifier_name][category].std(),
     
123.             time.time() - t0))

复制代码

1. """
   
2. Voxel-Based Morphometry on Oasis dataset
   
3. ========================================
   
4. This example uses Voxel-Based Morphometry (VBM) to study the relationship
   
5. between aging and gray matter density.
   
6. The data come from the `OASIS <http://www.oasis-brains.org/>`_ project.
   
7. If you use it, you need to agree with the data usage agreement available
   
8. on the website.
   
9. It has been run through a standard VBM pipeline (using SPM8 and
   
10. NewSegment) to create VBM maps, which we study here.
    
11. Predictive modeling analysis: VBM bio-markers of aging?
    
12. --------------------------------------------------------
    
13. We run a standard SVM-ANOVA nilearn pipeline to predict age from the VBM
    
14. data. We use only 100 subjects from the OASIS dataset to limit the memory
    
15. usage.
    
16. Note that for an actual predictive modeling study of aging, the study
    
17. should be ran on the full set of subjects. Also, parameters such as the
    
18. smoothing applied to the data and the number of features selected by the
    
19. Anova step should be set by nested cross-validation, as they impact
    
20. significantly the prediction score.
    
21. Brain mapping with mass univariate
    
22. -----------------------------------
    
23. SVM weights are very noisy, partly because heavy smoothing is detrimental
    
24. for the prediction here. A standard analysis using mass-univariate GLM
    
25. (here permuted to have exact correction for multiple comparisons) gives a
    
26. much clearer view of the important regions.
    
27. ____
    
28. """
    
29. # Authors: Elvis Dhomatob, <elvis.dohmatob@inria.fr>, Apr. 2014
    
30. #          Virgile Fritsch, <virgile.fritsch@inria.fr>, Apr 2014
    
31. #          Gael Varoquaux, Apr 2014
    
32. import numpy as np
    
33. from scipy import linalg
    
34. import matplotlib.pyplot as plt
    
35. from nilearn import datasets
    
36. from nilearn.input_data import NiftiMasker
    
37. 
    
38. n_subjects = 100  # more subjects requires more memory
    
39. 
    
40. ### Load Oasis dataset ########################################################
    
41. oasis_dataset = datasets.fetch_oasis_vbm(n_subjects=n_subjects)
    
42. gray_matter_map_filenames = oasis_dataset.gray_matter_maps
    
43. age = oasis_dataset.ext_vars['age'].astype(float)
    
44. 
    
45. # print basic information on the dataset
    
46. print('First gray-matter anatomy image (3D) is located at: %s' %
    
47.       oasis_dataset.gray_matter_maps[0])  # 3D data
    
48. print('First white-matter anatomy image (3D) is located at: %s' %
    
49.       oasis_dataset.white_matter_maps[0])  # 3D data
    
50. 
    
51. ### Preprocess data ###########################################################
    
52. nifti_masker = NiftiMasker(
    
53.     standardize=False,
    
54.     smoothing_fwhm=2,
    
55.     memory='nilearn_cache')  # cache options
    
56. gm_maps_masked = nifti_masker.fit_transform(gray_matter_map_filenames)
    
57. n_samples, n_features = gm_maps_masked.shape
    
58. print("%d samples, %d features" % (n_subjects, n_features))
    
59. 
    
60. ### Prediction with SVR #######################################################
    
61. print("ANOVA + SVR")
    
62. # Define the prediction function to be used.
    
63. # Here we use a Support Vector Classification, with a linear kernel
    
64. from sklearn.svm import SVR
    
65. svr = SVR(kernel='linear')
    
66. 
    
67. # Dimension reduction
    
68. from sklearn.feature_selection import VarianceThreshold, SelectKBest, \
    
69.         f_regression
    
70. 
    
71. # Remove features with too low between-subject variance
    
72. variance_threshold = VarianceThreshold(threshold=.01)
    
73. 
    
74. # Here we use a classical univariate feature selection based on F-test,
    
75. # namely Anova.
    
76. feature_selection = SelectKBest(f_regression, k=2000)
    
77. 
    
78. # We have our predictor (SVR), our feature selection (SelectKBest), and now,
    
79. # we can plug them together in a *pipeline* that performs the two operations
    
80. # successively:
    
81. from sklearn.pipeline import Pipeline
    
82. anova_svr = Pipeline([
    
83.             ('variance_threshold', variance_threshold),
    
84.             ('anova', feature_selection),
    
85.             ('svr', svr)])
    
86. 
    
87. ### Fit and predict
    
88. anova_svr.fit(gm_maps_masked, age)
    
89. age_pred = anova_svr.predict(gm_maps_masked)
    
90. 
    
91. # Visualization
    
92. # Look at the SVR's discriminating weights
    
93. coef = svr.coef_
    
94. # reverse feature selection
    
95. coef = feature_selection.inverse_transform(coef)
    
96. # reverse variance threshold
    
97. coef = variance_threshold.inverse_transform(coef)
    
98. # reverse masking
    
99. weight_img = nifti_masker.inverse_transform(coef)
    
100. 
     
101. # Create the figure
     
102. from nilearn.plotting import plot_stat_map, show
     
103. bg_filename = gray_matter_map_filenames[0]
     
104. z_slice = 0
     
105. from nilearn.image.resampling import coord_transform
     
106. affine = weight_img.get_affine()
     
107. _, _, k_slice = coord_transform(0, 0, z_slice,
     
108.                                 linalg.inv(affine))
     
109. k_slice = np.round(k_slice)
     
110. 
     
111. fig = plt.figure(figsize=(5.5, 7.5), facecolor='k')
     
112. weight_slice_data = weight_img.get_data()[..., k_slice, 0]
     
113. vmax = max(-np.min(weight_slice_data), np.max(weight_slice_data)) * 0.5
     
114. display = plot_stat_map(weight_img, bg_img=bg_filename,
     
115.                         display_mode='z', cut_coords=[z_slice],
     
116.                         figure=fig, vmax=vmax)
     
117. display.title('SVM weights', y=1.2)
     
118. 
     
119. # Measure accuracy with cross validation
     
120. from sklearn.cross_validation import cross_val_score
     
121. cv_scores = cross_val_score(anova_svr, gm_maps_masked, age)
     
122. 
     
123. # Return the corresponding mean prediction accuracy
     
124. prediction_accuracy = np.mean(cv_scores)
     
125. print("=== ANOVA ===")
     
126. print("Prediction accuracy: %f" % prediction_accuracy)
     
127. print("")
     
128. 
     
129. ### Inference with massively univariate model #################################
     
130. print("Massively univariate model")
     
131. 
     
132. # Statistical inference
     
133. from nilearn.mass_univariate import permuted_ols
     
134. data = variance_threshold.fit_transform(gm_maps_masked)
     
135. neg_log_pvals, t_scores_original_data, _ = permuted_ols(
     
136.     age, data,  # + intercept as a covariate by default
     
137.     n_perm=2000,  # 1,000 in the interest of time; 10000 would be better
     
138.     n_jobs=1)  # can be changed to use more CPUs
     
139. signed_neg_log_pvals = neg_log_pvals * np.sign(t_scores_original_data)
     
140. signed_neg_log_pvals_unmasked = nifti_masker.inverse_transform(
     
141.     variance_threshold.inverse_transform(signed_neg_log_pvals))
     
142. 
     
143. # Show results
     
144. threshold = -np.log10(0.1)  # 10% corrected
     
145. 
     
146. fig = plt.figure(figsize=(5.5, 7.5), facecolor='k')
     
147. 
     
148. display = plot_stat_map(signed_neg_log_pvals_unmasked, bg_img=bg_filename,
     
149.                         threshold=threshold, cmap=plt.cm.RdBu_r,
     
150.                         display_mode='z', cut_coords=[z_slice],
     
151.                         figure=fig)
     
152. title = ('Negative $\log_{10}$ p-values'
     
153.          '\n(Non-parametric + max-type correction)')
     
154. display.title(title, y=1.2)
     
155. 
     
156. signed_neg_log_pvals_slice_data = \
     
157.     signed_neg_log_pvals_unmasked.get_data()[..., k_slice, 0]
     
158. n_detections = (np.abs(signed_neg_log_pvals_slice_data) > threshold).sum()
     
159. print('\n%d detections' % n_detections)
     
160. 
     
161. show()

复制代码

1. import os, sys; sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
   
2. 
   
3. from pattern.web import Google, plaintext
   
4. from pattern.web import SEARCH
   
5. 
   
6. # The pattern.web module has a SearchEngine class,
   
7. # with a SearchEngine.search() method that yields a list of Result objects.
   
8. # Each Result has url, title, text, language, author and date and properties.
   
9. # Subclasses of SearchEngine include:
   
10. # Google, Bing, Yahoo, Twitter, Facebook, Wikipedia, Wiktionary, Flickr, ...
    
11. 
    
12. # This example retrieves results from Google based on a given query.
    
13. # The Google search engine can handle SEARCH type searches.
    
14. # Other search engines may also handle IMAGE, NEWS, ...
    
15. 
    
16. # Google's "Custom Search API" is a paid service.
    
17. # The pattern.web module uses a test account by default,
    
18. # with a 100 free queries per day shared by all Pattern users.
    
19. # If this limit is exceeded, SearchEngineLimitError is raised.
    
20. # You should obtain your own license key at:
    
21. # https://code.google.com/apis/console/
    
22. # Activate "Custom Search API" under "Services" and get the key under "API Access".
    
23. # Then use Google(license=[YOUR_KEY]).search().
    
24. # This will give you 100 personal free queries, or 5$ per 1000 queries.
    
25. engine = Google(license=None, language="en")
    
26. 
    
27. # Veale & Hao's method for finding similes using wildcards (*):
    
28. # http://afflatus.ucd.ie/Papers/LearningFigurative_CogSci07.pdf
    
29. # This will match results such as:
    
30. # - "as light as a feather",
    
31. # - "as cute as a cupcake",
    
32. # - "as drunk as a lord",
    
33. # - "as snug as a bug", etc.
    
34. q = "as * as a *"
    
35. 
    
36. # Google is very fast but you can only get up to 100 (10x10) results per query.
    
37. for i in range(1, 2):
    
38.     for result in engine.search(q, start=i, count=10, type=SEARCH, cached=True):
    
39.         print plaintext(result.text) # plaintext() removes all HTML formatting.
    
40.         print result.url
    
41.         print result.date
    
42.         print

复制代码

1. import os, sys; sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
   
2. 
   
3. from pattern.web import Google, plaintext
   
4. 
   
5. # A search engine in pattern.web sometimes has custom methods that the others don't.
   
6. # For example, Google has Google.translate() and Google.identify().
   
7. 
   
8. # This example demonstrates the Google Translate API.
   
9. # It will only work with a license key, since it is a paid service.
   
10. # In the Google API console (https://code.google.com/apis/console/),
    
11. # activate Translate API.
    
12. 
    
13. g = Google(license=None) # Enter your license key.
    
14. q = "Your mother was a hamster and your father smelled of elderberries!"    # en
    
15. #   "Ihre Mutter war ein Hamster und euer Vater roch nach Holunderbeeren!"  # de
    
16. print q
    
17. print plaintext(g.translate(q, input="en", output="de")) # fr, de, nl, es, cs, ja, ...
    
18. print
    
19. 
    
20. q = "C'est un lapin, lapin de bois, un cadeau."
    
21. print q
    
22. print g.identify(q) # (language, confidence)

复制代码

1. import os, sys; sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
   
2. 
   
3. from pattern.web import Bing, asynchronous, plaintext
   
4. from pattern.web import SEARCH, IMAGE, NEWS
   
5. 
   
6. import time
   
7. 
   
8. # This example retrieves results from Bing based on a given query.
   
9. # The Bing search engine can retrieve up to a 1000 results (10x100) for a query.
   
10. 
    
11. # Bing's "Custom Search API" is a paid service.
    
12. # The pattern.web module uses a test account by default,
    
13. # with 5000 free queries per month shared by all Pattern users.
    
14. # If this limit is exceeded, SearchEngineLimitError is raised.
    
15. # You should obtain your own license key at:
    
16. # https://datamarket.azure.com/account/
    
17. engine = Bing(license=None, language="en")
    
18. 
    
19. # Quote a query to match it exactly:
    
20. q = "\"is more important than\""
    
21. 
    
22. # When you execute a query,
    
23. # the script will halt until all results are downloaded.
    
24. # In apps with an infinite main loop (e.g., GUI, game),
    
25. # it is often more useful if the app keeps on running
    
26. # while the search is executed in the background.
    
27. # This can be achieved with the asynchronous() function.
    
28. # It takes any function and that function's arguments and keyword arguments:
    
29. request = asynchronous(engine.search, q, start=1, count=100, type=SEARCH, timeout=10)
    
30. 
    
31. # This while-loop simulates an infinite application loop.
    
32. # In real-life you would have an app.update() or similar
    
33. # in which you can check request.done every now and then.
    
34. while not request.done:
    
35.     time.sleep(0.01)
    
36.     print ".",
    
37. 
    
38. print
    
39. print
    
40. 
    
41. # An error occured in engine.search(), raise it.
    
42. if request.error:
    
43.     raise request.error
    
44. 
    
45. # Retrieve the list of search results.
    
46. for result in request.value:
    
47.     print result.text
    
48.     print result.url
    
49.     print

复制代码

1. import os, sys; sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
   
2. 
   
3. from pattern.web import Twitter, hashtags
   
4. from pattern.db  import Datasheet, pprint, pd
   
5. 
   
6. # This example retrieves tweets containing given keywords from Twitter.
   
7. 
   
8. try:
   
9.     # We'll store tweets in a Datasheet.
   
10.     # A Datasheet is a table of rows and columns that can be exported as a CSV-file.
    
11.     # In the first column, we'll store a unique id for each tweet.
    
12.     # We only want to add the latest tweets, i.e., those we haven't seen yet.
    
13.     # With an index on the first column we can quickly check if an id already exists.
    
14.     # The pd() function returns the parent directory of this script + any given path.
    
15.     table = Datasheet.load(pd("cool.csv"))
    
16.     index = set(table.columns[0])
    
17. except:
    
18.     table = Datasheet()
    
19.     index = set()
    
20. 
    
21. engine = Twitter(language="en")
    
22. 
    
23. # With Twitter.search(cached=False), a "live" request is sent to Twitter:
    
24. # we get the most recent results instead of those in the local cache.
    
25. # Keeping a local cache can also be useful (e.g., while testing)
    
26. # because a query is instant when it is executed the second time.
    
27. prev = None
    
28. for i in range(2):
    
29.     print i
    
30.     for tweet in engine.search("is cooler than", start=prev, count=25, cached=False):
    
31.         print
    
32.         print tweet.text
    
33.         print tweet.author
    
34.         print tweet.date
    
35.         print hashtags(tweet.text) # Keywords in tweets start with a "#".
    
36.         print
    
37.         # Only add the tweet to the table if it doesn't already exists.
    
38.         if len(table) == 0 or tweet.id not in index:
    
39.             table.append([tweet.id, tweet.text])
    
40.             index.add(tweet.id)
    
41.         # Continue mining older tweets in next iteration.
    
42.         prev = tweet.id
    
43. 
    
44. # Create a .csv in pattern/examples/01-web/
    
45. table.save(pd("cool.csv"))
    
46. 
    
47. print "Total results:", len(table)
    
48. print
    
49. 
    
50. # Print all the rows in the table.
    
51. # Since it is stored as a CSV-file it grows comfortably each time the script runs.
    
52. # We can also open the table later on: in other scripts, for further analysis, ...
    
53. 
    
54. pprint(table, truncate=100)
    
55. 
    
56. # Note: you can also search tweets by author:
    
57. # Twitter().search("from:tom_de_smedt")

复制代码