Deep Neural Networks with GPU support

是 否 +2 论坛币

k人 参与回答

经管之家送您一份

应届毕业生专属福利!

求职就业群

赵安豆老师微信：zhaoandou666

经管之家联合CDA

送您一个全额奖学金名额~ !

立即领取

感谢您参与论坛问题回答

经管之家送您两个论坛币！

+2 论坛币

Deep Neural Networks with GPU Support

This is a Java implementation of some of the algorithms for training deep neural networks. GPU support is provided via the OpenCL and Aparapi. The architecture is designed with modularity, extensibility and pluggability in mind.

Git structure

I'm using the git-flow model. The most stable (but older) sources are available in the master branch, while the latest ones are in the develop branch.

If you want to use the previous Java 7 compatible version you can check out this release.

Neural network types

• Multilayer perceptron
• Restricted Boltzmann Machine
• Autoencoder
• Deep belief network
• Stacked autoencodeer
• Convolutional networks with max pooling, average poolng and stochastic pooling.
• Maxout networks (work-in-progress)
  

Training algorithms

• Backpropagation - supports multilayer perceptrons, convolutional networks and dropout.
• Contrastive divergence and persistent contrastive divergence implemented using these and these guidelines.
• Greedy layer-wise training for deep networks - works for stacked autoencoders and DBNs, but supports any kind of training.
  

All the algorithms support GPU execution.

Out of the box supported datasets are MNIST, CIFAR-10/CIFAR-100 (experimental, not much testing), IRIS and XOR, but you can easily implement your own.

Experimental support of RGB image preprocessing operations - affine transformations, cropping, and color scaling (see Generaltest.java -> testImageInputProvider).

Activation functions

• Logistic
• Tanh
• Rectifiers
• Softplus
• Softmax
• Weighted sum
  

All the functions support GPU execution. They can be applied to all types of networks and all training algorithms. You can also implement new activations.

How to build the library

• Java 8.
• To build the project you need gradle or maven. If you don't use any of these you can go to the project folder and execute thegradlew console command, which will automatically setup gradle environment for you.
• I'm also uploading the latest jar file (with bundled dependencies and sources) here.
• Depending on your environment you might need to download the relevant aparapi .dll or .so file (located in the root of each archive) from here and add it's location to the system PATH variable. (This)[https://code.google.com/p/aparapi/wiki/DevelopersGuideLinux] is a guide on how to set up OpenCL in linux environment.
  

How to run the samples

The samples are organized as unit tests. If you want see examples on various popular datasets you can go to nn-samples/src/test/java/com/github/neuralnetworks/samples/.

Library structure

There are two projects:

• nn-core - contains the full implementation.
• nn-samples - contains implementations of popular datasets and
  

The software design is tiered, each tier depending on the previous ones.

Network architecture

This is the first "tier". Each network is defined by a list of layers. Each layer has a set of connections that link it to the other layers of the network, making the network a directed acyclic graph. This structure can accommodate simple feedforwad nets, but also more complex architectures like http://www.cs.toronto.edu/~hinton/absps/imagenet.pdf. You can build your own specific network.

Data propagation

This tier is propagating data through the network. It takes advantage of it's graph structure. There are two main base components:

• LayerCalculator - propagates data through the graph. It receives target layer and input data clamped to a given layer (considered an input layer). It ensures that the data is propagated through the layers in the correct order and that all the connections in the graph are calculated. For example, during the feedforward phase of backpropagation the training data is clamped to the input layer and is propagated to the target layer (the output layer of the network). In the bp phase the output error derivative is clamped as "input" to the layer and the weights are updated using breadth-first graph traversal starting from the output layer. Essentially the role of the LayerCalculator is to provide the order, in which the network layers are calculated.
• ConnectionCalculator - base class for all neuron types (sigmoid, rectifiers, convolutional etc.). After the order of calculation of the layers is determined by the LayerCalculator, then the list of input connections for each layer is calculated by theConnectionCalculator.
  

GPU

Most of the ConnectionCalculator implementations are optimized for GPU execution. Aparapi imposes some important restrictions on the code that can be executed on the GPU. The most significant are:

• only one-dimensional arrays (and variables) of primitive data types are allowed. It is not possible to use complex objects.
• only member-methods of the Aparapi Kernel class itself are allowed to be called from the GPU executable code.
  

Therefore before each GPU calculation all the data is converted to one-dim arrays and primitive type variables. Because of this all Aparapi neuron types are using either AparapiWeightedSum (for fully connected layers and weighted sum input functions),AparapiSubsampling2D (for subsampling layers) or AparapiConv2D (for convolutional layers). Most of the data is represented as one-dimensional array by default (for example Matrix).

Training

All the trainers are using the Trainer base class. They are optimized to run on the GPU, but you can plug-in other implementations and new training algorithms. The training procedure has training and testing phases. Each Trainer receives parameters (for example learning rate, momentum, etc) via Properties (a HashMap). For the supported properties for each trainer please check the TrainerFactory class.

Input data

Input is provided to the neural network by the trainers via TrainingInputProvider interface. Each TrainingInputProvider provides training samples in the form of TrainingInputData (default implementation is TrainingInputDataImpl). The input can be modified by a list of modifiers - for example MeanInputFunction (for subtracting the mean value) and ScalingInputFunction (scaling within a range). Currently MnistInputProvider and IrisInputProvider are implemented.

Author

Ivan Vasilev (ivanvasilev [at] gmail (dot) com)

License

MIT License

本帖隐藏的内容

neuralnetworks-master (1).zip (5.36 MB)

2016-9-3 08:29:30 上传

扫码加我 拉你入群

请注明：姓名-公司-职位

以便审核进群资格，未注明则拒绝

分享0 收藏2 回帖

关键词：Networks network Support Neural Works available designed previous provided training

1. package com.github.neuralnetworks.samples.cifar;
   
2. 
   
3. import java.awt.image.BufferedImage;
   
4. import java.awt.image.DataBufferByte;
   
5. import java.io.File;
   
6. import java.io.FileNotFoundException;
   
7. import java.io.IOException;
   
8. import java.io.RandomAccessFile;
   
9. import java.util.ArrayList;
   
10. import java.util.List;
    
11. 
    
12. import com.github.neuralnetworks.input.ImageInputProvider;
    
13. import com.github.neuralnetworks.util.Util;
    
14. 
    
15. /**
    
16. * Input provider for the CIFAR-10 and CIFAR-100 datasets. Requires location of the CIFAR images
    
17. * files (not included in the library). Do not use this class directly, but use the subclasses instead
    
18. * Experimental
    
19. */
    
20. public abstract class CIFARInputProvider extends ImageInputProvider {
    
21. 
    
22.     private static final long serialVersionUID = 1L;
    
23. 
    
24.     protected RandomAccessFile files[];
    
25.     protected int labelSize;
    
26.     protected int inputSize;
    
27.     protected byte[] nextInputRaw;
    
28.     protected float[] nextTarget;
    
29.     private List<Integer> elementsOrder;
    
30. 
    
31.     private CIFARInputProvider() {
    
32.         super();
    
33.         this.elementsOrder = new ArrayList<>();
    
34.         this.nextInputRaw = new byte[3072];
    
35.     }
    
36. 
    
37.     @Override
    
38.     public int getInputSize() {
    
39.         return inputSize;
    
40.     }
    
41. 
    
42.     @Override
    
43.     public float[] getNextTarget() {
    
44.         return nextTarget;
    
45.     }
    
46. 
    
47.     @Override
    
48.     public float[] getNextInput() {
    
49.         // if no transformations are required and the data is grouped by color
    
50.         // channel the code can be optimized
    
51.         if (!requireAugmentation() && getProperties().getGroupByChannel()) {
    
52.             if (nextInput == null) {
    
53.                 nextInput = new float[3072];
    
54.             }
    
55. 
    
56.             float scaleColors = getProperties().getScaleColors() ? 255 : 1;
    
57.             for (int i = 0; i < nextInput.length; i++) {
    
58.                 nextInput[i] = (nextInputRaw[i] & 0xFF) / scaleColors;
    
59.             }
    
60. 
    
61.             return nextInput;
    
62.         }
    
63. 
    
64.         return super.getNextInput();
    
65.     }
    
66. 
    
67.     @Override
    
68.     protected BufferedImage getNextImage() {
    
69.         BufferedImage image = new BufferedImage(32, 32, BufferedImage.TYPE_3BYTE_BGR);
    
70.         byte[] pixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
    
71. 
    
72.         for (int i = 0; i < 1024; i++) {
    
73.             pixels[i * 3] = nextInputRaw[1024 * 2 + i];
    
74.             pixels[i * 3 + 1] = nextInputRaw[1024 + i];
    
75.             pixels[i * 3 + 2] = nextInputRaw[i];
    
76.         }
    
77. 
    
78.         return image;
    
79.     }
    
80. 
    
81.     @Override
    
82.     public void beforeSample() {
    
83.         if (elementsOrder.size() == 0) {
    
84.             resetOrder();
    
85.         }
    
86. 
    
87.         int currentEl = elementsOrder.remove(getProperties().getUseRandomOrder() ? getProperties().getRandom().nextInt(elementsOrder.size()) : 0);
    
88.         int id = currentEl % (getInputSize() / files.length);
    
89. 
    
90.         RandomAccessFile f = files[currentEl / (getInputSize() / files.length)];
    
91. 
    
92.         try {
    
93.             f.seek(id * (3072 + labelSize));
    
94.             if (labelSize > 1) {
    
95.                 f.readUnsignedByte();
    
96.             }
    
97. 
    
98.             Util.fillArray(nextTarget, 0);
    
99.             nextTarget[f.readUnsignedByte()] = 1;
    
100. 
     
101.             f.readFully(nextInputRaw);
     
102.         } catch (IOException e) {
     
103.             e.printStackTrace();
     
104.         }
     
105.     }
     
106. 
     
107.     @Override
     
108.     public void reset() {
     
109.         super.reset();
     
110.         resetOrder();
     
111.     }
     
112. 
     
113.     public void resetOrder() {
     
114.         elementsOrder = new ArrayList<Integer>(getInputSize());
     
115.         for (int i = 0; i < getInputSize(); i++) {
     
116.             elementsOrder.add(i);
     
117.         }
     
118.     }
     
119. 
     
120.     public static class CIFAR10TrainingInputProvider extends CIFARInputProvider {
     
121. 
     
122.         private static final long serialVersionUID = 1L;
     
123. 
     
124.         /**
     
125.          * @param directory - the folder where the CIFAR files are located
     
126.          */
     
127.         public CIFAR10TrainingInputProvider(String directory) {
     
128.             super();
     
129. 
     
130.             this.labelSize = 1;
     
131.             this.inputSize = 50000;
     
132.             this.nextTarget = new float[10];
     
133.             this.files = new RandomAccessFile[5];
     
134. 
     
135.             try {
     
136.                 if (!directory.endsWith(File.separator)) {
     
137.                     directory += File.separator;
     
138.                 }
     
139. 
     
140.                 files[0] = new RandomAccessFile(directory + "data_batch_1.bin", "r");
     
141.                 files[1] = new RandomAccessFile(directory + "data_batch_2.bin", "r");
     
142.                 files[2] = new RandomAccessFile(directory + "data_batch_3.bin", "r");
     
143.                 files[3] = new RandomAccessFile(directory + "data_batch_4.bin", "r");
     
144.                 files[4] = new RandomAccessFile(directory + "data_batch_5.bin", "r");
     
145.             } catch (FileNotFoundException e) {
     
146.                 e.printStackTrace();
     
147.             }
     
148.         }
     
149.     }
     
150. 
     
151.     public static class CIFAR10TestingInputProvider extends CIFARInputProvider {
     
152. 
     
153.         private static final long serialVersionUID = 1L;
     
154. 
     
155.         /**
     
156.          * @param directory - the folder where the CIFAR files are located
     
157.          */
     
158.         public CIFAR10TestingInputProvider(String directory) {
     
159.             super();
     
160. 
     
161.             this.labelSize = 1;
     
162.             this.inputSize = 10000;
     
163.             this.nextTarget = new float[10];
     
164.             this.files = new RandomAccessFile[1];
     
165. 
     
166.             try {
     
167.                 if (!directory.endsWith(File.separator)) {
     
168.                     directory += File.separator;
     
169.                 }
     
170. 
     
171.                 files[0] = new RandomAccessFile(directory + "test_batch.bin", "r");
     
172.             } catch (FileNotFoundException e) {
     
173.                 e.printStackTrace();
     
174.             }
     
175.         }
     
176.     }
     
177. 
     
178.     public static class CIFAR100TrainingInputProvider extends CIFARInputProvider {
     
179. 
     
180.         private static final long serialVersionUID = 1L;
     
181. 
     
182.         /**
     
183.          * @param directory - the folder where the CIFAR files are located
     
184.          */
     
185.         public CIFAR100TrainingInputProvider(String directory) {
     
186.             super();
     
187. 
     
188.             this.labelSize = 2;
     
189.             this.inputSize = 50000;
     
190.             this.nextTarget = new float[100];
     
191.             this.files = new RandomAccessFile[5];
     
192. 
     
193.             try {
     
194.                 if (!directory.endsWith(File.separator)) {
     
195.                     directory += File.separator;
     
196.                 }
     
197. 
     
198.                 files[0] = new RandomAccessFile(directory + "data_batch_1.bin", "r");
     
199.                 files[1] = new RandomAccessFile(directory + "data_batch_2.bin", "r");
     
200.                 files[2] = new RandomAccessFile(directory + "data_batch_3.bin", "r");
     
201.                 files[3] = new RandomAccessFile(directory + "data_batch_4.bin", "r");
     
202.                 files[4] = new RandomAccessFile(directory + "data_batch_5.bin", "r");
     
203.             } catch (FileNotFoundException e) {
     
204.                 e.printStackTrace();
     
205.             }
     
206.         }
     
207.     }
     
208. 
     
209.     public static class CIFAR100TestingInputProvider extends CIFARInputProvider {
     
210.        
     
211.         private static final long serialVersionUID = 1L;
     
212. 
     
213.         /**
     
214.          * @param directory - the folder where the CIFAR files are located
     
215.          */
     
216.         public CIFAR100TestingInputProvider(String directory) {
     
217.             super();
     
218. 
     
219.             this.labelSize = 2;
     
220.             this.inputSize = 10000;
     
221.             this.nextTarget = new float[100];
     
222.             this.files = new RandomAccessFile[1];
     
223. 
     
224.             try {
     
225.                 if (!directory.endsWith(File.separator)) {
     
226.                     directory += File.separator;
     
227.                 }
     
228. 
     
229.                 files[0] = new RandomAccessFile(directory + "test_batch.bin", "r");
     
230.             } catch (FileNotFoundException e) {
     
231.                 e.printStackTrace();
     
232.             }
     
233.         }
     
234.     }
     
235. }

复制代码

1. package com.github.neuralnetworks.architecture;
   
2. 
   
3. import java.io.Serializable;
   
4. import java.util.List;
   
5. import java.util.Set;
   
6. 
   
7. import com.github.neuralnetworks.calculation.LayerCalculator;
   
8. 
   
9. /**
   
10. * this interface is implemented by everything that wants to present itself as a
    
11. * black box with with a list of input/output layers for example these could be
    
12. * whole neural network taking part in committee of machines, single
    
13. * convolutional/subsamplingo layers or even a single connection between the layers
    
14. */
    
15. public interface NeuralNetwork extends Serializable {
    
16. 
    
17.     /**
    
18.      * input layer
    
19.      */
    
20.     public Layer getInputLayer();
    
21. 
    
22.     /**
    
23.      * @return output layer
    
24.      */
    
25.     public Layer getOutputLayer();
    
26. 
    
27.     /**
    
28.      * @return all the layers in this network
    
29.      */
    
30.     public Set<Layer> getLayers();
    
31. 
    
32.     /**
    
33.      * @return all the connections in this network
    
34.      */
    
35.     public List<Connections> getConnections();
    
36. 
    
37.     /**
    
38.      * LayerCalculator associated to this network
    
39.      */
    
40.     public LayerCalculator getLayerCalculator();
    
41. }

复制代码

1. package com.github.neuralnetworks.architecture;
   
2. 
   
3. import java.util.Collection;
   
4. import java.util.List;
   
5. import java.util.Set;
   
6. 
   
7. import com.github.neuralnetworks.calculation.LayerCalculator;
   
8. import com.github.neuralnetworks.util.Constants;
   
9. import com.github.neuralnetworks.util.Properties;
   
10. import com.github.neuralnetworks.util.UniqueList;
    
11. import com.github.neuralnetworks.util.Util;
    
12. 
    
13. /**
    
14. * Base class for all types of neural networks. A neural network is defined only
    
15. * by the layers it contains. The layers themselves contain the connections with
    
16. * the other layers.
    
17. */
    
18. public class NeuralNetworkImpl implements NeuralNetwork {
    
19. 
    
20.     private static final long serialVersionUID = 1L;
    
21. 
    
22.     private Set<Layer> layers;
    
23.     private Properties properties;
    
24. 
    
25.     public NeuralNetworkImpl() {
    
26.         super();
    
27.         this.layers = new UniqueList<Layer>();
    
28.     }
    
29. 
    
30.     @Override
    
31.     public LayerCalculator getLayerCalculator() {
    
32.         return properties != null ? properties.getParameter(Constants.LAYER_CALCULATOR) : null;
    
33.     }
    
34. 
    
35.     public void setLayerCalculator(LayerCalculator layerCalculator) {
    
36.         if (properties == null) {
    
37.             properties = new Properties();
    
38.         }
    
39. 
    
40.         properties.setParameter(Constants.LAYER_CALCULATOR, layerCalculator);
    
41.     }
    
42. 
    
43.     @Override
    
44.     public Set<Layer> getLayers() {
    
45.         return layers;
    
46.     }
    
47. 
    
48.     public void setLayers(Set<Layer> layers) {
    
49.         this.layers = layers;
    
50.     }
    
51. 
    
52.     public Properties getProperties() {
    
53.         return properties;
    
54.     }
    
55. 
    
56.     public void setProperties(Properties properties) {
    
57.         this.properties = properties;
    
58.     }
    
59. 
    
60.     /*
    
61.      * (non-Javadoc)
    
62.      *
    
63.      * @see com.github.neuralnetworks.architecture.NeuralNetwork#getInputLayer()
    
64.      * Default implementation - the input layer is that layer, which doesn't
    
65.      * have any inbound connections
    
66.      */
    
67.     @Override
    
68.     public Layer getInputLayer() {
    
69.         return layers.stream().filter(l -> l.getConnections(this).stream().noneMatch(c -> l == c.getOutputLayer() && !Util.isBias(c.getInputLayer()))).findFirst().orElse(null);
    
70.     }
    
71. 
    
72.     @Override
    
73.     public Layer getOutputLayer() {
    
74.         return getNoOutboundConnectionsLayer();
    
75.     }
    
76. 
    
77.     protected Layer getNoOutboundConnectionsLayer() {
    
78.         return layers.stream().filter(l -> l.getConnections(this).stream().noneMatch(c -> l == c.getInputLayer())).findFirst().orElse(null);
    
79.     }
    
80. 
    
81.     /*
    
82.      * (non-Javadoc)
    
83.      *
    
84.      * @see
    
85.      * com.github.neuralnetworks.architecture.NeuralNetwork#getConnections()
    
86.      * Returns list of all the connections within the network. The list is
    
87.      * retrieved by iterating over all the layers. Only connections that have
    
88.      * both layers in this network are returned.
    
89.      */
    
90.     @Override
    
91.     public List<Connections> getConnections() {
    
92.         List<Connections> result = new UniqueList<>();
    
93.         if (layers != null) {
    
94.             layers.forEach(l -> result.addAll(l.getConnections(this)));
    
95.         }
    
96. 
    
97.         return result;
    
98.     }
    
99. 
    
100.     /**
     
101.      * @param inputLayer
     
102.      * @param outputLayer
     
103.      * @return Connection between the two layers if it exists
     
104.      */
     
105.     public Connections getConnection(Layer inputLayer, Layer outputLayer) {
     
106.         return getConnections().stream().filter(c -> (c.getInputLayer() == inputLayer && c.getOutputLayer() == outputLayer) || (c.getInputLayer() == outputLayer && c.getOutputLayer() == inputLayer)).findFirst().orElse(null);
     
107.     }
     
108. 
     
109.     /**
     
110.      * Add layer to the network
     
111.      *
     
112.      * @param layer
     
113.      * @return whether the layer was added successfully
     
114.      */
     
115.     public boolean addLayer(Layer layer) {
     
116.         if (layer != null) {
     
117.             if (layers == null) {
     
118.                 layers = new UniqueList<>();
     
119.             }
     
120. 
     
121.             if (!layers.contains(layer)) {
     
122.                 layers.add(layer);
     
123.                 return true;
     
124.             }
     
125.         }
     
126. 
     
127.         return false;
     
128.     }
     
129. 
     
130.     /**
     
131.      * Remove layer from the network
     
132.      *
     
133.      * @param layer
     
134.      */
     
135.     public void removeLayer(Layer layer) {
     
136.         if (layer != null) {
     
137.             if (layers != null) {
     
138.                 // remove layer and bias layers
     
139.                 layers.remove(layer);
     
140.                 layer.getConnections(this).stream().map(Connections::getInputLayer).filter(l -> Util.isBias(l)).forEach(l -> layers.remove(l));
     
141.             }
     
142.         }
     
143.     }
     
144. 
     
145.     /**
     
146.      * Add layers to the network
     
147.      *
     
148.      * @param newLayers
     
149.      */
     
150.     public void addLayers(Collection<Layer> newLayers) {
     
151.         if (newLayers != null) {
     
152.             if (layers == null) {
     
153.                 layers = new UniqueList<>();
     
154.             }
     
155. 
     
156.             newLayers.stream().filter(l -> !layers.contains(l)).forEach(l -> layers.add(l));
     
157.         }
     
158.     }
     
159. 
     
160.     /**
     
161.      * Add connection to the network - this means adding both input and output
     
162.      * layers to the network
     
163.      *
     
164.      * @param connection
     
165.      */
     
166.     public void addConnections(Connections... connections) {
     
167.         if (connections != null) {
     
168.             for (Connections c : connections) {
     
169.                 addLayer(c.getInputLayer());
     
170.                 addLayer(c.getOutputLayer());
     
171.             }
     
172.         }
     
173.     }
     
174. }

复制代码

thanks for sharing

谢谢分享