Deep Learning library with GPU(CUDA/cuBLAS)

Chat / Discussions
,
21

Please give me some advice.

I’m adding CNN functionality and repeating tests on the CIFAR10 data set. But it’s not working. It will help me if I can see how the image is transformed by CNN. Please let me know if you have any graphics tools for Elixir.

22

There might be some ascii image libraries. Matrex library has a ascii heat maps function you could copy. I think there are some tools for ImageMagick but don’t know if they’d support bitmaps to images. There’s gnuplot wrapper too.

2 Likes
23

Thank you.

1 Like
24

I have installed Matrex and use heatmap. Thank you.

2 Likes
25

I have added code to check the gradient confirmation. I am checking whether the function of CNN is correct. The goal is to be able to recognize images with CIFAR10. If you have any advice, thank you.

2 Likes
26

I am verifying the operation of CNN. Many bugs have been fixed and now it is more stable than before. However, DP2 still can’t learn CIFAR10 well. I am still checking the gradient calculation.

3 Likes
27

DP2 has learned for MNIST, 28 * 28 image data.
The notation of convolution of neural network means the following.

e.g. f (5, 5, 1, 12, 1, 1, 0.5, 0.0001)
Filter size 5 * 5
Input channel 1
Output channel 12
Stride 1
padding 1
rate to multiply initial random number 0.5
Learning rate 0.0001

CNN test for MNIST
defnetwork init_network4(_x) do
_x
|> f(5, 5, 1, 12, 1, 1, 0.5, 0.0001)
|> pooling(2)
|> f(3, 3, 12, 12, 1, 1, 0.5, 0.0001)
|> f(2, 2, 12, 12, 1, 1, 0.5, 0.0001)
|> pooling(2)
|> f(3, 3, 12, 12, 1, 0, 0.5, 0.0001)
|> relu
|> full
|> w(300, 10, 0.1, 0.001)
|> softmax
end

iex(1)> Test.cnn(100,100)
preparing data
learning start
100 13.068114280700684
99 13.220925331115723
98 12.160013198852539
97 12.39694595336914
96 12.35841178894043
95 9.357890129089355
94 10.0624361038208
…
6 6.846850395202637
5 7.928662300109863
4 8.563251495361328
3 6.413776874542236
2 6.471324920654297
1 8.531964302062988
learning end
accuracy rate = 0.398
“time: 47.943411 second”
“-------------”
:ok

I have the code for CIFAR10 in the cifar.ex file. But so far DP2 is not well learned. If you have any advice, thank you.

3 Likes
28

A bug was found in convolution. I have fixed a bug and I am testing on CIFAR10. The GPU of my machine is GTX960. It takes 8 and a half hours to learn 20 epochs of 10,000 data in CIFAR10. I am looking for the proper hyperparameters. I would appreciate it if you could help with the test.

1 Like
29

In CIFAR’s code it is by SGD method. I presume that there is stagnation due to saddle points. I am improving by the Momentum method.

1 Like
30

I only understand about 1% of what you are doing but I think this work is incredibly valuable to the community. Please keep going!

4 Likes
31

Thank you very much.

2 Likes
32

I gave up CIFAR10 and tested DP2 with a different dataset. The dataset is Fashon-MNIST. Since the file has the same structure as MNIST, the code is simple. The correct answer rate is also good. I use this dataset to check and improve CNN functionality. DP2 will be released at the end of June 2020.

5 Likes
33

Although it is still incomplete, I registered DP2 on Hex. I will improve it further.

5 Likes
34

sorry. There was a bug. The Hex version of DP2 could not find the nifs.so file. I fixed it in ver1.1.2.

2 Likes
35

I posted to medium.

4 Likes
36

I found DeepPipe2 works well on TITAN RTX and Jetson Nano, though the latter requires much execution time.

2 Likes
37

I also registered Elxlog on the Hex. Elxlog is a Prolog interpreter and compiler written in Elixir. I am thinking of using Prolog, which is classical artificial intelligence, and Deep-Learning, which is modern artificial intelligence, in combination.

7 Likes
38

I implemented the Xavier method and the He method in ver1.1.6 for generating initial values. As a result, I obtained a Fashion-MNIST accuracy rate 86% for the two-layer CNN. I think CNN of DeepPipe2 is working.

Then I will work on improving execution speed. Convolution of DP2 is slow. It does not utilize GPU resources on executing multiple channels. The implementation of convolution was naive in the prototype. Needs improvement.

8 Likes
39

In ver1.1.7, I improved the speed of convolution and pooling. As a result, DP2 is approximately 4.5 times faster with the Fashion-MNIST dataset.

It will be in time for the official release on June 30, 2020. I am relieved.

8 Likes
40

In ver1.1.8, I added Adam and RMSprop optimizer. By using this, DP2 can be trained on the CIFAR10 dataset.

DP2 opens the way to deep-learning with practical training time in Elixir.

8 Likes