Deep Learning Module in TMVA

Intro

As part of the Google Summer of Code (GSoC) 2017, under the mentorship of CERN-HSF umbrella organization, is the development of a Deep Learning Module in TMVA (Toolkit for Multivariate Analysis) which includes almost every Machine Learning method, except the Deep Learning one. TMVA is a submodule in ROOT, a data-analysis tool for High Energy Physics.

Since this kind of a module is already too vast, we are a group of three students working on three different parts of the module. I am working on the Convolutional Neural Networks, one student is working on the Recurrent Neural Networks and the last one is working on the Auto Encoders.

Although these networks have a different kind of nature and goals, thinking on a Software Engineering plane, they all have a common ground and share to a big extent the same patterns.

For this reason, it was challenging to design such a module, that will satisfy and fit all needs, be easily maintainable and extensible and in the same time compatible with the TMVA standards. In the following blog post, I will give a general and brief overview of the Software Engineering solution for this Deep Learning Module within TMVA.

Solution

The solution and the implementation of this Deep Learning Module is motivated and it is based on the work done by Simon Pfreundschuh, GSoC 2016 student, who developed the foundations and enabled us to continue and add more complex and sophisticated solutions on top of it. The blog, describing his project can be found here.

The design of the entire Deep Learning Module is given in the figure below. As it can be seen, it is a big puzzle, so let me give you a more detailed view.

An overview of the Deep Learning Module in TMVA

Layers

In general, one deep neural network, is a configuration of several layers with different nature. For this reason, the layers are playing the central role. Each layer is specialized to perform one particular operation on the input in order to produce some output that can be used by some other layer and so on.

In order to enable this smooth chaining of the layers, they all extend the generic, pure virtual General Layer class, which design is shown in the diagram below. The introduction of the generics, decouples the algorithmic implementation of the layers and the hardware-specific implementations of the underlying numerical operations. There are three types of low-level interfaces or also called architectures: Reference, CPU and GPU. The Reference one, is only for testing purposes and it is not using any specialized library. The CPU architecture is based on the BLAS (Basic Linear Algebra Subprograms) library, which is highly optimized for matrix multiplications. CuDA is the only support for a GPU-based execution.

Since the generic General Layer class, lies at the bottom of this module, it entails almost every class to be generic. We can see the main methods and attributes of this class in the image below.

General Layer class

Thus, the generic Deep Net class contains a vector of pointers to General Layer class objects, therefore leveraging the polymorphic mechanism. It is responsible for the entire forward and backward propagation through the layers, as well as for the initialization, loss calculation and prediction.

Tensor Data Loader and DL Minimizer

The Tensor Data Loader and DL Minimizer classes are individual and independent classes, that represent crucial service classes. The Tensor Data Loader class is managing the streaming of the training and testing data to the accelerator device in a round-robin manner. It creates a set of batches by shuffling the training and testing examples. Therefore, a deep neural network is fed using these batches.

The DL Minimizer class represents a generic, architecture and input data independent implementation of the gradient descent minimization algorithm. It does that, by providing the functions, Step, StepMomentum and StepNesterov, that perform a single minimization step using different techniques.

Method DL class

This is the main class, assembling everything that was mentioned before. First, it parses the user input specifying the Deep Net configuration, and accordingly instantiating it. Then, given the training and testing settings it is managing the process of training, testing and evaluating the Deep Neural Net.

Conclucion

This blog post was only an outline and the general idea behind the Deep Learning Module in TMVA. There are a lot of details that were intentionally missed and that can be found in the official documentation. Working on this collaborative project was very challenging and in the same time funny. I learned a lot and I hope we established a solid foundation for the further development and advancing of this module.