Deep Learning Frameworks

What is a Deep Learning Framework?

Software framework or software library used by researchers and data scientists to design and train deep
neural network models. Deep learning frameworks offer building blocks for designing, training and validating deep neural networks, through a high level programming interface. Widely used deep learning frameworks such as Caffe2, MXNet, PyTorch, TensorFlow and others rely on GPU-accelerated libraries such as cuDNN and NCCL to deliver high-performance multi-GPU accelerated training.

Why Use a Deep Learning Framework?

  • Readily available libraries for defining layers, network types (CNNs, RNNs) and common model architectures
  • Support for computer vision applications, image, speech and natural language processing
  • Uses interfaces with popular programming languages such as Python, C, C++ and Scala
  • Most deep learning frameworks are accelerated by NVIDIA deep learning libraries such as cuDNN, NCCl and cuBLAS for GPU accelerated deep learning training

Example Frameworks

  • Easy to use – well defined APIs, documentation
  • Flexible – ideal for researching and prototyping new ideas
  • Several productivity layer options: TensorFlow Slim, SKflow, PrettyTensor, Keras, TFLearn and other
  • Great community engagement and support
  • Large body of existing TensorFlow samples and code, accelerates research
  • Computational graph visualizations via TensorBoard
  •  Python interface
Theano > Automatic differentiation as a symbolic expression
> Computation graph optimizations and on-the-fly code generation for speed,
numerical stability, and memory usage
> Keras productivity layer
> Low-level and flexible for research of new ideas
> Older framework with lot of existing code examples
> Python based, with numpy integration
> no-multiGPU
Caffe > Designed for computer vision framework problems
> Too rigid for researching new algorithms
> Caffe is in maintanence mode, users are expected to migrate to Caffe2
> NVIDIA fork of Caffe is maintained and updated by NVIDIA
> Delivers optimized performance on latest GPU hardware
Caffe 2 > High Performance, scalable framework designed for production use cases
> Cross-platform capability
> Modularity and extensibility
> Mobile deployment support
> Backend merged with PyTorch frontend for flexible user experience with PyTorch 1.0
> Production ready – widely deployed at Facebook for training and
Facebook mobile apps for inference
> Addresses shortcomings of Caffe, with Python support
and modularity


> PyTorch is based on Python. It is the successor of Torch which was based
on the Lua programming language
> Primary audience is researchers
> Supports dynamic computational graphs
> PyTorch 1.0 is a new iteration with PyTorch merged with Caffe2
> Used by Facebook FAIR for research
> Extremely flexible for research
> Shares the same backend as popular Torch framework
Preferred Networks


> Models that are fast to prototype and easier to debug
> CuPy: NumPy-equivalent multi-dimensional CUDA® array-library
> Extensions & Tools: ChainerRL, ChainerMN, for computer vision
> Dynamic computation graphs with a Python API are strengths
of Chainer and PyTorch
Microsoft Cognitive Toolkit > Flexible, Scalable computation graph on multiple-GPUs and cross machines
> Efficient RNN implementation for variable length sequence
> Keras productivity layer
> Multi-GPU and Multi-node scaling
> Fast LSTM RNN performance for speech and language problems
Matlab > Beyond just DL classifiers: combine image/vision processing with DL
> Automates distributed training and deployment to data center and embedded
without manual coding
> High productivity desktop IDE (integrated development environment)
that makes research, prototyping and debugging easy
> MATLAB expert technical support
> Easily integrates into existing MATLAB and simulink workflows