Suggest edit — Deep Learning Frameworks

Title

Name

Note

---
title: "Deep Learning Frameworks"
visibility: public
---

# Deep Learning Frameworks

Category: [[machine-intelligence|Machine Intelligence]]

[Read the original document](https://docs.google.com/document/d/1zvo_HQKdNhxqSt4SfQCyXoiC0RmqwvvXI3mFGi_Tiug/edit?usp=drivesdk&sa=D&ust=1596495076582000&usg=AOvVaw3aSDt1hgGZ20h9x8F-K1RN)

---

Major Frameworks

1. Tensorflow
2. Torch / PyTorch
3. Caffe / Caffe2
4. MXNet
5. CNTK
6. DL4J
7. Theano
8. Chainer

Minor Frameworks
1. CoreML
2. Nirvana Neon
3. DyNet
4. Dsstne 
5. PaddlePaddle
6. SystemML
7. BigDL

Higher Level APIs / Wrappers
1. Keras [Theano, Tensorflow, MXNet, CNTK]
2. Sonnet [Tensorflow]
3. TFLearn [Tensorflow]
4. TFSlim [Tensorflow]
5. Gluon [MXNet]
6. Lasagne [Theano]
7. ScalNet [DL4J]

Technical Differentiating Factors

1. Static vs. Dynamic Graph Declaration
2. Autograd (Reverse Mode)
3. Model Parallelism
4. Compiler
5. Language API
6. RNN Support
7. Productionizability / Model Serving
8. Long Tail Support
   1. Localization
   2. Segmentation
   3. Regression
   4. Generation
   5. ...
9. Custom Extensions
10. Speed
   1. Chainer Benchmarks
      1. https://chainer.org/general/2017/02/08/Performance-of-Distributed-Deep-Learning-Using-ChainerMN.html
   2. Convnet-benchmarks (Soumith)
      1. https://github.com/soumith/convnet-benchmarks
   3. DeepBench (Baidu)
      1. https://github.com/baidu-research/DeepBench
   4. Benchmarking DL Software Tools (Hong Kong Baptist University Paper)
      1. https://arxiv.org/pdf/1608.07249.pdf
11. Hardware
12. Model / Data Parallelism
13. Adoption

Other Overviews & Comparisons

1. Chainer Comparisons
   1. https://docs.chainer.org/en/latest/comparison.html
2. DL4J [2016]
   1. https://deeplearning4j.org/compare-dl4j-torch7-pylearn
3. DeepFrameworks [2016]
   1. https://github.com/zer0n/deepframeworks

Tensorflow

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. TPU support - important downside, code optimized for hardware unavailable without google cloud.
3. RNN support, though static graph makes variable structural inputs difficult. Hard-coded solutions for variable input lengths in RNNs. Substantially slower LSTM / GRU / RNNs than Torch / CNTK / others.
4. CNN support, full. Early to have new architectures implemented as custom kernels.
5. Slower than other frameworks in part due to lack of inplace matrix operations, forced to copy a matrix to operate on it
6. Python API. Too low level for most data scientists, leading to the popularity of many wrappers (TFlearn, Keras, Sonnet). Other APIs available - ex. C++ for production.
   1. Somewhat bloated API.
7. Strongly supported by the Google Brain team.
8. Compiler for linear algebra operations - XLA.
9. Computational graph tooling still closed-source.
10. Excellent visualization w/ Tensorboard.
11. Autograd for reverse mode automatic differentiation
12. Static computational graph. Upsides for production, downsides for flexible inputs - see bottom comparison for more information.
13. Model and data parallelism.
14. Extremely robust documentation
15. Broad and robust model serving options, but through Tensorflow Serving. Mobile serving for Android and IOS.
16. Extremely strong adoption in research and industry.

Torch / PyTorch

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. Strong, fast RNN / LSTM / GRU support.
3. Full CNN support.
4. API for Torch is in LuaJIT, API for PyTorch is in Python. Differing back-ends as well, PyTorch being more modern.
5. Supported by Facebook, Twitter, and various academic labs. Used to be supported by Deepmind.
6. No Linear Algebra Compiler, and unlikely due to dynamic computational graph.
7. Autograd for reverse mode automatic differentiation through Twitter Cortex implementation for Torch, and built into PyTorch core.
8. Dynamic computation graph. (See comparison below)
9. Model and Data parallelism.
10. Unsuited to production (need to run Python in production, rather than faster C / C++ / Java)
   1. Plan to build a converter to Caffe 2
11. Integrates cleanly with Numpy / Cython / Python Data Ecosystem
12. Somewhat spotty documentation
13. Strong adoption in research communities, weak adoption from industry.

Caffe / Caffe 2

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. Extremely weak RNN / LSTM / GRU support.
3. Extremely strong CNN support.
4. Python / C++ API.
   1. API is inelegant for very deep networks (kept in config file, with details for each layer)
5. Supported by Berkeley, Caffe 2 created by Facebook.
6. No Autograd / Reverse Mode Automatic Differentiation.
7. Static computation.
8. Model and Data Parallelism.
9. Caffe 2 optimized for production, including on mobile.
10. Strong adoption for image processing in industry and research.

MXNet

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. RNN / LSTM / GRU Support.
3. CNN Support.
4. Python API / others.
5. Supported by Amazon, created by researchers at University of Washington
6. Autograd through NNVM - ‘Neural Network Visual Model’ is a C++ based computational graph for networks. It’s abstracted out so that DL libraries can be customized and be made more modular.
7. Mixed Static / Dynamic computational graph, taking advantage of both paradigms.
8. Model and Data Parallelism.
9. Merged with production (rather than having a different library like Pytorch / Caffe 2), looking to provide a solution for both industry and research.
10. Limited adoption in industry.

CNTK

1. CPU, GPU, Distributed CPU, Distributed GPU support.
   1. 1bit-SGD (Distributed comput algorithm) is licensed, can still use standard Asynch-SGD
2. Strong RNN / LSTM / GRU Support.
3. Strong CNN Support.
4. Python / C++ API
5. Backed by Microsoft.
   1. Works well on Windows
6. No compiler.
7. Implements Autograd with reverse mode automatic differentiation.
8. Static Computational Graph.
9. Model and Data Parallelism.
10. Production through Azure, as well as Windows production options.
11. Limited adoption in industry.

DL4J

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. Weak RNN / LSTM / GRU Support.
3. Strong CNN Support.
4. Inelegant API.
5. Java / Scala / Python APIs, focused on Hadoop / Spark ecosystem.
6. Supported by Skymind. ~15 person startup, headed by Adam Gibson and Chris Nicholson.
7. No linear algebra compiler.
8. Backed by ND4J, linear algebra tensor library connected to BLAS and abstracting over hardware.
9. No autograd.
10. Static computation
11. Data Parallelism, no model parallelism.
12. Production through Skymind Intelligence Layer, which is not open source
13. Some adoption in Japan / China.

Theano

1. Old and original deep learning library, created by MILA
2. CPU and GPU support. No distributed CPU, no distributed GPU support.
3. Support for RNNs and LSTMs.
4. Python API. Very low level. Wrapped by Keras and Lasagne
5. Supported by MILA, but relatively inactive due to new frameworks.
6. Does support autograd.
7. No linear algebra compiler.
8. Static computation.
9. Poor for production, need to run python - you can pickle compiled theano function and load into a server.
10. Moderate adoption in research.

Chainer

1. CPU, GPU, Distributed CPU, Distributed GPU support.
2. Known as an innovator with dynamic computational graph and great speed.
3. Strong RNN / LSTM / GRU support.
4. CNN support.
5. Python API
6. Supported by Intel, in partnership with Preferred Networks (which created Chainer) in Japan.
7. No linear algebra compiler, and unlikely to get one due to dynamic computational graph.
8. Model and data parallelism.
9. Few to no examples of Chainer’s use in production.
10. Weak adoption.

Minor Frameworks

1. CoreML is Apple’s IOS ML library, optimized for mobile in production
2. Nirvana Neon was acquired by Intel, works on abstracting over hardware and has great speed.
3. DyNet is an NLP focused new library out of a number of universities but primarily CMU, applying a dynamic computation graph. 
4. Dsstne is an Amazon library focused on deep recommender systems in production, that has fallen out of favor relative to MXNet.
5. PaddlePaddle is Baidu’s ML library, with strong machine translation (Chinese <-> English) applications.
6. BigDL is Intel’s Spark focused distributed CPU library, working on Intel’s chips through MKL.

DyNet Fact List

1. Focused on NLP
2. Dynamic computational graph
   1. ‘Dy’ in the name stands for Dynamic
3. Developed by Carnegie Mellon, many others
4. Applications
   1. Syntactic Parsing [https://github.com/clab/lstm-parser]
   2. Machine Translation [https://github.com/neubig/lamtram]
   3. Morphological Inflection [https://github.com/mfaruqui/morph-trans]
5. Python bindings over C++

Positive Feedback

There’s a loop where being popular means that new research and custom kernels are implemented first in your framework. And as you accumulate more base code, more people what to use your framework. This leads to more new research and custom code being written for your framework, in a positive feedback loop. The fast / faster RCNN example is a clean one. Tensorflow is benefiting the most from this.

Static vs. Dynamic Graph Declaration

Static Declaration
1. Definition
   1. A computational architecture is defined (typically in a computational graph) which can be differentiated using autodiff and optimized. In a separate step, the graph is executed.
2. Advantages
   1. Computational graph can be optimized for performance
   2. Ease of scheduling computation across many workers

Dynamic Declaration
1. Definition
   1. No separate steps for graph definition and execution. The necessary computation graph is created on the fly, with a new graph for each training instance.
2. Advantages
   1. Variable sized inputs
      1. Naturally allows for different sizes to the input, say images of differing size or sequences of different lengths
      2. TF and Theano have special dynamic_rnn and scan operations for variable length sequences in RNNs
   2. Variably structured inputs
      1. Say, tree or graph inputs
   3. Nontrivial inference Algorithms
      1. Say, Bayes risk or marginal likelihood approximation
   4. Variably structured outputs
      1. The structure of the output can change as a function of computations earlier in the graph.
   5. Easy to debug
   6. Ability to inspect, visualize and generally interact with components of the graph
   7. Very difficult to implement Attentional Models without it, which are crucial for Seq2seq models in NLP. Difficult to do SOTA neural machine translation, question answering, dialogue.

---

*Source: [Original Google Doc](https://docs.google.com/document/d/1zvo_HQKdNhxqSt4SfQCyXoiC0RmqwvvXI3mFGi_Tiug/edit?usp=drivesdk&sa=D&ust=1596495076582000&usg=AOvVaw3aSDt1hgGZ20h9x8F-K1RN)*