Suggest edit — 17-10-14 Interesting Facts in Machine Learning (Neural Networks)

Title

Name

Note

---
title: "17-10-14 Interesting Facts in Machine Learning (Neural Networks)"
visibility: public
---

# 17-10-14 Interesting Facts in Machine Learning (Neural Networks)

Category: [[idea-lists-upon-request|Idea Lists (Upon Request)]]

[Read the original document](https://docs.google.com/document/d/1NTpMGZG45aHJqUNpAtmpvsPlUClrDHIFHC3TTTVHQeo/edit?usp=drivesdk&sa=D&ust=1596495076884000&usg=AOvVaw323F_Th6pv-cQmpK-IjVD0)

---

1. Learns compositional (bottom-up) hierarchical structure
2. Model complexity overcomes the curse of dimensionality
   1. Combinatorial in depth and in width
3. Requires high signal-to-noise ratio
4. ‘Just’ adaptive basis function regression
5. Optimizer improved by exponentially weighted average of the gradient, learning rate
6. Covariate Shift
7. Close-to-linear model leads to failure to generalize, ex. adversarial examples
8. Dimensionality of the representation increases with depth of a convnet.
9. Softmax leads to extreme solutions
10. Non-convex optimization surface is dominated by saddle points.
11. Convnets are:
   1. Parameter Sharing leads to translation equivariance
   2. Locality (Sparse Connectivity)
   3. Composition
   4. Not equivariant to scale or rotation.
12. Many machine learning libraries implement cross-correlation but call it convolution
13. Achieving the global minimum would overfit the training data.

---

*Source: [Original Google Doc](https://docs.google.com/document/d/1NTpMGZG45aHJqUNpAtmpvsPlUClrDHIFHC3TTTVHQeo/edit?usp=drivesdk&sa=D&ust=1596495076884000&usg=AOvVaw323F_Th6pv-cQmpK-IjVD0)*