Suggest edit — 19-11-08 Optimization Algorithms

Title

Name

Note

---
title: "19-11-08 Optimization Algorithms"
visibility: public
---

# 19-11-08 Optimization Algorithms

Category: [[idea-lists-upon-request|Idea Lists (Upon Request)]]

[Read the original document](https://docs.google.com/document/d/18PYPb9n2N6OOMIhlz2VMQRqIcBv6ryixAaDn1aMBrDQ/edit?usp=drivesdk&sa=D&ust=1596495076775000&usg=AOvVaw1U-lLktIUPL4rC1Stb4-Nn)

---

1. Gradient Descent
   1. Momentum
      1. Nesterov Momentum
   2. RMSProp
   3. Adadelta
   4. Adagrad
   5. Adam
2. Newton’s Method
   1. Newton-Rhapson
   2. Trust Region Methods
      1. Trust Newton
3. BFGS
   1. L-BFGS
4. Iteratively Reweighted Least Squares
5. Conjugate Gradients
6. Coordinate Descent
7. Line Search
8. Expectation Maximization
9. Lloyd’s Algorithm
10. Evolutionary Strategies
11. Finite Differences
12. Convex Optimization
   1. Linear Programing
      1. Simplex Method
         1. Branch, Bound and Cut
      2. Interior-Point Methods
   2. Quadratic Programming
      1. Factords Karush-Kuhn-Tucker (KKT) System
13. Forwards Backwards Algorithm
14. Matrix Factorization
   1. LU Decomposition
   2. QR Factorization
   3. Cholesky Decomposition
   4. Singular Value Decomposition
   5. Non-Negative Matrix Factorization?
15. MCMC
   1. Metropolis Hastings
   2. Gibbs Sampling
16. CART
17. Constrained Optimization
   1. Lagrange Multiplier Constrained Optimization
   2. Penalty Methods
18. Automatic Differentiation
19. Closed Form Solutions
   1. Normal Form Equations

---

*Source: [Original Google Doc](https://docs.google.com/document/d/18PYPb9n2N6OOMIhlz2VMQRqIcBv6ryixAaDn1aMBrDQ/edit?usp=drivesdk&sa=D&ust=1596495076775000&usg=AOvVaw1U-lLktIUPL4rC1Stb4-Nn)*