Suggest edit — Automatic Differentiation

Title

Name

Note

---
title: "Automatic Differentiation"
type: concept
source: https://www.jemoka.com/posts/kbhautomatic_differentiation/
confidence: high
status: active
---

Forward Accumulation First, make a computation graph.
Consider \(\ln (ab + \max (a,2))\)
Say we want \(\pdv{f}{a}(3,2)\).
Let&rsquo;s begin by tracking, left to right, both the value of each node and its derivative.
Layer 1:
\(b = 2, \pdv{b}{a} = 0\) \(a = 3, \pdv{a}{a} = 1\) Layer 2:
\(c_1 = a\times b = 6, \pdv{c_1}{a} = b\pdv{a}{a} + a \pdv{a}{b} = 2\) and so on; until we get to \(c_4\)
Dual Number Method Dual Number Consider:
\begin{equation} a+b \epsilon \end{equation}
Let&rsquo;s declare:
\begin{equation} \epsilon^{2} = 0 \end{equation}
The standard field operations still apply:
\begin{equation} (a+b\epsilon) + (c+d\epsilon) = (a+c) + (b+d) \epsilon \end{equation}
The Method We can write down a usual Taylor expansion:
\begin{equation} f(a+b\epsilon) = \sum_{k=0}^{\infty} \frac{f^{(k)}}{k!} (a+b \epsilon - a)^{k} \end{equation}
IMPORTANTLY:
\begin{equation} f(a+1\epsilon) = f(a) + f&rsquo;(a) \epsilon \end{equation}
This means that we can use Dual Numbers to directly compute derivatives.