Suggest edit — Generative Learning Algorithm

Title

Name

Note

---
title: "Generative Learning Algorithm"
type: concept
related: [Sigmoid, Naive Bayes, Logistic Regression]
source: https://www.jemoka.com/posts/kbhgenerative_learning_algorithm/
confidence: high
status: active
---

Gaussian Discriminant Analysis High level idea: 1) fit parameters to the positive and negative examples separately as a multi-variant Gaussian density 2) try to see if a new samples&rsquo; probablitity to 1) is greater or 2) is greater.
requirements fit a parameter to additional information multivariant gaussian See multi-variant Gaussian density. If it helps, here you go:
\begin{equation} p\qty(z) = \frac{1}{\qty(2\pi)^{\frac{|z|}{2}}|\Sigma|^{\frac{1}{2}}} \exp \qty(-\frac{1}{2} \qty(x-\mu)^{T} \Sigma^{-1} \qty(x - \mu)) \end{equation}
making predictions Suppose \(p\qty(y=1) = \phi\), \(p\qty(y=0) = 1-\phi\).
Now, this means that:
\begin{equation} p\qty(y) = \phi^{y} \qty(1-\phi)^{1-y} \end{equation}
Now, we can write using multi-variant Gaussian density that:
\begin{equation} p\qty(x\mid y = 0) = \frac{1}{\qty(2\pi)^{\frac{|z|}{2}}|\Sigma|^{\frac{1}{2}}} \exp \qty(-\frac{1}{2} \qty(x-\mu_{0})^{T} \Sigma^{-1} \qty(x - \mu_{0})) \end{equation}
\begin{equation} p\qty(x\mid y = 1) = \frac{1}{\qty(2\pi)^{\frac{|z|}{2}}|\Sigma|^{\frac{1}{2}}} \exp \qty(-\frac{1}{2} \qty(x-\mu_{1})^{T} \Sigma^{-1} \qty(x - \mu_{1})) \end{equation}
Finally, to predict \(p\qty(\cdot | x)\), we use Naive Bayes using the \(p\qty(y)\) and \(p\qty(x|y)\) above. In particular we can have:
\begin{align} y &amp;= \arg\max_{y} p\qty(y|x) \\ &amp;= \arg\max_{y} \frac{p\qty(x|y)p\qty(y)}{p\qty(x)} \\ &amp;= \arg\max_{y} p\qty(x|y)p\qty(y) \end{align}
applying Bayes rule to make prediction makes the sigmoid function Fun fact: applying the Bayes rule on making predictions will just hand you the sigmoid function; i.e. the predicted \(p\qty(y=1|x)\) is just going to be a sigmoid. That is, assuming:
\begin{equation} \begin{cases} x|y=0 \sim \mathcal{N}\qty(\mu_{0}, \Sigma) \\ x|y=1 \sim \mathcal{N}\qty(\mu_{1}, \Sigma) \\ y \sim \text{Ber}\qty(\phi) \end{cases} \end{equation}
results in the assumption that
\begin{equation} p\qty(y=1|x) \end{equation}
is logistic. Meaning, GDA makes stronger assumptions than logistic regression.
double fun fact:
\begin{equation} \begin{cases} x|y=0 \sim \text{ExFam}\qty(\eta_{0}) \\ x|y=1 \sim \text{ExFam}\qty(\eta_{1}) \\ y \sim \text{Ber}\qty(\phi) \end{cases} \end{equation}
will also be logistic
So why are we doing any of this?!
If you have a large dataset with lots of noise, logistic regression will do better. If you know your dataset is a Gaussian, this will fit faster (i.e. so good for data-constrained regimes).
fitting Our goal is to solve for \(\mu_{0}, \mu_{1}\) to make predictions above that maximizes the joint likelihood of our system.
\begin{equation} \mathcal{L}\qty(\phi, \mu_{0}, \mu_{1}, \Sigma) = \prod_{i=1}^{n} p\qty(x^{(i)}, y^{(i)}; \phi, \mu_{0}, \mu_{1}, \Sigma) \end{equation}
in particular:
\begin{equation} \mathcal{L}\qty(\phi, \mu_{0}, \mu_{1}, \Sigma) = \prod_{i=1}^{n} p\qty(x^{(i)}|y^{(i)}) p\qty(y^{(i)}) \end{equation}
And finally, apply the log trick, you can find parameters \(\mu_{0}\) and \(\mu_{1}\) such that:
\begin{equation} \max_{\phi, \mu_{0}, \mu_{1}, \Sigma} \sum_{i=1}^{n} \log \qty[p\qty(x^{(i)}|y^{(i)}) p\qty(y^{(i)})] \end{equation}
If you do the derivative thing and go a bunch of brrr and solving, we obtain:
\begin{equation} \phi = \frac{\sum_{i=1}^{n}y^{(i)}}{n} = \frac{\sum_{i=1}^{n} 1\qty {y^{(i)}=1}}{n} \end{equation}
and the means are just the mean of all samples in each class
\begin{equation} \mu_{0} = \frac{\sum_{i=1}^{n} 1 \qty {y^{(i)}=0} x^{(i)}}{\sum_{i=1}^{n}1 \qty {y^{(i)}=0}} \end{equation}
\begin{equation} \mu_{0} = \frac{\sum_{i=1}^{n} 1 \qty {y^{(i)}=1} x^{(i)}}{\sum_{i=1}^{n}1 \qty {y^{(i)}=1}} \end{equation}
and the covariance is a function of these means:
\begin{equation} \Sigma = \frac{1}{n} \sum_{i=1}^{n} \qty(x^{(i)}-\mu_{y^{(i)}}) \qty(x^{(i)}- \mu_{y(i)})^{T} \end{equation}
&ldquo;why do we have a single covariance for all classes&rdquo;? For #reasons, a single covariance matrix will result in a linear decision boundary. Otherwise, a custom covariance for each class would result in a non-linear boundary.