Suggest edit — Baysian Parameter Learning

Title

Name

Note

---
title: "Baysian Parameter Learning"
source: https://www.jemoka.com/posts/kbhbaysian_parameter_learning/
---

We treat this as an inference problem in Naive Bayes: observations are independent from each other.
Instead of trying to compute a $\theta$ that works for Maximum Likelihood Parameter Learning, what we instead do is try to understand what $\theta$ can be in terms of a distribution.
That is, we want to get some:
&ldquo;for each value of $\theta$, what&rsquo;s the chance that that is the actual value&rdquo;
To do this, we desire:
\begin{equation} p(\theta | D) \end{equation}
&ldquo;what&rsquo;s the probability of theta being at a certain value given the observations we had.&rdquo;
And to obtain the actual the actual value, we calculate the expectation of this distribution:
\begin{equation} \hat{\theta} = \mathbb{E}[\theta] = \int \theta p(\theta | D) \dd{\theta} \end{equation}
If its not possible to obtain such an expected value, we then calculate just the mode of the distribution (like where the peak probability of $\theta$ is) by:
\begin{equation} \hat{\theta} = \arg\max_{\theta} p(\theta | D) \end{equation}
Bayesian Parameter Learning on Binary Distributions We are working in a Naive Bayes environment, where we assume that $o_{1:m}$ are conditionally independent. Then, we essentially consider each class as carrying some parameter $\theta$ which contains the possibility of that class happening.
Using the same steps as inference with Naive Bayes and some algebra:
\begin{equation} p(\theta | o_{1:m}) \propto p(\theta, o_{1:m}) \end{equation}
Now, we would like to normalize this function for $\theta \in [0,1]$, so, we get:
\begin{equation} \int_{0}^{1} \theta^{n}(1-\theta)^{m-n}\dd{\theta} = \frac{\Gamma(n+1) \Gamma(m-n+1)}{\Gamma(m+2)} \end{equation}
where, $\Gamma$ is a real valued factorial generalization, and this entire integral is often called the &ldquo;Beta Function&rdquo;
Normalizing the output, we have that:
\begin{align} p(\theta | o_{1:m}) &amp;\propto p(\theta, o_{1:m}) \\ &amp;= \frac{\Gamma(m+2)}{\Gamma(n+1) \Gamma(m-n+1)} \theta^{n} (1-\theta)^{m-n} \\ &amp;= Beta(\theta | n+1, m-n +1) \end{align}
where $m$ is the sample size and $n$ is the number of events in the sample space.
Beta Distribution Suppose you had a non-uniform prior:
Prior: $Beta(\alpha, \beta)$ Observe: $m_1$ positive outcomes, $m_2$ negative outcomes Posterior: $Beta(\alpha+m_1, \beta+m_2)$ That is: for binary outcomes, the beta distribution can be updated without doing any math.
For instance, say we had:
\begin{equation} \theta_{t} = Beta(\alpha, \beta) \end{equation}
and we observed that $o_{i} = 1$, then:
\begin{equation} \theta_{t+1} = Beta(\alpha+1, \beta) \end{equation}
instead, if we observed that $o_{i} = 0$, then:
\begin{equation} \theta_{t+1} = Beta(\alpha, \beta+1) \end{equation}
Essentially: MAGNITUDE of beta distribution governs how small the spread is (higher magnitude smaller spread), and the balance between the two values represents how much skew there is.
Beta is a special distribution which takes parameters $\alpha, \beta$,and has mean:
\begin{equation} \frac{\alpha}{\alpha + \beta} \end{equation}
and variance:
\begin{equation} \frac{ab}{(a+b)^{2}(a+b+1)} \end{equation}
and has mode:
\begin{equation} \frac{\alpha -1 }{\alpha + \beta -2} \end{equation}
when $\alpha &gt; 1$ and $\beta &gt; 1$.
This means that, at $beta(1,1)$, we have a inform distribution
Laplace Smoothing Laplace Smoothing is a prior where:
\begin{equation} prior\ X \sim Beta(2,2) \end{equation}
so you just add $2$ to each of our output pseudo counts.
see also Laplace prior, where you use Laplace Smoothing for your prior
Total Probability in beta distributions Recall, for total probability, beta is a special distribution which takes parameters $\alpha, \beta$,and has expectation:
\begin{equation} \frac{\alpha}{\alpha + \beta} \end{equation}
and has mode:
\begin{equation} \frac{\alpha -1 }{\alpha + \beta -2} \end{equation}
Choosing a prior do it with only the problem and no knowledge of the data uniform typically works well, but if you have any reason why it won&rsquo;t be uniform (say coin flip), you should count accordingly such as making the distribution more normal with $Beta(1,1)$ Dirichlet Distribution We can generalize the Bayesian Parameter Learning on Binary Distributions with the Dirichlet Distribution.
For $n$ parameters $\theta_{1:n}$ ($n-1$ of which independent, because we know that $\sum \theta_{i} = 1$), where $\theta_{j}$ is the probability that the $j$ th case of the categorical distribution happening.
Now:
\begin{equation} Dir(\theta_{1:n} | \alpha) = \frac{\Gamma(\alpha_{0})}{\prod_{i=1}^{n} \Gamma(\alpha_{i})} \prod_{i=1}^{n} \theta_{i}^{\alpha_{i}-1} \end{equation}
whereby:
\begin{equation} \alpha_{j} = prior + count \end{equation}
for $j \geq 1$, and
\begin{equation} \alpha_{0} = prior + total_{}count \end{equation}
whereby prior is your initial distribution. If its uniform, then all prior equals one.
The expectation for each $\theta_{i}$ happening is:
\begin{equation} \mathbb{E}[\theta_{i}] = \frac{a_{i}}{\sum_{j=1}^{n} \alpha_{j}} \end{equation}
and, with $a_{i} &gt; 1$, the $i$th mode is:
\begin{equation} \frac{a_{i}-1 }{\sum_{j=1}^{n} a_{j}-n} \end{equation}
expectation of a distribution For Beta Distribution and Dirichlet Distribution, the expectation of their distribution is simply their mean.
if you say want to know what the probability of $P(thing|D)$, you can integrate over all $P(thing|\theta)$:
\begin{equation} \int^{1}_{0} P(thing|\theta)P(\theta)d\theta \end{equation}
The first thing is just the actual value of $\theta$ (because $\theta$ is literally the probability of $thing$ happening). The second thing is the probability of that $\theta$ actually happening.
This, of course, just add up to the expected value of $\theta$, which is given above:
\begin{equation} \frac{\alpha}{\alpha + \beta} \end{equation}