Suggest edit — neural network

Title

Name

Note

---
title: "neural network"
type: concept
related: [Neural Networks, Neural Network, Kernel Trick, Deep Learning]
source: https://www.jemoka.com/posts/kbhneural_network/
confidence: high
status: active
---

Neural networks are a non-linear learning architecture that involves a combination of matrix multiplication and entry-wise non-linear operations.
two layers constituents Consider a two layer neural network with:
\(m\) hidden units \(d\) dimensional input \(x \in \mathbb{R}^{d}\) requirements \begin{align} &amp;\forall j \in \qty {1, \dots, m}\\ &amp;z_{j} = w_{j}^{(1)}^{T} x + b_{j}^{(1)}\\ &amp;a_{j} = \text{ReLU}\qty(z_{j}) \\ &amp;a = \qty(a_1, \dots, a_{m})^{T} \in \mathbb{R}^{m} \\ &amp;h_{\theta} \qty(x) = w^{(2)}^{T} a + b^{(2)} \end{align}
\(z_{j}\) are hidden units, \(a_{j}\) are activated hidden units, \(h_{\theta}\) is the prediction function.
vectorized two-layer constituents \(m\) hidden units per layer \(d\) input dimension requirements \begin{equation} W^{(1)} = \mqty[w_1^{(1)}^{T} \\ \dots \\ w_m^{(1)}^{T} ] \end{equation}
which emits a \(m \times d\) matrix. So this gives:
\begin{equation} \mqty[z_1 \\ \dots \\ z_{M}] = \mqty[w_1^{(1)}^{T} \\ \dots \\ w_m^{(1)}^{T} ] \mqty[x_1 \\ \dots \\ x_{D}] + \mqty[b_1^{( 1 )} \\ \dots \\ b_m^{( 1 )}] \end{equation}
where \(z \in \mathbb{R}^{m \times 1}, w^{(1)} \in \mathbb{R}^{m \times d}, x \in \mathbb{R}^{d \times 1} , b^{(j)} \in \mathbb{R}^{m \times 1}\). Writing this as matrix operations:
\begin{equation} z = w^{(1)} x + b^{(1)} \end{equation}
and
\begin{equation} a = \text{ReLU}\qty(z) \end{equation}
with:
\begin{equation} h_{\theta}\qty(x) = w^{(2)} a + b^{(2)} \end{equation}
multi-layer \begin{equation} a^{(1)} = \text{ReLU}\qty(W^{(1)} x + b^{(1)}) \end{equation}
\begin{equation} a^{(2)} = ReLU\qty(W^{(2)} a^{(1)} + b^{(2)}) \end{equation}
and so on&hellip;
\begin{equation} a^{(r-1)} = \text{ReLU}\qty(W^{(r-1)} a^{(r-2)} + b^{(r-1)}) \end{equation}
\begin{equation} h_{\theta}\qty(x) = W^{r} a^{r-1} + b^{r} \end{equation}
metadata total number of neurons: \(m_1 + m_2 + &hellip; + m_{r}\) number of parameters: \(\qty(d+1) m_1 + \qty(m_{1}+1)m_{2} + &hellip; + \qty(m_{r-1}+1)m_{r}\) additional information Neural networks admit a local optima, and we cannot find a global optima.
neuron Consider first a single neuron neural network in one dimension. For instance, let&rsquo;s think of a slightly non-linear case first:
\begin{align} h_{\theta}\qty(x) &amp;= \max \qty(wx+b, 0) \end{align}
it admits two parameters, \(\theta = \qty(w, b) \in \mathbb{R}^{2}\). Such a function is called relu function. What if we have multiple input features? Consider: \(x \in \mathbb{R}^{d}\), \(w \in \mathbb{R}^{d}\), and \(b \in \mathbb{R}\). Now:
\begin{equation} h_{\theta} \qty(x) = \text{ReLU}\qty(w^{\top}x + b) \end{equation}
We refer to relu function as an &ldquo;Activation Function&rdquo;.
neurons We can write latent units in terms of the input units, as well as parameters to weight them; for instance:
\begin{equation} a_1 = \text{ReLU}\qty(\theta_{1} x_1 + \theta_{2} x_2 + \theta_{3}) \end{equation}
Instead of writing this directly, we can just make every neuron connected to every other neuron, resulting in:
\begin{equation} a_1 = \text{ReLU}\qty(w_1^{T} x + b_1) \end{equation}
\begin{equation} a_2 = \text{ReLU}\qty(w_2^{T} x + b_2) \end{equation}
and so on.
why would the neurons learn different things? Because random initializations will give local minims, and each node would specialize.
some activation functions see Activation Function
see also Neural Networks see Neural Networks
kernel methods vs deep learning Instead of using Kernel Trick and feature map to extract features yourselves, deep learning promises to learn the correct feature map after multiple, non-linear layers.
Consider \(\beta\) as the parameters of a fully-connected neural network, then the final hypothesis function of a neural network is:
\begin{equation} h_{\theta} = W^{r} \phi_{\beta} \qty(x) + b^{( r)} \end{equation}
in some sense, the entire damn neural network is a feature map for the final, linear output head. We can therefore think of training a neural network as automatically finding a feature map \(\phi_{\beta}\), as well as learning a classifier for that feature map.