Suggest edit — belief-state MDP

Title

Name

Note

---
title: "belief-state MDP"
type: concept
related: [Vector, Markov Decision Process, Conditional Plan, Belief]
source: https://www.jemoka.com/posts/kbhbelief_state_mdp/
confidence: high
status: active
---

Our belief can be represented as vectors as the probability of us being in each state. If we have that, we can just use our belief vector as our state vector. Now use MDP any solving you&rsquo;d like, keeping in mind that the reward is just the expected reward:
\begin{equation} \mathbb{E}[R(b,a)] = \sum_{s} R(s,a) b(s) \end{equation}
we can estimate our transition between belief-states like so:
\begin{align} T(b&rsquo;|b,a) &amp;= P(b&rsquo;|b,a) \\ &amp;= \sum_{o}^{} P(b&rsquo;|b,a,o) P(o|b,a) \\ &amp;= \sum_{o}^{} P(b&rsquo; = Update(b,a,o)) \sum_{s&rsquo;}^{}O(o|a,s&rsquo;) \sum_{s}^{}T(s&rsquo;|s,a)b(s) \end{align}
&ldquo;the probability of the next belief being \(b&rsquo;\) is equal to how probable it is to get state b&rsquo; from conditions b,a,o, times the probability of getting that particular observation.&rdquo;.
However, this expression is quite unwheldy if your state-space is large. Hence, we turn to a technique like conditional plans which foregos considering individual states altogether.