Suggest edit — Probability of Failure

Title

Name

Note

---
title: "Probability of Failure"
type: concept
related: [Probability Of Failure, Failure Distribution, Adaptive Importance Sampling, Proposal Distribution, Direct Estimation]
source: https://www.jemoka.com/posts/kbhprobability_of_failure/
confidence: high
status: active
---

\begin{align} p_{\text{fail}} &amp;= \mathbb{E}_{\tau \sim p\qty(\cdot)} \qty [1 \qty{\tau \not \in \psi}] \\ &amp;= \int 1 \qty {\tau \not\in \psi} p\qty(\tau) \dd{\tau } \end{align}
that is, the Probability of Failure is just the normalizing constant of the Failure Distribution. Like with Failure Distribution itself, computing this is quite intractable. We have a few methods to solve this, namely:
direct estimation: directly approximate your failure probability from nominal distribution \(p\) &mdash; \(\tau_{j} \sim p\qty(\cdot)\), \(\hat{p}_{\text{fail}} = \frac{1}{m} \sum_{i=1}^{m} 1\qty {\tau_{i} \not \in \psi}\) Importance Sampling: design a distribution to probe failure, namely proposal distribution \(q\), and then reweight by how different it is from \(p\) &mdash; \(\tau_{j} \sim q\qty(\cdot)\), \(\hat{p}_{\text{fail}} = \frac{1}{m}\sum_{i=1}^{m} w_{i} \mathbb{1} \qty {\tau_{i}\not \in \psi}\), call \(w_{i} = \frac{p\qty(\tau_{i})}{q\qty(\tau_{i})}\) (the &ldquo;importance weight&rdquo;) adaptive importance sampling multiple importance sampling sequential monte-carlo How do you pick a proposal distribution? See proposal distribution.
evaluating estimators bias: an estimator is unbiased if it predicts the true value in expectation; namely \(\mathbb{E}\qty [\hat{p}_{\text{fail}}] = p_{\text{fail}}\) consistency: an estimator is consistent if it converges to infinity at infinite samples, namely that \(\text{lim}_{m \to \infty} \hat{p}_{\text{fail}} = p_{\text{fail}}\) variance: we want the variance of the estimates around the true value to be low evaluation of direct estimation \begin{equation} \text{Var}\qty [\hat{p}] = \frac{p_{\text{fail}} \qty(1-p_{\text{fail}})}{m} \end{equation}
as the number of samples increases, the variance in the estimate decreases (yay!) as the probability of failure deceases, the variance in the failure actually icreases!!! baaaad vibes!