Suggest edit — SU-CS238V JAN072025

Title

Name

Note

---
title: "SU-CS238V JAN072025"
source: https://www.jemoka.com/posts/kbhsu_cs238v_jan072024/
date: 2024-01-07
---

Alignment Problem autonomous systems will do exactly what we tell them to do&hellip; so we need to give them good instructions. This is the Alignment Problem
imperfect objective&mdash;underspecified objective imperfect model&mdash;understanding of the world is underspecified imperfect optimization&mdash;the model just didn&rsquo;t solve the problem correctly Validation Framework High level structure:
validation_algorithm(system, spec) system environment: state of the world, \(T(s&rsquo;|s,a)\) sensor, \(O(o|s)\) agent, policy \(\pi\qty(a | o)\) example: inverted pendulum state: \(\qty (\theta, \omega)\) of the pendulum observation: \(O(o|s) = \mathcal{N}\qty (o|s,\Sigma)\), Gaussian noise policy: consider the following proportional controller policy \begin{equation} \pi \qty(a | o) = \begin{cases} 1, \text{if} a = -15 \tau - 8 \omega \\ 0 \end{cases} \end{equation}
environment: a \(T(s&rsquo;|s,a)\) given by physics specification \(\psi\) Rules of the system&mdash;&ldquo;do not let the pendulum tip over&rdquo;. Specifications are usually given in formal specification language such as Linear Temporal Logic or Signal Temporal Logic.
validation algorithm With input a system + specification, a Validation Algorithm provides an output in the form of one of&hellip;
Failure Analysis Falsification: search for failures of a particular system Failure distribution: identify what are the distirbutions of failures Failure probability estimation: estimate the probability of failures Formal Methods &ldquo;reachability&rdquo;
Linear System reachability Nonlinear system reachability Discrete system reachability Other stuff Explanation Runtime Assurances