Suggest edit — SU-CS120 OCT082024

Title

Name

Note

---
title: "SU-CS120 OCT082024"
source: https://www.jemoka.com/posts/kbhsu_cs120_oct082024/
date: 2024-10-08
---

evaluation is quite hard&mdash;you need Classical Test Theory &ldquo;just average each test&rdquo; (think MUC, b3, etc.) test-dependent ability estimation BAD: because each test maybe different difficulty Item Response Theory (IRT) model item and test taker characteristics test-invariant ability estimation (subset invariant) adaptive testing problem requires calibration first &hellip;which is quite costly Flash-HELM HELM, prioritizing higher-ranked models. Evaluate good model more.
Sang&rsquo;s Method We want to estimate \(\theta\) with a budget of \(K\) questions.
test taker ability is fixed, but unknown: \(\theta \sim p(\theta)\) there&rsquo;s some function \(z(q) \to Z \in \triangle\), for some question \(q \in Q\) our response model, then, is \(p(y=1 | z; \theta) = \sigma(\theta - z)\) Then for ever question we have we ask what the fisher information is.
You then update for every test result the response model using MLE.
amortized calibration compute the calibration difficulty \(z\) advantages more reliable and efficient across emprical setting incorporates amortized (learned) calibration to reduce calibration costs introduces conditional question generation to generate questions of specific difficulties