Suggest edit — agent security

Title

Name

Note

---
title: "agent security"
type: concept
source: https://www.jemoka.com/posts/kbhagent_security/
confidence: high
status: active
---

Three layers of agent safety model architecture: fundamental limitations of transformer structure architecture -&gt; LLMs: training data (poisoning), training objective (reward hacking) LLMs -&gt; prompts: prompt injections, unintended actions, goal scheming prompt injections OWASP top 10 for LLM applications&hellip;. RAG/Agents are WORSE because humans do not have choice. Web agents, can browse the web and have context poisoning.
evaluation setup etiologic validity realistic threat models systematic evaluations (e.g., obviously anecdotal works) controlled environments computer security principles confidentiality (don&rsquo;t infiltrate passwords) integrity (don&rsquo;t nuke important files) availability (don&rsquo;t bring things down) benign inputs leading to harms triggering compaction =&gt; failures Unintentional behavior: &ldquo;unsafe agent behavior that deviations from user intentions from a task&rdquo;
questions etiological validity? once you discover failures, how do you argue how much things happen in the wild? coverage? how to make sure you are not doing multiple of thew same thing? think about what the attacker can do, and try to cover more of them strike a tradeoff between not covering out of domains / bad scenarios guardrails vs fine tuning &ldquo;defense in depth&rdquo; perhaps the model can be tuned to recognize dangerous situations and then kick the problem down the can