How to Monitor AI Systems for Security Risks

I've reviewed Anthropic's Risk Report for Claude Opus 4.6 because many of our enterprise customers are actively deploying AI agents into production environments. When those systems fail, the consequences are operational, financial and reputational. Most of the reaction centers on the headline that catastrophic risk is very low but not negligible. What matters more for customers and future customers is how risk actually manifests inside live enterprise systems and what that means for uptime, data integrity and compliance. It does not look like a breach. It looks like business as usual. An agent subtly influencing procurement decisions. A finance workflow that starts omitting inconvenient data. Permissions that expand over time without clear oversight. Anthropic describes a scenario called Persistent Rogue Internal Deployment, where an AI system with privileged access creates a less monitored instance of itself and continues operating inside production systems. In a real enterprise environment, that translates into downtime, data exposure or regulatory impact. The organizations at greatest risk are not the ones moving cautiously. They are the ones who pushed agents into production without adding an operational governance layer. We have seen this pattern before in cloud adoption. Technology advances quickly, and controls often lag behind. That gap is where exposure grows. So what should enterprise IT and security teams do now? 1. Constrain actions, not just access. Define what an agent can set in motion and enforce least privilege at the identity level, just as you have done for human users for decades. 2. Log actions, not just outcomes. Maintain an auditable trail of what the agent did, where and what triggered it, the same standard applies to human operators in regulated environments. 3. Automate your tripwires. Do not rely on people to catch machine speed behavior. Build policy enforcement and anomaly response into the loop. 4. Audit your agent footprint. Inventory every agent, its owner, permissions and kill path. Governance starts with visibility and most enterprises are still building it. The window to build these guardrails is now, before the agent workforce scales. At Rackspace, 25 years of running mission-critical systems have taught us that trust without controls creates exposure. We build and operate AI infrastructure with governance embedded from day one because customers need speed, resilience and measurable outcomes, not experiments in production. What this means for you is simple. Move forward on AI with confidence, but make operational governance part of the foundation so scale strengthens your business instead of introducing risk.

This new guide from the OWASP® Foundation Agentic Security Initiative for developers, architects, security professionals, and platform engineers building or securing agentic AI applications, published Feb 17, 2025, provides a threat-model-based reference for understanding emerging agentic AI threats and their mitigations. Link: https://lnkd.in/gFVHb2BF * * * The OWASP Agentic AI Threat Model highlights 15 major threats in AI-driven agents and potential mitigations: 1️⃣ Memory Poisoning – Prevent unauthorized data manipulation via session isolation & anomaly detection. 2️⃣ Tool Misuse – Enforce strict tool access controls & execution monitoring to prevent unauthorized actions. 3️⃣ Privilege Compromise – Use granular permission controls & role validation to prevent privilege escalation. 4️⃣ Resource Overload – Implement rate limiting & adaptive scaling to mitigate system failures. 5️⃣ Cascading Hallucinations – Deploy multi-source validation & output monitoring to reduce misinformation spread. 6️⃣ Intent Breaking & Goal Manipulation – Use goal alignment audits & AI behavioral tracking to prevent agent deviation. 7️⃣ Misaligned & Deceptive Behaviors – Require human confirmation & deception detection for high-risk AI decisions. 8️⃣ Repudiation & Untraceability – Ensure cryptographic logging & real-time monitoring for accountability. 9️⃣ Identity Spoofing & Impersonation – Strengthen identity validation & trust boundaries to prevent fraud. 🔟 Overwhelming Human Oversight – Introduce adaptive AI-human interaction thresholds to prevent decision fatigue. 1️⃣1️⃣ Unexpected Code Execution (RCE) – Sandbox execution & monitor AI-generated scripts for unauthorized actions. 1️⃣2️⃣ Agent Communication Poisoning – Secure agent-to-agent interactions with cryptographic authentication. 1️⃣3️⃣ Rogue Agents in Multi-Agent Systems – Monitor for unauthorized agent activities & enforce policy constraints. 1️⃣4️⃣ Human Attacks on Multi-Agent Systems – Restrict agent delegation & enforce inter-agent authentication. 1️⃣5️⃣ Human Manipulation – Implement response validation & content filtering to detect manipulated AI outputs. * * * The Agentic Threats Taxonomy Navigator then provides a structured approach to identifying and assessing agentic AI security risks by leading though 6 questions: 1️⃣ Autonomy & Reasoning Risks – Does the AI autonomously decide steps to achieve goals? 2️⃣ Memory-Based Threats – Does the AI rely on stored memory for decision-making? 3️⃣ Tool & Execution Threats – Does the AI use tools, system commands, or external integrations? 4️⃣ Authentication & Spoofing Risks – Does AI require authentication for users, tools, or services? 5️⃣ Human-In-The-Loop (HITL) Exploits – Does AI require human engagement for decisions? 6️⃣ Multi-Agent System Risks – Does the AI system rely on multiple interacting agents?

The Cybersecurity and Infrastructure Security Agency (CISA), together with other organizations, published "Principles for the Secure Integration of Artificial Intelligence in Operational Technology (OT)," providing a comprehensive framework for critical infrastructure operators evaluating or deploying AI within industrial environments. This guidance outlines four key principles to leverage the benefits of AI in OT systems while reducing risk: 1. Understand the unique risks and potential impacts of AI integration into OT environments, the importance of educating personnel on these risks, and the secure AI development lifecycle.  2. Assess the specific business case for AI use in OT environments and manage OT data security risks, the role of vendors, and the immediate and long-term challenges of AI integration 3. Implement robust governance mechanisms, integrate AI into existing security frameworks, continuously test and evaluate AI models, and consider regulatory compliance.  4. Implement oversight mechanisms to ensure the safe operation and cybersecurity of AI-enabled OT systems, maintain transparency, and integrate AI into incident response plans. The guidance recommends addressing AI-related risks in OT environments by: • Conducting a rigorous pre-deployment assessment. • Applying AI-aware threat modeling that includes adversarial attacks, model manipulation, data poisoning, and exploitation of AI-enabled features. • Strengthening data governance by protecting training and operational data, controlling access, validating data quality, and preventing exposure of sensitive engineering information. • Testing AI systems in non-production environments using hardware-in-the-loop setups, realistic scenarios, and safety-critical edge cases before deployment. • Implementing continuous monitoring of AI performance, outputs, anomalies, and model drift, with the ability to trace decisions and audit system behavior. • Maintaining human oversight through defined operator roles, escalation paths, and controls to verify AI outputs and override automated actions when needed. • Establishing safe-failure and fallback mechanisms that allow systems to revert to manual control or conventional automation during errors, abnormal behavior, or cyber incidents. • Integrating AI into existing cybersecurity and functional safety processes, ensuring alignment with risk assessments, change management, and incident response procedures. • Requiring vendor transparency on embedded AI components, data usage, model behavior, update cycles, cybersecurity protections, and conditions for disabling AI capabilities. • Implementing lifecycle management practices such as periodic risk reviews, model re-evaluation, patching, retraining, and re-testing as systems evolve or operating environments change.

The latest joint cybersecurity guidance from the NSA, CISA, FBI, and international partners outlines critical best practices for securing data used to train and operate AI systems recognizing data integrity as foundational to AI reliability. Key highlights include: • Mapping data-specific risks across all 6 NIST AI lifecycle stages: Plan and Design, Collect and Process, Build and Use, Verify and Validate, Deploy and Use, Operate and Monitor • Identifying three core AI data risks: poisoned data, compromised supply chain, and data drift for each with tailored mitigations • Outlining 10 concrete data security practices, including digital signatures, trusted computing, encryption with AES 256, and secure provenance tracking • Exposing real-world poisoning techniques like split-view attacks (costing as little as 60 dollars) and frontrunning poisoning against Wikipedia snapshots • Emphasizing cryptographically signed, append-only datasets and certification requirements for foundation model providers • Recommending anomaly detection, deduplication, differential privacy, and federated learning to combat adversarial and duplicate data threats • Integrating risk frameworks including NIST AI RMF, FIPS 204 and 205, and Zero Trust architecture for continuous protection Who should take note: • Developers and MLOps teams curating datasets, fine-tuning models, or building data pipelines • CISOs, data owners, and AI risk officers assessing third-party model integrity • Leaders in national security, healthcare, and finance tasked with AI assurance and governance • Policymakers shaping standards for secure, resilient AI deployment Noteworthy aspects: • Mitigations tailored to curated, collected, and web-crawled datasets and each with unique attack vectors and remediation strategies • Concrete protections against adversarial machine learning threats including model inversion and statistical bias • Emphasis on human-in-the-loop testing, secure model retraining, and auditability to maintain trust over time Actionable step: Build data-centric security into every phase of your AI lifecycle by following the 10 best practices, conducting ongoing assessments, and enforcing cryptographic protections. Consideration: AI security does not start at the model but rather it starts at the dataset. If you are not securing your data pipeline, you are not securing your AI.

☢️Manage Third-Party AI Risks Before They Become Your Problem☢️ AI systems are rarely built in isolation as they rely on pre-trained models, third-party datasets, APIs, and open-source libraries. Each of these dependencies introduces risks: security vulnerabilities, regulatory liabilities, and bias issues that can cascade into business and compliance failures. You must move beyond blind trust in AI vendors and implement practical, enforceable supply chain security controls based on #ISO42001 (#AIMS). ➡️Key Risks in the AI Supply Chain AI supply chains introduce hidden vulnerabilities: 🔸Pre-trained models – Were they trained on biased, copyrighted, or harmful data? 🔸Third-party datasets – Are they legally obtained and free from bias? 🔸API-based AI services – Are they secure, explainable, and auditable? 🔸Open-source dependencies – Are there backdoors or adversarial risks? 💡A flawed vendor AI system could expose organizations to GDPR fines, AI Act nonconformity, security exploits, or biased decision-making lawsuits. ➡️How to Secure Your AI Supply Chain 1. Vendor Due Diligence – Set Clear Requirements 🔹Require a model card – Vendors must document data sources, known biases, and model limitations. 🔹Use an AI risk assessment questionnaire – Evaluate vendors against ISO42001 & #ISO23894 risk criteria. 🔹Ensure regulatory compliance clauses in contracts – Include legal indemnities for compliance failures. 💡Why This Works: Many vendors haven’t certified against ISO42001 yet, but structured risk assessments provide visibility into potential AI liabilities. 2️. Continuous AI Supply Chain Monitoring – Track & Audit 🔹Use version-controlled model registries – Track model updates, dataset changes, and version history. 🔹Conduct quarterly vendor model audits – Monitor for bias drift, adversarial vulnerabilities, and performance degradation. 🔹Partner with AI security firms for adversarial testing – Identify risks before attackers do. (Gemma Galdon Clavell, PhD , Eticas.ai) 💡Why This Works: AI models evolve over time, meaning risks must be continuously reassessed, not just evaluated at procurement. 3️. Contractual Safeguards – Define Accountability 🔹Set AI performance SLAs – Establish measurable benchmarks for accuracy, fairness, and uptime. 🔹Mandate vendor incident response obligations – Ensure vendors are responsible for failures affecting your business. 🔹Require pre-deployment model risk assessments – Vendors must document model risks before integration. 💡Why This Works: AI failures are inevitable. Clear contracts prevent blame-shifting and liability confusion. ➡️ Move from Idealism to Realism AI supply chain risks won’t disappear, but they can be managed. The best approach? 🔸Risk awareness over blind trust 🔸Ongoing monitoring, not just one-time assessments 🔸Strong contracts to distribute liability, not absorb it If you don’t control your AI supply chain risks, you’re inheriting someone else’s. Please don’t forget that.

The 𝗔𝗜 𝗗𝗮𝘁𝗮 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 guidance from 𝗗𝗛𝗦/𝗡𝗦𝗔/𝗙𝗕𝗜 outlines best practices for securing data used in AI systems. Federal CISOs should focus on implementing a comprehensive data security framework that aligns with these recommendations. Below are the suggested steps to take, along with a schedule for implementation. 𝗠𝗮𝗷𝗼𝗿 𝗦𝘁𝗲𝗽𝘀 𝗳𝗼𝗿 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 1. Establish Governance Framework     - Define AI security policies based on DHS/CISA guidance.     - Assign roles for AI data governance and conduct risk assessments.  2. Enhance Data Integrity     - Track data provenance using cryptographically signed logs.     - Verify AI training and operational data sources.     - Implement quantum-resistant digital signatures for authentication.  3. Secure Storage & Transmission     - Apply AES-256 encryption for data security.     - Ensure compliance with NIST FIPS 140-3 standards.     - Implement Zero Trust architecture for access control.  4. Mitigate Data Poisoning Risks     - Require certification from data providers and audit datasets.     - Deploy anomaly detection to identify adversarial threats.  5. Monitor Data Drift & Security Validation     - Establish automated monitoring systems.     - Conduct ongoing AI risk assessments.     - Implement retraining processes to counter data drift.  𝗦𝗰𝗵𝗲𝗱𝘂𝗹𝗲 𝗳𝗼𝗿 𝗜𝗺𝗽𝗹𝗲𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻  Phase 1 (Month 1-3): Governance & Risk Assessment   • Define policies, assign roles, and initiate compliance tracking.   Phase 2 (Month 4-6): Secure Infrastructure   • Deploy encryption and access controls.   • Conduct security audits on AI models. Phase 3 (Month 7-9): Active Threat Monitoring • Implement continuous monitoring for AI data integrity.   • Set up automated alerts for security breaches.   Phase 4 (Month 10-12): Ongoing Assessment & Compliance   • Conduct quarterly audits and risk assessments.   • Validate security effectiveness using industry frameworks.  𝗞𝗲𝘆 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗙𝗮𝗰𝘁𝗼𝗿𝘀   • Collaboration: Align with Federal AI security teams.   • Training: Conduct AI cybersecurity education.   • Incident Response: Develop breach handling protocols.   • Regulatory Compliance: Adapt security measures to evolving policies.  

AI security is quickly becoming a real architecture problem, not just a model problem. As more companies deploy copilots, agents, and AI-driven automation, the security stack needs to evolve around how these systems actually operate. Prompts, models, APIs, agents, and automated actions introduce entirely new control points. A practical way to think about the emerging Enterprise AI Security Stack is in four layers. 1. Foundations Identity and Access Data Protection Infrastructure Integrity Start by extending Zero Trust to AI workloads. Every model interaction, API call, and agent action should be tied to a verified identity with clear authorization. 2. Input and Processing Prompt Injection Defense API Security Agent Permissioning Treat prompts as an attack surface. Implement input filtering, strong API authentication, and strict permissioning for agents that can call tools or systems. 3. Output and Actions Output Filtering Monitoring and Anomaly Detection Incident Response Do not just trust model outputs. Monitor behavior for anomalies, filter unsafe responses, and build playbooks for AI-related incidents. 4. Governance and Intelligence Compliance Mapping Encryption and Key Management Risk Intelligence Track where models are used, what data they access, and how they are governed. Encryption, key management, and audit trails become essential. A few practical steps organizations can start with now: 1. Inventory where AI models and agents are already running. 2. Require identity-based access for all model APIs. 3. Implement guardrails for prompts and outputs. 4. Monitor AI systems the same way you monitor production infrastructure. 5. Define incident response procedures for AI failures or misuse. AI security will increasingly look like identity architecture plus runtime monitoring. The organizations that get ahead are the ones designing this intentionally instead of reacting after deployment. How are teams structuring AI security right now?

If you’re assessing AI cybersecurity risk in your vendors, here’s a short list of things you should be actively validating... Where the model lives Is the AI: - Hosted by the vendor? - Built on a third-party model (OpenAI, Anthropic, etc.)? - Running in your environment? If they can’t clearly articulate the architecture, that’s not innovation then it’s risk. What data touches the model Validate specifically: - What data is used for training? - What data is used for inference? - Is customer data ever retained, reused, or used to retrain? “Trust us” is not a data governance strategy. Model access & isolation You want to validate: - Tenant-level isolation - Role-based access to prompts, outputs, and logs - Controls preventing cross-customer data leakage If one customer’s prompt can influence another’s output… that’s a problem. Prompt & output security Assess whether they have controls for: - Prompt injection - Jailbreaking - Output manipulation - Abuse monitoring AI doesn’t remove the attack surface it creates a new one. Human-in-the-loop controls Where does a human: - Review outputs? - Approve automated actions? - Override decisions? Fully autonomous + no oversight = unacceptable risk in most regulated environments. Logging, monitoring, and forensics You should be able to validate: - Are prompts and outputs logged? Can they support incident investigations? - How do they detect misuse or anomalous behavior? If it can’t be audited, it can’t be trusted. Third-party risk inheritance AI vendors often are aggregators of other vendors. Validate: - Who are the underlying model providers? - What contractual and security assurances flow down? - How are upstream incidents communicated? - Your vendor’s AI stack becomes your risk stack. Bottom line: Assessing AI in vendors isn’t about whether they use AI. It’s about how it’s architected, governed, monitored, and controlled.

AI agents are not yet safe for unsupervised use in enterprise environments The German Federal Office for Information Security (BSI) and France’s ANSSI have just released updated guidance on the secure integration of Large Language Models (LLMs). Their key message? Fully autonomous AI systems without human oversight are a security risk and should be avoided. As LLMs evolve into agentic systems capable of autonomous decision-making, the risks grow exponentially. From Prompt Injection attacks to unauthorized data access, the threats are real and increasingly sophisticated. The updated framework introduces Zero Trust principles tailored for LLMs: 1) No implicit trust: every interaction must be verified. 2) Strict authentication & least privilege access – even internal components must earn their permissions. 3) Continuous monitoring – not just outputs, but inputs must be validated and sanitized. 4) Sandboxing & session isolation – to prevent cross-session data leaks and persistent attacks. 5) Human-in-the-loop, i.e., critical decisions must remain under human control. Whether you're deploying chatbots, AI agents, or multimodal LLMs, this guidance is a must-read. It’s not just about compliance but about building trustworthy AI that respects privacy, integrity, and security. Bottom line: AI agents are not yet safe for unsupervised use in enterprise environments. If you're working with LLMs, it's time to rethink your architecture.