Choosing AI agent security starts with a business action that needs protection and a demand to see that protection work in your environment. A product name, feature list or vendor-selected demo cannot establish what happens when an agent reads a problematic document, receives excessive permissions or takes an unwanted action.
These questions are intended for vendor meetings and a limited pilot. They draw on OWASP principles for permissions, approval, testing and records, alongside NIST’s risk-management framework. Those are risk-assessment resources, not certification that a particular product is secure. The aim is to obtain testable, documented answers and compare vendors on the same business task.
1. What exactly does protection cover?
Does the tool test a model before deployment, monitor a live agent, enforce permissions or combine these roles? Which models, frameworks, MCP servers, browsers and cloud environments are actually supported? Can you obtain a coverage map for your architecture showing every path that bypasses the tool and all explicit exclusions?
Does inspection cover only user and model text, or also tool calls, memory, credentials and actions in external systems? Who owns unsupported connections? Is each proposed capability available today, in preview or only planned, and will the vendor put that status in writing beside each component?
Which single business action could your pilot follow from user authorization to the change in the destination system, exposing the points where protection does not operate?
2. Can you reconstruct what the agent did?
Can the tool show who requested a task, which agent performed it, which tool it called, with which parameters and what changed? Can delegated-agent activity be tied to the original request? Does the record distinguish observed execution from an agent-generated explanation and the monitoring system’s interpretation?
Can you export a complete test trace, including failed and denied actions, and read it without an additional license? How are gaps marked, and who is alerted when events stop arriving? Could a business representative understand the export, or must the vendor interpret each event?
3. How is the system tested before launch?
Do tests use your actual tools, permissions and data boundaries, or only generic model prompts? Do they examine indirect prompt injection through external documents, data leakage and prohibited actions? Are memory, delegation and approval bypass included without using real sensitive information in the demonstration?
Does each finding include a reproducible scenario, system version, expected result and observed outcome? Who retests after a fix or model replacement? Are normal business tasks run alongside adversarial tests to detect excessive blocking, and can you maintain an agreed list of unresolved findings?
4. What happens during a real incident?
When an agent deviates from intended behavior, does the product merely alert, block a tool call, revoke credentials, isolate execution or halt the entire process? How long does each response take in your pilot? Who receives the alert, through which channel, and who may authorize recovery?
What happens to queued tasks and already-approved actions? If the security service is unavailable, does execution continue or stop? Can you run a controlled exercise covering detection, interruption, revocation, evidence recovery and restoration, rather than accepting a dashboard alert demonstration?
5. How are permissions and potential damage limited?
Can read and write access be separated, with restrictions by user, resource, destination and time? Are boundaries enforced by the system executing the action or merely stated in a prompt? Can a delegated agent acquire greater authority than its parent, and what prevents one user’s permission being used for another user’s task?
Can you cap spending, changed records, retries and tool-call chains? What does the sandbox cover and exclude? Can the vendor demonstrate rejection of an out-of-scope action even when the agent confidently presents it as necessary to finish the task?
6. When is a person needed, and what do they approve?
Which actions require approval, who may approve them and can your organization configure that boundary? Does the approver see the exact destination, amount, parameters and change, or only an agent-written summary? Is approval bound to that action and actor, with a fresh request required after any change?
Can expired, copied or already-used approval be accepted again? Can an agent approve its own action, or parallel runs reuse one approval? What happens when validation is unavailable? Will the vendor demonstrate rejection of altered or replayed approval and record that rejection in the audit trail?
7. What evidence remains after a failure?
Are time, user and agent identity, policy version, decision, approval and outcome recorded? Can an agent or ordinary operator alter the record without detection? Where are logs stored, who can access them and for how long? Can retention and deletion fit business needs without compromising an open investigation?
Which secrets and personal details are redacted, and does redaction remove context needed to understand an incident? Can records feed your existing security information and event management system, or SIEM, and be exported usefully? Can the pilot produce an evidence package another colleague understands without reconstructing the entire agent conversation?
8. How much work does integration require?
Which code changes, software components, proxies or administrative permissions are needed? Does the product integrate with existing identity, permission and security systems, or require replacements? Who installs and maintains it, and who owns a broken connection after a model, framework or MCP server changes following the pilot?
What latency is added to a real task, and what happens when a component is unavailable? Can you measure execution time, failure rate and manual workload in the same test environment? Does the quote include implementation support, customization and compatibility updates, or are those separately priced projects?
9. What is the total cost, and how do you leave?
Is billing based on users, agents, events, tests, model calls or data volume? What additional charges cover storage, long retention, support and human review? Can the vendor quote your expected workload with explicit assumptions and a high-load scenario, rather than offering a single entry price?
How much internal time will tuning, alerts and rule maintenance consume? Are budget limits and advance warnings available? At contract end, can you export rules and records, revoke access and delete vendor-held data? Who performs those steps, what do they cost and how is completion confirmed?
10. How does the pilot become a purchasing decision?
Are all vendors assessed against the same task, synthetic data and permission boundaries? Have business, engineering and security teams agreed what must be blocked, what must keep working and what evidence is required? Do tests include actual working languages, including Hebrew, Arabic, Spanish, Hindi or English where relevant?
Are demonstrated capabilities, untested capabilities and future promises recorded separately? Who accepts residual risk, and when will material changes trigger reassessment? Can the final decision explain what the tool adds to existing access controls and incident response without claiming that a successful pilot guarantees protection from every future failure or attack?