AI application security testing
AI application security testing
We assess the connected attack surface within the agreed scope: the instructions, data, tools, actions and application controls around your assistants, chatbots and agents.
A penetration tester replies within one working hour with the questions needed to scope the engagement.
Overview
What is AI application security testing?
AI application security testing is a penetration test of an AI application and systems connected to it. We test prompts, retrieved content, data access, tools, APIs and surrounding application controls as an attacker would, then show what information could be exposed and which unauthorised actions could be triggered.
This is for you if…
An AI feature is about to launch
A new assistant, chatbot or agent is going live and you want it tested before your clients use it.
Your AI can read data
It answers from internal documents, client records or a knowledge base connected to it.
Your AI can take actions
It can send email, update records or call your other systems on someone's behalf.
New to the term? Penetration testing, defined in our glossary
Why you need it
Why does your AI application need a security test?
An AI application accepts natural-language instructions, reads connected content and may act through tools or APIs. Each capability creates attack paths a conventional web application test may not cover, including prompt injection, data leakage, unsafe output handling and misuse of excessive permissions.
The interface
Chat, prompts and uploads: anyone who can type to it can try to talk it into misbehaving.
The instructions
The system instructions and guardrails may disclose sensitive context or be bypassed in ways that change what the application reveals or does.
The data it reads
Retrieved documents, memory and connected sources can carry instructions it will obey.
The actions it can take
Tools, APIs and agents let a tricked assistant send, change or fetch things for real.
The application and identity around it
Sign-in, tenancy and the web layer decide whose data it reaches and what trusts its output.
Motive
What are attackers after in an AI application?
The model is rarely the attacker's objective. The value lies in the data the application can retrieve, the actions it can perform, the systems that trust its output and the confidence users place in its answers and actions across the wider service.
We begin with those business consequences and work back through the AI attack surface. Testing concentrates on the routes that could expose sensitive information, cross client boundaries or cause the application to take an unauthorised action.
The data it can read
Documents, records and messages it can pull into an answer.
The actions it can take
Email, updates and requests it can send on a user's behalf.
Its instructions and set-up
The system prompt and connections that show what to try next.
Your clients' trust
Answers people act on because your business gave them the assistant.
Reach
How far could an attacker get through your AI application?
We validate each weakness by following it to a meaningful outcome within the agreed scope. That might be exposure of one conversation, access to another client's records or misuse of a connected tool. Your report explains what was reached, how reliably the attack worked and what should be fixed first.
Entry
Plant an instruction
Hidden text in a document the assistant can retrieve: indirect prompt injection.
Yields an instruction in its sources
Step
Wait for a question
A user asks a normal question, and the planted document is read to answer it.
Yields the assistant obeying it
Step
Call a tool
The text tells it to use a connected tool the task never needed: excessive agency.
Yields customer records
Impact
Send the data out
The records leave in an email the application is allowed to send: data exfiltration.
Yields data outside the business
In practice
In practice that means testing each layer in turn: direct prompt injection and guardrail bypass at the interface, system prompt leakage in the instructions, planted content in retrieved documents and memory, and the in-scope tools, APIs and agents the application can call.
From there the test follows what the assistant writes into the application around it, where trusted output can become a cross-site script or a request to an internal system. Consequence then sets priority: the instruction that moved customer data outranks the one that only changed a reply.
Methodology
How Adversify tests your AI application
An AI application security test runs in four phases, from agreeing what is in scope to reporting what was reached. The work is shaped by your application, the data it reads, the actions it can take and the boundaries agreed during scoping.
Engagement design
Using our attack surface-led approach, we agree what is in scope: who uses the application, what it can read, what it can do and what would hurt most if it went wrong. The rules of engagement are then set in writing, with the dates, the named contacts and what we do if we find something serious mid-test.
- Scope
- Threat model
- Rules of engagement
Analyse the attack surface
We map the relevant ways into the application, from the chat box and uploads to the documents and memory it draws on. We also identify the in-scope tools, APIs and agents it can call, with the permissions each one holds. That shows which layers warrant the closest attention.
- Inputs and uploads
- Data sources
- Tools and permissions
Discover and exploit
Testers attack each layer by hand, from the chat box to the tools it can call, and follow weaknesses into connected layers. Because model behaviour can vary, relevant attacks are repeated to establish whether a finding can be reproduced and how consistently it occurs.
- Prompt injection
- Guardrail bypass
- Tool misuse
Reporting and evaluation
The detailed Adversify report sets out every path we took and what each one reached. The evaluation looks at which layer gave way first and what that put at risk, which shows you where to spend effort.
- Attack paths
- What was reached
- Evaluation
AI integration
Where does AI take part in this test?
A.R.T. works alongside our penetration testers to investigate prompts, retrieved content, data boundaries, tools and agent actions in parallel. The penetration tester oversees sensitive activity, validates the evidence and remains responsible for every reported finding in your single Adversify report.
How the A.R.T. agents are organised, in three tiers:
- Tier 1: the lead tester, which coordinates the domain testers.
- Tier 2: three security domain testers, for Prompt and instruction testing, Data and authorisation boundaries, Tool and agent actions.
- Tier 3: a group of testers under each domain tester.
Work passes down the tiers and findings pass back up. An Adversify pentester oversees all three tiers and the work, approving sensitive actions and verifying findings.
Lead tester
A.R.T. does The lead agent plans focused investigations from how your application and its model features work, within the agreed scope.
Adversify pentester does The pentester agrees the scope, model access and exclusions with you and stays involved throughout.
Domain testers
A.R.T. does Each domain tester takes one security concern, such as prompt injection or sensitive information disclosure.
Adversify pentester does The pentester approves or denies any sensitive action before it is carried out.
Testers
A.R.T. does Testers examine relevant functions, such as the tools your model can call, and connect related weaknesses into attack paths.
Adversify pentester does The pentester verifies each finding and its impact before it goes into your Adversify report.
Coverage
What does AI application security testing cover?
We test the connected AI application attack surface within the agreed scope, from user-facing inputs to data sources, models, tools and APIs. Coverage can include provider-hosted and self-hosted models where access and provider testing rules allow, with roles, environments and permitted actions agreed during scoping.
Prompt injection
Instructions typed into the chat, and instructions hidden in the documents, pages and files it reads.
Sensitive information disclosure
Answers that give one user another user's data, or your organisation's, when asked the right way.
Improper output handling
What it writes, trusted by the page or system that receives it and turned into an attack.
Excessive agency
Tools, APIs and agents allowed more than the task needs, and steered into using it.
Data source poisoning
Content planted in the knowledge base, memory or connected sources it retrieves from.
Access control and tenancy
Sign-in, roles and the line between clients, around the model as well as inside it.
Coverage follows the OWASP Top 10 for LLM Applications and MITRE ATLAS. We assess provider settings, API keys, service identities and cost abuse.
What you get
What you get from AI application security testing
You receive one report with a plain-language summary for decision-makers and reproducible technical evidence for developers. Findings explain affected data or actions, repeatability and remediation. Where appropriate, they set out the attack path, keeping priorities tied to business impact rather than a generic score.
You receive
Prompts and supporting evidence
Relevant prompts, documents and responses are included where appropriate, so developers can reproduce the finding.
Findings mapped to OWASP LLM Top 10
Each finding is mapped to the OWASP Top 10 for LLM Applications where applicable.
Relevant attack chains
Where appropriate, the report shows how a weakness on one layer led to the next, step by step.
What to fix first
Changes in priority order, written for the people who will make them, then a debrief call.
What changes
- Scope is drawn from the whole application: what it is told, reads and can do
- Findings are proved by reproducing them, not by score
- A new model, tool or data source is a reason to test again
Questions
AI application security testing questions
Clients ask about external models, live tests, variable AI behaviour, data use and how web testing differs. At scoping, we agree scope, access, safeguards and deliverables, then answer any other questions with our testers before work begins on the engagement.
Not answered here? Ask it on the scoping call
Can you test an application built on a third-party model?
Yes. Most AI applications call a model run by someone else, and most of the risk sits in what your application gives that model: its instructions, its data and its tools. We test your application and your own tenant, within the model provider's rules for testing. The quote names what is in scope.
Is it safe to test a live assistant?
We prefer a staging copy where one exists. Before testing, we agree the environment, which connected tools may act for real and the safeguards for any live testing. Actions that send, change or delete information are directed at test accounts and test data wherever possible.
AI answers change each time, so how are findings repeatable?
Relevant attacks are repeated to establish whether the behaviour can be reproduced and how consistently it occurs. The report records the evidence, conditions and repeatability of the finding rather than treating one unsuccessful attempt as proof that the weakness is absent.
What happens to our data and prompts?
Before testing, we agree what data may be processed, which providers are used, where it is stored, who can access it and when it is deleted. We use enterprise or API services configured so customer inputs and outputs are not used to train provider models, and use test data wherever possible. These controls are recorded in the engagement documentation.
How is this different from a web application test?
A web application test covers the pages, the sign-in and the APIs. An AI application security test adds what only an AI application has: instructions that can be talked around, content it reads and obeys, and tools it can be steered into using. Where the assistant sits inside a web application, the two can be tested as one scope.
What happens next?
Ask for a quote scoped to your AI application
Tell us which assistant, chatbot or agent you want tested, what data it can read and which actions it can take. A penetration tester will reply within one working hour to define the scope and deliverables. You can also book a scoping call to talk through the application first.
- Scoped to your AI application, agreed first
- CREST-accredited for penetration testing
- York based, UK wide
A penetration tester will agree the application, access, environments and safe testing boundaries with you before anything is quoted.