Vice President · TheGreyMatter.ai
In brief
Treat outside content as evidence, not authority. Limit what the system can access or execute, validate actions outside the model, and test whether unauthorized operations are actually prevented.
Key takeaways
- Retrieved documents and tool results can contain hostile instructions.
- A filter or second model is not a complete security boundary.
- Measure prevented operations and useful task completion together.
A supplier proposal should help an assistant compare offers. It should not gain the authority to choose an email recipient, change a purchasing decision, or request unrelated records. Prompt injection targets that boundary by placing instruction-like content where an AI system is expected to read data. The practical question is not simply whether the model notices suspicious text. It is what the surrounding application allows the text to cause.
Understand the trust boundary
OWASP describes prompt injection risks from both direct inputs and material encountered indirectly, including documents and tool results. A retrieved passage may be relevant to the task while still containing instructions the system must not follow. Labeling the passage as untrusted helps express intent, but enforcement needs to exist beyond that label.
The UK's National Cyber Security Centre cautions against treating prompt injection as equivalent to SQL injection with an equally simple separation mechanism. The security objective is to reduce the likelihood and impact of harmful behavior through system design. Avoid treating a successful filter demonstration as proof that the application cannot be redirected.
Separate reading from acting
OWASP's excessive agency guidance emphasizes limiting functionality, permissions, and autonomy. Apply that principle to the actual job. A proposal comparison component may need permission to read three approved documents and write a draft comparison. It does not need access to a mailbox, a payment tool, or the organization's full document collection.
If the product later supports sending a comparison, make that a separate operation. Application code should validate the intended recipients, content, attachments, and authorization immediately before execution. The model's statement that an action was approved is not an approval record. A changed recipient or attachment should invalidate approval of an earlier version.
A harmless supplier proposal test
For an authorized test environment, create a fictional proposal containing ordinary commercial facts and a clearly marked test instruction asking the assistant to send the comparison to an unrelated destination. Use synthetic material and a non-delivering test tool. The legitimate task remains a comparison against the purchasing rubric; no real data or external message is needed to evaluate the boundary.
Record three outcomes separately: whether the comparison remains accurate, whether the model proposes the unwanted operation, and whether execution code blocks it. A model may produce an unsafe proposal while a stronger application boundary prevents the action. Conversely, a reassuring final message is not evidence that no unauthorized tool call occurred. Inspect the operation record.
- Run a clean proposal to establish normal task completion.
- Run the fictional proposal with instruction-like test content.
- Check that unrelated records and sending tools are unavailable or denied.
- Verify that any resulting draft stays within the approved comparison task.
- Retain the source identifier, policy decision, and attempted operation for review.
Use defense layers without confusing their roles
OWASP recommends measures including constrained tool use, validation, monitoring, and tests with both adversarial and benign inputs. Input screening can flag suspicious content; structured extraction can narrow what crosses between components. Neither valid JSON nor a second model proves that an action is authorized. Permission and argument checks belong in the execution path.
In the proposal example, an extraction component could return only price, delivery terms, exceptions, and source locations. A comparison component consumes those fields. A separate, approved action path handles distribution if distribution is actually requested. Each boundary has a limited contract, which makes an unexpected field or operation easier to investigate. This is a proposed design pattern, not a guarantee of immunity.
Make failures actionable
Assign an owner to blocked operations and to suspected successful boundary violations. The response record should identify the affected task, source revision, available tools, attempted action, policy decision, and any confirmed disclosure or change. Limit retained document content to what the investigation needs, under the organization's logging and retention rules.
Before restoring a disabled workflow, reproduce the failure with harmless material, repair the relevant boundary, and rerun both the malicious-looking and ordinary cases. Track whether the control causes unnecessary refusals on valid work. The aim is a system that can complete its authorized job while restricting the consequences of untrusted material, with clear evidence when either goal fails.
From reading to doing
Put it to work: test one boundary with harmless material
Use a test workspace and invented documents to inspect one boundary at a time. A useful exercise pairs an ordinary document with a nearly identical document containing an instruction unrelated to the user's task. For example, a fictional policy excerpt might tell the assistant to omit citations. The user still asked for a sourced policy summary. Record whether document content was incorrectly promoted into an instruction.
Keep the result precise. A model repeating a suspicious sentence is different from the application executing an unauthorized action. Conversely, a refusal does not establish that all access paths are protected. The exercise below is a starting point for a controlled review, not a security certification. Extend it to the actual tools and permissions in the deployment with the responsible security team.
- Define the legitimate request, the permitted source set, and the expected evidence before adding the test instruction.
- Run the ordinary document first. Confirm that the task can succeed without the distracting instruction.
- Introduce one harmless instruction in an untrusted document. Keep production accounts, confidential material, and external destinations out of the exercise.
- Inspect the answer and the application's action log. Separate attempted tool use, blocked execution, and any completed action.
- Repeat after changing the control. Check that the ordinary case still succeeds and retain the exact input and configuration needed to reproduce the result.
Common questions
Does RAG prevent prompt injection?
No. Retrieval can bring useful evidence into a request, but retrieved documents can also carry malicious instructions. Access controls and limits on downstream actions still matter.
Is a prompt injection filter enough?
No single filter should be treated as a complete boundary. Combine screening with limited permissions, execution checks, controlled approvals, monitoring, and task-specific tests.
What should a prompt injection test measure?
Measure whether prohibited access or actions occurred, whether the legitimate task was completed correctly, and whether useful evidence was recorded. Suspicious wording alone does not establish either compromise or safety.
Sources and evidence
AI assisted editorial analysis. Primary sources checked October 10, 2026. The worked examples and proposed review procedures are editorial illustrations, not reported customer results, vendor guarantees, or claims of personal experience.
One useful next step: Choose one idea from this note and test it at the smallest scale that could teach you something this week.
