Defending against prompt injection
Securing LLM apps. This post digs into the topic, drawing on recent discussion from Hacker News, Reddit, and the wider community.
The conversation that prompted this post is linked in the sources. I have tried to capture the substance of the discussion rather than just summarize it, because the comments are usually where the real insight lives.
What happened
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Structured output via JSON schema constraints makes LLMs usable as components in larger systems. Unstructured text is a liability in a pipeline.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Structured output via JSON schema constraints makes LLMs usable as components in larger systems. Unstructured text is a liability in a pipeline.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Why it matters
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Model routing sends simple queries to cheap models and hard ones to expensive models. It is the easiest cost optimization most teams have not done.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Model routing sends simple queries to cheap models and hard ones to expensive models. It is the easiest cost optimization most teams have not done.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Structured output via JSON schema constraints makes LLMs usable as components in larger systems. Unstructured text is a liability in a pipeline.
Structured output via JSON schema constraints makes LLMs usable as components in larger systems. Unstructured text is a liability in a pipeline.
Practical takeaways
- Retrieval quality dominates RAG quality.
- The hardest part of building with LLMs is not the model call.
- Prompt engineering is real engineering.
- Structured output via JSON schema constraints makes LLMs usable as components in larger systems.
- Retrieval quality dominates RAG quality.
- Fine-tuning is overused.
- Fine-tuning is overused.
- Structured output via JSON schema constraints makes LLMs usable as components in larger systems.
- Evaluation is a product, not a test.
- Structured output via JSON schema constraints makes LLMs usable as components in larger systems.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Fine-tuning is overused. Most teams should exhaust prompting and RAG before considering a fine-tune, because fine-tuning is expensive to evaluate and hard to iterate on.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Model routing sends simple queries to cheap models and hard ones to expensive models. It is the easiest cost optimization most teams have not done.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Fine-tuning is overused. Most teams should exhaust prompting and RAG before considering a fine-tune, because fine-tuning is expensive to evaluate and hard to iterate on.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Fine-tuning is overused. Most teams should exhaust prompting and RAG before considering a fine-tune, because fine-tuning is expensive to evaluate and hard to iterate on.
Fine-tuning is overused. Most teams should exhaust prompting and RAG before considering a fine-tune, because fine-tuning is expensive to evaluate and hard to iterate on.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Sources
This post is informed by discussion on owasp.org and related threads on Hacker News and Reddit. Primary source: https://owasp.org/www-project-top-10-for-large-language-model-applications/.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Model routing sends simple queries to cheap models and hard ones to expensive models. It is the easiest cost optimization most teams have not done.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Evaluation is a product, not a test. A good eval suite encodes what your users actually care about and becomes the spec your model is measured against.
Large language models are best understood as very good pattern matchers with a large but finite context. They are not databases, not search engines, and not oracles, and treating them as any of those leads to predictable failures.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Large language models are best understood as very good pattern matchers with a large but finite context. They are not databases, not search engines, and not oracles, and treating them as any of those leads to predictable failures.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Structured output via JSON schema constraints makes LLMs usable as components in larger systems. Unstructured text is a liability in a pipeline.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Fine-tuning is overused. Most teams should exhaust prompting and RAG before considering a fine-tune, because fine-tuning is expensive to evaluate and hard to iterate on.
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Large language models are best understood as very good pattern matchers with a large but finite context. They are not databases, not search engines, and not oracles, and treating them as any of those leads to predictable failures.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Latency is a feature. Users abandon slow AI features faster than slow traditional ones because the expectation of instant response is higher.
Caching is the single biggest cost lever for LLM applications. Prompt caching and response caching can cut bills by an order of magnitude for workloads with repeated prefixes.
Large language models are best understood as very good pattern matchers with a large but finite context. They are not databases, not search engines, and not oracles, and treating them as any of those leads to predictable failures.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Retrieval quality dominates RAG quality. A perfect model with bad retrieval gives bad answers, and a mediocre model with great retrieval gives acceptable ones.
Observability for LLM apps means logging prompts, completions, tool calls, and costs. Without it you are flying blind.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Tool use turns a chat model into an agent. The model decides which tool to call, with what arguments, and interprets the result. Getting this loop right is where most of the engineering effort goes.
The hardest part of building with LLMs is not the model call. It is the evaluation. Without a solid eval suite, you cannot tell whether a change helped or hurt, and you end up shipping on vibes.
Prompt engineering is real engineering. The difference between a mediocre and an excellent prompt is often the difference between a demo and a product, and it deserves the same rigor as code.
Context windows keep growing, but attention is not free. Stuffing more tokens into the prompt dilutes the signal and increases cost and latency. Retrieval and summarization usually beat brute force.
Guardrails are not optional in production. Models will eventually produce something you did not want, and the question is whether your system catches it before the user does.
Multi-agent systems are promising but add coordination complexity. Start with a single agent and a good tool set before splitting into specialists.