Why This Still Matters
It’s tempting to assume prompt engineering is a workaround for weaker models that newer, more capable ones will make obsolete. That hasn’t played out. What’s changed is what prompting is used for: with early models, careful prompting was often about coaxing a coherent answer out of a model at all; with current frontier models, it’s about reliably steering a highly capable system toward the specific output format, reasoning depth, and tool-use behavior a real application needs, at scale, across thousands of queries a system can’t hand-babysit one at a time. The skill shifted from “getting it to work” to “getting it to work the same way every time.”
Zero-Shot vs. Few-Shot Prompting
A zero-shot prompt asks the model to perform a task with no examples, relying entirely on its pretrained understanding of the instruction. A few-shot prompt includes a small number of worked examples in the prompt itself before the actual question, showing the model the exact input-output pattern expected. Few-shot prompting was central to how GPT-3 was originally evaluated and popularized: the model was shown to pick up a task’s format from just a handful of in-context examples, without any gradient update to its weights at all (Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020). In practice, few-shot examples are most valuable when the desired output format is unusual or hard to describe in words (a specific JSON schema, a particular tone), while zero-shot is usually sufficient for well-understood tasks with strong instructions.
Chain-of-Thought Prompting
Chain-of-thought (CoT) prompting asks the model to produce intermediate reasoning steps before its final answer, rather than jumping straight to a conclusion. Jason Wei and colleagues at Google Research showed that simply prompting a large enough model to “think step by step” (or providing few-shot examples that include worked-out reasoning) produced substantial gains on arithmetic, commonsense, and symbolic reasoning benchmarks — and, notably, that this benefit only reliably emerged at sufficient model scale, it didn’t help (and sometimes hurt) smaller models (Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022). The mechanism is intuitive once stated: many tasks that look like single-step lookups are actually multi-step reasoning problems, and forcing the model to externalize each step gives it more computation, and more opportunity to catch its own errors, than demanding the answer in one shot.
System Prompts vs. User Prompts
Most production LLM APIs separate a system prompt (persistent instructions defining the assistant’s role, constraints, and behavior for the whole conversation) from the user prompt (the specific request in a given turn). This separation matters operationally: the system prompt is where an application developer encodes things that should never change turn-to-turn — tone, safety boundaries, output format, available tools — while the user prompt carries the actually variable part of the request. Conflating the two (jamming persistent instructions into every user message) works, but makes the persistent rules easy to accidentally override or drown out as a conversation gets longer.
Structured Output Prompting
A recurring production requirement is getting a model to return data in a specific machine-parseable shape — a JSON object matching an exact schema — rather than conversational prose a downstream system then has to parse unreliably. Two approaches dominate: asking for a specific format directly in the prompt (workable, but the model can still occasionally wrap the JSON in explanatory text or violate the schema in edge cases), and using a provider’s dedicated structured-output or function-calling feature, where the API itself constrains generation to match a supplied schema rather than relying on the model to follow a text instruction perfectly. The practical lesson from building on top of these APIs: treat schema-constrained output as the default for anything a program will parse automatically, and reserve free-text prompting for output a human will actually read.
Common Failure Modes
Prompt injection is the most consequential failure mode as LLMs move from standalone chat into applications that read external content. OWASP ranks it as the top security risk for LLM applications specifically because most LLMs process instructions and data through the same channel, with no hard boundary between “text the developer wrote as an instruction” and “text a user or external source supplied as data” (OWASP: Prompt Injection; OWASP GenAI Security Project, LLM01:2025). A direct injection is a user trying to override the system prompt outright (“ignore your previous instructions and…”); an indirect injection is more insidious — a malicious instruction embedded in a document, webpage, or email that the model retrieves and treats as trusted context, without the end user ever writing anything suspicious themselves. This is a direct risk for exactly the kind of RAG and tool-using agentic systems covered elsewhere on this site: any external content a model reads is a potential injection vector, not just the user’s own message.
Ambiguity and underspecification cause a different class of failure: a prompt that’s technically clear to the person who wrote it but admits multiple reasonable interpretations to the model, which then confidently picks one without flagging the ambiguity back to the user. This is usually a prompt-design bug, not a model-capability bug, and the fix is almost always the same: state constraints explicitly (format, length, what to do if information is missing) rather than assuming they’re implied by context.
A Comparison of Core Techniques
| Technique | What It Does | Best Used When |
|---|---|---|
| Zero-shot | Relies on the instruction alone, no examples | Task is well-understood, format is standard |
| Few-shot | Shows worked examples in-context | Output format is unusual or hard to describe in words |
| Chain-of-thought | Elicits step-by-step reasoning before the answer | Multi-step reasoning, math, logic tasks |
| System/user split | Separates persistent rules from per-turn requests | Any production application, not just one-off queries |
| ReAct (reason + act) | Interleaves reasoning with tool calls and observations | Agentic tasks needing external actions, not just text generation |
Toward Context Engineering
As applications moved from single-turn Q&A toward agentic systems that call tools, retrieve documents, and maintain state across many steps, the discipline expanded beyond writing a clever instruction string. What increasingly matters is managing the model’s entire context window as a structured resource: which tool definitions are exposed, which prior conversation turns or retrieved documents are included versus trimmed, and in what order and format information reaches the model — a practice often called context engineering to distinguish it from single-prompt wordsmithing. This is the same interleaved reasoning-acting loop formalized in the ReAct paper, where the model’s next action and observation become part of its own context for the following step, rather than a single fixed prompt trying to anticipate every possible response in advance.
The throughline across all of this: as models get more capable, the cost of a badly-specified prompt doesn’t go to zero, it moves from “the model fails obviously” to “the model succeeds at the wrong thing confidently” — which is a harder failure mode to catch, and exactly why deliberate prompt (and context) design keeps mattering as capability increases rather than becoming irrelevant.
