Does Prompting Actually Work? — The Science of Asking AI Well

Anyone who uses ChatGPT or Claude every day wonders at least once: "How should I ask to get a better answer?" The text put in front of the model to get such an answer is a prompt. The internet is overflowing with sentences claiming, "Add just this one line and the AI becomes a genius." Researchers have measured these claims using benchmarks, which are fixed sets of problems. This article summarizes which techniques showed effects in which models and tasks, and where they diverged.

What Has Been Shown to Work: Showing Examples with Intermediate Steps

Having the AI write out intermediate solution steps one by one before giving an answer is called Chain-of-Thought (CoT) (exactly as defined in "Does AI Really 'Think'?"). The most widely tested method for eliciting such solutions is chain-of-thought prompting. Chain-of-thought prompting is a method where examples that write out intermediate reasoning steps leading to the answer are included in the prompt, causing the model to also answer while writing out intermediate steps. The 2022 original paper showed a notable improvement in the arithmetic, commonsense, and symbolic reasoning performance of large models. When just 8 examples were given to a model with 540 billion parameters (the numbers inside the model determined by learning), it achieved the highest performance up to that point on a grade-school math word problem benchmark (GSM8K) (arXiv:2201.11903, preprint).

However, this effect appeared in models that were sufficiently large. That paper interprets that this reasoning ability emerges naturally in large models, and chain-of-thought prompting elicits it.

Techniques built on top of chain-of-thought prompting have also been tested. Because chatbots pick the next token based on probabilities (which is the sampling explained in "How Generative AI Answers"), the same problem can produce different solutions every time it is solved. Self-Consistency is a method where a single problem is solved in multiple directions, and then the most frequently occurring answer is chosen. In this 2022 paper, GSM8K scores rose by +17.9% compared to using chain-of-thought prompting alone (arXiv:2203.11171). Tree of Thoughts has the model explore multiple directions and evaluate itself instead of reasoning in a single straight line. In a 2023 experiment, GPT-4 could only solve 4% of the "24 game" (a number puzzle requiring planning and exploration) with chain-of-thought prompting, but solved up to 74% using this method (arXiv:2305.10601). There is also "thought boosting," which automatically fixes prompts through error analysis gained from trial and error (arXiv:2402.11140).

Few-shot prompting is a method where a few examples with correct answers are put into the prompt so the model follows that pattern. It is a method that uses in-context learning—as discussed in "How Guessing the Next Word Becomes a Conversation"—accomplishing new tasks using only the examples in the preceding text without modifying the parameters. However, results fluctuate depending on how the examples are included and in what order, and this point is examined in the next section.

The name "few-shot" is also used when handling models with a small amount of training data. A 2024 study that categorized paragraphs of German medical opinion letters in low-resource language environments is such a case. As a result of handling a lightweight model tuned for medical text (a small model of a different type than conversational chatbots) using just 20 examples as a small amount of training data and a prompt, accuracy was 30.5% higher than traditional classification models (as reported in the paper) (arXiv:2403.13369). Here, the examples were used as a small amount of training data, unlike examples put into a chat window.

Efforts to improve reasoning also extend outside the prompt, namely toward retraining the model. Methods that score and reward each step of a solution (arXiv:2312.08935) and methods that train on pairs where the better side is chosen at each step (arXiv:2406.18629) have boosted math accuracy. While a 2025 survey on Long CoT views deep reasoning, broad exploration, and looking back as keys to solving complex problems, it notes that debates surrounding "overthinking"—thinking for excessively long periods—continue (arXiv:2503.09567).

Subtle or Exaggerated: Prompts Shift Easily Than You Think

Even with prompts containing examples, minor formatting changes caused performance to swing significantly. Across several open-source models widely used in 2023, prompts with identical meanings that differed only in using a colon or where line breaks were placed altered performance. In LLaMA-2-13B, this formatting change alone created a gap in accuracy of up to 76 points. This sensitivity did not disappear even when scaling up the model, adding more examples, or performing instruction tuning—fine-tuning with paired examples of instructions and responses. Therefore, the same paper questions the "practice of reporting performance measured by a single format" (arXiv:2310.11324, preprint). This means a single performance number is just a value measured in that single format.

The order of examples also shakes performance. It has been reported that simply rearranging the order of the exact same examples can swing performance from "nearly random" to "near state-of-the-art" (arXiv:2402.11447). There is also an analysis showing that causal models—where each token attends only to the preceding tokens—are particularly sensitive to this order (arXiv:2402.15637). These studies also offer countermeasures. A tool that automatically scans multiple formats to report a range of expected performance, principles for arranging examples, and fine-tuning to reduce order sensitivity have each been proposed, with their effectiveness reported (arXiv:2310.11324, arXiv:2402.11447, arXiv:2402.15637).

Consequently, research interest is shifting away from "magic single lines" toward robustness, the property of performance fluctuating less even when expressions change slightly. A 2024 survey points out the limitations of designs that expect "AI to think like a human." Reviewing 50 representative studies, the authors report that goal-directed prompt construction—guiding AI to follow human logical reasoning flows—significantly improves performance (arXiv:2401.14043). Other researchers in 2025 argue that prompt development still remains stuck in ad-hoc trial and error, coining it a "promptware crisis," and assert that software engineering principles must be brought in (arXiv:2503.02400).

Politeness and Emotional Stimulation — Results Are Mixed

There is a story that adding sentences like "Please, I'm begging you" or "This depends on my career" makes AI do better. EmotionPrompt is a method of appending these emotion-mixed sentences to the original prompt. An early 2023 paper automatically evaluated 45 tasks across various models of that time—such as Flan-T5, Llama 2, ChatGPT, and GPT-4—and reported that emotional stimulation raised performance (8% relative improvement in instruction induction, 115% in BIG-Bench — the ratio against the original scores). In a human evaluation involving 106 participants, performance, truthfulness, and responsibility metrics for generation tasks rose by an average of 10.9% (arXiv:2307.11760, preprint). A follow-up study the same year tested emotional sentences chosen to raise performance (EmotionPrompt) alongside ones chosen to lower it (EmotionAttack), showing that emotional stimulation can work in either direction (arXiv:2312.11111).

A 2026 study (preprint) reports the possibility of side effects. In this study's evaluation, positive emotional stimulation tended to increase accuracy while reducing toxicity—that is, harsh or harmful expressions. At the same time, it also tended to increase sycophancy, which is the tendency to agree along with the user's words (arXiv:2604.07369). This is a result showing that sycophancy can increase alongside accuracy. There are also attempts to combine emotional stimulation with structural prompts (arXiv:2404.10500). Depending on which emotional sentence was used, the direction of the effect differed, and side effects were reported alongside gains.

Same Technique, Different Results — Models and Tasks Change the Outcome

The exact same technique may or may not work depending on the model. A 2026 controlled experiment had models classify text using multi-shot prompts that included several examples, and measured how closely their classifications matched those of human researchers. This approach increased agreement by a statistically significant, small margin in Claude Haiku, but not in DeepSeek-Chat or Gemini 2.5 Flash, and the stability upon repetition also varied across models (arXiv:2605.07422, preprint). A 2025 study classifying German survey responses also found large performance gaps between models, with the effectiveness of the prompting method depending on which model was used. In that study, no model reached satisfactory prediction performance through prompting alone, and only fine-tuned models reached that level (arXiv:2506.14634).

The nature of the task also changes the results. Zero-shot chain-of-thought is a method that instructs the model to write intermediate steps without examples by simply adding instructions like "think step by step." When a 2023 study applied this method to moral scenario tasks from a benchmark called MMLU, accuracy dropped by about 4% compared to simply asking, unlike in mathematics. The same study designed a prompt template that had the model self-generate and answer counterfactual questions of the "what if?" type, boosting accuracy by 9 to 16% over other methods that asked without examples. Adding five examples raised it by up to 80% (arXiv:2306.14308). In this study, "think step by step" without examples did not work for that task, while a template tailored to that task raised accuracy.

Conversely, there are tasks where the effects were substantial. In a 2025 secure code generation study, security-focused prompt prefixes reduced vulnerabilities by up to 56% in some GPT models. In all models tested, using a prompt-echoing approach where the model repeated back its own code, it found and fixed 41.9 to 68.7% of the vulnerabilities it had previously created (arXiv:2502.06039). In multilingual environments, strategies such as whether to pre-translate prompts into a single language or ask queries in mixed languages greatly influenced classification performance (arXiv:2507.22923).

Finally, a surface-plausible answer does not mean it is correct underneath. A 2026 evaluation study ran experiments where certain prompt items were deliberately omitted. Even among answers that received a perfect score in total points, 25.7% of Chinese outputs and 58.6% of English outputs missed the user's specific intent (arXiv:2605.14517, preprint). This means that just because an answer has the right shape does not mean the meaning of the request has been followed.

So the Bottom Line is

Both those who argue that "rewriting prompts is no longer meaningful" and those who say "it is still important" can provide evidence. Chain-of-thought prompting, providing examples, and self-consistency have been reported to be effective as benchmarks in the models and tasks those researchers tested. At the same time, because performance fluctuated wildly with minor changes in format, order, or phrasing, the promise that "this single line is all you need" is mostly an exaggeration. The effect of any technique must be read alongside when, on which model, and in which task it was measured.

There were exceptions to these effects as well. In the MMLU moral scenarios task, zero-shot chain-of-thought with only instructions attached and no examples lowered accuracy by about 4% compared to asking directly, and examples caused results to fluctuate depending on their order. Emotional stimuli showed different directions of effect depending on the sentence, and one study found that positive emotional stimuli increased sycophancy. This is why knowing how to ask well must go hand in hand with doubting the answers you receive.

---

Disclaimer: AI is a fast-moving field. The results gathered here are from research published between 2022 and 2026 measured on models of that time, and the same techniques worked differently across different models. A significant portion of the cited papers are preprints before peer review, so the results may change. Please read these not as prescriptions, but as evidence under the conditions tested by each study.

Source Papers (arXiv)

• arXiv:2201.11903 — Chain-of-Thought Prompting Elicits Reasoning in LLMs
• arXiv:2203.11171 — Self-Consistency Improves Chain of Thought Reasoning
• arXiv:2305.10601 — Tree of Thoughts: Deliberate Problem Solving
• arXiv:2402.11140 — Boosting of Thoughts
• arXiv:2403.13369 — Clinical Information Extraction with Few-shot/Prompting
• arXiv:2312.08935 — Math-Shepherd: step-by-step verify and reinforce
• arXiv:2406.18629 — Step-DPO: step-wise preference optimization
• arXiv:2503.09567 — Survey of Long Chain-of-Thought
• arXiv:2310.11324 — Sensitivity to Spurious Features in Prompt Design (formatting)
• arXiv:2402.11447 — In-Context Example Ordering Guided by Label Distributions
• arXiv:2402.15637 — Order Sensitivity of In-Context Demonstrations
• arXiv:2401.14043 — Goal-oriented Prompt Engineering (anthropomorphic limit)
• arXiv:2503.02400 — Promptware Engineering (promptware crisis)
• arXiv:2307.11760 — EmotionPrompt: Emotional Stimuli
• arXiv:2312.11111 — The Good, The Bad, and Why: Emotions in Generative AI
• arXiv:2604.07369 — Emotional Stimuli and Intensity (sycophancy)
• arXiv:2404.10500 — Auto-Prompt Graphical Paradigm (emotional + framework)
• arXiv:2605.07422 — Prompt Engineering for Qualitative Coding (model-dependent)
• arXiv:2506.14634 — LLMs for Coding German Survey Responses
• arXiv:2306.14308 — Thought Experiments / Counterfactuals (zero-shot CoT hurts moral reasoning)
• arXiv:2502.06039 — Prompt Engineering for Secure Code Generation
• arXiv:2507.22923 — Translation Strategies in Cross-lingual Prompting
• arXiv:2605.14517 — Dimension-Level Intent Fidelity Evaluation