Thinking Inertia: LLMs Keep Thinking When Told Not To

What the paper says. Language models increasingly ship with a thinking mode and a switch to turn it off. A paper from Tsinghua University, the University of Oxford and Stanford University asks what a model does when the switch is off. The authors split every reply into pre-answer text and final answer, then grade the text at 3 levels: is it empty, is it relevant, and does it contain an explicit reasoning step. A blinded GPT-5.5 judge does the last check, and 5 human annotators agreed with it on 98.8 percent of labels. They ran 6 models (3 sizes of Qwen3, DeepSeek-V4-Flash, GPT-5.4 and Gemini-3-Flash) under 6 prompt settings on yes/no, multiple-choice and open-ended questions.

They call the result thinking inertia: visible reasoning survives the off switch and follows the question type more than the switch. It is lowest on yes/no questions, higher on multiple choice and highest on open-ended questions. With thinking disabled, DeepSeek-V4-Flash still showed explicit reasoning in 99.9 percent of open-ended replies. A strict answer-only prompt comes at a price. Across the 5 models with thinking fully disabled, answer-only replies rose from near zero to roughly 40 percent and mean accuracy fell by about 15 percentage points on open-ended tasks. Rewriting the same question with answer choices made answer-only replies much easier to get.

Why it matters for a company. If your team turns thinking off to save tokens or time, check the replies. The setting is a request, and the model may reason anyway. Where you can, give the model answer choices. On open questions, forcing a bare answer can cost accuracy.

One caveat. The reasoning label comes from a model judge, GPT-5.5. The authors checked it against 600 human-reviewed outputs and a second judge model. The open-ended tests are 2 maths benchmarks, GSM8K and MATH, so the trade-off numbers describe maths questions. The study measures visible text only and makes no claim about what happens inside the model.

Where to read it. arXiv 2610.11765. The paper is in the week 42 list at aihorizonnetwork.com/papers.