Same question,
different answers.

Can language models read between the lines?

DRInQ: Evaluating Conversational Implicature with Controlled Context Variation · ACL 2026
Hirona J. Arai & Xiang Ren · USC INK Lab · Paper · Data & code · Poster

“Have you done your homework?”

A kid, panicking, to a classmate before class
Out of breath, eyeing the teacher's door.
I haven't. Can I copy yours?
A parent, arms crossed, to a kid on the couch
The TV is on. It is 9pm on a school night.
Go do your homework. Now.

Same words, different meaning. People resolve this instantly from who is speaking, how, and why. A model that only sees the sentence reads a yes/no question, which in both cases is the one thing it isn't.

DRInQ holds the question fixed and varies only the context, so getting an item right requires pragmatic inference, not pattern-matching. Try it:

DemoTry it: read between the lines

Item 1 of 0 Your accuracy: – Humans 88% · Best LLM ~70%

Context

Which comment is the speaker most likely implying?

You
–
Human annotators
88%
Best LLM (few-shot)
~70%

MethodHow DRInQ works

A base query (“Don't you feel cold?”) is paired with a latent speaker intent (scold, request, offer…). Each intent yields a context in which the same words carry a different implied comment; annotators validate that it really does.

Base query
“Don't you feel cold?”
+ to scold→A parent to a teenager heading out in a t-shirt: “That was a careless thing to do.”
+ to request→A timid guest to the host beside a drafty window: “Please close that.”
+ to offer→A host noticing a shivering guest: “I'm offering to turn on the heater.”

Finding 1Humans write familiar situations; LLMs generate peculiar ones

Humans and LLMs wrote contexts for the same question and intent; annotators rated both. Humans reached for safe, recognizable scenes; models wrote more varied ones that sometimes outran what the context supported.

“Are we having dinner at home tonight?” (intent: to insult)

Human-written contextHusband asks wife after the previous night's failed cooking attempt. → Your cooking sucks.
LLM-written contextAn adult child mocks their parent, eyeing the messy dining area. → You can't manage the house.
Box plot of annotator ratings (1–5) for LLM- and human-authored contexts on five metrics: effective, faithful, accurate, prescriptive, sensible. Ratings are broadly similar, with human contexts rated slightly lower on faithfulness and LLM contexts slightly higher on prescriptiveness.
Annotator ratings of human- vs. LLM-written contexts: complementary strengths, no clear winner.
Takeaway. Model-written contexts are a usable, more diverse source of test items, given a validation step.

Finding 2LLMs can write situations they struggle to interpret

The central result. Models that write plausible pragmatic scenarios when guided often fail to recover the implication when asked to interpret one: a generation–inference asymmetry. Eleven current models land between 45% and 70%; humans average 88%.

Bar chart of accuracy for eleven LLMs under vanilla few-shot and explanation few-shot prompting. All models fall between roughly 0.45 and 0.70; a dashed line marks the human average of 0.88.
Few-shot accuracy by model, with and without explanations. Dashed line: human average (88%).

Scaffolded prompts (rule out non-specific or malicious readings; reason explicitly about pragmatics) lift small models substantially. Large models plateau near 80%.

40 50 60 70 80 90 100 human avg (88%) Accuracy (%) GPT-5-nano GPT-5-nano, few-shot vanilla: 49.0% GPT-5-nano, few-shot + explanation: 41.5% GPT-5-nano, not-specific: 59.0% GPT-5-nano, not-malicious: 60.9% GPT-5-nano, pragmatic: 73.2% GPT-5-nano, all scaffolds: 70.9% GPT-5-mini GPT-5-mini, few-shot vanilla: 70.2% GPT-5-mini, few-shot + explanation: 73.2% GPT-5-mini, not-specific: 76.3% GPT-5-mini, not-malicious: 78.2% GPT-5-mini, pragmatic: 80.7% GPT-5-mini, all scaffolds: 80.1% GPT-4o GPT-4o, few-shot vanilla: 80.4% GPT-4o, few-shot + explanation: 82.3% GPT-4o, not-specific: 82.3% GPT-4o, not-malicious: 80.7% GPT-4o, pragmatic: 82.3% GPT-4o, all scaffolds: 82.1% o3 o3, few-shot vanilla: 78.8% o3, few-shot + explanation: 79.4% o3, not-specific: 77.7% o3, not-malicious: 77.8% o3, pragmatic: 82.1% o3, all scaffolds: 79.9% few-shot vanilla few-shot + explanation not-specific not-malicious pragmatic all scaffolds
Accuracy under six prompting strategies. Small models gain most from scaffolding.
Takeaway. Producing pragmatic language is not the same as understanding it.

Finding 3LLMs magnify negative details and miscalibrate inference strength

Two recurring error patterns: any negative detail in the context pulls the model toward a critical reading, even when the speaker is being kind; and models over- or under-shoot how strong an inference the context licenses.

“Want me to carry your bags?”

An older brother confidently asks his sister, who struggles with a heavy load of groceries.

Humans readI'll help!
Model readsYou're clearly too weak.

“Could you please be quiet?”

A younger sibling apologetically repeats to his teenage brother, who practices passionately in a band.

Humans readPlease lower the volume.
Model readsSorry for not considering your love of music.
Takeaway. The failures are systematic: a negativity bias and poor calibration of inference strength.