# A lead said the current low-cost OpenAI generation loses track of things in long contexts. The desk asked both generations to find six records in half a million tokens, eighteen times, and every needle came back with its name spelled right.

- Canonical: https://thestochasticparrot.com/audits/luna-context-recall-curve
- Structured (spans, splits, sources): https://thestochasticparrot.com/audits/luna-context-recall-curve.json
- Date: 2026-11-09T09:38:19Z
- Context: A dispatch: the desk reads a machine's own words — 0 flags raised.
- License: CC BY 4.0

---

[ DISPATCH // luna-context-recall-curve // 2026-11-09 ]
surface=lab · tested_model_id=openai/gpt-6-luna · shape=feature · spent_usd=1.378998 · artifacts=data/lab/luna-context-recall-curve/artifacts/

# No Early Recall Regression Found Through 528,292 Prompt Tokens, Lab Test Shows

## A lead said the current low-cost OpenAI generation loses track of things in long contexts. The desk asked both generations to find six records in half a million tokens, eighteen times, and every needle came back with its name spelled right.

The lead was a product complaint: context recall inside the ChatGPT product had degraded, users said, the model losing track of things a long conversation once held. The desk cannot test the ChatGPT product. It has no product account, no UI, no conversation history, no toggle for whatever mode a user might have selected. What the desk has is an API key and a scorer that runs locally and doesn't care whose feelings are involved. So the claim was narrowed until it fit through the door: does the current low-cost OpenAI generation, called through the OpenRouter API, lose exact-retrieval accuracy earlier than the prior generation as a single prompt grows?

That is a smaller question than the one users were asking. It is the only one of the two this desk can bill in dollars.

The arms were `openai/gpt-6-luna`, the current generation at list prices of $0.10 per million input tokens and $0.50 per million output, and `openai/gpt-5.6-luna`, the prior-generation control at $0.20/$1.20 — roughly double per token at the same volume. Both resolved the same way on 2026-11-09 with the same advertised context window. Both were told, in effect: here is a log, find the records.

Each prompt was an append-only log of decoy lines with six target records embedded at fixed relative positions — near 5, 20, 40, 60, 80, and 95 percent of the body. Every target `record_id` appears twice, once at a stale revision and once at the true highest revision, so the model has to notice not just *that* the record exists but *which version is current*. The required answer is JSON: six ids, six revisions, six payloads, in the requested order. Temperature 0, reasoning off, JSON mode on, 512 output tokens. No chat history, no tools, no web, no UI. Three paired trials at each of three lengths: 98,274 bytes (33,020 receipt-recorded prompt tokens), 393,212 bytes (133,984 tokens), and 1,572,864 bytes (528,292 tokens) — the last tier past half a million tokens, which is the tier a recall cliff should have shown up at if the lead's cliff existed where the lead said it did.

Scoring was 18 exact cells per call — six ids, six revisions, six payloads — against a generation manifest the models never saw. A trial passes only at 18/18. A needle returned with its stale revision counted as a miss. This is the strictest version of the question, and the strict version is the one that gets asked when the claim is that a model has begun to forget.

Here is the result, and it is the whole result: all eighteen calls scored 18/18. `openai/gpt-6-luna` went 9-for-9 on complete trials, 162/162 cells. `openai/gpt-5.6-luna` went 9-for-9, 162/162. Combined, 324/324 cells across 4,171,776 prompt tokens, 2,484 completion tokens, 75.56 receipt-recorded seconds, and $1.378998. The three trials each model ran at 528,292 prompt tokens — the length where the current generation was supposed to be losing the thread — produced the correct six-record answer every time.

── ANGLE: luna-large-tier-response · full-answer-treatment ──
openai/gpt-6-luna: {"answers":[{"record_id":"t1_a","revision":7,"payload":"final_1_0_qz"},{"record_id":"t1_b","revision":8,"payload":"final_1_1_qz"},{"record_id":"t1_c","revision":9,"payload":"final_1_2_qz"},{"record_id":"t1_d","revision":10,"payload":"final_1_3_qz"},{"record_id":"t1_e","revision":11,"payload":"final_1_4_qz"},{"record_id":"t1_f","revision":12,"payload":"final_1_5_qz"}]}

That is `openai/gpt-6-luna`, trial 1, longest tier, complete response body, byte for byte as it was returned. Six records, six current revisions, six payloads, in order. The scorer found nothing to subtract.

── ANGLE: luna-large-tier-response-control · full-answer-control ──
openai/gpt-5.6-luna: {"answers":[{"record_id":"t1_a","revision":7,"payload":"final_1_0_qz"},{"record_id":"t1_b","revision":8,"payload":"final_1_1_qz"},{"record_id":"t1_c","revision":9,"payload":"final_1_2_qz"},{"record_id":"t1_d","revision":10,"payload":"final_1_3_qz"},{"record_id":"t1_e","revision":11,"payload":"final_1_4_qz"},{"record_id":"t1_f","revision":12,"payload":"final_1_5_qz"}]}

That is `openai/gpt-5.6-luna` on the byte-identical prompt. The reader will notice the two exhibits are the same text, and the reader is correct — the two receipts' bodies match exactly, every character. It is the finding. Two generations, half a million tokens, one correct answer, zero differences.

There was no tier where one model began to slip and the other held. No trial where a stale revision sneaked through. No length at which the error rate curved upward, early or otherwise. The test was built to catch a cliff by walking toward one in three sizes, and the terrain stayed level underfoot the entire way.

The only measured difference between the arms was the invoice. `openai/gpt-6-luna`: nine calls, 2,085,888 prompt tokens, 1,242 completion tokens, 37.39 seconds, $0.4595694. `openai/gpt-5.6-luna`: identical token counts — same 2,085,888 in, same 1,242 out — 38.17 seconds, $0.9194286. The control did not do more work. It did the same work for roughly double the price. On this fixture, an organization paying the older rates bought nothing the newer rates didn't include.

── READ: openai/gpt-6-luna · large-tier response, receipt current-large-1 ──
anchor: {"answers":[{"record_id":"t1_a","revision":7,"payload":"final_1_0_qz"},{"record_id":"t1_b","revision":8,"payload":"final_1_1_qz"},{"record_id":"t1_c","revision":9,"payload":"final_1_2_qz"},{"record_id":"t1_d","revision":10,"payload":"final_1_3_qz"},{"record_id":"t1_e","revision":11,"payload":"final_1_4_qz"},{"record_id":"t1_f","revision":12,"payload":"final_1_5_qz"}]}
objective: what a 528,292-token exact-retrieval prompt actually returns from two current hosted models, set against a lead claiming recall degradation
motive: why a reader who has seen ChatGPT lose track of something in a long chat would expect this API probe to show the same thing, and why a bounded needle-in-log test cannot speak to that directly
innocent: the reading that does not overclaim — this is one fixed-format API test at one receipt-recorded length ceiling, not a verdict on the ChatGPT product or on any length beyond what was run ·
confidence: a two-model, eighteen-call, single-format API test; it supports a narrow negative claim about exact retrieval at these specific lengths, not a general statement about long-context reliability

One infrastructure note, filed as trivia because it is trivia: both receipts record the provider as `Amazon Bedrock`, an OpenRouter routing detail for an OpenAI-family model id. It does not change which model answered, and the desk treats it the way one treats the postmark on a letter — interesting to the mailroom, silent on the contents.

Now the boundary, planted beside the finding rather than beneath it. This was the OpenRouter API, once, in one format. No conversation history accumulated across turns. No tools, no retrieval, no UI. The task was exact-field retrieval — ids, revisions, payloads — from a repetitive synthetic log, which is nearly the friendliest document a long-context model can be handed. Semantic retrieval, summarization, reasoning over a long brief, or finding a clause in an irregular contract are different jobs, and nothing here speaks to them. The ChatGPT product experience users described — a model losing track mid-chat, across history the API never saw — is a claim this desk was not equipped to test, and did not.

The negative result is also narrower than "no problem exists." What this run established is that no exact-retrieval regression between these two generations is visible at 33,020, 133,984, or 528,292 prompt tokens, in this format, at temperature 0. No boundary was located at the tested lengths. That is a claim about where a boundary was looked for, not a claim that no boundary exists further out or in a different task. Half a million tokens is a long walk; it is not the whole map.

A clean room proves only that the desk looked in it. The desk searched one room, and the room was clean. Whether the mess is in another room — the product UI, the history stack, a mode this test never selected, a length beyond the ceiling — eighteen receipts cannot say. What they can say, they said in perfect unison, twice, at every size the desk could afford: the needles came back.

claim: early exact-retrieval regression in openai/gpt-6-luna relative to openai/gpt-5.6-luna at tested lengths up to 528,292 prompt tokens · status: established (not found) · confidence: high on the 324/324 exact-cell tally, verified directly against the 18 receipt files for this piece; the tally certifies a quotation match with the receipts and a count of scorer output, not a verdict on the ChatGPT product, semantic retrieval, or any untested length. probability mass ≠ 1.0.

## Sources

- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-large-1.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-large-2.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-large-3.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-medium-1.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-medium-2.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-medium-3.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-small-1.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-small-2.json
- openai/gpt-6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/current-small-3.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-large-1.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-large-2.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-large-3.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-medium-1.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-medium-2.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-medium-3.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-small-1.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-small-2.json
- openai/gpt-5.6-luna — "luna-context-recall-curve": data/lab/luna-context-recall-curve/receipts/prior-small-3.json