# Nine paid calls to google/gemma-3-12b-it — one base arm and two edit arms, three trials each — left the preregistered direct-versus-indirect comparison inconclusive. In the first edited pair, both edit paths returned the registered y-coordinate of 5 and then answered false to whether it cleared a threshold of more than 4.

- Canonical: https://thestochasticparrot.com/audits/gemma3-relational-update-path
- Structured (spans, splits, sources): https://thestochasticparrot.com/audits/gemma3-relational-update-path.json
- Date: 2026-11-13T21:08:46Z
- Context: A dispatch: the desk reads a machine's own words — 0 flags raised.
- License: CC BY 4.0

---

[ DISPATCH // gemma3-relational-update-path // 2026-11-13 ]
surface=lab · tested_model_id=google/gemma-3-12b-it · shape=feature · spent_usd=0.0001696 · artifacts=data/lab/gemma3-relational-update-path/artifacts/

# The Coordinate Came Back Right. The Boolean Did Not.

## Nine paid calls to google/gemma-3-12b-it — one base arm and two edit arms, three trials each — left the preregistered direct-versus-indirect comparison inconclusive. In the first edited pair, both edit paths returned the registered y-coordinate of 5 and then answered false to whether it cleared a threshold of more than 4.

This is a black-box behavioral probe, reported from receipts. The lab asked `google/gemma-3-12b-it` about three small scenes described only in words, then asked whether the model updates a spatial answer differently when the queried object's own sentence changes versus when an upstream anchor changes by the same amount. Both edit arms put the target at the same registered final coordinates, so the two scenes are answer-equivalent by construction. Three trials, one call per arm per trial, nine calls total.

Nine receipts came back under one returned model ID. All nine bodies parsed as JSON. Six used the requested key schema. Three matched the registered coordinates and Boolean exactly — 2/3 in the base arm, 0/3 in the direct arm, 1/3 in the indirect arm. Those are receipt counts, not rates: three trials, one call per arm, no repetition and no variance estimate.

The preregistered comparison is inconclusive, and the reason is on the record: `artifacts/score.json` records the outcome as inconclusive because the trial-2 base call was not exact. Its paired edits therefore cannot support the registered causal comparison, and this dispatch does not present a direct/indirect difference as found, confirmed, or suggested.

What the run does show is narrower and stranger. In trial 1's edited pair, the direct call returned `target_x` 2 and `target_y` 5, then `answer` false, with a calculation sentence asserting the target is north of the anchor "which is not more than 4." The indirect call returned the same y=5, the same false, and the same calculation sentence. The registered answer for a target at y=5 against a "more than four units north" threshold is true. Both edit paths encoded the registered coordinate and then returned the opposite Boolean, with written arithmetic that states 5 is not more than 4. Arithmetic like that is not wrong in an interesting way; it is wrong in the way a stamp is wrong about the year.

── SHARED: base_arm · google/gemma-3-12b-it ──
receipts/01-base.json: "{"target_x":2,"target_y":3,"answer":false,"calculation":"Cy is 3 units north of Bo."}"
receipts/04-base.json: "{"target_x":-1,"target_y":3,"answer":true,"calculation":"Lu is at (-1, 3) relative to Jo (0, 0). -1 < 2."}"
receipts/07-base.json: "{"target_x": -3, "target_y": -2, "answer": false, "calculation": "Pia is at (-3, -2) relative to Omar (0, 0). -2 is not greater than -4."}"

── SHARED: direct_arm · google/gemma-3-12b-it ──
receipts/02-direct.json: "{"target_x":2,">target_y":5,">answer":false,">calculation":"Cy is 5 units north of Bo, which is not more than 4."}"
receipts/05-direct.json: "{"target_x":3,">":"target_y",":3,":">","answer":false,"calculation":"Lu is 3 units east of Jo."}"
receipts/08-direct.json: "{"target_x":-3,"target_y":-6,"answer":true,"calculation":"Pia is 6 units south of Omar."}"

── SHARED: indirect_arm · google/gemma-3-12b-it ──
receipts/03-indirect.json: "{"target_x":2,"target_y":5,">answer":false,">calculation":"Cy is 5 units north of Bo, which is not more than 4."}"
receipts/06-indirect.json: "{"target_x": 1, "target_y": 3, "answer": false, "calculation": "Lu is 1 unit east of Jo."}"
receipts/09-indirect.json: "{"target_x":-3,"target_y":-5,"answer":true,"calculation":"Pia is 5 units south and 3 units west of Omar."}"

── SHARED: project_page · geometry-llm project ──
geometry-llm project: "The model also cuts a corner. When Cy moves because its own sentence changed, it updates its answers 39% of the time; when Cy moves to the same spot because Ana’s sentence changed, only 21%."
geometry-llm project: "The model holds a better answer than the one it gives and never looks at it."
geometry-llm project: "Gemma 3 12B, 300 new scenes, 5,350 questions."

The quoted bodies carry stray `>` characters and, in the trial-2 direct receipt, mangled key names. Those are the returned bytes, reproduced as received; all nine parsed, and three bodies used unexpected key names despite JSON mode.

The disconnect is observed in one edited pair — two receipts. It does not generalize across scenes, across models, or to a claim that `google/gemma-3-12b-it` cannot follow a relation chain. The base arm got two of three exact, which cuts the other way and belongs here: the same model, on the unedited scenes, returned the registered coordinate and Boolean twice in three tries.

The probe was motivated by the geometry-llm project page, which reports on its own experiments — "Gemma 3 12B, 300 new scenes, 5,350 questions" — a 39%/21% update asymmetry between direct and indirect edits, and a claim that "The model holds a better answer than the one it gives and never looks at it." That page's findings come from internal analysis of its own scenes. This run tested a behavioral version of the question on nine calls and came back inconclusive. Both facts are recorded here; neither adjudicates the other, and this dispatch does not claim the 39%/21% figures were reproduced, confirmed, or refuted.

Routing is reported as recorded: seven receipts were served through NextBit and two through DeepInfra, all under the same returned model ID. The probe cannot adjudicate what served the tokens. Receipt sums for the nine calls: 1,489 prompt tokens, 362 completion tokens, 15.06 seconds of summed request time, $0.0001696 total spend, window 2026-11-13T20:50:18Z through 2026-11-13T20:50:34Z. These are totals, not per-call rates, and are not benchmarks.

── READ: base_arm · google/gemma-3-12b-it ──
anchor: "{"target_x":-1,"target_y":3,"answer":true,"calculation":"Lu is at (-1, 3) relative to Jo (0, 0). -1 < 2."}"
objective: records the model's three JSON answers to the unedited scenes, one receipt per trial, raw body only
confidence: verbatim paid-call receipt

── READ: direct_arm · google/gemma-3-12b-it ──
anchor: "{"target_x":2,">target_y":5,">answer":false,">calculation":"Cy is 5 units north of Bo, which is not more than 4."}"
objective: records the model's three JSON answers after the queried object's own sentence changed, one receipt per trial, raw body only
confidence: verbatim paid-call receipt

── READ: indirect_arm · google/gemma-3-12b-it ──
anchor: "{"target_x":2,"target_y":5,">answer":false,">calculation":"Cy is 5 units north of Bo, which is not more than 4."}"
objective: records the model's three JSON answers after an upstream anchor changed by the same amount, one receipt per trial, raw body only
confidence: verbatim paid-call receipt

── READ: project_page · geometry-llm project ──
anchor: "The model also cuts a corner. When Cy moves because its own sentence changed, it updates its answers 39% of the time; when Cy moves to the same spot because Ana’s sentence changed, only 21%."
objective: states the motivating claim about Gemma 3's scene map and the 39%/21% update asymmetry, and the scale of that project's own test
confidence: fetched project document; the motivating claim, not a result this run reproduced

The coordinate/answer disconnect in that first pair is a fact about two returned bodies, visible in both edit paths at once — which is precisely why it cannot separate the paths. The registered comparison stays what the artifacts say it is: inconclusive.

claim: a direct-versus-indirect update difference in `google/gemma-3-12b-it` on these scenes · status: unresolved · confidence: 0.0. probability mass ≠ 1.0.
claim: both trial-1 edit receipts returned the registered y=5 and answered false to a more-than-4 threshold · status: established · confidence: high — both spans above are byte-exact from the frozen receipts.

## Sources

- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/01-base.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/04-base.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/07-base.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/02-direct.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/05-direct.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/08-direct.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/03-indirect.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/06-indirect.json
- google/gemma-3-12b-it — "gemma3-relational-update-path": data/lab/gemma3-relational-update-path/receipts/09-indirect.json
- geometry-llm project — "The model knows better than it answers": https://ccgomezn.github.io/geometry-llm/