Turkish Document Haystack

How the benchmark works

A short Turkish statement — the needle — is drawn onto one page of a real Turkish annual report — the haystack — in a random font, colour, size and position. The model receives the document and is asked for the needle's value. This is a native Turkish re-creation of Document Haystack (Huybrechts et al., 2025), following its construction rules.

1The needle, in two variants

Text+image needle: Gizli meyve, then a photo of an orange

Text needle: Gizli meyve "portakal". — “Secret fruit: orange.” Text+image needle: the same label, with the value shown as a picture. There are 25 keys (fruit, currency, landmark, five animals, five objects, …), each with one value per report.

2The haystack

25 Turkish annual reports (2025, from banks, retailers, airlines, energy and industrial companies). A document is a report's first 5, 10, 25, 50, 75, 100, 150 or 200 pages. Documents of up to 25 pages hold one needle per page; longer ones hold 25 needles, one in each of 25 equal page ranges.

3The question, after the document

Belgedeki gizli meyve nedir?

“What is the secret fruit in the document?” Nothing else is sent — no instructions, no hint of the page — as in the original's inference script. One question per needle.

4How an answer is scored

✓portakaldırvalue or an accepted alias, with any Turkish suffixes
✕ayvadıra different fruit (“quince”)
✕orangeEnglish only: not credited (a lenient score that accepts it is in the data)
✕sütunwhole words only: “column” does not count for the value süt, “milk”

A document longer than the model's context counts as wrong. A stricter score, in the data, also requires the first needle value an answer names to be the right one.

Six tests differ only in what the model receives: page images or extracted text, a text or a picture needle, the whole document or the needle alone. They are explained one by one below, all with this same orange needle. 25 reports × 8 lengths × 5 sets = 20,625 questions per model.

The six tests, with one example each

Each panel is one column of the leaderboard below. Every example is real: question 13 of Akbank's report, the hidden fruit portakal (orange), with the models' answers verbatim (cut with … where long).

Text needles · page images

paper Table 5

The model receives all 25 page images of the report, in order, then the question. The needle is a line of text on page 13. That is 6,722 image tokens for Gemma, 31,571 for Qwen.

all 25 pages are sent · page 13 highlighted

QuestionBelgedeki gizli meyve nedir?

✓
gemma-4-31B-it
Belgedeki gizli meyve "portakal"dır.
All seven models found this needle.

Measures: Can the model find one short line of text among many page images, and read it?

All seven models on this test, % correct

Q3.8 27B92Q3.6 35B89Q3.5 9B89G4 26B63G4 31B77G4 12B39G3 12B47

Text needles · parsed text

paper Table 6

The model receives the text extracted from all 25 pages (no images), joined, then the question: 11,915 tokens. The needle is just a sentence in the text.

[… text of pages 1–12 …] … çekilebildiği ve öngörülebilirliğin arttığı bir makroekonomik istikrar ortamının tesis edilmesi, ülke risk priminin gerilemesi, dış finansman imkânlarının genişlemesi ve yatırım ortamının iyileşmesi finans sektörünün uzun vadeli büyüme potansiyelini destekleyecektir. Suzan Sabancı Yönetim Kurulu Başkanı Gizli meyve "portakal".needle · page 13 [… text of pages 14–25 …]

QuestionBelgedeki gizli meyve nedir?

✓
gemma-4-12B-it
Belgedeki gizli meyve **"portakal"**dır. (Sayfa 14'te "Gizli meyve 'portakal'" ifadesi yer almaktadır.)
✕
gemma-4-31B-it, same question in the 200-page document
no answer: the 200 pages of text are 129,071 tokens, more than its 60K context, so the question counts as wrong (the † in the tables)

Measures: Long-context retrieval in plain text, with no vision involved. Easiest, unless the text exceeds the model's context.

All seven models on this test, % correct

Q3.8 27B98†Q3.6 35B97†Q3.5 9B92†G4 26B90†G4 31B52†G4 12B93†G3 12B54†

Text+image needles · page images

paper Table 7

Again all 25 page images, but this time the needle's value is a picture: the label Gizli meyve next to a photo of an orange. The model must find the label and name what the picture shows.

Zoom: Gizli meyve, then a photo of an orange

QuestionBelgedeki gizli meyve nedir?

✓
gemma-4-26B-A4B-it
Belgedeki gizli meyve, 13. sayfada "Gizli meyve" etiketiyle gösterilen **portakal**dır.
✕
gemma-3-12b-it
Belgedeki gizli meyve elmadır.
✕
gemma-4-12B-it
Belgedeki gizli meyve **ayva**dır.

Measures: Finding the label among many pages and recognising the picture. Hardest of the three tasks.

All seven models on this test, % correct

Q3.8 27B92Q3.6 35B80Q3.5 9B77G4 26B31G4 31B37G4 12B20G3 12B13

Text+image needles · 200 pages

the longest documents

The same task in the report's 200-page version: 200 page images (53,622 tokens for Gemma, 252,421 for Qwen) holding 25 needles, one per 8-page range. The orange is now on page 101.

a needle (25 in all) the orange, page 101

QuestionBelgedeki gizli meyve nedir?

✓
Qwen3.8-27B
Belgenin 101. sayfasında, bir toplantı masasının üzerinde duran bir portakalın fotoğrafı yer almaktadır. Fotoğrafın yanında "Gizli meyve" şeklinde bir etiket bulunmaktadır. Bu nedenle, gizli…
✕
gemma-3-12b-it
Gizli meyve, belgede bir çizim olarak gösterilen **avokado**'dur.

Measures: How well picture-needle retrieval survives at the maximum length. The scores pool all 625 questions of the 25 200-page documents.

All seven models on this test, % correct

Q3.8 27B90Q3.6 35B69Q3.5 9B70G4 26B17G4 31B24G4 12B9G3 12B5

Crop · text

control

The model receives only the needle, cut out of the page with a small margin. There is no document and nothing to search. The question is the same.

QuestionBelgedeki gizli meyve nedir?

✓
gemma-4-12B-it
Belgedeki gizli meyve **"portakal"**dır.
All seven models read it.

Measures: A control: can the model read this needle at all (font, size, colour)? Near 100% for every model, so reading is not what fails in the long documents.

All seven models on this test, % correct

Q3.8 27B99Q3.6 35B100Q3.5 9B100G4 26B100G4 31B100G4 12B100G3 12B99

Crop · picture

control

The model receives only the picture needle, cut out of the page. Again there is nothing to search.

QuestionBelgedeki gizli meyve nedir?

✓
gemma-4-26B-A4B-it
Belgedeki gizli meyve **portakal**dır.
✕
gemma-4-12B-it
Görseldeki "Gizli meyve" (Hidden fruit) sorusunun cevabı, sağ taraftaki fotoğrafta açıkça görünen **mandalina**dır.

Measures: A control: does the model recognise the picture and know its Turkish name? Gemma models do so 93–99% of the time, yet find these needles only 13–37% of the time inside documents: searching, not recognising, is what fails. ‡ inflated, since Gemma-4-31B helped choose the pictures; § unreliable, since Qwen3.5-9B tends to explain the key instead.

All seven models on this test, % correct

Q3.8 27B93Q3.6 35B88Q3.5 9B56§G4 26B95G4 31B99‡G4 12B96G3 12B94

Leaderboard

Accuracy over all lengths, in %. Ranked by the mean of the three tasks — Gemma-4-31B's is pulled down by its 60K context on parsed text; on page images it is the best Gemma model. Click a column to sort.

#	Model	Tokens / page	Mean	1Text needles images	2Text needles parsed text	3Text+image images	4Text+image 200 pages	5Crop text	6Crop picture
1	Qwen3.8-27Bdense · 262K context	1,250	94.0	92.1%	97.7%^†	92.2%	90.4%	98.7%	92.9%
2	Qwen3.6-35B-A3B-FP8MoE, 3B active · 262K context	1,250	89.0	89.2%	97.4%^†	80.4%	69.4%	99.7%	88.0%
3	Qwen3.5-9Bdense · 262K context	1,250	85.8	88.9%	91.7%^†	76.8%	70.2%	99.8%	56.5%^§
4	gemma-4-26B-A4B-itMoE, 4B active · 262K context	268	61.1	63.0%	89.8%^†	30.6%	17.0%	99.9%	94.8%
5	gemma-4-31B-itdense · 60K context	268	55.3	77.1%	51.5%^†	37.2%	23.7%	100.0%	99.4%^‡
6	gemma-4-12B-itdense · 262K context	268	50.6	38.5%	92.8%^†	20.4%	9.0%	99.7%	96.1%
7	gemma-3-12b-itdense · 128K context	259	38.0	47.0%	53.9%^†	13.0%	4.6%	99.2%	93.8%

^† Some documents are longer than the model's context and count as wrong (parsed text of dense 200-page reports exceeds even 262K tokens; Gemma-4-31B fits 60K, Gemma-3-12B 128K). ^‡ Gemma-4-31B helped choose the needle pictures during dataset QA, so its picture-crop score is inflated. ^§ Qwen3.5-9B tends to explain the key instead of naming the picture when shown the crop alone. Tokens / page: measured image tokens per page; Qwen pages were capped at 1,310,720 pixels. Columns 1–6 are the six tests explained above, with an example each.

Heatmaps

Each cell pools the questions of one condition across the 25 reports. Hover any cell for its exact share.

Accuracy by document length for each task, as in the paper's Tables 5–7. Rows are ordered by the mean of the three tasks. All pools every length.

Text needles · page images paper Table 5

model	5	10	25	50	75	100	150	200	All
Qwen3.8-27B	98	98	95	94	92	91	88	89	92
Qwen3.6-35B-A3B-FP8	98	99	96	92	89	86	82	84	89
Qwen3.5-9B	98	99	94	92	88	85	84	84	89
gemma-4-26B-A4B-it	93	90	80	68	65	56	51	42	63
gemma-4-31B-it	94	94	87	81	78	73	69	64	77
gemma-4-12B-it	73	71	57	46	38	29	24	18	39
gemma-3-12b-it	84	74	62	52	48	38	34	29	47

document length (pages) →

Text needles · parsed text paper Table 6

model	5	10	25	50	75	100	150	200	All
Qwen3.8-27B	100	100	100	99	98	98	98	92^†	98^†
Qwen3.6-35B-A3B-FP8	100	100	99	98	97	98	98	93^†	97^†
Qwen3.5-9B	100	99	94	92	91	91	92	87^†	92^†
gemma-4-26B-A4B-it	100	100	96	94	91	90	85	76^†	90^†
gemma-4-31B-it	100	100	99	95^†	70^†	16^†	beyond context	beyond context	52^†
gemma-4-12B-it	100	100	97	95	94	92	89	85^†	93^†
gemma-3-12b-it	99	100	88	73	59	43^†	27^†	7^†	54^†

document length (pages) →

Text+image needles · page images paper Table 7

model	5	10	25	50	75	100	150	200	All
Qwen3.8-27B	93	96	96	93	92	92	89	90	92
Qwen3.6-35B-A3B-FP8	90	91	88	86	82	79	72	69	80
Qwen3.5-9B	78	82	85	80	76	76	72	70	77
gemma-4-26B-A4B-it	53	58	44	37	28	22	20	17	31
gemma-4-31B-it	60	60	49	44	38	30	25	24	37
gemma-4-12B-it	46	41	31	24	18	14	12	9	20
gemma-3-12b-it	30	30	21	18	10	8	7	5	13

document length (pages) →

Accuracy by where the needle sits (10 equal depth ranges) and document length, as in the paper's Figure 3: text needles on page images, lengths 25–200, about 62 needles per cell.

Qwen3.8-27B

3,434 / 3,750 correct · 91.6%

depth	25	50	75	100	150	200	Avg
0–10%	96	94	99	95	89	95	95
10–20%	94	98	98	97	92	81	93
20–30%	97	98	95	95	95	92	96
30–40%	94	98	98	98	93	98	97
40–50%	95	94	92	90	86	89	91
50–60%	98	97	90	94	93	92	94
60–70%	91	97	91	85	88	84	90
70–80%	94	92	84	83	79	92	87
80–90%	93	91	89	91	83	84	89
90–100%	94	85	85	84	83	80	85

document length (pages) →

Qwen3.6-35B-A3B-FP8

3,307 / 3,750 correct · 88.2%

depth	25	50	75	100	150	200	Avg
0–10%	99	98	100	98	91	98	97
10–20%	98	98	96	93	95	88	95
20–30%	97	98	97	91	91	90	94
30–40%	96	97	95	95	88	84	92
40–50%	100	92	97	85	81	74	89
50–60%	100	90	85	90	82	76	87
60–70%	95	94	91	84	87	91	90
70–80%	92	85	81	73	70	82	80
80–90%	95	86	74	66	68	72	77
90–100%	90	78	75	82	70	80	79

document length (pages) →

Qwen3.5-9B

3,299 / 3,750 correct · 88.0%

depth	25	50	75	100	150	200	Avg
0–10%	92	94	96	92	86	97	93
10–20%	92	97	93	95	90	87	92
20–30%	96	98	94	91	95	84	93
30–40%	94	95	90	92	81	84	89
40–50%	96	98	92	92	81	86	91
50–60%	98	93	92	84	87	81	89
60–70%	99	97	91	82	84	89	91
70–80%	96	86	86	81	75	82	84
80–90%	89	83	79	72	81	78	81
90–100%	90	78	71	69	74	75	76

document length (pages) →

gemma-4-26B-A4B-it

2,258 / 3,750 correct · 60.2%

depth	25	50	75	100	150	200	Avg
0–10%	87	88	81	66	66	78	78
10–20%	84	70	74	83	61	52	70
20–30%	80	80	77	69	56	40	67
30–40%	76	78	68	51	54	48	62
40–50%	84	74	57	50	48	44	60
50–60%	86	47	58	60	48	32	54
60–70%	69	61	56	50	46	30	52
70–80%	76	54	53	43	46	31	50
80–90%	77	48	61	44	44	30	51
90–100%	78	77	61	49	39	36	56

document length (pages) →

gemma-4-31B-it

2,828 / 3,750 correct · 75.4%

depth	25	50	75	100	150	200	Avg
0–10%	91	91	88	82	77	76	84
10–20%	92	84	84	87	75	63	80
20–30%	83	85	88	84	77	75	82
30–40%	84	92	80	70	68	60	75
40–50%	91	86	78	79	62	68	78
50–60%	92	63	70	71	69	58	70
60–70%	80	80	72	66	68	55	71
70–80%	84	68	82	59	64	62	69
80–90%	88	74	71	67	69	56	72
90–100%	88	85	66	69	62	67	72

document length (pages) →

gemma-4-12B-it

1,320 / 3,750 correct · 35.2%

depth	25	50	75	100	150	200	Avg
0–10%	64	72	60	48	34	34	53
10–20%	64	49	42	52	31	22	42
20–30%	65	57	48	39	32	25	45
30–40%	58	50	36	31	32	26	38
40–50%	55	40	35	27	19	17	33
50–60%	54	32	30	19	16	14	26
60–70%	53	45	32	26	17	16	32
70–80%	58	32	37	11	20	7	26
80–90%	47	34	21	17	22	14	26
90–100%	52	45	36	20	12	7	27

document length (pages) →

gemma-3-12b-it

1,646 / 3,750 correct · 43.9%

depth	25	50	75	100	150	200	Avg
0–10%	80	78	69	60	50	48	65
10–20%	78	66	60	67	39	31	56
20–30%	71	66	58	42	36	30	51
30–40%	64	65	44	43	34	37	47
40–50%	65	46	46	31	29	23	41
50–60%	62	42	38	30	27	27	37
60–70%	52	42	37	24	35	20	36
70–80%	56	32	37	22	25	23	32
80–90%	49	29	41	19	34	20	33
90–100%	44	57	46	43	29	33	41

document length (pages) →

As above for needles that are a picture (paper Figure 4).

Qwen3.8-27B

3,450 / 3,750 correct · 92.0%

depth	25	50	75	100	150	200	Avg
0–10%	95	91	90	91	88	91	91
10–20%	98	98	95	93	90	88	94
20–30%	99	95	98	98	89	90	95
30–40%	96	93	95	92	90	95	93
40–50%	95	92	92	94	90	92	93
50–60%	98	95	95	92	91	86	93
60–70%	93	91	90	92	90	89	91
70–80%	96	92	91	87	84	93	90
80–90%	99	91	89	91	88	88	91
90–100%	86	92	90	89	88	90	89

document length (pages) →

Qwen3.6-35B-A3B-FP8

2,976 / 3,750 correct · 79.4%

depth	25	50	75	100	150	200	Avg
0–10%	88	89	87	86	83	78	85
10–20%	90	92	93	85	84	75	86
20–30%	93	91	89	91	85	83	89
30–40%	92	88	92	77	73	68	81
40–50%	92	80	75	79	67	77	79
50–60%	84	87	85	84	78	59	79
60–70%	85	88	79	82	74	64	79
70–80%	90	85	81	75	57	72	76
80–90%	84	78	77	64	68	53	71
90–100%	74	80	66	62	55	66	67

document length (pages) →

Qwen3.5-9B

2,865 / 3,750 correct · 76.4%

depth	25	50	75	100	150	200	Avg
0–10%	73	78	69	72	72	72	73
10–20%	80	74	79	75	66	72	74
20–30%	91	88	83	78	79	81	83
30–40%	84	87	75	80	85	77	81
40–50%	87	88	72	77	60	62	75
50–60%	88	82	78	78	79	68	79
60–70%	89	77	82	74	70	69	77
70–80%	90	86	79	84	70	72	80
80–90%	85	74	77	64	71	70	74
90–100%	82	67	64	72	65	59	68

document length (pages) →

gemma-4-26B-A4B-it

1,050 / 3,750 correct · 28.0%

depth	25	50	75	100	150	200	Avg
0–10%	57	64	35	45	47	33	47
10–20%	50	49	44	40	25	25	38
20–30%	45	40	45	20	27	13	32
30–40%	44	47	20	20	29	15	28
40–50%	49	37	31	19	5	17	27
50–60%	46	20	23	17	18	17	23
60–70%	43	26	25	16	12	14	23
70–80%	48	22	26	11	20	11	22
80–90%	31	29	17	16	8	9	19
90–100%	24	32	17	15	9	16	18

document length (pages) →

gemma-4-31B-it

1,309 / 3,750 correct · 34.9%

depth	25	50	75	100	150	200	Avg
0–10%	64	73	43	46	47	45	53
10–20%	60	52	56	47	30	28	45
20–30%	44	45	56	25	27	22	37
30–40%	46	50	25	34	36	19	35
40–50%	53	45	42	24	9	27	34
50–60%	52	33	30	27	21	19	30
60–70%	45	39	34	23	16	17	29
70–80%	56	36	37	24	27	21	33
80–90%	35	32	26	22	20	20	26
90–100%	36	28	29	33	20	18	27

document length (pages) →

gemma-4-12B-it

681 / 3,750 correct · 18.2%

depth	25	50	75	100	150	200	Avg
0–10%	37	30	21	23	17	17	25
10–20%	40	41	33	25	16	13	28
20–30%	31	26	26	12	15	10	20
30–40%	28	27	10	11	22	5	17
40–50%	35	28	17	21	7	8	20
50–60%	26	8	12	17	15	7	14
60–70%	31	21	18	8	7	6	16
70–80%	42	20	14	3	9	8	15
80–90%	24	23	15	6	7	11	15
90–100%	12	20	15	15	9	5	13

document length (pages) →

gemma-3-12b-it

424 / 3,750 correct · 11.3%

depth	25	50	75	100	150	200	Avg
0–10%	29	34	18	23	17	12	23
10–20%	32	36	18	20	8	4	19
20–30%	21	17	15	5	12	3	13
30–40%	26	25	8	5	7	10	13
40–50%	21	26	8	6	5	2	12
50–60%	10	5	5	5	1	3	5
60–70%	19	11	6	3	3	6	8
70–80%	24	12	7	2	4	5	8
80–90%	11	6	8	6	5	2	6
90–100%	14	7	8	2	6	0	6

document length (pages) →

As above for text needles in the parsed page text (not shown in the paper).

Qwen3.8-27B

3,656 / 3,750 correct · 97.5%†

depth	25	50	75	100	150	200	Avg
0–10%	100	100	100	98	100	95^†	99^†
10–20%	100	100	98	100	98	94^†	98^†
20–30%	100	100	98	100	98	95^†	99^†
30–40%	98	100	100	100	100	94^†	99^†
40–50%	99	98	94	100	97	89^†	96^†
50–60%	100	98	100	98	97	95^†	98^†
60–70%	100	100	99	100	94	91^†	97^†
70–80%	100	100	96	92	98	89^†	96^†
80–90%	100	97	94	95	97	88^†	95^†
90–100%	100	100	97	100	97	95^†	98^†

document length (pages) →

Qwen3.6-35B-A3B-FP8

3,644 / 3,750 correct · 97.2%†

depth	25	50	75	100	150	200	Avg
0–10%	99	97	99	98	98	93^†	97^†
10–20%	100	98	100	97	97	96^†	98^†
20–30%	100	98	97	98	97	90^†	97^†
30–40%	98	98	97	100	100	92^†	97^†
40–50%	99	100	98	97	97	94^†	97^†
50–60%	96	98	100	98	99	92^†	97^†
60–70%	99	98	96	98	99	91^†	97^†
70–80%	94	98	95	100	98	95^†	97^†
80–90%	100	95	95	95	95	89^†	95^†
90–100%	100	100	98	98	100	97^†	99^†

document length (pages) →

Qwen3.5-9B

3,412 / 3,750 correct · 91.0%†

depth	25	50	75	100	150	200	Avg
0–10%	93	97	93	91	89	84^†	91^†
10–20%	94	92	89	92	82	81^†	88^†
20–30%	97	94	95	92	91	86^†	93^†
30–40%	94	83	92	90	95	89^†	90^†
40–50%	99	94	92	87	95	85^†	92^†
50–60%	92	95	95	97	97	92^†	95^†
60–70%	92	91	90	92	96	89^†	92^†
70–80%	90	90	84	89	88	87^†	88^†
80–90%	92	89	88	89	86	88^†	89^†
90–100%	94	90	92	92	95	90^†	92^†

document length (pages) →

gemma-4-26B-A4B-it

3,331 / 3,750 correct · 88.8%†

depth	25	50	75	100	150	200	Avg
0–10%	97	100	99	94	89	88^†	95^†
10–20%	96	97	96	97	93	93^†	95^†
20–30%	97	94	94	91	88	84^†	91^†
30–40%	98	93	97	97	93	73^†	91^†
40–50%	97	97	89	95	90	76^†	91^†
50–60%	96	92	87	89	90	76^†	88^†
60–70%	96	89	85	85	75	66^†	83^†
70–80%	86	92	82	84	75	69^†	81^†
80–90%	97	94	86	77	73	69^†	83^†
90–100%	98	97	98	89	85	70^†	89^†

document length (pages) →

gemma-4-31B-it

1,750 / 3,750 correct · 46.7%†

depth	25	50	75	100	150	200	Avg
0–10%	99	95^†	72^†	15^†	beyond context	beyond context	49^†
10–20%	100	97^†	72^†	17^†	beyond context	beyond context	45^†
20–30%	100	92^†	76^†	19^†	beyond context	beyond context	49^†
30–40%	100	97^†	66^†	13^†	beyond context	beyond context	44^†
40–50%	100	95^†	71^†	16^†	beyond context	beyond context	49^†
50–60%	98	97^†	70^†	16^†	beyond context	beyond context	44^†
60–70%	99	94^†	71^†	16^†	beyond context	beyond context	48^†
70–80%	100	95^†	68^†	16^†	beyond context	beyond context	45^†
80–90%	99	94^†	67^†	16^†	beyond context	beyond context	48^†
90–100%	100	90^†	68^†	16^†	beyond context	beyond context	43^†

document length (pages) →

gemma-4-12B-it

3,455 / 3,750 correct · 92.1%†

depth	25	50	75	100	150	200	Avg
0–10%	95	94	94	94	94	91^†	94^†
10–20%	100	98	95	95	97	87^†	95^†
20–30%	100	97	100	92	91	84^†	94^†
30–40%	98	97	95	93	83	81^†	91^†
40–50%	100	98	98	98	97	92^†	97^†
50–60%	96	95	92	90	87	85^†	91^†
60–70%	97	91	96	94	84	80^†	90^†
70–80%	94	92	89	84	86	80^†	87^†
80–90%	93	94	88	91	90	83^†	90^†
90–100%	98	93	95	92	88	85^†	92^†

document length (pages) →

gemma-3-12b-it

1,851 / 3,750 correct · 49.4%†

depth	25	50	75	100	150	200	Avg
0–10%	96	84	75	55^†	39^†	12^†	62^†
10–20%	92	85	58	52^†	30^†	3^†	51^†
20–30%	95	72	52	45^†	18^†	5^†	49^†
30–40%	82	58	49	28^†	15^†	8^†	39^†
40–50%	84	71	55	27^†	21^†	0^†	45^†
50–60%	88	72	63	38^†	21^†	10^†	47^†
60–70%	87	70	63	42^†	25^†	6^†	50^†
70–80%	78	71	49	33^†	30^†	8^†	44^†
80–90%	81	66	47	44^†	29^†	11^†	48^†
90–100%	94	80	75	62^†	39^†	10^†	59^†

document length (pages) →

Accuracy by how busy the needle's page is (words of report text on it), lengths 25–200. Reports open with sparse design pages and end with dense financial tables.

Text needles · page images

model	sparse	medium	dense
Qwen3.8-27B	95	92	86
Qwen3.6-35B-A3B-FP8	94	87	83
Qwen3.5-9B	94	86	82
gemma-4-26B-A4B-it	89	57	26
gemma-4-31B-it	95	76	48
gemma-4-12B-it	64	26	13
gemma-3-12b-it	75	35	19

words on the needle's page →

Text needles · parsed text

model	sparse	medium	dense
Qwen3.8-27B	99^†	97^†	96^†
Qwen3.6-35B-A3B-FP8	97^†	97^†	97^†
Qwen3.5-9B	92^†	91^†	90^†
gemma-4-26B-A4B-it	92^†	88^†	86^†
gemma-4-31B-it	62^†	42^†	35^†
gemma-4-12B-it	95^†	91^†	90^†
gemma-3-12b-it	61^†	46^†	39^†

words on the needle's page →

Text+image needles · page images

model	sparse	medium	dense
Qwen3.8-27B	94	93	88
Qwen3.6-35B-A3B-FP8	87	77	73
Qwen3.5-9B	81	76	70
gemma-4-26B-A4B-it	65	14	3
gemma-4-31B-it	75	22	5
gemma-4-12B-it	39	10	5
gemma-3-12b-it	30	3	2

words on the needle's page →

Accuracy for each of the 25 needle keys, all lengths. yiyecek (food) overlaps fruit and vegetable exactly as in the original; ünlü yapı contains yapı, a very common word in reports.

Text needles · page images

key	Q3.8 27B	Q3.6 35B	Q3.5 9B	G4 26B	G4 31B	G4 12B	G3 12B
el aleti tool	95	91	97	57	71	39	38
geometrik şekil shape	98	94	99	82	89	52	48
giysi clothing	96	97	92	59	81	43	35
içecek drink	97	91	90	70	82	36	42
kırtasiye malzemesi office supply	95	97	95	76	81	45	44
meyve fruit	99	86	95	66	81	40	63
mutfak cihazı kitchen appliance	97	98	100	83	86	54	68
para birimi currency	97	98	99	72	88	58	62
sebze vegetable	99	87	91	69	81	42	62
spor sport	99	93	94	68	75	45	55
taşıt transportation	94	93	94	68	73	36	40
yiyecek food	94	91	87	63	81	46	53
çalgı instrument	98	93	97	72	79	37	35
çiçek flower	90	91	94	77	72	33	46
ünlü yapı landmark	98	99	98	49	69	38	38
hayvan #1	91	89	81	65	83	30	37
hayvan #2	94	89	85	62	85	42	36
hayvan #3	93	84	87	55	75	37	44
hayvan #4	92	78	78	58	76	33	43
hayvan #5	93	77	83	62	74	26	50
nesne #1	81	84	73	43	71	29	34
nesne #2	81	83	73	38	70	29	53
nesne #3	81	79	78	49	73	27	51
nesne #4	70	81	76	49	66	29	49
nesne #5	71	75	72	47	62	25	43

model →

Text needles · parsed text

key	Q3.8 27B	Q3.6 35B	Q3.5 9B	G4 26B	G4 31B	G4 12B	G3 12B
el aleti tool	99^†	99^†	98^†	95^†	55^†	98^†	53^†
geometrik şekil shape	98^†	99^†	95^†	92^†	55^†	98^†	61^†
giysi clothing	99^†	99^†	98^†	85^†	54^†	98^†	44^†
içecek drink	99^†	99^†	99^†	98^†	56^†	99^†	64^†
kırtasiye malzemesi office supply	99^†	99^†	97^†	98^†	55^†	98^†	66^†
meyve fruit	99^†	99^†	96^†	93^†	55^†	93^†	71^†
mutfak cihazı kitchen appliance	99^†	99^†	99^†	98^†	55^†	99^†	75^†
para birimi currency	99^†	99^†	94^†	99^†	56^†	99^†	72^†
sebze vegetable	99^†	99^†	99^†	97^†	55^†	99^†	61^†
spor sport	99^†	99^†	98^†	94^†	54^†	98^†	68^†
taşıt transportation	99^†	99^†	96^†	98^†	54^†	98^†	52^†
yiyecek food	99^†	95^†	87^†	78^†	54^†	93^†	58^†
çalgı instrument	99^†	99^†	99^†	97^†	56^†	98^†	65^†
çiçek flower	99^†	99^†	99^†	99^†	55^†	99^†	71^†
ünlü yapı landmark	99^†	99^†	98^†	83^†	55^†	80^†	75^†
hayvan #1	99^†	97^†	89^†	92^†	46^†	97^†	50^†
hayvan #2	99^†	99^†	93^†	87^†	47^†	92^†	50^†
hayvan #3	99^†	99^†	91^†	88^†	47^†	90^†	43^†
hayvan #4	98^†	96^†	87^†	89^†	47^†	93^†	43^†
hayvan #5	99^†	99^†	89^†	89^†	47^†	94^†	37^†
nesne #1	89^†	93^†	80^†	83^†	43^†	78^†	21^†
nesne #2	93^†	90^†	81^†	68^†	43^†	71^†	27^†
nesne #3	92^†	88^†	73^†	79^†	45^†	88^†	34^†
nesne #4	93^†	97^†	71^†	79^†	44^†	81^†	34^†
nesne #5	89^†	89^†	73^†	77^†	46^†	79^†	27^†

model →

Text+image needles · page images

key	Q3.8 27B	Q3.6 35B	Q3.5 9B	G4 26B	G4 31B	G4 12B	G3 12B
el aleti tool	98	91	90	33	38	24	10
geometrik şekil shape	92	91	90	40	45	59	18
giysi clothing	99	94	94	32	38	22	11
içecek drink	93	84	61	33	38	19	13
kırtasiye malzemesi office supply	86	83	77	36	46	27	13
meyve fruit	99	84	88	44	47	43	19
mutfak cihazı kitchen appliance	96	89	81	33	31	18	11
para birimi currency	86	74	67	43	49	22	32
sebze vegetable	93	79	67	39	46	24	13
spor sport	97	62	80	23	31	16	8
taşıt transportation	97	90	92	36	44	28	14
yiyecek food	91	91	91	29	39	15	11
çalgı instrument	99	96	89	30	35	16	13
çiçek flower	94	69	32	39	43	18	5
ünlü yapı landmark	97	95	80	27	37	12	18
hayvan #1	98	69	80	32	44	20	17
hayvan #2	85	77	86	27	36	7	7
hayvan #3	97	73	63	22	23	5	13
hayvan #4	95	75	71	31	32	21	10
hayvan #5	93	67	68	29	35	13	12
nesne #1	87	68	79	21	35	17	17
nesne #2	89	81	83	19	28	21	14
nesne #3	86	75	77	26	35	13	9
nesne #4	78	66	65	18	27	14	14
nesne #5	75	75	65	13	19	7	3

model →

Findings

92.2%

Qwen3.8-27B on text+image needles, all lengths — the best model on every task

90% vs 5%

text+image needles in 200-page documents: Qwen3.8-27B vs Gemma-3-12B

65% → 3%

Gemma-4-26B-A4B on picture needles, sparse → dense pages

144,375

questions answered: 7 models × 25 reports × 8 lengths × 5 tasks

Qwen3.8-27B leads every task and barely degrades with length. It finds 92.1% of text needles on page images, 97.7% in parsed text and 92.2% of text+image needles, still 90.4% at 200 pages. All three Qwen models beat all four Gemma models on page images.
The paper's ordering holds in Turkish. Parsed text is easiest wherever the document fits the context, text needles on page images are harder, and needles that are pictures are hardest — for every model except Qwen3.8-27B, which finds pictures as well as text.
Finding the needle, not reading it, is the bottleneck. Shown only the needle crop, the Gemma models name the picture 93%+ of the time; inside 200-page documents they find it only 5–24% of the time.
Page resolution appears to matter. Gemma sees a page as about 268 image tokens, Qwen as about 1,250 (capped so 200 pages fit in 262K). Gemma-4-26B-A4B finds 65% of picture needles on sparse pages but 3% on dense ones: a small picture among dense text is nearly invisible at low resolution. This is a confound, not a controlled result: an equal-resolution control has not been run.
Depth matters on page images, not in parsed text. Qwen3.6-35B-A3B finds 97% of text needles in the first tenth of a document and 79% in the last tenth; with parsed text, 98% and 99%. The drop holds within pages of the same density, so it is not just the dense financial tables at the end of a report.
Context limits dominate some parsed-text scores. Gemma-4-31B fits only 60K tokens next to its bf16 weights (98.8% on the documents that fit, 51.5% overall); Gemma-3-12B has 128K, and also answers in English (215 parsed-text answers are right only in English and are not credited).

Methodology

Haystacks. 25 Turkish annual reports for fiscal year 2025 (published 2026 on KAP), five per needle set, from banking, insurance, retail, energy, industry, aviation, telecom and media. Reports printed as two-page spreads are split into single pages, each keeping only its own text. A document of N pages is the report's first N pages.

Needles. 25 keys × 5 value sets (fruit, currency, landmark, five animals, five objects, …), written as Gizli <key> "<value>". — the original's “The secret <key> is <value>” recast as a Turkish nominal sentence so no suffix depends on the value. Pictures were generated with Z-Image-Turbo and checked by a VLM. Placement follows the original exactly: one needle per page up to 25 pages, otherwise one in each of 25 equal page ranges; 5- and 10-page documents use only the 15 non-animal/object keys; 10 fonts (metric-compatible Liberation fonts, since the PDF base fonts cannot encode ğ, ş, ı), 10 colours on a white or black box, 7–14 pt text, pictures about 100 pt wide, uniform random positions. Pages are rendered at 200 DPI (JPEG quality 75); text is extracted with pypdf.

Inference. One user turn: all page images in order, then the bare question (for parsed text, the page texts concatenated, a newline and the question). Greedy decoding, up to 256 new tokens, no system prompt. vLLM on one A100-80GB with prefix caching, so a document is processed once and its 25 questions reuse it. Qwen models answer with thinking disabled (enable_thinking: false) and their pages are capped at 1,310,720 pixels (about 1,250 tokens) so that 200 pages fit in 262K tokens; every other model uses its own image processing, as in the original's Table 4.

Scoring. An answer is correct when it contains the value, or an accepted alias, as a whole Turkish word with any chain of suffixes that obeys vowel harmony (including softened stems such as kitab-ı). English answers are not credited; the data includes a lenient score that also accepts the English name. Documents longer than a model's context count as wrong.

Example question and needle (as sent)

Gizli meyve "portakal".
Belgedeki gizli meyve nedir?

Known limitations

Resolution is confounded with model family. Qwen pages cost about 1,250 tokens, Gemma pages about 270; part of the Qwen–Gemma gap on page images may be resolution rather than the model.
Context limits. Gemma-4-31B ran with 60K tokens (bf16 weights leave no room for more on one A100) and Gemma-3-12B with 128K, so their long parsed-text cells mostly measure the limit, not retrieval.
Crop baselines. Gemma-4-31B helped choose the needle pictures, inflating its picture-crop score; Qwen3.5-9B often explains the key instead of naming the picture when shown the crop alone.
Two keys overlap the reports' language. yiyecek (food) also covers the fruit and vegetable needles, as “food” does in the original; ünlü yapı (landmark) contains yapı, which appears about 280 times in a 200-page report — it costs the Gemma models about one point on long parsed text.
One run per question. Repeating a run moved single cells by 2–3 points; treat small differences as noise.
Short documents are opening pages. 5- and 10-page documents are each report's cover and opening pages, so their scores reflect those pages as well as the length.

Data

One row per question with the model's full answer and every score, the summaries behind the heatmaps, and the paper-style figures and LaTeX tables.

Answers, 7 models7 CSV files · 33.2 MB Summary tablesleaderboard, by length, depth, key · 101 KB Paper-style figuresPNG, PDF and LaTeX tables · 2.7 MB