The record for this issue
Rewarded for guessing.
104 claims in the draft, each tested by a second agent against the page it cites. 99 held; 5 did not and were fixed or cut before this went out.
-
01
titleconfirmed
Rewarded for guessing.
The page says “Saying "I don't know" guarantees zero points ... guessing when uncertain improves test performance”
-
02
dekconfirmed
A federal appeals court wants a person to check every AI-drafted citation
The page says “certify that a human reviewed filings prepared with generative AI ... independently verify the authorities cited”
-
03
dekconfirmed
Days later, two AI labs said their new models make fewer things up
The page says “GPT-6 Sol makes about half as many mistakes as its predecessor ... Anthropic released a new version of Opus 5.5 just 90 minutes before”
-
04
dekconfirmed
chatbots are scored like students who never lose a point for guessing
The page says “If you do not know the answer but take a wild guess, you might get lucky and be right ... leaving it blank guarantees a zero”
-
05
summaryconfirmed
Training and scoring rewards guessing; false things stated with confidence
The page says “We understand the statistical mechanisms through which hallucinations arise and are rewarded in evaluations”
-
06
art svg textconfirmed
1 IN 365
The page says “If it guesses "September 10," it has a 1-in-365 chance of being right”
-
07
art altconfirmed
Answer sheet labelled 1 in 365
The page says “1-in-365 chance of being right”
-
08
The essay, section 1confirmed
On Sept. 18 the Tenth Circuit proposed the rule
The page says “proposed a new rule September 18”
-
09
The essay, section 1confirmed
Tenth Circuit hears federal appeals from Colorado, Kansas, New Mexico, Oklahoma, Utah and Wyoming
The page says “hears federal appeals from Colorado, Kansas, New Mexico, Oklahoma, Utah and Wyoming”
-
10
The essay, section 1confirmed
Lawyers and self-represented people would have to certify human review before filing
The page says “Lawyers and self-represented parties would need to confirm human review”
-
11
The essay, section 1confirmed
Quote: "independently verified the existence of all cited authorities" (rule text, 7 words)
The page says “a reviewer who has "independently verified the existence of all cited authorities."”
-
12
The essay, section 1confirmed
and checked every quotation and citation
The page says “check quotations and citations”
-
13
The essay, section 1confirmed
Judges could strike a filing, treat an issue as waived or, in some cases, dismiss the appeal
The page says “strike a filing, treat an affected issue as waived ... in some circumstances, dismiss or affirm an appeal”
-
14
The essay, section 1confirmed
Comments are due Oct. 18
The page says “Public comments are due by October 18, 2026”
-
15
The essay, section 1confirmed
If approved, the rule takes effect Jan. 1, 2027
The page says “If approved, the rule would start on January 1, 2027”
-
16
The essay, section 1confirmed
In February the court sanctioned a lawyer whose brief cited seven cases that don't exist
The page says “sanctioned Maryland attorney Kusmin L. Amarsingh after finding seven nonexistent cases”
-
17
The essay, section 1confirmed
she said she had used ChatGPT to research and draft it
The page says “acknowledged using ChatGPT while researching and drafting the appeal”
-
18
The essay, section 1confirmed
In June, filings from a self-represented litigant who used AI contained quotations and propositions missing from cited cases
The page says “quotations and legal propositions missing from the cited cases”
-
19
The essay, section 1confirmed
Sept. 22, four days after the proposal, OpenAI released a new model
The page says “September 22, 2026 ... now available in ChatGPT Work and Codex”
-
20
The essay, section 1confirmed
Sept. 22 Anthropic released a new model (Claude Opus 5.5)
The page says “Release date: September 22, 2026”
-
21
The essay, section 1confirmed
Both pointed to fewer made-up facts
The page says “makes about half as many mistakes as its predecessor”
-
22
The essay, section 1confirmed
OpenAI said GPT-6 Sol makes about half as many mistakes as its predecessor
The page says “makes about half as many mistakes as its predecessor”
-
23
The essay, section 1confirmed
measured on real conversations in which users had flagged an error
The page says “based on de-identified real-world conversations where users flagged mistakes by our models”
-
24
The essay, section 1confirmed
Internal test where any invented figure or quote meant failure
The page says “where any invented figure or quote would have failed”
-
25
The essay, section 1confirmed
16 of 18 reports by Claude Opus 5.5 passed
The page says “16 out of 18 of Opus 5.5's reports cleared our quality bar”
-
26
The essay, section 1confirmed
two other Anthropic models passed none
The page says “Neither Fable 5.1 nor Opus 5 cleared that bar in any attempt”
-
27
The essay, section 1confirmed
Researchers call a plausible, confident, false answer a hallucination
The page says “Hallucinations are plausible but false statements generated by language models”
-
28
The essay, section 1confirmed
It isn't a mysterious glitch; build and grading make guessing natural
The page says “We understand the statistical mechanisms through which hallucinations arise and are rewarded in evaluations”
-
29
The essay, section 1confirmed
From fake citations to a proposed rule
The page says “sanctioned ... seven nonexistent cases ... proposed a new rule September 18”
-
30
The essay, section 1confirmed
Feb.: lawyer sanctioned for brief citing seven nonexistent cases
The page says “sanctioned Maryland attorney ... seven nonexistent cases”
-
31
The essay, section 1confirmed
Sept. 18: court proposes human review of AI-drafted filings
The page says “proposed a new rule September 18”
-
32
The essay, section 1confirmed
Sept. 22: OpenAI and Anthropic release models, both pointing to fewer made-up facts
The page says “Anthropic released a new version of Opus 5.5 just 90 minutes before OpenAI's release”
-
33
The essay, section 1confirmed
Oct. 18: comments due
The page says “Public comments are due October 18”
-
34
The essay, section 1confirmed
Jan. 1, 2027: rule takes effect if approved
The page says “scheduled to take effect January 1, 2027”
-
35
The essay, section 2confirmed
An LLM is trained to predict the next word; pretraining on enormous text
The page says “Pretraining means predicting the next word in large amounts of text”
-
36
The essay, section 2confirmed
None of that text comes labelled true or false
The page says “there are no "true/false" labels attached to each statement”
-
37
The essay, section 2confirmed
Spelling and matching parentheses follow consistent rules, mistakes fade as models grow
The page says “Spelling and parentheses follow consistent patterns, so errors there disappear with scale”
-
38
The essay, section 2confirmed
OpenAI researchers' term 'arbitrary low-frequency facts' (birthday, dissertation title)
The page says “arbitrary low-frequency facts, like a pet's birthday, cannot be predicted from patterns alone”
-
39
The essay, section 2confirmed
Asked for an author's birthday, a widely used chatbot gave three different dates, all wrong
The page says “gave three different dates, likewise all wrong”
-
40
The essay, section 2confirmed
Paper published in peer-reviewed form in Nature in April
The page says “published 22 April 2026 ... Nature volume 653, pages 1047-1051”
-
41
The essay, section 2confirmed
If 20% of birthday facts appear once, a pretrained model should get at least 20% wrong
The page says “pretrained models should hallucinate on at least 20% of birthday facts”
-
42
The essay, section 2confirmed
If a model can't tell true statements from plausible false ones, it will produce false ones
The page says “incorrect statements cannot be distinguished from facts”
-
43
The essay, section 2confirmed
Same question can get different answers; known facts agree, guesses scatter
The page says “sampling several possible answers to each question and clustering them ... higher semantic entropy indicates higher uncertainty”
-
44
The essay, section 2confirmed
In 2024 Oxford researchers described made-up answers that shift with irrelevant details, such as the random draw
The page says “the answer is sensitive to irrelevant details such as random seed (published 19 June 2024; authors at OATML, University of Oxford)”
-
45
The essay, section 2unsupported
Legal citations are close to a worst case: format is a pattern seen endlessly, case existence a one-off fact; the model learned what a citation looks like far better than which exist
Checked for consistency within the issue itself
-
46
The essay, section 2confirmed
In 2025 Anthropic interpretability researchers reported on one of its Claude models
The page says “March 27, 2025 ... examined Claude 3.5 Haiku”
-
47
The essay, section 2confirmed
Declining to answer is the default
The page says “refusal to answer is the default behavior”
-
48
The essay, section 2confirmed
A feature that fires on familiar names switches that default off
The page says “A "known entities" feature inhibits that default”
-
49
The essay, section 2confirmed
Ask about Michael Jordan and it answers; unknown name and it declines
The page says “asked about basketball player Michael Jordan ... answers; unknown name, Michael Batkin, declines”
-
50
The essay, section 2confirmed
Some hallucinations happen when recognition misfires
The page says “such misfires can occur when Claude recognizes a name but doesn't know anything else about that person”
-
51
The essay, section 2confirmed
LOOKS RIGHT; rubber stamp over citation template
The page says “Illustrative only; no factual assertion”
Checked for consistency within the issue itself
-
52
The essay, section 2unsupported
A model has seen the shape of a citation endlessly; whether a case exists is a one-off fact
Checked for consistency within the issue itself
-
53
The essay, section 3confirmed
Models compared on benchmarks whose scores feed leaderboards
The page says “benchmarks that are misaligned but dominate leaderboards”
-
54
The essay, section 3confirmed
Researchers argue models end up optimized to be good test-takers
The page says “language models are optimized to be good test-takers”
-
55
The essay, section 3confirmed
Nature paper examined 10 influential benchmarks
The page says “the ten evaluations selected here”
-
56
The essay, section 3confirmed
vast majority grade binary: point for right, nothing for wrong or IDK
The page says “the vast majority of popular evaluations use binary grading”
-
57
The essay, section 3confirmed
Only one, WildBench, gives even minimal credit for uncertainty
The page says “offers minimal credit given for indicating uncertainty”
-
58
The essay, section 3confirmed
Model answering 'September 10' has a 1-in-365 chance of a point
The page says “If it guesses "September 10," it has a 1-in-365 chance of being right”
-
59
The essay, section 3confirmed
Saying 'I don't know' guarantees zero
The page says “Saying "I don't know" guarantees zero points”
-
60
The essay, section 3confirmed
Over thousands of questions the guesser looks better on the scoreboard
The page says “guessing when uncertain improves test performance”
-
61
The essay, section 3confirmed
Comparison OpenAI published last year
The page says “Publication date: September 5, 2025 (table from the GPT-5 System Card)”
-
62
The essay, section 3confirmed
SimpleQA test of short factual questions
The page says “SimpleQA table (from the GPT-5 System Card)”
-
63
The essay, section 3confirmed
gpt-5-thinking-mini: 22 right, 26 wrong, 52 abstained
The page says “Abstention 52% | Accuracy 22% | Error 26%”
-
64
The essay, section 3confirmed
o4-mini: 24 right, 75 wrong, declined once
The page says “Abstention 1% | Accuracy 24% | Error 75%”
-
65
The essay, section 3confirmed
By accuracy the older model wins, 24 to 22
The page says “accuracy slightly favours OpenAI's o4-mini, which answers almost all questions”
-
66
The essay, section 3confirmed
With a one-point penalty: 22-26 = -4; 24-75 = -51
The page says “22/26 and 24/75 per table; arithmetic checked”
-
67
The essay, section 3confirmed
Leaderboard-tuned model learns to guess when unsure
The page says “guessing when uncertain improves test performance”
-
68
The essay, section 3confirmed
2023 GPT-4 technical report: pretrained model well calibrated
The page says “the pre-trained model is highly calibrated ... its predicted confidence in an answer generally matches the probability of being correct”
-
69
The essay, section 3confirmed
Quote: post-training "hurts calibration significantly"
The page says “The post-training hurts calibration significantly”
- 70 unsupported
-
71
The essay, section 3confirmed
Same test, two models; SimpleQA per 100 questions
The page says “SimpleQA table, percentages summing to 100%”
-
72
The essay, section 3confirmed
Right: 22 / 24
The page says “Accuracy rate 22% | 24%”
-
73
The essay, section 3confirmed
Wrong: 26 / 75
The page says “Error rate 26% | 75%”
-
74
The essay, section 3confirmed
Said it didn't know: 52 / 1
The page says “Abstention rate 52% | 1%”
-
75
The essay, section 3confirmed
o4-mini wins 24 to 22 on right answers; loses -51 to -4 with penalty
The page says “table values; arithmetic checked”
-
76
The essay, section 4unsupported
'It's lying' myth: model making a plausible guess isn't hiding anything; it doesn't have the fact
Checked for consistency within the issue itself
-
77
The essay, section 4confirmed
RAG means the model looks things up in documents before answering
The page says “RAG comprises two primary steps ... retrieval ... generation”
-
78
The essay, section 4confirmed
Study published last year by Stanford researchers
The page says “Journal of Empirical Legal Studies, 2025 ... accepted 14 March 2025; most authors at Stanford”
-
79
The essay, section 4confirmed
Each tool still hallucinated between 17% and 33% of the time
The page says “each hallucinate between 17% and 33% of the time”
-
80
The essay, section 4confirmed
Counting false statements and real sources cited for claims they don't support
The page says “A response is considered hallucinated if it is either incorrect or misgrounded”
-
81
The essay, section 4confirmed
Accuracy will never reach 100%
The page says “Accuracy will never reach 100% because, regardless of model size, search and reasoning capabilities”
-
82
The essay, section 4confirmed
Hallucinations aren't inevitable because a model can decline
The page says “They are not, because language models can abstain when uncertain”
-
83
The essay, section 4confirmed
Easier for a small model to know its limits
The page says “It can be easier for a small model to know its limits”
-
84
The essay, section 4confirmed
Nature tested Gemini 3 Pro, GPT-5, Grok 4 and Claude Opus 4.5
The page says “tested Gemini 3 Pro, GPT-5, Grok 4, and Claude Opus 4.5 on SimpleQA”
-
85
The essay, section 4confirmed
four of the most capable models of the time
The page says “four frontier models”
-
86
The essay, section 4confirmed
Each question stated how answers would be scored
The page says “state the scoring system alongside the question itself”
-
87
The essay, section 4confirmed
Wrong answer cost three or nine points vs one for right
The page says “penalties of L = 0, 1, 3, and 9 ... assigns scores 1 (correct)”
-
88
The essay, section 4confirmed
All four would have scored higher answering IDK to everything
The page says “so always abstaining yields a higher score”
-
89
The essay, section 4confirmed
Consistency check (two answers, judge agreement, decline) cut errors at some cost to accuracy
The page says “generate two responses, judge whether they agree, and abstain ... reduces incorrect responses ... but also decreases correct responses”
-
90
The essay, section 5confirmed
Comments close Oct. 18; approved changes scheduled to take effect Jan. 1
The page says “Approved changes are scheduled to take effect January 1, 2027”
-
91
The essay, section 5confirmed
Fifth Circuit proposed an AI certification rule in 2023 and dropped it in 2024
The page says “The Fifth Circuit proposed an AI certification requirement in 2023 ... dropped the proposal in 2024”
-
92
The essay, section 5unsupported
The Fifth Circuit dropped it saying existing rules already require filings to be carefully checked for accuracy
-
93
The essay, section 5confirmed
OpenAI's Sept. 22 test drew on conversations where users flagged mistakes
The page says “based on de-identified real-world conversations where users flagged mistakes”
-
94
The essay, section 5confirmed
which it says aren't typical of everyday use, where errors are rarer
The page says “These error-inducing conversations are not representative of typical usage, where factual errors are more rare”
-
95
The essay, section 5confirmed
Oxford built a detector sampling several answers and measuring how far their meanings disagree
The page says “sampling several possible answers ... clustering ... entropy computed over meanings of sentences”
-
96
The essay, section 5confirmed
Nature study suggests telling a model IDK is acceptable won't be enough on its own
The page says “For the larger penalties ... always abstaining yields a higher score (with the rubric stated in each question)”
-
97
The essay, section 5confirmed
Stanford study counted real sources cited for unsupported claims as hallucinations
The page says “misgrounded ... key factual propositions are cited but misinterpret the source”
-
98
The essay, section 5confirmed
Oxford's detector samples several answers and measures how far meanings disagree
The page says “computing uncertainty at the level of meaning rather than specific sequences of words”
-
99
The essay, section 5confirmed
Ask-twice steps (advice)
The page says “sampling several possible answers to each question”
-
100
Further reading, item 1confirmed
OpenAI 'Why language models hallucinate' includes the two-model comparison
The page says “SimpleQA table ... gpt-5-thinking-mini | OpenAI o4-mini”
-
101
Further reading, item 2confirmed
Nature paper includes test of telling models what a wrong answer costs
The page says “state the scoring system alongside the question itself”
-
102
Further reading, item 3confirmed
Anthropic 'Tracing the thoughts' covers whether a model knows an answer
The page says “refusal to answer is the default behavior”
-
103
Further reading, item 4confirmed
Stanford paper: tools searching real case law still misground citations
The page says “key factual propositions are cited but misinterpret the source”
-
104
Further reading, item 5confirmed
Oxford semantic entropy detector
The page says “semantic entropy ... signals a likely confabulation”
The checker's notes
- Back-issue window check: all cited dated sources are on or before D (Law Commentary 2026-09-23, Lawyer.exe 2026-09-24, JD Journal 2026-09-18, TechCrunch/Anthropic 2026-09-22). Nothing requires knowledge after D. 'What to watch' items (Oct. 18 comments, Jan. 1, 2027 effective date) were announced Sept. 18, before D.
- No compliance issues: no securities, markets, valuations or employer references. Anthropic's test concerned company performance reports, but the issue doesn't mention that.
- No first-person statements found. Quotes are within 14 words and attributed.
- Nature s41586-026-10549-w and s41586-024-07421-0 failed on direct fetch; both loaded via their doi.org redirects (same pages). A third-party PDF copy of the Nature paper was not used.