Governance

Problem 2: AI that confidently makes things up (and the real bills it has run up)

25 October 2026 · 7 min read · Hidden Node

In late 2022 a man in Canada asked Air Canada's website chatbot about bereavement fares after his grandmother died. The chatbot told him he could buy a full-price ticket and claim the discount within 90 days. That wasn't the airline's policy.

When he asked for the refund, Air Canada argued it couldn't be held responsible for what its chatbot said. The tribunal disagreed. It found the airline "did not take reasonable care to ensure its chatbot was accurate" and ordered it to pay (Moffatt v. Air Canada, 2024 BCCRT 149). The tribunal's reasoning was simple: a company is responsible for everything on its website, whether it comes from a static page or a chatbot.

The amount was small. The principle wasn't.

Two more expensive lessons

The lawyers and the six fake cases. In 2023, two New York lawyers filed a brief citing court decisions that ChatGPT had invented. Judge P. Kevin Castel fined them and their firm $5,000 in Mata v. Avianca, finding bad faith partly because they kept standing by the fake citations after being questioned.

The consultancy's footnotes. In 2025, Deloitte Australia agreed to partially refund a A$440,000 government contract after a researcher found the report contained a fabricated quote from a court judgment and references to academic papers that don't exist. The revised version disclosed that a generative AI system had been used in writing it.

Three very different organizations. Same root cause: AI output reached a customer, a court or a client without anyone checking it against a source.

Why it happens

Language models generate text that sounds right. Most of the time sounding right and being right overlap. When they don't, the model has no built-in alarm. It will invent a case citation with the same confidence as it states the boiling point of water.

That's why "just use a better model" isn't a strategy. Newer models hallucinate less, but not zero, and you can't tell from the output which answers are the invented ones.

What actually prevents it

Ground answers in your own documents. Instead of asking the model what it knows, retrieve the relevant passages from your policies, contracts or records and tell it to answer only from those. This approach, usually called retrieval-augmented generation (RAG), turns the model into a reader rather than a guesser.

Require citations, and check them automatically. Every factual statement should point to the passage it came from. Then verify mechanically that the cited passage exists and actually contains the claim. A citation that doesn't resolve is a red flag you can catch before anyone reads the answer.

Teach it to say "I don't know." A good assistant refuses when the sources don't contain the answer. That feels less impressive in a demo and is far more valuable in production. Air Canada's bot would have done better saying "please check our refund policy page".

Put a person between the AI and anything binding. Prices, refunds, legal statements, medical information and anything sent to a regulator or client should be reviewed before it leaves the building, at least until you have evidence the system is reliable on that task.

Test with questions designed to trip it up. Before launch, ask it things that aren't in your documents and things that sound similar to real policies but aren't. Count how often it invents an answer. That number should be close to zero before customers see it.

The uncomfortable takeaway

The courts are settling the question of responsibility: if your AI says it, you said it. That doesn't mean avoiding AI. It means designing for the fact that it will sometimes be wrong. Then the errors get caught by your process instead of by your customers, a judge or the newspapers.

Sources

• Moffatt v. Air Canada, 2024 BCCRT 149, summarised by Manatt and Kiplinger
• Mata v. Avianca, Inc., reported by Bloomberg Law; background on Wikipedia
• Deloitte Australia refund, reported by the Associated Press via CTV News