Fluent writing is not evidence, and a citation you have never opened is not proof. Why chatbots invent confident answers, and the mistakes that make it costly.
This guide is general information, not technical or legal advice. If you’ve lost money or personal information, contact your bank, IDCARE on 1800 595 160, or report it through ReportCyber at cyber.gov.au.
Quick summary
- Chatbots predict the next likely word, so they optimise for plausibility rather than truth and can state invented facts with total confidence.
- AI-generated citations can attach real authors to articles that do not exist, so search the exact title in quotation marks and open the DOI yourself.
- Numbers, dates and quotes are frequent casualties; take them from the original source, never from a chatbot summary.
- Chatbots that search the web and show links still need checking: open the links and confirm they say what the chatbot claims.
- A two-minute routine of isolating the claim, searching independently and reading the original source catches most invented answers.
Ask an AI chatbot a question and you will get a smooth, confident answer, whether or not it knows what it is talking about. These invented answers have a name, hallucinations, and they are built into how the technology works rather than being a bug a software update will fix. Knowing why they happen, and which habits let them slip into your work, is the difference between a useful assistant and a well-dressed source of nonsense.
Why Chatbots Invent Confident-Sounding Answers
Popular chatbots are built on large language models, which generate text by predicting the next likely word based on patterns in their training data. They optimise for plausibility, not correctness, so a model can produce text that reads as authoritative while matching nothing in the real world. The training data comes largely from crawls of the public web, which brings in its share of misinformation, and the compression needed to fit that much text into a model blurs details further. The model ends up producing the most plausible statement it can, and it has no built-in way to judge whether that statement is true.
The industry is open about this. Meta warned that one of its chatbots was prone to making confident statements that are not true when it released an updated version in 2021, and engineers treat hallucinations as an unsolved research problem that can be reduced but not eliminated. That is not a reason to avoid chatbots, but it is a reason to build a checking habit around them. Our timeline for trying an AI chatbot for the first time covers how to ease in.
Mistake 1: Treating Fluent Writing as Evidence
What goes wrong: the answer reads well, so it is assumed to be right. Tidy paragraphs, headings and a confident summary all signal care and expertise to human readers, and a chatbot produces those signals automatically no matter how solid the underlying facts are.
Fluency is a feature of the writing engine, not evidence about the world. Treat every output as a draft that needs checking, the way you would treat notes from an enthusiastic but unreliable colleague.
What to do instead: read for claims rather than tone. Pull out each factual statement and ask where it could be confirmed. If you cannot name a source that would confirm it, that is your cue to look. The stakes matter too; health, legal and money answers deserve a qualified professional regardless of how good the prose is.
Mistake 2: Trusting Citations You Never Opened
Citations from AI tools can carry every mark of real scholarship, journal, volume, page numbers, and still have no actual article behind them. Studies of the problem describe several versions: articles that were never published, real authors attached to titles they never wrote, and correct-looking entries with wrong page numbers or DOIs. In one documented case, an AI-generated article was published under a real researcher’s name in December 2023, complete with a valid DOI, and a review of that journal found that at least 48 of 53 articles with the fewest citations appeared to be AI-generated. In a separate legal case, a court discovered that several cases a lawyer had cited, quotes and all, had never been issued; the chatbot had stitched plausible case names and reasoning together from patterns in real legal texts.
The red flags are worth memorising: a title that sounds unusually perfect for your question, a journal that exists but does not contain the article, an author who exists but shows no evidence of writing it, and a DOI that resolves to some other paper than the one cited.
What to do instead: never assume a citation is accurate because it includes an author, journal, volume and page numbers. Search the exact title in quotation marks, confirm the listed authors actually appear on the publication, and open the DOI link to see where it lands. If you cannot find the source after a genuine search, do not use it.
Mistake 3: Using AI for Numbers, Dates and Quotes
Chatbots frequently get titles and years slightly wrong, and the same applies to figures, deadlines and quoted passages. Because the model is assembling likely-sounding text, a statistic can be almost right, a date can be off by a year, and a quotation can read exactly like something someone would say without ever having been said.
What to do instead: any number, date, quote or legal detail you plan to use or repeat should come from the original source, not from a chatbot summary. Find the statistic in the organisation’s own report, check the date on the agency’s page, and confirm a quote by locating where it was said. Chatbots are fine for working out what to look for; the answer itself needs a second source.
Handy tip
If an answer matters enough to repeat, it matters enough to check. Keep nothing from a chatbot without a source you have opened yourself.
Mistake 4: Letting the Chatbot Outrank the Official Source
Some chatbots search the web and display links to their sources, which feels like built-in fact-checking. It is not. Showing a link does not mean the source is any good, or that it says what the chatbot claims, so the links need to be opened and read. Tools that ground answers in an external database or draw on scholarly literature lower the risk, but they do not remove it.
What goes wrong in practice: a chatbot describes a government rule or a company policy in confident terms, and the reader never visits the page that actually sets it. Details drift, and the official source may have changed since the model’s training data was collected. Invented links are part of the problem too, because made-up outputs can include web addresses that lead nowhere or somewhere unexpected. Our guide to spotting scams covers what to check when a link or sender seems off.
What to do instead: for anything official, go to the source. Government agencies for rules and entitlements, the provider’s own help pages for product settings, and an organisation’s website for its policies. Use the chatbot to work out the question; let the official source answer it.
A Two-Minute Way to Check Any Answer
The routine fits into about two minutes:
- Isolate the claim. Write down the one fact you would actually be acting on.
- Search independently. Use the exact title in quotation marks if a citation is involved, or search the author and keywords, because AI often gets titles or years slightly wrong.
- Open the original source. Read the relevant part yourself instead of trusting the chatbot’s description of it.
- Compare. If the source contradicts the chatbot, the source wins. If you cannot find the source at all, treat the claim as unverified.
Fact-checkers use variations of this constantly: investigate the source behind a claim, then look for better coverage from a trusted outlet or the original document. The same routine works whether the answer is for a work email, a purchase or a school assignment.
None of this makes chatbots useless. They are good at brainstorming, explaining a concept in plain language and suggesting angles you had not thought of. The mistake is treating them as a source rather than a starting point. Checking costs two minutes; acting on a confident-sounding falsehood usually costs more.
Pick the last chatbot answer you acted on without checking, and run it through the four steps above.
Your questions answered
Q: Can AI chatbots ever be made fully accurate?
A: Hallucinations are inherent to how generative models work. Grounding answers in an external database reduces them, but engineers treat the problem as unsolved and expect it to be reduced rather than eliminated.
Q: What is a fabricated citation?
A: A reference that looks scholarly but does not exist, or one that attaches a real author to an article they never wrote. Search the exact title in quotation marks and open the DOI link to check.
Q: Does paying for an AI subscription reduce mistakes?
A: The causes of hallucination are built into how the models work, so checking answers matters on every plan. What a subscription buys is mostly usage limits and extra features, not correctness.
Q: Should I use chatbot answers for medical or legal questions?
A: Take those decisions to a qualified professional. A chatbot can help you work out what to ask, but its answer should not be the one you act on.