We built PMOpack because this happened to us. On 30 April 2026 we asked an AI assistant for a SteerCo pack from a real programme’s data. It told us the deck was done. It was not. It looked right. It read well. Nothing in the tool told us what was wrong. A figure that is wrong in the same confident tone is the same problem, and it is harder to see.
Why it happens
OpenAI published a paper in September 2025 called “Why language models hallucinate”. The short version: the way these models are trained and scored rewards a confident guess over an honest “I do not know”. If a model does not know a date, a guess has a small chance of being marked right. Saying “unknown” scores zero. So it guesses. The paper says this “remains a fundamental challenge for all large language models”.
That is the vendor’s own description of every model of this kind, not a fault in one product. A better prompt helps. It does not remove the problem.
What it looks like in a project report
- A budget figure that is close to the real one but not the same. Rounded, or from a different line.
- A go-live date from an older version of the plan, because that date was in the text you pasted.
- A RAG colour that is one step kinder than the one in the RAID log.
- A milestone described as “on track” when the plan shows it two weeks late.
- A percentage complete that does not add up from the rows below it.
None of these look like errors. They look like a tidy report. That is the problem.
How to spot it: a five-minute check
- Pick every number in the summary. Write them down. Five minutes, one list.
- Find each one in the source file. The plan, the budget tracker, the RAID log. Not in another report. In the source.
- Check the dates against the plan. Every date. Especially the ones that sound good.
- Check the colours against each other. The Status Report, the Milestone Report and the RAID log should agree on what is red.
- Read the adjectives. “Broadly on track”, “minor slippage”, “largely complete”. Each one is hiding a number. Find it.
If you cannot find a figure in the source, it does not go in the report. Not “I think it is about that”. Out. The full pre-issue routine is in What checks should I run before I issue a project report?
A real example
A PM used an AI tool to draft a month-end summary. The budget tracker showed spend to date at 61% of the approved figure. The summary said 58%. Where did 58% come from? A paragraph in the previous month’s report that the PM had pasted in for context. The tool used the old number because it was in the text in front of it.
Three points of difference is not a crisis. A board member who spots that you do not know your own spend figure is.
If you are weighing up whether to use a chat tool at all, start with Is it safe to use ChatGPT or Claude for project reports?
Where PMOpack fits
PMOpack uses AI to draft the narrative in a reporting pack, so the same risk applies to its draft. The difference is what runs after. Fixed checks compare the figures in the pack with each other at generation time: the SteerCo pack against the Milestone Report, the RAG colours against the RAID log, the roll-ups against their rows. A critical mismatch blocks publish until a person fixes the data or accepts it with a written reason, and that acceptance is stored. The checks are not AI. They run the same way every time. They do not prove a figure is true against your finance system, and a check skips, and says so, when the partner document is missing. A person decides. PMOpack makes the disagreement visible first. See how that differs from drafting in ChatGPT.
Questions people ask
- Why do AI tools make up numbers?
- Research published by OpenAI in September 2025 says the way models are trained and scored rewards a confident guess over an honest 'I do not know'. The paper says this applies to all large language models.
- Will a better prompt stop it?
- It helps, but it does not remove the problem. The only reliable fix is to check every figure in the output against the source file.
- What is the fastest check I can do?
- List every number and date in the summary, then find each one in the source file. Anything you cannot find comes out.
- Does PMOpack stop AI from making up numbers?
- No tool can promise that. PMOpack runs fixed checks across the pack after the draft and blocks publish on a critical mismatch until a person deals with it. It does not prove a figure is true against the source system.
Sources
Last checked 28 October 2026.
- OpenAI, “Why language models hallucinate” (September 2025): training and evaluation reward guessing over admitting uncertainty; hallucinations “remain a fundamental challenge for all large language models.”openai.com/index/why-language-models-hallucinate
- PMOpack behaviour statements are drawn from PMOpack’s own claims register.