How do you know whether an AI use is drifting over time?
By monitoring it like a process: indicators, samples reviewed, alerts when behaviour changes. An AI use is never finished on the day it goes live.
Updated on 24 October 2026
What is going on.
An AI use put into production works well in the first few weeks. Then, without anyone deciding so, its behaviour changes: answers get longer, quality drops on certain cases, errors appear where there were none.
The causes are many. The supplier has updated its model. The data received no longer quite looks like the data at the start. Users have found other ways of using it. None of this shows unless you look. An AI use is therefore monitored like a process: with indicators, samples reviewed regularly and alerts when something moves.
What we observe.
- Models change without warning.An update on the supplier’s side can change the results of a use that has not changed on yours.
- Drift is gradual.No single incident reveals it: only measurement over time shows it.
- Nobody is in charge of looking.The project team has left, and the business assumes the tool is working.
What you can do right now.
- Keep a set of reference cases and run it through the tool regularly to compare the answers.
- Every month, have a sample of real results reviewed by someone from the business.
- Decide who receives the alerts, and what they are allowed to decide, up to suspending the use.