I believe the eight week paid pilot has become a way of avoiding a decision rather than making one. Pilots are expensive, they occupy your best engineers, and they frequently prove only that a vendor can make one cherry picked part work under controlled conditions.
Send Your Worst Images First
Before any commercial conversation, send three sets of images. Twenty clearly good parts, twenty clearly defective parts, and twenty borderline cases that your own inspectors disagree about. That third set is the one that matters, because it is where every system earns or loses its keep.
A vendor who returns results on the borderline set within a week is showing you something real. A vendor who wants a site visit before touching the images is selling a process.
Keep a further ten images back and never send them. Once a supplier reports strong numbers, ask them to run that held back set live on a call. The difference between the two results tells you whether you are looking at genuine generalisation or careful tuning against the sample you provided.
Ask About False Positives, Not Accuracy
A 99% accuracy claim means little without the split between missed defects and false rejects. A line producing 10,000 units per shift at a 2% false reject rate throws 200 good parts into the rework bin every shift, which operators will stop tolerating quickly.
Ask for both numbers at a stated confidence threshold, and ask what happens to each when the threshold moves.
Test the Change Request Path
Products change. Ask precisely what happens when a new variant arrives on Monday. Some platforms let a plant engineer add it in an afternoon. Others require a vendor visit, a fresh data collection round and a four week retraining cycle, which quietly becomes a recurring cost line.
Comparing how established machine vision companies handle retraining and variant onboarding tells you more about five year cost than any price sheet, because that workflow is where the ongoing spend actually lives.
Check Who Owns the Data and the Model
Get ownership in writing. Your defect images are a genuine asset, and a trained model built on them is arguably derivative of that asset. Contracts vary far more than buyers expect, and the answer becomes urgent only when you want to change supplier.
Two clauses are worth reading closely. The first covers whether your images can be used to train models sold to other customers, including competitors. The second covers what you receive if the relationship ends, since a model you cannot export or run without an active licence is a rental rather than a purchase.
Reference Calls Beat Case Studies
Ask for two references in your industry with comparable line speeds, and ask them one question. What did you find out in month six that you wish you had known in month one. The answers are consistently more useful than any published case study.
Ask the vendor as well for an installation that did not go to plan. Anyone with real deployment history has one, and the willingness to describe it is informative.
Ask the reference about support response as well. A system that stops a production line at 2 am is only as good as the person who answers the phone, and response times vary enormously between suppliers of otherwise similar technology.
Understand What You Are Actually Buying
Some suppliers sell software that runs on hardware you source. Others sell a complete cell including cameras, lighting, enclosure and integration. The second costs more upfront and removes the integration risk that sinks a surprising number of projects. Reviewing how the ai visual inspection market splits between software platforms and turnkey systems helps clarify which model suits your internal engineering capacity.
Then Run a Very Short Pilot
Once two suppliers survive that screening, run a two week pilot with a defined pass mark agreed in advance. Short, scoped and measurable beats an open ended evaluation that drifts for a quarter.
Screening on images, false reject rates and retraining workflow removes most of the field before anybody visits the plant.
