Skip to content

Blog · Buyer's guide

How to benchmark a machine vision vendor: 12 questions and the numbers behind them.

Compact machine-vision camera aimed at a sample part on a test conveyor

Updated July 2026 · 8 min read · Adente Vision Engineering Team

Benchmark a vision vendor on numbers, not adjectives. Ask for F1-score and false-negative rate on a defect like yours, per-part time, how many training images you need, and where your images go. A real vendor answers with figures, for example 99.65% F1 and a 0.69% false-negative rate on a live cap line.

Most vision proposals are written in adjectives: highly accurate, real-time, easy to deploy. None of those can be checked, and the specifics that can, the false-negative rate, the per-part time, the training-data need and the data path, are exactly what a weak proposal leaves out. The fix is to make every vendor answer in the same units you would use to judge your own quality, and to refuse to score a proposal on adjectives.

The 12 questions below split into three groups: the metric questions that expose real accuracy, the practical questions that expose the real workload, and the architecture questions that decide where your data lives. The answers Adente Vision gives run through as a worked example of what a specific reply looks like.

Why is "99% accurate" a non-answer from a vision vendor?

Accuracy counts every correct decision, and on a line where 1 in 100 parts is defective, a system that passes every part is 99% accurate and catches zero defects. That single figure hides the two errors that actually cost money: the escape that reaches your customer and the good part wrongly rejected. The number that exposes this is the false-negative rate, the fraction of bad parts that pass, read together with F1, precision and recall. Ask for those, and "99% accurate" stops being an answer. For why the false-negative rate is the metric to demand, see the false-negative rate as an inspection metric.

Which metric questions expose real accuracy?

Questions 1 to 4 turn a vague "highly accurate" claim into something you can compare. Ask each one, and ask for the parts it was measured on.

  • 1. What is your F1-score on a defect like mine, and on whose parts was it measured? F1 folds escapes and false rejects into one honest number. As a reference point, Adente Vision reached 99.65% F1 on a live cap-inspection line, and the context matters as much as the figure.
  • 2. What is your false-negative rate, the escape rate? This is the number a quality manager loses sleep over. The same cap line ran a 0.69% false-negative rate. A vendor who cannot state this on your defect type has not measured it.
  • 3. What are precision and recall separately? F1 is a summary; precision (how many rejects were real) and recall (how many defects were caught) tell you which way the system errs.
  • 4. What is the per-part inspection time? Ask for a measured figure against a stated bound. Adente Vision commits a conservative catalog bound of 0.5 s per part and 100+ parts per minute, and notes that your committed number needs an application-specific measurement on your own parts.

If a vendor answers any of these four with "very accurate" or "real-time," treat it as a missing data point, not a pass.

Which practical questions expose the real workload?

Questions 5 to 8 decide how much of your time and money a deployment actually costs, which is where many quiet surprises live.

  • 5. How many reference images do you need to start? A modern anomaly approach trains on good parts only. Adente Vision starts from about 20 good reference images, so you are not collecting a defect catalogue for months before go-live.
  • 6. How long is model training, and who runs it? On the cap line, training completed in under 48 hours. Ask whether you must ship data out to a vendor's data-science team, or whether it happens on your site.
  • 7. How long is install, and does it need a vision engineer? Adente Vision mounts in about 30 minutes by one person because the unit is under 9 kg, with a four-step Mount, Aim, Configure, Wire routine and an on-device preview, so aiming needs no laptop.
  • 8. How do you handle a new variant or a new defect after go-live? Ask whether that is a training task or a re-engineering task. Learning from examples means adding a variant is a capture-and-train job, not a rewrite.

Which architecture questions decide where your data lives?

Questions 9 to 12 decide where your images go and how the system fits your controls. For a plant with sensitive part IP, these can outweigh the metric answers.

  • 9. Does inference run on the edge or in the cloud? Adente Vision runs on a fanless Jetson-class board, 8-16 GB, inside the enclosure, so the decision is made on the line with no round-trip.
  • 10. Where do my part images go, and do they leave the line? On-device inference means raw imagery stays inside the plant. Ask any cloud vendor to state, in writing, the residency and retention of your images.
  • 11. What protocols do you speak to my PLC or robot? Adente Vision carries pass/fail and program-select over PROFINET, EtherNet/IP, Modbus TCP, EtherCAT or OPC UA, with 4 inputs and 4 outputs at 24V for discrete signalling.
  • 12. How are models updated on an air-gapped line? Updates by USB stick support lines with no network, so a hardened cell is not forced online to stay current.

What does a good answer look like versus a red flag?

Score each answer on whether it is a checkable figure or an adjective. The pattern is consistent: a good answer names a number and the conditions it was measured under, while a red flag substitutes a feeling for a fact.

Buyer questionA good answerRed flag
How accurate is it?F1 and false-negative rate on a defect like yours, e.g. 99.65% F1 and 0.69% FNR on a live cap line"99% accurate" or "highly accurate"
How fast per part?A measured per-part time with its conditions, e.g. measured against a 0.5 s catalog bound"real-time," no number
How much data to start?About 20 good images, training under 48 hours, on good parts only"thousands of labelled defects" or "it depends"
Where do my images go?On-device on the edge, images stay on the line, USB updates for air-gapped cells"uploaded to our cloud," no residency answer
How does it reach my PLC?PROFINET, EtherNet/IP, Modbus TCP, EtherCAT, OPC UA, 4-in/4-out 24V"we will sort out the interface later"

The single red flag that cuts across all twelve is an adjective where a number belongs. Highly accurate, real-time, advanced, with no figure behind them, are the tell that a claim was never measured.

How should you score the answers?

Score on proof, not on presentation. Give a point for every answer that is a number with named conditions, half a point for a number with no context, and zero for an adjective. Weight the false-negative rate and the data-residency answers highest, because an escape reaches your customer and a data leak reaches your legal team, and neither is easy to reverse.

Run the same 12 questions past every vendor and one automation integrator, and the scores become directly comparable; the vendor with the strongest proof is rarely the one with the most adjectives, which is the point of asking. Adente Vision is an edge-AI visual inspection unit built by ADENTE Advanced Engineering Technologies, part of the Aden Group, sold through automation system integrators, and the specific answers above show what a checkable reply looks like. See the system for the full spec, and the pillar guide on AI visual inspection for the method behind few-shot and anomaly detection.

Frequently asked questions

Evaluating vision vendors right now?

Send us a sample part or a short video of your line, and we return the F1, false-negative rate and per-part time on your defect before any quote, so you can benchmark us on numbers. See how Adente Vision answers the 12 questions in specifics.