How to Validate a Vendor's AI Inspection Claims: The Questions Nobody Wants You to Ask

To validate an AI inspection vendor's accuracy claim, ask for four things in writing: the test method, the sample size, the confidence interval, and who ran the test. If any of the four is missing, the number is marketing, not measurement. Then insist the test be repeated on your product, your packaging, and your line speed — because the widely quoted accuracy and false-reject figures circulating in this industry trace back to vendor marketing blogs, with no peer-reviewed study or independent third-party validation behind them.

Where the famous numbers actually come from

If you have shopped for X-ray inspection in the last two years, you have seen two claims. One says deep-learning AI cuts false rejects dramatically — a figure in the high tens of percent. The other says AI-enhanced systems achieve detection above 99.5% with false rejects below 0.2%. They appear in buyer's guides, in LinkedIn posts, in RFP boilerplate, and increasingly in AI-generated answers to purchasing questions.

We tried to source them. Both trace back to equipment-vendor and SaaS marketing blogs — the kind of content published to rank for a keyword, not to report a result. We searched specifically for peer-reviewed validation of those figures. What came back was more vendor technical documentation. There is no independent study we could find that supports either number, under any stated test condition.

That is the verifiable fact in this whole discussion, and it is worth stating plainly: the fact is not "AI doesn't reduce false rejects." The fact is that these specific numbers have no independent source. They may be roughly right for some product on some line. They may be optimistic. Nobody outside the vendor knows, because nobody outside the vendor has checked.

Here is the part that should bother a buying engineer more than the numbers themselves: not one vendor we reviewed discloses the test method, the sample size, or the confidence interval. Those three items are the minimum required to interpret any detection statistic. Without them, "99.5%" is not a specification. It is a sentence.

Common misconceptions vs. what is actually true

Misconception 1: "AI" is a detection capability

It isn't. AI is a classification layer that sits on top of a physical measurement. In X-ray, the physics comes first: the machine images density differences, and whatever the detector cannot resolve, no model can recover. A neural network can help you decide whether an ambiguous grey blob is a bone fragment or a fold in the product. It cannot conjure contrast that the detector never captured. When a vendor answers a physics question with an AI answer, that is the moment to slow down.

Misconception 2: A higher detection rate is automatically better

Detection rate and false-reject rate move together. You can push detection up on any system by loosening the threshold — and you will pay in good product thrown in the reject bin. Any accuracy claim quoted without its matching false-reject figure, at the same threshold, on the same samples, is meaningless. Always demand the pair. If a vendor gives you one number, they have chosen the flattering half.

Misconception 3: The hard problem is the model

This is where the peer-reviewed literature disagrees with the marketing directly. Zeegers and colleagues, in published academic work on deep-learning X-ray foreign-object detection, identify the core bottleneck as training-data annotation — not model architecture. Human labelling of X-ray images is slow, subjective, and inconsistent between annotators, and that noise propagates straight into the model's ceiling. Their proposed workflow avoids manual labelling altogether: CT-scan a set of representative objects, reconstruct them in 3D, segment the reconstruction, then generate virtual 2D projections whose ground truth is known by construction. Their notable finding is that fewer than ten representative objects are typically sufficient. In other words: the credible published route to better AI inspection is better ground truth, not a bigger network. So when a vendor's pitch is entirely about their model, ask them where their labels came from.

Misconception 4: A successful demo predicts line performance

A demo run on samples the vendor selected, at a speed the vendor chose, with contamination the vendor inserted in a known position, tells you the machine functions. It tells you close to nothing about your recall rate on your product at your throughput. Demos are a filter for gross incompetence, not a validation.

Misconception 5: More sensitivity beats the right technology

X-ray is a density-contrast method. Low-density contaminants — plastics, rubber, wood — are a structural difficulty, not a tuning problem. This is not our opinion; it is visible in what the market leaders are building. Mettler-Toledo's global launch on 7 May 2026 of the X56 DXD+, a dual-energy photon-counting X-ray system with integrated AI, is aimed squarely at low-density foreign bodies under overlapping products, mixed product types, and varying product position — alongside the M50 R-Series AdvancedLine metal detector and ProdX data management software, shown at Interpack 2026 in Hall 11, Stand A60. When the largest players engineer entire new detector architectures around low-density detection, that tells you it is a real physical limit — and that an AI checkbox on a conventional detector does not remove it.

What the technology direction actually looks like

Two things are worth knowing so you can hold a technical conversation rather than a sales one.

First, detector hardware is moving. Photon-counting detectors and dual-energy acquisition are where the serious engineering effort is going, because they attack the contrast problem at the physical layer instead of the inference layer. Ishida's IX-PD series, per the vendor's own claim, combines a photon-counting detector with proprietary genetic-algorithm imaging — we cite that as a vendor claim, because that is what it is; we have not seen independent verification of it, and neither have you.

Second, the model architecture story in adjacent fields is informative. In X-ray security screening for prohibited items — a neighbouring problem with far more open research than food inspection — the field has been shifting from pure CNNs toward CNN-Transformer hybrids, and 2025 brought lightweight real-time approaches such as TinyRay (YOLOv7-tiny with a FasterNet backbone) and YOLO-SRW (a modified YOLOv8 with dynamically adjusted spatial receptive fields). Note what that trend implies: the research direction is toward models light enough to run in real time on constrained hardware. If your vendor is vague about inference latency at line speed, that is a fair thing to press on.

The validation checklist: questions to put in writing

Print this. Send it as part of your RFQ. A vendor's willingness to answer is itself a data point — arguably the most useful one you will collect.

On the number itself

1. What exactly does your accuracy figure measure — detection rate, false-reject rate, or a blend? State both, at the same threshold. 2. What was the sample size? 3. What is the confidence interval, and at what confidence level? 4. Who ran the test: you, the customer, or an independent third party? Can we see the report? 5. What product, packaging, and line speed was it measured on? 6. On how many distinct production lines has this been reproduced?

On the AI specifically

7. What is the AI doing — image classification, thresholding, image reconstruction, or reject decisioning? Name the step. 8. Where did your training data come from, and how was the ground truth established: manual annotation, or a construction-based method? 9. How many images, and how many distinct contaminant types? 10. What happens when we change product recipe or packaging — does the model need retraining, who does it, how long does it take, and what does it cost? 11. Does the model run locally or in the cloud? If cloud, what is the line's behaviour when the link drops? 12. What is the inference latency at our stated line speed, and what is the throughput ceiling?

On accountability

13. Will you write the detection performance for our product into the contract as an acceptance criterion, tested on our samples at our site? 14. What is your remedy if the accepted performance is not met after installation? 15. Who owns the trained model and the images from our line? 16. What certifications does the machine actually hold, and can we see the certificate number — not the logo on a brochure?

Question 13 is the one that separates the field. Any vendor can produce a number in a blog post. Very few will sign it.

How to run the test yourself

You do not need a laboratory to do better than a vendor demo. You need discipline.

Step 1: Build your own sample set. Use your product, in your packaging, at your fill weight. Include the contaminants that actually appear in your process — not just the certified stainless spheres. If you have ever pulled a piece of conveyor belt or a plastic clip out of a line, that is a test sample. Step 2: Insert contaminants across positions, not just centre. Position variation is a known weakness; test it deliberately, including edges, seams, and the densest part of the product. Step 3: Include a substantial run of clean product — this is how you measure false rejects, and it is the half vendors skip. Step 4: Run blind. The operator should not know which packs are seeded. Step 5: Run at production speed, not demo speed. Step 6: Record every result and compute both rates with the sample size attached. A detection rate from 20 packs is a rumour, not a result. Step 7: Repeat after a product changeover — the number that matters is the one that survives a recipe change.

Do this once and you will know more about the machine's real performance than any published figure will tell you.

Where MIQI stands, stated plainly

We build X-ray and metal detection equipment, so we have an interest here. We will state ours openly rather than pretend to neutrality.

Our MQ-XR series is a standard-configuration X-ray foreign-object inspection machine for bagged products, and MQ-XR-P is the high-precision variant. Our MQ-MD-C series metal detectors are available in standard and non-standard custom configurations. We are a source factory, and we support free sample testing — send us your actual product and your actual contaminants and we will run them and send you the results, including the failures. That offer exists precisely because of everything above: the only accuracy number worth anything to you is the one measured on your product.

Two things we will not do. We will not quote you an AI accuracy percentage we cannot substantiate with a method, a sample size, and a confidence interval. And we will not imply certifications we do not hold: MIQI does not currently hold CE or ISO 9001 certification. If a supplier's certification status matters to your procurement rules, you should ask every vendor for certificate numbers and verify them independently — including ours. A supplier who tells you the uncomfortable thing before you sign is more likely to tell you the uncomfortable thing after.

Talk to an engineer, not a salesperson. WhatsApp +1 (213) 563-6234 or email 897874196@qq.com. Send your product spec, your line speed, and the contaminants you are actually worried about, and we will tell you honestly whether X-ray, metal detection, or a combination is the right answer for you — including when the answer is that none of them solves your problem.