How AI Chat Agents Process Product Images — Scoped to What Your Store Sells
A shopper takes a photo of a jacket they already own and drops it into the chat: "Do you have this?" Another sends a picture of a cracked phone case: "This arrived broken." A third snaps a lamp they saw at a friend's place: "Anything like this?" These are three different jobs, and a store chat agent that handles images well does all of them — while quietly refusing a fourth: the random selfie, meme, or screenshot that has nothing to do with what you sell.
That last part is the whole trick. General-purpose vision models will happily describe any image. A store agent should not. Its camera is pointed at your catalog and nowhere else. This post is about how to build that: the pipeline, the gate that keeps random images out, and the three things it does once an image is allowed through.
What "image processing" means for a store agent
It does not mean "the AI can see." It means the agent can turn a shopper's photo into a question about your inventory and answer it with real SKUs. The unit of truth is the same as it is for text search: your live catalog. An uploaded image is only useful to the degree it can be matched against something you actually stock.
So the design goal is narrow on purpose. Three catalog-grounded jobs, one hard guardrail:
- Match — "Is this one of ours?" Find the SKU (or the closest few) that the photo depicts.
- Condition — "Is it broken?" Compare the shopper's photo against the pristine catalog image to flag damage, wear, or a missing part.
- Discover — "Find me something like it." Return visually similar items you do sell.
- Refuse — anything that is not plausibly a product in your catalog is declined, not processed.
Start with the gate, not the model
The instinct is to wire the image straight into a multimodal model and ask "what is this?" Resist it. The first component you build is the relevance gate: a cheap check that answers a single question — could this image be something we sell? — and drops everything that fails before any expensive reasoning happens.
The gate is not a separate AI mood; it is the natural by-product of the matching step. You embed the uploaded image into the same vector space as your product images, look up the nearest catalog item, and read the similarity score. If the best match is strong, process it. If the best match is weak, the image is out-of-catalog by definition — a face, a landscape, a competitor's packaging, a screenshot of a spreadsheet — and the agent declines.
A reasonable starting threshold is a cosine similarity around 0.75 against your catalog embeddings — then tune it on real uploads. Set it too low and the agent tries to "match" a houseplant to a hoodie; set it too high and it rejects legitimate but badly-lit photos. The failure mode you want is the safe one: when in doubt, decline and ask for a clearer product shot. A polite "I couldn't recognise a product in that image — mind sending a photo of the item itself?" beats a confident wrong answer every time.
Why the gate matters more than the model
| Without a gate | With a gate |
|---|---|
| Describes any image — including private photos and memes | Only reasons about images that resemble your products |
| Hallucinates a "match" for things you don't sell | Says "not in our catalog" and moves on |
| Burns tokens and latency on every upload | Drops junk with one cheap vector lookup |
| Becomes a general image chatbot bolted to your store | Stays a shopping assistant that happens to accept photos |
Job 1 — "Is this one of ours?"
This is visual search, and it runs on the same machinery as the gate. Encode the uploaded image into a vector with an image embedding model, then find the nearest product vectors in your store's catalog collection. The top hit — if it clears the threshold — is your answer; the next few are useful for "or did you mean one of these?"
Two details make this production-grade rather than a demo. First, the results are real SKUs with current price and stock, not a model's description of what it thinks the photo shows. Second, you return a ranked shortlist, not a single guess — shoppers are forgiving of "here are the three closest" and unforgiving of a wrong single answer stated with confidence.
Job 2 — "Is it broken?"
Post-purchase, the most valuable image a shopper can send is a photo of something wrong. Because your catalog already holds the pristine reference image for every SKU, the agent has a baseline to compare against. Once the item is matched (Job 1), a condition check looks for the difference between "how we ship it" and "what arrived": a crack, a tear, a stain, a missing accessory, the wrong colour.
This is where scoping pays off twice. A general model asked "is this damaged?" has no idea what undamaged looks like. An agent that knows the exact product can say "the strap is torn — that's not how this ships" and route straight into your returns or warranty flow. Keep a human in the loop for the actual refund decision; let the agent do the triage that otherwise eats a support queue.
Job 3 — "Find me something like it"
Sometimes the photo is not one of yours — and that is a sales opportunity, not a dead end. If the image passes the gate (it clearly depicts a product in a category you sell) but doesn't match a specific SKU closely, pivot to discovery: return the visually and categorically nearest items you do stock.
The line between "no match" and "here are similar options" is the same threshold, read differently. A near-zero score means out-of-catalog → decline. A middling score means in-category but not this exact item → "we don't carry that one, but these three are close." That single number drives all three outcomes:
| Best catalog match | Interpretation | Agent behaviour |
|---|---|---|
| High (≈ 0.85+) | This is one of ours | Show the SKU · add to cart · check condition |
| Medium (≈ 0.75–0.85) | Same category, different item | "Not that exact one — here are similar options" |
| Low (< 0.75) | Not plausibly in the catalog | Decline · ask for a product photo |
How the matching actually works
Under the hood there is no magic — just embeddings and a vector search, the same pattern that grounds text-based product search. The build has three moving parts:
- Index your catalog once. Run every product image through an image embedding model and store the vectors in a per-store collection. This is a background job that re-runs when the catalog changes — exactly like a text search index.
- Embed the upload at request time. When a shopper sends a photo, encode it with the same model into the same space, so distances are comparable.
- Search and threshold. Nearest-neighbour lookup returns the closest SKUs with scores; the score decides match vs. similar vs. decline.
# Sketch of the request-time path
vec = image_embed(upload) # same model as the index
hits = catalog.search(vec, top_k=5) # per-store vector collection
top = hits[0]
if top.score < 0.75:
return decline("I couldn't spot a product in that image.")
elif top.score < 0.85:
return similar_items(hits) # in-category, not this SKU
else:
return matched_product(top) # add-to-cart, condition checkEverything downstream of the match — add to cart, check stock, start a return — is the store's existing toolkit. The image step just gives those tools a new way in. A photo becomes a SKU; a SKU is something your agent already knows how to act on.
A worked example
One upload, all four behaviours in a single conversation:
- Shopper sends a photo of a navy hoodie.
- Gate → best catalog match 0.94 → process.
- Agent: "That's our Cozy Hoodie in Navy — $59, in stock in M and L. Want me to add one?"
- Shopper: "Mine came with a hole in the sleeve." (sends a second photo)
- Agent compares against the catalog image → flags a torn sleeve → "That's a defect, not how it ships. I can start a warranty return — want me to?"
- Shopper: "Also, anything like this lamp?" (sends a photo of a lamp you don't sell)
- Gate → 0.14, no lighting category → decline: "We don't carry lighting, so I can't match that one."
- Shopper: (sends a selfie by accident)
- Gate → 0.08 → "I couldn't find a product in that image — send a photo of the item and I'll take a look."
Match, condition, decline, decline — the agent never once described the shopper's face or guessed at the lamp. That restraint is the feature.
Guardrails worth building in from day one
- Decline is the default. Below the threshold, do nothing but ask for a better product photo. Never narrate arbitrary images.
- Don't retain uploads longer than the turn. A shopper's photo may contain faces, homes, or other PII. Process it, answer, and discard — don't quietly build a photo archive.
- Keep money moves human-gated. An image can trigger a return; a person should still approve the refund.
- Log the score, not the image. For debugging, keep the match score and the decision — that's enough to tune the threshold without hoarding shopper photos.
- Make it operator-controllable. Image handling should be a toggle a store owner can turn off, and the threshold should be adjustable per store — a boutique with 40 SKUs tunes differently than a catalog of 40,000.
How this fits WisWes
WisWes already grounds every answer in a per-store catalog: product search, cart actions, and policy answers all run against your live inventory rather than a model's training data. Catalog-scoped image processing is the same idea with a camera pointed at it. The store-level vector collection that powers text search is exactly the index an uploaded photo gets compared against — which is what makes the "ignore anything we don't sell" behaviour fall out naturally rather than needing a bolt-on content filter.
The practical takeaway holds whether you build it yourself or turn it on: an image feature for a store agent is not "let the AI see." It is "let a photo become a SKU, and refuse everything that can't." Build the gate first, keep the jobs narrow, and the agent stays a shopping assistant — one that now takes pictures.