Image-similarity scoring with CLIP for marketplace listings

A listing can omit the brand name and still reuse the brand’s product photography, sometimes literally the same JPEG. This piece explains how Brand Protector records CLIP-based whole-image similarity on configured marketplace scans, what the signal can and cannot support, and why inference runs in a separate Cloud Run service.

The signal: visual replicas with sanitised text

The useful case is narrow and important: the listing title is generic, while the listing reuses a brand’s near-duplicate product photo. The image comparison gives a reviewer evidence that title-only matching does not. Brand Protector has not validated this whole-image score for a product composited into a materially different or AI-generated scene, and it is not proof that the physical good is counterfeit.

Why CLIP, not perceptual hashing

Two technical options for “is this image similar to a reference image”:

• Perceptual hashing (pHash, dHash) — fast, local, deterministic. Catches exact-copy and minor crop / color variations. Misses anything more complex than that.
• CLIP embeddings — a vision-language model that maps images and text into a shared 512-D space. We use whole-image cosine for near-duplicate photo comparison. It costs a service call and is not assumed to recognize the same product in a different composition.

We landed on CLIP for this evidence path. Reference embeddings are cached in the service per tenant. The separate cross-listing clusterer has a 30-day embedding cache at tenants/{tid}/embedding_cache/{sha}; per-listing reference comparisons still make one logical service call for each eligible image.

The architecture

Three pieces:

• imgsim_service — a long-running Cloud Run service that hosts the CLIP model in memory (~600MB resident). Receives requests over HTTPS with an ID-token auth check. The model loads once at cold start; subsequent requests are pure inference. We run it under a dedicated service account (imgsim-runner@) with read-only Firestore access — separation of concerns from the scanners that call it.
• image_clusterer_job — a nightly Cloud Run Job that pulls all detection images from the prior day, embeds them via imgsim_service, clusters them by cosine-similarity threshold (0.92 default per tenant, configurable). Same-counterfeit-photo-different-seller patterns surface here.
• per-scanner inline call — configured marketplace scanners (Amazon, eBay, Walmart, Google Shopping, Apify) call imgsim_service inline while building evidence. Per-tenant opt-in: only fires for tenants who’ve uploaded reference brand assets. Graceful fallback if the service is unreachable — the scanner persists no numeric image score rather than inventing one or failing the run.

Cost guards (real numbers)

Image-similarity isn’t free. Per-tenant per-day caps are the structural defense against runaway cost (a tenant who suddenly uploads 10,000 reference images and asks us to score against the entire Amazon US catalog will burn the per-call budget). Default cap:

• 1,000 imgsim calls per tenant per day (overrideable per tenant via tenants/{tid}/usage_caps/imgsim).
• Separate clusterer cache — cross-listing clustering reuses image embeddings for 30 days; this does not make per-listing reference-comparison calls free.
• Global kill switch at system_config/imgsim_kill_switch — single operator-flippable doc that halts ALL imgsim calls platform-wide. Used during incident response.

No confidence threshold for the reference score

Brand Protector stores the computed reference score as raw reviewer evidence. It does not change detection confidence or trigger an enforcement branch while it remains unvalidated against labeled detections. A computed 0.0 is a real measurement; a missing score means the comparison was not computed.

The separate cross-listing clusterer has an operational grouping threshold, configurable at tenants/{tid}/config/image_clustering.threshold . That threshold groups already-detected listings; it does not turn a reference score into counterfeit evidence or affect marketplace confidence.

What we’d build differently next time

• Labelled validation before scoring. CLIP is general-purpose. Any future use in the confidence rubric must start with labeled Brand Protector detections and measured error rates, not a plausible-looking cutoff.
• Vector DB rather than per-tenant embedding cache. Today the cache is a Firestore document per image hash. Works at current scale; will become a read-amplification problem at 100+ tenants. Vertex AI Vector Search is the obvious upgrade path.
• Per-region CLIP models. The base CLIP model is trained on English-leaning web data. Performance on Asian-marketplace listings (Mercari JP, Shopee SE Asia) is slightly worse. A multilingual variant or a per-region fine-tune is the medium-term move.

If you’re building something similar and want to compare notes, engineering at brandprotector dot io.