How Etsy taught a model what "looks like this" means

Some things are hard to type. "Searching by image is being rapidly adopted throughout the e-commerce industry, and nowhere does it make more sense than on Etsy, where the uniqueness of our sellers’ creations can’t always be expressed easily with words." On 26 May 2023 Eden Dolev and Alaa Awad described what Etsy built for that: "Tap the camera icon in the search bar to take a picture, and in a fraction of a second we’ll surface visually similar results from our inventory of nearly 100 million listings." The lesson is the three tries it took to teach a model what "similar" means.

The problem: listing photos are not shopper photos

Search by photo turns every picture into an embedding, a list of numbers, so that pictures that look alike sit close together. But the query is not a listing photo: "we're not expecting users to search from Etsy listing photos - the whole point of this is that the user is holding a camera."

The two kinds of photo differ. "Photos uploaded by sellers are typically high quality, have professional lighting, and are taken over a white or premeditated background. But photos from a user's phone might be blurry, or poorly lit, or on a diversity of backgrounds that distract from the object the user is searching for."

And "similar" means two things at once: the same kind of item and the same look. For a photo of a red mug with white dots, a good answer is a red dotted mug. Any mug is wrong, and so is any red dotted thing.

How they built it

Try one: sort photos into categories. Etsy trained a model to predict a listing's category and took its inner layer as the embedding. "Not surprisingly, our tests showed that these embeddings were particularly useful in surfacing listings from the same taxonomy." The look was off, though: "items were surfaced that didn't match well with the query image in color, material or pattern."

Try two: triplet loss. Triplet loss trains on three photos at a time: an anchor, a positive that should land close to it, and a negative that should land far away. "In our case, we used pairs of images from the same item as the anchor and positive examples, and an image from a different listing as the negative example." The look came back: "listings were now visually cohesive, displaying similar colors and patterns." The kind slipped: "these embeddings lacked categorical accuracy compared to the classification approach." Training was harder to watch, too: triplet metrics track distances, not accuracy. The team's verdict: "Taxonomy is not something we can tolerate our model being sloppy about."

Try three: many labels at once. Back to classification, with several heads on one shared model: "Item category (e.g. accessories, home & living), fine-grained item category (belt buckles, dining chairs), primary color, and other item attributes." Category keeps the kind right; colour and attributes bring the look. The base is EfficientNet, a pretrained image model; only its last layers and the heads were trained on Etsy data, and the embedding is read just before the heads.

Seller labels are uneven: "some optional seller input attributes, such as item color and material, can be sparse." So every task got its own dataset, and a sampler "combines an equal number of examples from each dataset into every minibatch." The "loss from each classification head is calculated only for examples from the head’s corresponding source dataset." A listing with no colour label still teaches the category head.

The missing piece: buyer photos. Etsy already had phone photos of its own products. Reviews "can have photos attached: photos taken by buyers, often with their phone cameras, very much like the images we expect to see used when searching by image." They became one more task. "And indeed with this new component in place we saw significant improvement in the model’s ability to surface visually relevant results."

Index once, embed the photo live. In batch, the main photo of each of "nearly 100M active listings" becomes an embedding in an approximate nearest neighbour index built with an inverted file (IVF), which groups listings into clusters. At query time "we only look at the nearest subset of clusters to the query embedding, which greatly reduces search latency while only marginally impairing accuracy." The shopper's photo cannot be computed ahead of time, and a model this size is slow on a CPU, so Etsy worked with its ML platform team "to bring the first use case of real-time GPU inferencing at Etsy."

What it delivered

A camera search over the whole catalogue, and no numbers: no accuracy, no latency, no A/B test, no effect on sales. The three tries are compared by eye. These are Etsy-reported claims; read the post as a design, not as proof.

Some numbers came a year later. A June 2024 follow-up by the same two authors and Denisa Roberts says "The same efficient embedding model backed the first search-by-image shopping experience at Etsy." It was EfficientNet B0, "with ~5M parameters and around 1.7ms latency on iPhone 12", with a 256-number embedding. Its A/B results are from other places: visually similar ads gave "a +1.92% increase in ad return-on-spend on iOS and a +1.18% increase in post-click purchase rate on Android". There is still no figure for photo search itself.

What has changed since

Etsy's launch news of November 2022 says the beta went to "all iOS app users", with Android "soon"; we could not confirm it got there.

Since then Etsy's AI story has moved to words and to other companies' apps. Its September 2025 partnership post with OpenAI says that for a hat seen on social media, "AI-powered visual search will connect them to similar items on Etsy." In January 2026 it announced agentic shopping with Google and Microsoft, per its Q4 2025 results, and in May 2026 an Etsy app in ChatGPT went live in beta. A May 2026 engineering post names a gap: "A product’s visual style might be clear in its images, but not all of our systems can process images."

The parts are now off the shelf. Etsy trained its own heads on labelled data. Open image-text models such as SigLIP 2, Apache-2.0 on its model card, put photos and words in one embedding space, so "like this, but green" becomes one query. Vector search is a library such as Faiss or a Postgres extension such as pgvector.

The big platforms moved to a camera that keeps looking. Amazon's Lens Live, from September 2025, uses "an accurate, lightweight computer vision object detection model running on-device to identify products in real time as customers pan their cameras across scenes".

Why it matters for your business

Indonesian shoppers already know the camera icon. Shopee's help page explains it: tap the camera in the search bar, take a photo or pick one from the gallery, and see similar products, one image per search.

Etsy's lesson travels: a batik shirt shot on a bed in a dim room looks nothing like the catalogue's studio shot. Train and test with photos like the ones buyers take.

Two rules apply. The data protection law, UU 27/2022, gives "gambar wajah" (a face image) as its example of biometric data, and a shopper's photo can contain a face. It asks for a legal basis, explicit consent first on the list, and deletion after the retention period. For marketplaces, Permendag 19/2026, in force since June 2026 and replacing Permendag 31/2023, asks search, recommendation and ranking to start by "mengutamakan penayangan Produk Dalam Negeri", domestic products first. Merchants must be told before ranking changes, and a marketplace that uses AI needs a complaint route. Whether photo search counts is a question for counsel. That is our reading as engineers, not legal advice.

How we would build it today

1. Start from an open image-text model of the SigLIP 2 class. Fine-tune only if a test on your own catalogue shows a gap.
2. Build the test set from buyer photos first. A few hundred real phone photos, with a person marking the acceptable matches, split by lighting and clutter. The one gain Etsy calls significant came from this side.
3. Embed on the server, shrink on the phone. Etsy's shape: one model version, any phone. Shrink the photo to tens of kilobytes and let the shopper crop it. Try on-device embedding later, measured on three mid-range Android phones.
4. Index by size. pgvector up to a few hundred thousand items; Faiss, Qdrant or Milvus past that. Check recall against an exact search before you choose, and index more than the main photo.
5. Re-rank with business facts. Stock, price band, seller quality, and on a marketplace the domestic-first rule.
6. Keep words in the loop. A box to refine the result ("yang ini, tapi warna hijau") and a "bukan ini" button, which is also a complaint route and free labelled data.
7. Do not keep the photo. Embed it and drop it; ask consent before keeping any, blur faces on the phone, delete on request.
8. Measure what Etsy did not publish. Recall on buyer photos, photo search to cart against text search, and how often a photo search is followed by a typed one.

For a single brand with a few thousand products, steps 1, 2 and 4 with pgvector are a fair start. Add the re-rank and on-device embedding when the traffic asks for them.