Visual Image Search Relevance

Marjory Meechan - April 3, 2008

Image search relevance is not something that we think about very often, but as Karen Umpierre noted recently, image search is getting smarter. Google can look inside images and tell the difference between faces and non-faces … well, usually. Occasionally a non-face slips in but it’s pretty impressive. This brings up the question of the future of search. Will there come a day when we won’t have to put 200 words of text on a page for it to rank well for relevance in a search query? The answer may be yes because new search engine technologies are emerging that take search into the images themselves. Instead of image search using text, we have visual search using images.

Maybe the most exciting new visual search idea is Nokia’s wireless Point & Find where a user just takes a picture and gets relevant information about the image based on information on the internet — including location, price, etc. While this is very much an emerging technology, it could very well change the way people shop. Imagine seeing some shoes you like at a party and being able to find them using your wireless device’s built-in search. Nokia claims this technology is at least three years away, but in the meantime, there are some visual search engines available now.

One that deserves special mention is Like.com because it is the first engine that is actually using visual search to help shoppers find what they want. For example, let’s say I want to buy some shoes. I go to the shoes section of Like.com and find a style that I like. Then I just click on the button marked Visual Search. This gives me a display of shoes that all resemble the shoes that I chose grouped by “overall matches”, “style matches”, “brand matches” and “color matches”. The most interesting aspect is the opportunity to refine my search by selecting a portion of the original image.

I just draw a box around the part of the image that I most want to match and Like.com gives me a new result. Like.com has just recently launched a new site after a lengthy beta period so we’re interested in seeing how well the new site is received. Will shoppers take to it? One problem may be that, as we noted with Google’s face search, the results aren’t perfect. For example, one of the top ten results for my search for matches to that spiky heel was this wedge heeled number. Of course, it is a high wedge — sort of spiky as wedges go and roughly the same shape as the heel of the shoe I selected so maybe this is just the visual search algorithm’s interpretation of my query. This brings me to the point of my post today — image relevance.

Just because a search engine uses a different point of reference (shape, color, style as opposed to words) doesn’t mean that we can stop worrying about relevance. If visual search becomes more prevalent, search engine optimization may actually become even more complicated with considerations of image quality, angle and other visually based factors coming to the fore. Furthermore, even Like.com relies on linguistic navigation to subsidize their product search and this is unlikely to change. So, if you were waiting around for the image search technology to get better before you optimized your site for keyword relevance, we would recommend that you not wait because whether it’s images or text, making your pages relevant to search queries will always matter.