How Pinterest’s visual search went from a moonlight project to a real-world search engine

Sometime around 2013 and 2014, deep learning was going through a revolution that required pretty much everyone to reset their expectations as to how things worked, and leveled the playing field for what people were doing with computer vision. At least thats the philosophy that Pinterest engineer Andrew Zhai and his team have taken, because around that time he and a few others began working on some internal moonlightproject to build computer vision models within Pinterest. Machine learning tools and techniques had really been around for some time, but thanks to revelations in how deep learning worked and the increasing use of GPUs, the companywas able to take a fresh look at computer vision and see how it would work in the context of Pinterest. From a computer vision perspective we have a lot of images where visual search makes sense, Zhai said. Theres this product/data-set fit. Users that come to Pinterest, theyre often in this visual discovery experience mode. We were in the right place at the right time where the technology was in the middle of a revolution, and we had our data set, and were very focused on iterating as quickly as we can and get user feedback as fast as we can. The end result was Lens, a product Pinterest launched earlier this month that allows users to basically point at an object in the real world with their camera and return search results for Pinterest. While a semi-beta was launched last year, Lens was the result of years of scrapped prototypes and product experimentation that eventually produced something that would hopefully turn the world collectively into a bunch of pins that were searchable through your camera, creative lead Albert Pereta said. When a user looks at something through Lens, Pinterests visual detection kicks in and determines what objects are in the photo. Pinterests technology can then frame the image around, say, a chair, and use that to ask a query using Pinterests existing search technology. It uses certain heuristics, like a confidence score of what kind of object it is, and the context of it like whether it is the dominant object, the largest one, the one the most in focus or something along the lines. Zhai said part of the priority was leveraging as much of Pinterests existing technology, like search, to build its visual search products.

Read Full Article at

Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,