Google AI open-sourced mood board search, a new ML-powered tool for subjective or conceptual queries over images.
Mood board search helps users to define conceptual and subjective queries like peaceful, and beautiful, over images. Advances in using deep learning in computer vision enabled engineers and researchers to provide different functionalities such as similar image search, object detection, tagging, etc. One of the main challenges in this area is how to define and query images with conceptual intent. Mood board search helps people train and personalize the deep learning model in a way that they see the world. The following snapshot shows how an artist sees the world by categorizing images into different artistic concepts.
personal classification of the images into abstract and subjective concepts
In mood board search, researchers used pre-trained computer vision models like GoogLeNet and MobileNet, and a machine learning approach called Concept Activation Vectors (CAVs).
CAV is a technique to measure how a trained model is sensitive to the concept presented by the user. The following picture shows how CAV or tested CAV is working.
Getting the tested CAV score which quantifies the sensitivity of the classifier to the concept
As an example, a deep learning model trained to classify images as zebra or not zebra. We want to quantify how important the stripe concept is for the classifier. By simply running TCAV and getting the score we can answer the question. CAV is used as one of the general techniques for the explainability of deep learning models. As mentioned in the blog post :
In Mood Board Search, we use CAVs to find a model's sensitivity to a mood board created by the user. In other words, each mood board creates a CAV — a direction in embedding space — and the tool searches an image dataset, surfacing images that are the closest match to the CAV. However, the tool takes it one step further, by segmenting each image in the dataset in 15 different ways, to uncover as many relevant compositions as possible.
Working with the mood board search GUI is straightforward. As it is explained in the blog post :
To get started, simply drag and drop a small number of images that represent the idea you want to convey. Mood Board Search returns the best results when the images share a consistent visual quality, so results are more likely to be relevant with mood boards that share visual similarities in color, pattern, texture, or composition.
Some of the design studios like morrama got excited about this tool and tweeted :
Amazing work, can't wait to be able to play with it.
Google AI open-sourced the code for researchers and developers for more contributions in this area. Also, there is an experimental app by design invention studio Nord Projects, which uses mood board search.