Style-aware Mid-level Representation for Discovering
Visual Connections in Space and Time
People
Yong Jae Lee
Alexei A. Efros
Martial Hebert


Abstract
We present a weakly-supervised visual data mining approach that discovers connections between recurring midlevel visual elements in historic (temporal) and geographic (spatial) image collections, and attempts to capture the underlying visual style.  In contrast to existing discovery methods that mine for patterns that remain visually consistent throughout the dataset, our goal is to discover visual elements whose appearance changes due to change in time or location; i.e., exhibit consistent stylistic variations across the label space (date or geo-location).  To discover these elements, we first identify groups of patches that are stylesensitive.  We then incrementally build correspondences to find the same element across the entire dataset.  Finally, we train style-aware regressors that model each element’s range of stylistic differences.  We apply our approach to date and geo-location prediction and show substantial improvement over several baselines that do not model visual style.  We also demonstrate the method’s effectiveness on the related task of fine-grained classification.


Paper & Presentation



ICCV Paper (pdf)
Supplemental material (pdf)
Talk slides (pptx)
Video Presentation
Citation
Y. J. Lee, A. A. Efros, and M. Hebert.  Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time.  Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013.


Code
Code can be downloaded here.


Data
Our historical car dataset (CarDb as referenced in the paper) can be downloaded here.

The database contains historical car images from 1920s to 1990s crawled from cardatabase.net. There are 10130 training and 3343 test images. Note that the entire dataset is 2.8 GB.

Annotations include: model, manufacturer, year, and the date, city, and country that the photo was taken. These can be accessed in matlab by:
>> imgInfo = imfinfo(imgfile); % imgfile is the file name of the image; e.g., '1.jpg'
>> annotations = imgInfo.Comment; % get annotations 
>> year = str2double(annotations{3}(6:end)); % get the model year of the car

 
Acknowledgements
This work was supported in part by Google, ONR MURI N000141010934, and the Intelligence Advanced Research Projects Activity (IARPA) via Air Force Research Laboratory.  The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation thereon.  Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, AFRL or the U.S. Government.