MM

Leveraged wisely, new datasets can inspire new multimedia methods and algorithms, as well as catalyze innovations in how their efficacy, efficiency, and generalizability can be evaluated. The availability of very large multimedia datasets like the Yahoo-Flickr Creative Commons 100 Million (YFCC100M)---which spans 99.2 million images and 0.8 million videos---has offered unique opportunities for advancing the state of the art in multimedia processing, analysis, search, and visualization.

The Multimedia Commons Initiative has been developing a community around the YFCC100M, including associated annotation and evaluation efforts. Computed features, human-generated annotations, and analysis tools have been released into the public domain, hosted via Amazon's Public Data Sets program. In addition to research in several multimedia subfields, including computer vision, image processing, and video content analysis, the YFCC100M and Multimedia Commons resources have been used in various competitions and benchmarks, such as the MediaEval Placing Task and the ACM Multimedia Grand Challenge competition.

As use of the YFCC100M and the Multimedia Commons resources broadens across the multimedia community, the MMCommons'16 workshop offers an opportunity for participants to share new research results, compare approaches, and coordinate efforts to maximize the scientific benefit of the initiative. In particular, this massive, open dataset challenges us to pursue some important "meta-research" questions, such as how to measure the scalability, generalizability, and reproducibility of methods across datasets; whether we need to rethink our evaluation paradigms as the field moves in new directions, in particular to better approximate "in the wild" conditions; and how annotation strategies affect the impact of benchmarks and data challenges using that data.

Participants in MMCommons'16 will share novel research using the YFCC100M dataset, particularly focusing on solving multimedia problems in ways that were not possible with previous data collections. Themes that will receive particular focus in the paper sessions include improving the understanding and representation of multimedia content; leveraging user-supplied metadata to bootstrap analysis and benchmarking; enabling web-scale distributed search and indexing; and defining strategies for performance evaluation, with an eye towards maximizing generalizability.

These themes will also be explored in special sessions and discussions on dataset bias, reproducibility, and task-driven annotation. The workshop will kick off with a keynote by Roeland Ordelman on the importance of the benchmark development process in shaping our understanding of the research problems being addressed, with examples from audiovisual search evaluations.