Preview: Example Images in MetaShift
For each image class (e.g. Dogs), the MetaShift contains different sets of dogs under different contexts to represent diverse data distributions. The contexts include presence/absence of other objects (e.g. dog with frisbee). Contexts can also reflect attributes (e.g. black dogs) and general settings (e.g. dogs in sunny weather). These concepts thus captures diverse and real-world distribution shifts. We list the object presense, attribute and general location contexts below.
Presence/Absence of Other Objects
MetaShift considers the presence/absence of 1,702 distinct object classes. We use the pre-processed and cleaned version of Visual Genome by Hudson and Manning which contains 113,018 distinct images across 1,702 object classes. MetaShift is a collection of 12,868 sets of natural images from 410 classes. Each class has 31.4 subsets, and each subset has 200.4 images on average.
Different subsets may share common images—e.g. a dog image can have both grass and frisbee would occur in both emph{dog with grass} and emph{dog with frisbee}. Therefore, the total number of subsets times the average number of images in each subset is much larger than the original dataset. The distance between subsets capture this overlap, but also broader similarities.
General Contexts (Locations and Weather)
MetaShift covers 37 general contexts including location (e.g., indoor, outdoor, ocean, snow) and weather (e.g., couldy, sunny, rainy).
The attributes and their ontology in MetaShift are as follows:
GENERAL_CONTEXT_ONTOLOGY = {
'indoor/outdoor': ['indoors', 'outdoors'],
'weather': ['clear', 'overcast', 'cloudless', 'cloudy', 'sunny', 'foggy', 'rainy'],
'room': ['bedroom', 'kitchen', 'bathroom', 'living room'],
'place': ['road', 'sidewalk', 'field', 'beach', 'park', 'grass', 'farm', 'ocean', 'pavement',
'lake', 'street', 'train station', 'hotel room', 'church', 'restaurant', 'forest', 'path',
'display', 'store', 'river', 'yard', 'snow', 'airport', 'parking lot']
}
Object Attributes
MetaShift covers attributes including activity (e.g., sitting, jumping), color (e.g., orange, white), material (e.g., wooden, metallic), shape (e.g., round, square) and so on.
The general contexts and their ontology in MetaShift are as follows:
ATTRIBUTE_CONTEXT_ONTOLOGY = {
'darkness': ['dark', 'bright'], 'dryness': ['wet', 'dry'],
'colorful': ['colorful', 'shiny'], 'leaf': ['leafy', 'bare'],
'emotion': ['happy', 'calm'], 'sports': ['baseball', 'tennis'],
'flatness': ['flat', 'curved'], 'lightness': ['light', 'heavy'],
'gender': ['male', 'female'], 'width': ['wide', 'narrow'],
'depth': ['deep', 'shallow'], 'hardness': ['hard', 'soft'],
'cleanliness': ['clean', 'dirty'], 'switch': ['on', 'off'],
'thickness': ['thin', 'thick'], 'openness': ['open', 'closed'],
'height': ['tall', 'short'], 'length': ['long', 'short'],
'fullness': ['full', 'empty'], 'age': ['young', 'old'],
'size': ['large', 'small'], 'pattern': ['checkered', 'striped', 'dress', 'dotted'],
'shape': ['round', 'rectangular', 'triangular', 'square'],
'activity': ['waiting', 'staring', 'drinking', 'playing', 'eating', 'cooking', 'resting',
'sleeping', 'posing', 'talking', 'looking down', 'looking up', 'driving',
'reading', 'brushing teeth', 'flying', 'surfing', 'skiing', 'hanging'],
'pose': ['walking', 'standing', 'lying', 'sitting', 'running', 'jumping', 'crouching',
'bending', 'smiling', 'grazing'],
'material': ['wood', 'plastic', 'metal', 'glass', 'leather', 'leather', 'porcelain',
'concrete', 'paper', 'stone', 'brick'],
'color': ['white', 'red', 'black', 'green', 'silver', 'gold', 'khaki', 'gray',
'dark', 'pink', 'dark blue', 'dark brown',
'blue', 'yellow', 'tan', 'brown', 'orange', 'purple', 'beige', 'blond',
'brunette', 'maroon', 'light blue', 'light brown']
}
More Examples of Meta-graphs
Citation
@InProceedings{liang2022metashift,
title={MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts},
author={Weixin Liang and James Zou},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=MTex8qKavoS}
}