We study verbs in image--text corpora, contrasting caption corpora, where texts are explicitly written to characterize image content, with depiction corpora, where texts and images may stand in more general relations. Captions show a distinctively limited distribution of verbs, with strong preferences for specific tense, aspect, lexical aspect, and semantic field. These limitations, which appear in data elicited by a range of methods, restrict the utility of caption corpora to inform image retrieval, multimodal document generation, and perceptually-grounded semantic models. We suggest that these limitations reflect the discourse constraints in play when subjects write texts to accompany imagery, so we argue that future development of image--text corpora should work to increase the diversity of event descriptions, while looking explicitly at the different ways text and imagery can be coherently related.
If you use this data or code, please cite the following paper:
Malihe Alikhani and Matthew Stone. "Caption" as a Coherence Relation: Evidence and Implications. In Proceedings of NAACL19, Workshop on Shortcomings in Vision and Language.
BibTeX entry:
@inproceedings{alikhani2018arrows,
title={Arrows are the Verbs of Diagrams},
author={Alikhani, Malihe and Stone, Matthew},
booktitle={Proceedings of the 27th International Conference on Computational Linguistics},
pages={3552--3563},
year={2018}
}