Chuofan Ma Yi Jiang Jiannan Wu Zehuan Yuan Xiaojuan Qi

Abstract

We introduce Groma, a Multimodal Large Language Model (MLLM) with groundedand fine-grained visual perception ability. Beyond holistic imageunderstanding, Groma is adept at region-level tasks such as region captioningand visual grounding. Such capabilities are built upon a localized visualtokenization mechanism, where an image input is decomposed into regions ofinterest and subsequently encoded into region tokens. By integrating regiontokens into user instructions and model responses, we seamlessly enable Gromato understand user-specified region inputs and ground its textual output toimages. Besides, to enhance the grounded chat ability of Groma, we curate avisually grounded instruction dataset by leveraging the powerful GPT-4V andvisual prompting techniques. Compared with MLLMs that rely on the languagemodel or external module for localization, Groma consistently demonstratessuperior performances in standard referring and grounding benchmarks,highlighting the advantages of embedding localization into image tokenization.Project page: https://groma-mllm.github.io/.