Abstract

Utilizing Graphic User Interface (GUI) for human-computer interaction isessential for accessing a wide range of digital tools. Recent advancements inVision Language Models (VLMs) highlight the compelling potential to developversatile agents to help humans finish GUI navigation tasks. However, currentVLMs are challenged in terms of fundamental abilities (OCR and grounding) andGUI knowledge (the functions and control methods of GUI elements), preventingthem from becoming practical GUI agents. To solve these challenges, wecontribute GUICourse, a suite of datasets to train visual-based GUI agents fromgeneral VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR andgrounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChatdatasets to enrich their knowledge of GUI components and interactions.Experiments demonstrate that our GUI agents have better performance on commonGUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1Bparameters) can still work well on single-step and multi-step GUI tasks.Finally, we analyze the different varieties in the training stage of this agentby ablation study. Our source codes and datasets are released athttps://github.com/yiye3/GUICourse.