Zeng Zijie ; Liu Shiqi ; Sha Lele ; Li Zhuang ; Yang Kaixun ; Liu Sannyuya ; Gašević Dragan ; Chen Guanliang

Abstract

This study explores the challenge of sentence-level AI-generated textdetection within human-AI collaborative hybrid texts. Existing studies ofAI-generated text detection for hybrid texts often rely on synthetic datasets.These typically involve hybrid texts with a limited number of boundaries. Wecontend that studies of detecting AI-generated content within hybrid textsshould cover different types of hybrid texts generated in realistic settings tobetter inform real-world applications. Therefore, our study utilizes theCoAuthor dataset, which includes diverse, realistic hybrid texts generatedthrough the collaboration between human writers and an intelligent writingsystem in multi-turn interactions. We adopt a two-step, segmentation-basedpipeline: (i) detect segments within a given hybrid text where each segmentcontains sentences of consistent authorship, and (ii) classify the authorshipof each identified segment. Our empirical findings highlight (1) detectingAI-generated sentences in hybrid texts is overall a challenging task because(1.1) human writers' selecting and even editing AI-generated sentences based onpersonal preferences adds difficulty in identifying the authorship of segments;(1.2) the frequent change of authorship between neighboring sentences withinthe hybrid text creates difficulties for segment detectors in identifyingauthorship-consistent segments; (1.3) the short length of text segments withinhybrid texts provides limited stylistic cues for reliable authorshipdetermination; (2) before embarking on the detection process, it is beneficialto assess the average length of segments within the hybrid text. Thisassessment aids in deciding whether (2.1) to employ a text segmentation-basedstrategy for hybrid texts with longer segments, or (2.2) to adopt a directsentence-by-sentence classification strategy for those with shorter segments.