Li Yan Kim So-Eon Park Seong-Bae Han Soyeon Caren

Abstract

Although Large Language Models (LLMs) can generate coherent text, they oftenstruggle to recognise user intent behind queries. In contrast, Natural LanguageUnderstanding (NLU) models interpret the purpose and key information of userinput for responsive interactions. Existing NLU models typically map utterancesto a dual-level semantic frame, involving sentence-level intent (SI) andword-level slot (WS) labels. However, real-life conversations primarily consistof multi-turn dialogues, requiring the interpretation of complex and extendedexchanges. Researchers encounter challenges in addressing all facets ofmulti-turn dialogue using a unified NLU model. This paper introduces MIDAS, anovel approach leveraging multi-level intent, domain, and slot knowledgedistillation for multi-turn NLU. We construct distinct teachers for SIdetection, WS filling, and conversation-level domain (CD) classification, eachfine-tuned for specific knowledge. A multi-teacher loss is proposed tofacilitate the integration of these teachers, guiding a student model inmulti-turn dialogue tasks. Results demonstrate the efficacy of our model inimproving multi-turn conversation understanding, showcasing the potential foradvancements in NLU through multi-level dialogue knowledge distillation. Ourimplementation is open-sourced on https://github.com/adlnlp/Midas.