Md. Ashraful Islam Mohammed Eunus Ali Md Rizwan Parvez

Abstract

Large Language Models (LLMs) have made significant strides in code generationand problem solving. Current approaches employ external tool-based iterativedebuggers that use compiler or other tool-based runtime feedback to refinecoarse programs generated by various methods. However, the effectiveness ofthese approaches heavily relies on the quality of the initial code generation,which remains an open challenge. In this paper, we introduce CodeSim, a novelmulti-agent code generation framework that comprehensively addresses the stagesof program synthesis-planning, coding, and debugging-through a human-likeperception approach. As human verifies their understanding of any algorithmsthrough visual simulation, CodeSim uniquely features a method of planverification and internal debugging through the step-by-step simulation ofinput/output. Extensive experiments across seven challenging competitiveproblem-solving and program synthesis benchmarks demonstrate CodeSim'sremarkable code generation capabilities. Our framework achieves newstate-of-the-art (pass@1) results-(HumanEval 95.1%, MBPP 90.7%, APPS 22%, andCodeContests 29.1%). Furthermore, our method shows potential for even greaterenhancement when cascaded with external debuggers. To facilitate furtherresearch and development in this area, we have open-sourced our framework inthis link (https://kagnlp.github.io/codesim.github.io/).