Hu Yaojie ; Zhou Qiang ; Chen Qihong ; Li Xiaopeng ; Liu Linbo ; Zhang Dejiao ; Kachroo Amit ; Oz Talha ; Tripp Omer

Abstract

We introduce QualityFlow, a dynamic agentic workflow for program synthesis.Given the English description of a programming problem and a set of unit tests,the model's goal is to synthesize the correct program that solves the problemand passes the tests. QualityFlow includes large language model (LLM) agentsresembling a software development team, including code generation, testing, andself-debugging. We propose the LLM Quality Checker, which explicitly "imagines"whether the synthesized programs' execution would conform to the unit tests.The Quality Checks dynamically control the workflow, including actions tosubmit the final answer, clarify the problem statement, and revert previousworkflow steps. Our experiments show that the Quality Checker can preciselyaccept any correct program, mitigate faulty synthesized tests, and preventpotential workflow deviation. QualityFlow establishes the state-of-the-artresults on four program synthesis benchmarks: MBPP, HumanEval, and stricterevaluations from MBPP-EvalPlus and HumanEval-EvalPlus.