Abstract

Finetuning large language models (LLMs) on instructions leads to vastperformance improvements on natural language tasks. We apply instruction tuningusing code, leveraging the natural structure of Git commits, which pair codechanges with human instructions. We compile CommitPack: 4 terabytes of Gitcommits across 350 programming languages. We benchmark CommitPack against othernatural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16Bparameter StarCoder model, and achieve state-of-the-art performance amongmodels not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmarkto a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack amongall permissive models, demonstrating CommitPack's benefits in generalizing to awider set of languages and natural coding tasks. Code, models and data arefreely available at https://github.com/bigcode-project/octopack.