Mistral AI has unveiled Codestral, its first code-focused AI model. Codestral helps the developers with coding tasks offering efficiency and accuracy in code generation.
Codestral is designed to address the requirements of developers across programming tasks like generating code snippets, completing functions, writing tests, or filling in incomplete code segments.
Codestral supports over 80 programming languages, including Python, Java, C, C++, JavaScript, and Bash. It has a context length of 32k, much larger than other coding AI models. This larger context helps Codestral generate code structures, providing more detailed and complete solutions to complex coding problems.
Codestral has performed well on various benchmarks, outperforming other code-centric models like CodeLlama 70B, Deepseek Coder 33B, and Llama 3 70B in accuracy and efficiency. It scored 81.1% on HumanEval for Python code generation and 51.3% on CruxEvall for Python output prediction. To further measure its performance, several additional benchmarks were used: HumanEval pass@1, sanitized MBPP pass@1, CruxEval, and RepoBench EM for Python, the Spider benchmark for SQL, and evaluations in C++, Bash, Java, PHP, Typescript, and C#. Its fill-in-the-middle performance was assessed with HumanEval pass@1 in Python, JavaScript, and Java, compared to DeepSeek Coder 33B.
Codestral integrates with popular development environments like VSCode and JetBrains, making it accessible and easy to use for developers. Additionally, it integrates with LlamaIndex and LangChain, allowing users to build agent applications with ease.
Mikhail Evtikhiev, researcher at JetBrains, said:
We used Codestral to run a test on our Kotlin-HumanEval benchmark and were impressed with the results. For instance, in the case of the pass rate for T=0.2, Codestral achieved a score of 73.75, surpassing GPT-4-Turbo’s score of 72.05 and GPT-3.5-Turbo’s score of 54.66.
User @1littlecoder posted on X:
Mistral's codestral just solved a HARD Bash problem from HackerRank - just 0-shot!
Codestral has strong capabilities but requires high-performance computing resources and has some usage restrictions. Codestral is a 22B open-weight model licensed under the new Mistral AI Non-Production License, which means that you can use it for research and testing purposes. Codestral can be downloaded on HuggingFace.
