Databricks launched DBRX, a new open-source large language model (LLM) that aims to redefine the standards of open models and outperform well-known competitors on industry benchmarks.
With 132 billion parameters, DBRX has demonstrated in its own run of industry benchmarks that this model outperforms popular open-source LLMs such as LLaMA 2 70B, Mixtral, and Grok-1 across various language understanding, programming, and math tasks. The new model even competes favorably against Anthropic’s closed-source model Claude on specific benchmarks.
The AI Community expressed excitement about the release of DBRX, with Clem Delangue, CEO at Hugging Face posting on X:
Not a surprise but DBRX is already #1 trending on HF!
DBRX's performance is attributed to its more efficient mixture-of-experts architecture, making it up to 2x faster at inference than LLaMA 2 70B despite having fewer active parameters. Databricks claims that training the model was also approximately 2x more compute-efficient than dense alternatives.
The model was pretrained on 12 trillion tokens of curated text and code data, leveraging advanced technologies like rotary position encodings and curriculum learning during pretraining. Developers can interact with DBRX via APIs or utilize Databricks' tools to fine-tune the model on their proprietary data. Integration into Databricks' AI products is already underway. DBRX is available on GitHub and Hugging Face.
Databricks anticipates that developers will adopt the model as a foundation for their own LLMs, potentially enhancing customer chatbots or internal question-answering systems. This approach also provides insight into how DBRX was constructed using Databricks' proprietary tools.
To create the dataset utilized in developing DBRX, Databricks utilized Apache Spark and Databricks Notebooks for data processing, Unity Catalog for data management and governance, and MLflow for experiment tracking.
DBRX sets a new standard for open-source AI models, offering customizable and transparent generative AI solutions for enterprises. A recent survey from Andreessen Horowitz, indicates a growing interest among AI leaders to increase open-source adoption as fine-tuned models approach closed-source performance levels. Databricks expects DBRX to accelerate the shift from closed to open-source solutions.