BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

TL;DR

BioMistral is an open-source LLM tailored for the medical domain, built on Mistral and further pre-trained on PubMed Central. It achieves state-of-the-art performance on 10 medical QA benchmarks and conducts the first multilingual evaluation across 7 languages.

Problem

General-purpose LLMs face challenges when applied to the medical domain due to lack of domain-specific knowledge and performance degradation. Additionally, limited multilingual support in the medical field restricts accessibility for non-English users.

Approach

We used Mistral 7B as the base model and performed additional pre-training on biomedical literature from PubMed Central. We then evaluated on 10 medical QA tasks (in English) and explored quantization and model merging techniques. We also automatically translated the benchmark into 7 languages for multilingual evaluation.

Results & Contribution

BioMistral outperforms existing open-source medical models (including GPT-4) and shows competitive results even with quantized models. It provides the first large-scale multilingual evaluation of medical LLMs across 7 languages, promoting research on multilingual generalization. All data, models, and scripts are released as open-source.