The LF AI & Data Foundation, under the Linux Foundation, has formed a working group with IBM, NVIDIA, and others to develop an AI-friendly document format called 'DocLang'. Existing formats like PDF and Markdown are ill-suited for AI parsing, causing structural information loss and high token costs.
The LF AI & Data Foundation, under the Linux Foundation, has formed a working group with IBM, NVIDIA, Red Hat, ABBYY, HumanSignal, and Forgis to develop 'DocLang', an AI-friendly document format. DocLang is designed to help enterprises feed their files to AI systems efficiently.
Existing formats like PDF, Markdown, HTML, and LaTeX were designed for human readability, causing loss of semantic information, structural relationships, and geometric context when parsed by AI models. Based on IBM's open-source toolkit Docling, DocLang provides a standard for exchanging structured output across systems. It uses a limited XML vocabulary optimized for LLM tokenizers, enabling 1:1 mapping and lossless conversion.
DocLang can improve AI parsing accuracy and reduce token consumption compared to formats like PDF, leading to cost savings. According to ABBYY, ambiguous structure forces models into guesswork, increasing hallucination risk and wasting tokens. DocLang addresses these issues, potentially boosting enterprise AI efficiency. However, adoption and ecosystem growth remain key challenges.
Comments point out that the proposed format is essentially XML, suggesting it's not a new idea but a rediscovery of existing technology. There was no debate for or against, merely an observation of its similarity to past formats.