AI Open Source · RAG 与检索
microsoft/markitdown
微软维护的 Python 工具,把 PDF、Word、Excel、PPT 等 Office 文档 和常见格式统一转成 markdown。做 RAG 数据预处理、给 LLM 喂结构 干净的文本时是趁手的第一步,也常配合 LangChain、AutoGen 使用。
Python tool for converting files and office documents to Markdown.
- Stars
- ★ 123k
- Language
- Python
- License
- MIT
- Last push
- 27d ago
- Created
- 2024-11-13
- Topics
- autogenautogen-extensionlangchainmarkdownmicrosoft-officeopenai
README
MarkItDown
[!IMPORTANT] MarkItDown performs I/O with the privileges of the current process. Like open() or requests.get(), it will access resources that the process itself can access. Sanitize your inputs in untrusted environments, and call the narrowest
convert_*function needed for your use case (e.g.,convert_stream(), orconvert_local()). See the Security Considerations section of the documentation for more information.
MarkItDown is a lightweight Python utility for converting various files to Markdown for use with LLMs and related text analysis pipelines. To this end, it is most comparable to textract, but with a focus on preserving important document structure and content as Markdown (including: headings, lists, tables, links, etc.) While the output is often reasonably presentable and human-friendly, it is meant to be consumed by text analysis tools -- and may not be the best option for high-fidelity document conversions for human consumption.
MarkItDown currently supports the conversion from:
- PowerPoint
- Word
- Excel
- Images (EXIF metadata and OCR)
- Audio (EXIF metadata and speech transcription)
- HTML
- Text-based formats (CSV, JSON, XML)
- ZIP files (iterates over contents)
- Youtube URLs
- EPubs
- ... and more!
Why Markdown?
Markdown is extremely close to plain text, with minimal markup or formatting, but still provides a way to represent important document structure. Mainstream LLMs, such as OpenAI's GPT-4o, natively "speak" Markdown, and often incorporate Markdown into their responses unprompted. This suggests that they have been trained on vast amounts of Markdown-formatted text, and understand it well. As a side benefit, Markdown conventions are also highly token-efficient.
Prerequisites
MarkItDown requires Python 3.10 or higher. It is recommended to use a virtual environment to avoid dependency conflicts.
With the standard Python installation, you can create and activate a virtual environment using the following commands:
python -m venv .venv
source .venv/bin/activate
If using uv, you can create a virtual environment with:
uv venv --python=3.12 .venv
source .venv/bin/activate
# NOTE: Be sure to use 'uv pip install' rather than just 'pip install' to install packages in this virtual environment
If you are using Anaconda, you can create a virtual environment with:
conda create -n markitdown python=3.12
conda activate markitdown
Installation
To install MarkItDown, use pip: pip install 'markitdown[all]'. Alternatively, you can install it from the source:
git clone git@github.com:microsoft/markitdown.git
cd markitdown
pip install -e 'packages/markitdown[all]'
Usage
Command-Line
markitdown path-to-file.pdf > document.md
Or use -o to specify the output file:
markitdown path-to-file.pdf -o document.md
You can also pipe content:
cat path-to-file.pdf | markitdown
Optional Dependencies
MarkItDown has optional dependencies for activating various file formats. Earlier in this document, we installed all optional dependencies with the [all] option. However, you can also install them individually for more control. For example:
pip install 'markitdown[pdf, docx, pptx]'
will install only the dependencies for PDF, DOCX, and PPTX files.
At the moment, the following optional dependencies are available:
[all]Installs all optional dependencies[pptx]Installs dependencies for PowerPoint files[docx]Installs dependencies for Word files[xlsx]Installs dependencies for Excel files[xls]Installs dependencies for older Excel files[pdf]Installs dependencies for PDF files[outlook]Installs dependencies for Outlook messages[az-doc-intel]Installs dependencies for Azure Document Intelligence[audio-transcription]Installs dependencies for audio transcription of wav and mp3 files[youtube-transcription]Installs dependencies for fetching YouTube video transcription
Plugins
MarkItDown also supports 3rd-party plugins. Plugins are disabled by default. To list installed plugins:
markitdown --list-plugins
To enable plugins use:
markitdown --use-plugins path-to-file.pdf
To find available plugins, search GitHub for the hashtag #markitdown-plugin. To develop a plugin, see packages/markitdown-sample-plugin.
markitdown-ocr Plugin
The markitdown-ocr plugin adds OCR support to PDF, DOCX, PPTX, and XLSX converters, extracting text from embedded images using LLM Vision — the same llm_client / llm_model pattern that Mark
同一分类的其他项