87 lines
2.0 KiB
Markdown
87 lines
2.0 KiB
Markdown
# MinerU v2.0 多GPU服务器
|
||
|
||
[English](README.md)
|
||
|
||
这是一个精简的多GPU服务器实现。
|
||
|
||
## 快速开始
|
||
|
||
### 1. 安装 MinerU
|
||
|
||
```bash
|
||
pip install --upgrade pip
|
||
pip install uv
|
||
uv pip install -U "mineru[core]"
|
||
uv pip install litserve aiohttp loguru
|
||
```
|
||
|
||
### 2. 启动服务器
|
||
|
||
```bash
|
||
python server.py
|
||
```
|
||
|
||
### 3. 启动客户端
|
||
|
||
```bash
|
||
python client.py
|
||
```
|
||
|
||
现在，`[demo](../../demo/)` 文件夹下的PDF文件将并行处理。假设您有2个GPU，如果您将 `workers_per_device` 更改为 `2`，则可以同时处理4个PDF文件！
|
||
|
||
## 自定义
|
||
|
||
### 服务器
|
||
|
||
以下示例展示了如何启动带有自定义设置的服务器：
|
||
```python
|
||
server = ls.LitServer(
|
||
MinerUAPI(output_dir='/tmp/mineru_output'), # 自定义输出文件夹
|
||
accelerator='auto', # 您可以指定 'cuda'
|
||
devices='auto', # "auto" 使用所有可用的GPU
|
||
workers_per_device=1, # 每个GPU启动一个工作实例
|
||
timeout=False # 禁用超时，用于长时间处理
|
||
)
|
||
server.run(port=8000, generate_client_file=False)
|
||
```
|
||
|
||
### 客户端
|
||
|
||
客户端支持同步和异步处理：
|
||
|
||
```python
|
||
import asyncio
|
||
import aiohttp
|
||
from client import mineru_parse_async
|
||
|
||
async def process_documents():
|
||
async with aiohttp.ClientSession() as session:
|
||
# 基本用法
|
||
result = await mineru_parse_async(session, 'document.pdf')
|
||
|
||
# 带自定义选项
|
||
result = await mineru_parse_async(
|
||
session,
|
||
'document.pdf',
|
||
backend='pipeline',
|
||
lang='ch',
|
||
formula_enable=True,
|
||
table_enable=True
|
||
)
|
||
|
||
# 运行异步处理
|
||
asyncio.run(process_documents())
|
||
```
|
||
|
||
### 并行处理
|
||
同时处理多个文件：
|
||
```python
|
||
async def process_multiple_files():
|
||
files = ['doc1.pdf', 'doc2.pdf', 'doc3.pdf']
|
||
|
||
async with aiohttp.ClientSession() as session:
|
||
tasks = [mineru_parse_async(session, file) for file in files]
|
||
results = await asyncio.gather(*tasks)
|
||
|
||
return results
|
||
``` |