feat: complete FunASR realtime integration
Bifang
parent
24b4b62fa9
commit
1a239cffbf
|
|
@ -8,6 +8,13 @@ FUNASR_ASR_MODEL=paraformer-zh-streaming
|
|||
FUNASR_VAD_MODEL=fsmn-vad
|
||||
FUNASR_DEVICE=cuda:0
|
||||
FUNASR_VAD_DEVICE=cpu
|
||||
# Silence duration in milliseconds before FSMN VAD finalizes a speech segment.
|
||||
# Strategy 0 (semantic sentence) defaults to 800 ms; increase to preserve pauses.
|
||||
FUNASR_VAD_MAX_END_SILENCE_MS=800
|
||||
# Strategy 1 (paragraph) keeps the longer 5 s pause. Adjust independently if needed.
|
||||
FUNASR_VAD_PARAGRAPH_MAX_END_SILENCE_MS=5000
|
||||
# Higher margins reject more weak background noise but can suppress quiet speech.
|
||||
FUNASR_VAD_SPEECH_NOISE_THRES=0.6
|
||||
|
||||
# Native FunASR WSS chunk settings. The middle chunk is sent as 10 x 60 ms.
|
||||
FUNASR_CHUNK_SIZE=0,10,5
|
||||
|
|
@ -22,6 +29,8 @@ FUNASR_NATIVE_WS_PORT=10095
|
|||
AUXILIARY_SERVICE_URL=http://127.0.0.1:8010
|
||||
AUXILIARY_DEVICE=cpu
|
||||
AUXILIARY_PRELOAD_KINDS=speaker_verification
|
||||
# Optional punctuation runs on CPU by default so the speaker model can retain GPU memory.
|
||||
FUNASR_PUNC_DEVICE=cpu
|
||||
|
||||
# Frontend and backend run independently. The frontend proxies /ws and /api/stop.
|
||||
FRONTEND_HOST=127.0.0.1
|
||||
|
|
|
|||
|
|
@ -20,11 +20,10 @@ If directories have different names, set `FUNASR_ASR_MODEL` and `FUNASR_VAD_MODE
|
|||
|
||||
## Install and start
|
||||
|
||||
Install a torch/torchaudio build suitable for the host, then install the project dependencies and models:
|
||||
The root requirements.txt includes FunASR and auxiliary-service dependencies plus the Qwen vLLM stack pinned for a GB10 CUDA 13 host. Adjust those CUDA-specific pins for other hosts before installing:
|
||||
|
||||
~~~powershell
|
||||
python -m pip install -r backend/requirements-funasr.txt
|
||||
python -m pip install -r backend/requirements-auxiliary.txt
|
||||
python -m pip install -r requirements.txt
|
||||
python scripts/download_models.py --funasr-runtime
|
||||
if (-not (Test-Path .env)) { Copy-Item .env.funasr.example .env }
|
||||
~~~
|
||||
|
|
|
|||
16
README.md
16
README.md
|
|
@ -1,21 +1,21 @@
|
|||
# FunASR realtime ASR demo
|
||||
|
||||
This project keeps the frontend and backend in separate directories. The frontend serves the original Tencent demo assets from frontend/static/ unchanged.
|
||||
The project keeps the frontend and backend in separate directories. The frontend serves the original Tencent demo assets from frontend/static/ unchanged.
|
||||
|
||||
## Project layout
|
||||
|
||||
- backend/: FunASR WebSocket adapter, native engine launcher, CAM++ service, and backend dependencies.
|
||||
- frontend/: Tencent demo files, static HTTP/WebSocket proxy, and frontend dependencies.
|
||||
- scripts/: shared model download tools and the model manifest.
|
||||
- backend/: FunASR WebSocket adapter, native engine launcher, CAM++ service, and backend code.
|
||||
- frontend/: Tencent demo files and the static HTTP/WebSocket proxy.
|
||||
- scripts/: shared model download tools.
|
||||
- requirements.txt: combined dependencies for both ASR engines and the frontend.
|
||||
- model_manifest.json: shared model registry.
|
||||
|
||||
## Install and download models
|
||||
|
||||
Install a host-compatible PyTorch/torchaudio build first. Then install backend dependencies, copy the example environment file, and download the FunASR runtime models:
|
||||
The combined requirements file includes the GB10 CUDA 13 PyTorch and vLLM stack used by the Qwen3-ASR branch. Adjust the CUDA index and torch-family pins before installing on a different host.
|
||||
|
||||
~~~powershell
|
||||
python -m pip install -r backend/requirements-funasr.txt
|
||||
python -m pip install -r backend/requirements-auxiliary.txt
|
||||
python -m pip install -r frontend/requirements.txt
|
||||
python -m pip install -r requirements.txt
|
||||
if (-not (Test-Path .env)) { Copy-Item .env.funasr.example .env }
|
||||
python scripts/download_models.py --funasr-runtime
|
||||
~~~
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ Qwen/Qwen3-ASR-1.7B
|
|||
cd D:\github-project\ASR\Qwen-Asr\demo
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install modelscope==1.34.0
|
||||
pip install -r requirements.txt
|
||||
python scripts\download_models.py
|
||||
```
|
||||
|
||||
|
|
@ -70,7 +70,7 @@ python scripts\download_models.py
|
|||
需要宿主机具备与 VLLM 兼容的 Python、CUDA 和 NVIDIA 驱动环境。安装部署依赖：
|
||||
|
||||
```bash
|
||||
python -m pip install -r backend/requirements-qwen-legacy.txt
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
先复制并按服务器实际路径修改 `.env`，启动器会自动读取该文件：
|
||||
|
|
@ -125,7 +125,7 @@ curl "http://${VLLM_DISPLAY_HOST:-127.0.0.1}:9950/v1/audio/transcriptions" \
|
|||
|
||||
```powershell
|
||||
cd D:\github-project\ASR\Qwen-Asr\demo
|
||||
pip install -r backend/requirements-auxiliary.txt
|
||||
pip install -r requirements.txt
|
||||
python -m backend.auxiliary_server
|
||||
```
|
||||
|
||||
|
|
@ -145,7 +145,7 @@ WebSocket demo 默认连接 `9950` 的 ASR VLLM，辅助服务使用 `8010`。`/
|
|||
|
||||
- `scripts/download_models.py`：下载选定 ASR 和全部辅助模型资产。
|
||||
- `backend/serve_qwen_legacy.py`：读取 `.env`，解析模型选择、宿主机参数并启动新版 `vllm serve`。
|
||||
- `backend/requirements-qwen-legacy.txt`：安装宿主机部署所需的官方 Qwen3-ASR VLLM 依赖。
|
||||
- `requirements.txt`：安装宿主机部署所需的官方 Qwen3-ASR VLLM 依赖。
|
||||
- `tests/`：只验证本项目自己的模型清单和选择逻辑，不依赖原项目。
|
||||
|
||||
模型服务就绪后，新的实时 ASR demo 放在同级 `demo` 项目中继续开发，但不得通过 Python import 或 HTTP/WebSocket 调用原项目服务。
|
||||
|
|
|
|||
|
|
@ -18,11 +18,9 @@ from aiohttp.web_request import FileField
|
|||
from dotenv import load_dotenv
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
# Support direct script startup while sharing the root model manifest.
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from scripts.model_manifest import auxiliary_models, load_manifest, model_directory
|
||||
from backend.model_manifest import auxiliary_models, load_manifest, model_directory
|
||||
|
||||
|
||||
# 将辅助服务端口固定在代码变量中，服务器启动时只需执行脚本，便于部署和排查。
|
||||
|
|
@ -109,12 +107,18 @@ class AuxiliaryRuntime:
|
|||
def _load_asset(self, model_id: str, config: dict[str, Any], path: Path) -> Any | None:
|
||||
"""只加载当前运行接口需要的模型；依赖模型和对齐模型先保持本地资产就绪。"""
|
||||
kind = str(config.get("kind") or "")
|
||||
if kind == "vad":
|
||||
if kind in {"vad", "punctuation"}:
|
||||
from funasr import AutoModel
|
||||
|
||||
# Keep punctuation on CPU by default so CAM++ can retain the GPU.
|
||||
device = (
|
||||
os.getenv("FUNASR_PUNC_DEVICE", "cpu")
|
||||
if kind == "punctuation"
|
||||
else AUXILIARY_DEVICE
|
||||
)
|
||||
return AutoModel(
|
||||
model=str(path),
|
||||
device=AUXILIARY_DEVICE,
|
||||
device=device,
|
||||
disable_update=True,
|
||||
disable_pbar=True,
|
||||
disable_log=True,
|
||||
|
|
@ -250,6 +254,23 @@ class AuxiliaryRuntime:
|
|||
async with self.inference_lock:
|
||||
return await asyncio.to_thread(model.generate, input=audio_path, cache={})
|
||||
|
||||
async def punctuate(self, text: str) -> str:
|
||||
"""Load CT-Transformer lazily and punctuate one completed ASR turn."""
|
||||
async with self.inference_lock:
|
||||
try:
|
||||
model = self._find_model("punctuation")
|
||||
except RuntimeError:
|
||||
# Load while holding the lock to prevent duplicate loads across sessions.
|
||||
model = await asyncio.to_thread(self._load_optional_kind, "punctuation")
|
||||
output = await asyncio.to_thread(model.generate, input=text, cache={})
|
||||
if isinstance(output, (list, tuple)) and output:
|
||||
output = output[0]
|
||||
if isinstance(output, Mapping):
|
||||
result = output.get("text")
|
||||
else:
|
||||
result = getattr(output, "text", None)
|
||||
return str(result).strip() if result else text
|
||||
|
||||
async def diarization(self, audio_path: str) -> Any:
|
||||
"""使用 CAM++ 对完整会话执行说话人聚类，保持跨片段的标签一致性。"""
|
||||
try:
|
||||
|
|
@ -447,6 +468,10 @@ async def health_handler(request: web.Request) -> web.Response:
|
|||
"device": AUXILIARY_DEVICE,
|
||||
"speaker_embedding_model": speaker_model_id,
|
||||
"speaker_embedding_ready": speaker_model_id is not None,
|
||||
"punctuation_ready": any(
|
||||
config.get("kind") == "punctuation" and model_id in runtime.models
|
||||
for model_id, config in runtime.assets.items()
|
||||
),
|
||||
"models": runtime.status,
|
||||
}
|
||||
)
|
||||
|
|
@ -471,6 +496,27 @@ async def vad_handler(request: web.Request) -> web.Response:
|
|||
Path(temp_path).unlink(missing_ok=True)
|
||||
|
||||
|
||||
async def punctuation_handler(request: web.Request) -> web.Response:
|
||||
"""Return punctuation when the optional local CT-Transformer is available."""
|
||||
runtime: AuxiliaryRuntime = request.app[MODEL_SERVICE_KEY]
|
||||
try:
|
||||
payload = await request.json()
|
||||
except (ValueError, web.HTTPException):
|
||||
return web.json_response({"error": "request body must be JSON"}, status=400)
|
||||
text = payload.get("text") if isinstance(payload, dict) else None
|
||||
if not isinstance(text, str):
|
||||
return web.json_response({"error": "JSON field 'text' must be a string"}, status=400)
|
||||
if not text.strip():
|
||||
return web.json_response({"text": text, "available": True})
|
||||
try:
|
||||
punctuated = await runtime.punctuate(text)
|
||||
return web.json_response({"text": punctuated, "available": True})
|
||||
except Exception as exc:
|
||||
# Punctuation is optional; keep ASR usable and expose the reason to the bridge.
|
||||
print(f"[punctuation] unavailable: {exc}", flush=True)
|
||||
return web.json_response({"text": text, "available": False, "error": str(exc)})
|
||||
|
||||
|
||||
def _normalize_diarization_segments(result: Any) -> list[dict[str, Any]]:
|
||||
"""将不同 ModelScope 版本的聚类输出统一为 start/end/speaker 字段。"""
|
||||
# ModelScope 通常返回 {'text': [[start_sec, end_sec, speaker_id], ...]}；
|
||||
|
|
@ -599,6 +645,7 @@ async def create_app() -> web.Application:
|
|||
app[MODEL_SERVICE_KEY] = runtime
|
||||
app.router.add_get("/health", health_handler)
|
||||
app.router.add_post("/v1/vad", vad_handler)
|
||||
app.router.add_post("/v1/punctuation", punctuation_handler)
|
||||
app.router.add_post("/v1/diarization", diarization_handler)
|
||||
app.router.add_post("/v1/speaker/resolve", speaker_resolve_handler)
|
||||
app.router.add_post("/v1/speaker/reset", speaker_reset_handler)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,79 @@
|
|||
"""独立 VLLM 部署项目的模型清单与路径解析辅助函数。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
MANIFEST_PATH = PROJECT_ROOT / "model_manifest.json"
|
||||
|
||||
|
||||
def load_manifest(path: Path = MANIFEST_PATH) -> dict[str, Any]:
|
||||
"""读取本项目自己的模型清单，整个过程不导入原项目代码。"""
|
||||
with path.open("r", encoding="utf-8") as manifest_file:
|
||||
manifest = json.load(manifest_file)
|
||||
if not isinstance(manifest.get("models"), dict) or not manifest["models"]:
|
||||
raise ValueError("model_manifest.json 必须包含非空的 models 对象")
|
||||
return manifest
|
||||
|
||||
|
||||
def resolve_model_id(model: str | None, manifest: dict[str, Any]) -> str:
|
||||
"""将默认值、短别名或完整模型 ID 解析为一个 ASR 模型。"""
|
||||
models = manifest["models"]
|
||||
requested = (model or "default").strip()
|
||||
if requested == "default":
|
||||
requested = str(manifest["default_model"])
|
||||
|
||||
if requested in models:
|
||||
return requested
|
||||
|
||||
for model_id, config in models.items():
|
||||
if requested.lower() == str(config.get("alias", "")).lower():
|
||||
return model_id
|
||||
raise ValueError(f"不支持的 ASR 模型 '{model}'，可选模型：{', '.join(models)}")
|
||||
|
||||
|
||||
def model_directory(model_id: str, manifest: dict[str, Any], models_dir: Path) -> Path:
|
||||
"""根据清单返回 ASR 或辅助模型实际使用的本地目录。"""
|
||||
config = manifest.get("models", {}).get(model_id)
|
||||
if config is None:
|
||||
config = manifest.get("auxiliary_models", {}).get(model_id)
|
||||
if not isinstance(config, dict) or not config.get("directory"):
|
||||
raise ValueError(f"模型 '{model_id}' 在清单中没有配置本地目录")
|
||||
return models_dir / str(config["directory"])
|
||||
|
||||
|
||||
def auxiliary_models(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||
"""返回可独立部署的 VAD、说话人和对齐模型资产。"""
|
||||
models = manifest.get("auxiliary_models", {})
|
||||
if not isinstance(models, dict):
|
||||
raise ValueError("model_manifest.json 的 auxiliary_models 必须是对象")
|
||||
return {str(model_id): config for model_id, config in models.items() if isinstance(config, dict)}
|
||||
|
||||
def resolve_auxiliary_model_id(
|
||||
model: str | None,
|
||||
manifest: dict[str, Any],
|
||||
kind: str | None = None,
|
||||
) -> str:
|
||||
"""Resolve a configured auxiliary asset by ID, alias, or FunASR alias."""
|
||||
assets = auxiliary_models(manifest)
|
||||
requested = (model or "").strip()
|
||||
if requested in assets and (kind is None or assets[requested].get("kind") == kind):
|
||||
return requested
|
||||
|
||||
for model_id, config in assets.items():
|
||||
if kind is not None and config.get("kind") != kind:
|
||||
continue
|
||||
aliases = [config.get("alias"), config.get("funasr_alias")]
|
||||
if requested.lower() in {
|
||||
str(alias).lower() for alias in aliases if isinstance(alias, str)
|
||||
}:
|
||||
return model_id
|
||||
expected = ", ".join(
|
||||
model_id for model_id, config in assets.items()
|
||||
if kind is None or config.get("kind") == kind
|
||||
)
|
||||
raise ValueError(f"Unsupported auxiliary model '{model}'; configured assets: {expected}")
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
# Demo 说话人链路修复与验收
|
||||
|
||||
本次仅修改 `demo/`。参考原 `app/services/qwen3_websocket_asr.py` 的前滚音频、同句异步说话人更新和结束提交方式，保留独立 vLLM ASR + 辅助声纹服务 + WebSocket 三个进程。当前机器没有显卡，验证采用模拟模型，真实声纹准确率和模型加载状态需在部署机器上验收。
|
||||
|
||||
## 已确认的代码问题
|
||||
|
||||
1. 页面只处理 `sentences`，忽略已实现的 `display_state`，因此后端合并结果没有展示。未知片段还会临时附在上一位已确认说话人的气泡里。
|
||||
2. 页面点击停止后 5 秒强制断开，但辅助 HTTP 请求的超时是 45 秒，可能截掉迟到的声纹更新。
|
||||
3. 无 embedding、低置信度、协议字段不完整等情况被静默丢弃，用户无法区分等待、短音频、服务错误和证据拒绝。
|
||||
4. 旧声纹长度检查把句尾 800ms 静音一起计算，可能让短插话通过长度要求；WAV 固定去掉 44 字节也会破坏带额外元数据的输入。
|
||||
5. 仅 vLLM 启动器加载 `demo/.env`，另两个进程没有加载；`0.6b` 下载别名也可能被直接当作公开模型名发送。
|
||||
|
||||
这些是代码中可复现的问题，不能据此断言部署中的声纹模型一定已经正常加载。新版本将具体原因直接显示在片段气泡和日志里。
|
||||
|
||||
## 修复后的行为
|
||||
|
||||
- partial/final 和声纹更新继续使用同一 `sentence_id`。页面显示完整聚合快照，过时的 `revision` 不覆盖新快照。
|
||||
- 仅合并相邻且身份可信的片段。未知短插话独立显示；A→B→A 保留顺序。不同实名或实名与弱匿名身份不会因为簇 ID 相同而合并。
|
||||
- 每条声纹证据必须来自当前片段；同步、异步两种入口都拒绝 `short_attach` / `embedding_attach`。不复制上一段 embedding。声纹向量拒绝零向量、NaN、Infinity 和多样本矩阵。
|
||||
- WebSocket 以有效有声帧判断长度。少于 800ms 的语音保持 pending；800ms～1.6s 也必须独立提取特征，不能直接继承前一位身份。该长度门槛属于保守保护，不能保证短样本识别准确率。
|
||||
- 保留 200ms 前滚，提交时去除尾部静音。增量解析 WAV 的 RIFF/fmt/data/JUNK 等头，拒绝非 16kHz 单声道 PCM16 文件。
|
||||
- `stop/eof` 返回 `draining`，排空 ASR 和声纹队列后才发送最终快照及 `end`。声纹失败不阻断 ASR。abort、断线、异常和正常结束均清理聚类会话。
|
||||
- ASR worker 异常会立即发送 `error`，不会一直等待客户端停止。
|
||||
|
||||
## 如何判断卡在哪里
|
||||
|
||||
页面顶部显示已确认数量；未匹配到说话人的气泡统一显示“未知说话人”，详细等待或失败原因通过标签悬停提示和原始日志查看。原始日志保留 `speaker_status`、`speaker_reason`、`speaker_strategy` 和置信度。
|
||||
|
||||
| `speaker_status` | 含义与排查方向 |
|
||||
|---|---|
|
||||
| `waiting_final` | 讲话仍在进行，等待静音或最大时长切段 |
|
||||
| `queued` / `processing` | 文本已完成，声纹正在排队或推理 |
|
||||
| `confirmed` | 当前片段声纹已确认，应该显示说话人标签 |
|
||||
| `insufficient_audio` | 有效语音过短，不继承上一位；使用较长发言复测 |
|
||||
| `service_unavailable` / `service_error` | 未配置、无法访问或模型推理失败；检查辅助服务及完整错误 |
|
||||
| `no_embedding` | 服务没有产生可用特征 |
|
||||
| `evidence_rejected` | 缺少 fresh/confirmed 证据、置信度不足或使用了继承策略 |
|
||||
| `disabled` | 本次未开启说话人分离 |
|
||||
|
||||
健康检查为 `http://辅助服务器:8010/health`。新版本有 `speaker_protocol_version: 2`，重点检查 `speaker_embedding_ready` 与 `speaker_embedding_model`。若没有版本字段，检查是否重启了更新后的辅助进程。HTTP 健康检查不执行真实声纹推理，不能代替音频验收。
|
||||
|
||||
辅助服务目前返回匿名的“说话人 1、2……”；demo 没有接入原应用的声纹注册库，因此不会自动识别人员实名。vLLM 仅输出转写文本，声纹标签由 `scripts/auxiliary_server.py` 负责。
|
||||
|
||||
模型职责要区分：`iic/speech_campplus_sv_zh-cn_16k-common` 是实时 turn 的 CAM++ embedding 模型，必须加载；`iic/speech_campplus_speaker-diarization_common` 是完整音频分离 pipeline，包含额外的 change locator/VAD 依赖，当前实时 WebSocket 不在启动阶段调用它。WebSocket 自身仍用轻量 RMS 帧门控切句，辅助服务的 FunASR VAD 对外提供 `/v1/vad`，并供完整 diarization 依赖使用；因此启动辅助服务是为了 CAM++ 声纹和聚类，不能把整段 diarization 的加载失败误认为 vLLM 失败。
|
||||
|
||||
## 部署后操作
|
||||
|
||||
在部署机更新这些文件后，已有 vLLM 可继续运行。重新安装增补的 `python-dotenv` 依赖，并重启辅助服务及 WebSocket。辅助服务启动时强制依赖 VAD + CAM++ `speaker_verification`；完整 CAM++ diarization 不再阻断实时启动；以下命令都从 `demo/` 目录执行：
|
||||
|
||||
```text
|
||||
pip install -r requirements-auxiliary.txt
|
||||
pip install -r realtime_asr_optimization_demo/requirements.txt
|
||||
python scripts/auxiliary_server.py
|
||||
```
|
||||
|
||||
如果仍提示核心模型缺失或加载失败，日志会列出模型 ID、实际查找路径、状态和底层异常。先执行 `python scripts/download_models.py --auxiliary-only`，或设置 `.env` 的 `MODEL_DIR` 指向同时包含 `damo/speech_fsmn_vad_zh-cn-16k-common-pytorch` 与 `iic/speech_campplus_sv_zh-cn_16k-common` 的目录。不要用 `AUXILIARY_ALLOW_MISSING=true` 掩盖 VAD/CAM++ 核心模型缺失；该选项只适合临时查看可选模型状态。
|
||||
|
||||
另一个终端执行：
|
||||
|
||||
```text
|
||||
python realtime_asr_optimization_demo/server.py --no-browser
|
||||
```
|
||||
|
||||
三个进程现在均读取 `demo/.env`，系统环境变量优先。确认 `MODEL_SERVICE_URL` 指向现有 vLLM 的 `/v1`，`AUXILIARY_SERVICE_URL` 指向辅助服务。`QWEN3_ASR_MODEL` 支持部署清单别名，设置 `VLLM_SERVED_MODEL_NAME` 时优先使用该公开名称。更新后刷新浏览器；脚本 URL 已更新版本号。
|
||||
|
||||
`127.0.0.1` 指各 Python 服务运行的机器，不是浏览器所在机器。跨服务器部署时填写对应服务器 IP。浏览器麦克风访问远程页面需要安全上下文（HTTPS）；本机 localhost 可用于测试。
|
||||
|
||||
## 验收顺序
|
||||
|
||||
1. 单人讲话 2～4 秒后停顿：先出现文本，随后同句变为“说话人 1”。
|
||||
2. 同一人再次讲话并停顿：确认后相邻块应合并；取消页面“合并相邻”可对照物理片段。
|
||||
3. A→B→A，各说 2 秒以上：应保持三个时间顺序块。标签准确率需要真实声纹模型验证。
|
||||
4. A 后 B 说一个很短的“嗯”：应显示“有效语音不足”，不能进入 A 的气泡。
|
||||
5. 辅助服务关闭时测试：ASR 仍完成，片段明确显示服务错误。
|
||||
6. 讲话中点击停止：等待最终结果，不能在五秒时丢失说话人更新。
|
||||
|
||||
无显卡回归命令：
|
||||
|
||||
```text
|
||||
cd demo
|
||||
python -m unittest discover -s tests -v
|
||||
cd realtime_asr_optimization_demo
|
||||
python -m unittest discover -s tests -v
|
||||
node --test tests/test_frontend.cjs
|
||||
```
|
||||
|
||||
这些测试覆盖状态机、模拟 HTTP/WebSocket、延迟更新、前端脚本和有效向量校验，不执行模型下载或 GPU 推理。当前 VLLM 适配器默认使用 `/v1/realtime` 原生流式，旧同步 HTTP 保留为兼容回退。单个内部片段中无停顿的多人换话或重叠讲话仍需真实模型和更细粒度切段验证。
|
||||
|
|
@ -1,135 +0,0 @@
|
|||
# Realtime ASR WebSocket Optimization Demo
|
||||
|
||||
这是一个独立的实时 ASR WebSocket 验证项目，放在模型部署项目 `demo` 下，但不导入、不启动、也不调用仓库根目录的原始 `app/`。
|
||||
|
||||
## 验证目标
|
||||
|
||||
- 浏览器麦克风或按实时速度发送的 PCM/WAV 音频通过本项目 WebSocket 发送。
|
||||
- 同一 `sentence_id` 的 partial、final 和后续更新覆盖同一条 raw segment。
|
||||
- raw segment 与前端 display block 分离，避免把物理切段直接等同于展示换行。
|
||||
- 小于 1.6 秒且带有 `short_attach` / `embedding_attach` 策略的实名结果降级为 pending。
|
||||
- 不把 embedding 字段写入 demo 状态池。
|
||||
- 只有相邻且身份可信的 segment 才合并；A→B→A 保持时间顺序。
|
||||
- 记录 partial 首次延迟、final 延迟、partial 修订次数和服务端返回时间范围。
|
||||
- 每个实时 turn 单独提交声纹特征，由辅助服务维护本 WebSocket session 的在线聚类中心。
|
||||
|
||||
本次修复、诊断状态和部署验收步骤见 [FIXES.md](FIXES.md)。ASR 继续使用已部署的独立 vLLM；已有 vLLM 服务时无需重复启动或下载模型。
|
||||
|
||||
当前默认使用 vLLM `/v1/realtime` WebSocket：PCM 音频帧只发送一次，模型
|
||||
持续返回 `transcription.delta`，VAD 切段时发送 `input_audio_buffer.commit`。
|
||||
因此 partial 不再重复上传不断增长的整段音频，适合长时间会议运行。
|
||||
启动脚本默认启用 `Qwen3ASRRealtimeGeneration`；如果部署端点不支持 realtime，
|
||||
本项目会自动回退到原同步 HTTP 接口，外部 WebSocket 参数不变。
|
||||
|
||||
## 启动
|
||||
|
||||
需要启动两个模型服务和一个 WebSocket 页面：VLLM 只负责 ASR，辅助服务负责
|
||||
VAD、CAM++ 声纹模型及在线聚类，WebSocket 只做音频流编排，不导入原项目代码。
|
||||
|
||||
先在 `demo` 目录下载 ASR 和辅助模型，并启动 VLLM：
|
||||
|
||||
```powershell
|
||||
cd D:\github-project\ASR\Qwen-Asr\demo
|
||||
python scripts\download_models.py
|
||||
python backend/serve_qwen_legacy.py
|
||||
```
|
||||
|
||||
另一个终端启动辅助模型服务：
|
||||
|
||||
```powershell
|
||||
cd D:\github-project\ASR\Qwen-Asr\demo
|
||||
pip install -r backend/requirements-auxiliary.txt
|
||||
python -m backend.auxiliary_server
|
||||
```
|
||||
|
||||
另开一个终端启动 WebSocket 页面：
|
||||
|
||||
```powershell
|
||||
cd D:\github-project\ASR\Qwen-Asr\demo\realtime_asr_optimization_demo
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\Activate.ps1
|
||||
pip install -r requirements.txt
|
||||
python server.py --no-browser
|
||||
```
|
||||
|
||||
页面服务默认监听 `0.0.0.0:8082`，端口在 `server.py` 顶部的 `WEB_PORT` 内部变量中维护。
|
||||
VLLM 默认地址为 `http://127.0.0.1:9950/v1`，辅助服务默认地址为
|
||||
`http://127.0.0.1:8010`。可通过环境变量切换到远程服务：
|
||||
|
||||
```powershell
|
||||
$env:MODEL_SERVICE_URL = 'http://127.0.0.1:9950/v1'
|
||||
$env:AUXILIARY_SERVICE_URL = 'http://127.0.0.1:8010'
|
||||
python server.py --no-browser
|
||||
```
|
||||
|
||||
如果服务器端口已通过 VS Code Remote/端口转发映射到本机，保持上述两个
|
||||
`127.0.0.1` 地址即可：本地 WebSocket 只负责编排，ASR 和 VAD/CAM++ 推理仍在
|
||||
服务器 GPU 服务中完成。先访问 `http://127.0.0.1:9950/v1/models` 与
|
||||
`http://127.0.0.1:8010/health`，分别确认 VLLM 模型和辅助模型服务可达且 `ready=true`。
|
||||
|
||||
服务器部署时使用 `--no-browser`，然后在客户端浏览器访问 `http://服务器IP:8082`。如需让启动日志显示服务器域名或 IP，可设置 `WEB_DISPLAY_HOST`；它只影响提示文本，不改变监听地址。
|
||||
|
||||
辅助服务启动后可用 `http://服务器IP:8010/health` 检查模型状态。WebSocket
|
||||
收到聚类服务错误时仍会继续输出 ASR，但对应片段会显示“未知说话人”；详细的
|
||||
`speaker_reason` 可将鼠标悬停在标签上查看，事件日志仍会显示 `speaker_warning`，
|
||||
便于区分“模型未归类”和“ASR 失败”。
|
||||
|
||||
声纹服务的实时路径必须通过 ModelScope pipeline 的公开接口提取 embedding：
|
||||
`pipeline([wav_path], output_emb=True)`。不能绕过 pipeline 预处理后直接调用
|
||||
`pipeline.model`，否则采样率、声道和 waveform 预处理不会执行，部分 ModelScope
|
||||
版本会直接抛异常，WebSocket 仍会继续输出 ASR 并把说话人保留为 pending。
|
||||
更新辅助服务代码后需要重启 `python -m backend.auxiliary_server`，仅重启页面
|
||||
服务不会替换已经驻留在 GPU 中的旧辅助服务进程。
|
||||
|
||||
## 页面操作
|
||||
|
||||
1. 选择 Mic 或 File。
|
||||
2. 点击开始，浏览器通过 `/ws` 建立本项目 WebSocket。
|
||||
3. 页面展示腾讯 Demo 风格的气泡；麦克风或 PCM/WAV 文件按流式方式输入，partial 会在讲话过程中实时刷新。
|
||||
4. VAD 检测到静音后提交当前 turn，先返回 pending，再异步更新说话人。
|
||||
5. 停止会发送 `stop`，服务端完成当前 turn 和 speaker 队列后再发送 `end`。
|
||||
6. `abort` 只取消会话，不提交当前片段。
|
||||
|
||||
## WebSocket 消息
|
||||
|
||||
客户端首条消息：
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "start",
|
||||
"source": "mic",
|
||||
"model_service_url": "http://127.0.0.1:9950/v1",
|
||||
"model": "Qwen/Qwen3-ASR-0.6B",
|
||||
"speaker_diarization": 1,
|
||||
"sentence_strategy": 0,
|
||||
"enable_native_partial_stream": true,
|
||||
"partial_interval_ms": 1200,
|
||||
"max_segment_sec": 12,
|
||||
"speaker_gap_ms": 400,
|
||||
"display_merge": true
|
||||
}
|
||||
```
|
||||
|
||||
随后持续发送 16kHz、单声道、PCM16 二进制音频；文件模式仅支持 PCM/WAV，结束发送 `{"type":"eof"}`，停止发送
|
||||
`{"type":"stop"}`，取消发送 `{"type":"abort"}`。每个已完成 turn 会向辅助服务
|
||||
发送一次 `/v1/speaker/resolve`，只包含当前 turn 音频和 session_id，不会重复上传整段会话。
|
||||
|
||||
服务端会发送 `start`、`sentences`、`display_state`、`metrics`、`speaker_warning`、`draining`、`end` 和 `error`。
|
||||
页面用带 `revision` 的 `display_state` 渲染，以 `block_id` 标识展示块；`sentences` 保留原始片段及诊断状态。
|
||||
停止后必须等待 `end`，其中包含完整 `sentences` 和 `display_blocks`，不能提前关闭连接。
|
||||
`sentences` 中 `sentence_type=0` 是 partial，`sentence_type=1` 是 final；同一个
|
||||
`sentence_id` 必须覆盖更新而不是追加。`sentence_strategy=0` 使用约 800ms
|
||||
静音切句，`sentence_strategy=1` 使用约 1400ms 静音切段，更适合段落模式。
|
||||
|
||||
说话人模式下，`speaker_gap_ms`（默认 400ms）会在有效语音达到 800ms 后
|
||||
提前结束短停顿话轮，用于拆开 A→B 的交接；它不触发额外声纹请求，同一说话人的
|
||||
相邻片段仍由辅助服务聚类为同一说话人。服务未就绪时该策略会自动关闭。
|
||||
|
||||
## 目录
|
||||
|
||||
- `server.py`：本地 HTTP 页面和 WebSocket 会话编排。
|
||||
- `model_service.py`：独立 VLLM OpenAI 音频接口适配层。
|
||||
- `auxiliary_service.py`：独立辅助模型 HTTP 接口适配层。
|
||||
- `speaker_assembler.py`：raw segment、speaker evidence 和 display block 状态机。
|
||||
- `static/`：麦克风/文件测试页面。
|
||||
- `tests/`：只测试本项目状态合并和音频转换逻辑。
|
||||
|
|
@ -64,6 +64,23 @@ class AuxiliaryModelService:
|
|||
raise RuntimeError("auxiliary health check returned a non-object JSON value")
|
||||
return decoded
|
||||
|
||||
async def punctuate(self, text: str) -> dict[str, Any]:
|
||||
"""Apply optional FunASR punctuation without making it a startup dependency."""
|
||||
if self._session is None:
|
||||
raise RuntimeError("auxiliary model service is not started")
|
||||
endpoint = self.config.base_url.rstrip("/") + "/v1/punctuation"
|
||||
async with self._session.post(endpoint, json={"text": text}) as response:
|
||||
body = await response.text()
|
||||
if response.status >= 400:
|
||||
raise RuntimeError(f"auxiliary punctuation failed ({response.status}): {body[:500]}")
|
||||
try:
|
||||
decoded = await response.json(content_type=None)
|
||||
except ValueError as exc:
|
||||
raise RuntimeError(f"auxiliary punctuation returned invalid JSON: {body[:500]}") from exc
|
||||
if not isinstance(decoded, dict):
|
||||
raise RuntimeError("auxiliary punctuation returned a non-object JSON value")
|
||||
return decoded
|
||||
|
||||
async def resolve_speaker(
|
||||
self,
|
||||
pcm_bytes: bytes,
|
||||
|
|
|
|||
|
|
@ -123,7 +123,7 @@ class FunASRModelService:
|
|||
from funasr import AutoModel
|
||||
except ImportError as exc: # pragma: no cover - deployment-only branch
|
||||
raise RuntimeError(
|
||||
"FunASR is not installed; run pip install -r backend/requirements-funasr.txt"
|
||||
"FunASR is not installed; run pip install -r requirements.txt"
|
||||
) from exc
|
||||
|
||||
def load_models() -> tuple[Any, Any]:
|
||||
|
|
|
|||
|
|
@ -21,6 +21,41 @@ from scipy.spatial.distance import cosine
|
|||
import torch # 保留不影响
|
||||
|
||||
|
||||
def _bounded_env_float(name: str, default: float, minimum: float, maximum: float) -> float:
|
||||
"""Read and validate a numeric FunASR tuning value from the environment."""
|
||||
raw_value = os.getenv(name, str(default)).strip()
|
||||
try:
|
||||
value = float(raw_value)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"{name} must be a number; got {raw_value!r}") from exc
|
||||
if not minimum <= value <= maximum:
|
||||
raise ValueError(f"{name} must be between {minimum} and {maximum}; got {value}")
|
||||
return value
|
||||
|
||||
|
||||
def _positive_env_int(name: str, default: int) -> int:
|
||||
"""Read a positive integer environment value with a clear startup error."""
|
||||
raw_value = os.getenv(name, str(default)).strip()
|
||||
try:
|
||||
value = int(raw_value)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"{name} must be a positive integer; got {raw_value!r}") from exc
|
||||
if value <= 0:
|
||||
raise ValueError(f"{name} must be greater than zero; got {value}")
|
||||
return value
|
||||
|
||||
|
||||
# An explicit value disables FunASR's duration-based silence schedule for testing.
|
||||
VAD_MAX_END_SILENCE_MS = _positive_env_int("FUNASR_VAD_MAX_END_SILENCE_MS", 800)
|
||||
VAD_PARAGRAPH_MAX_END_SILENCE_MS = _positive_env_int(
|
||||
"FUNASR_VAD_PARAGRAPH_MAX_END_SILENCE_MS", 5000
|
||||
)
|
||||
# Higher margins classify weak background noise as non-speech more readily.
|
||||
VAD_SPEECH_NOISE_THRESHOLD = _bounded_env_float(
|
||||
"FUNASR_VAD_SPEECH_NOISE_THRES", 0.6, 0.0, 1.0
|
||||
)
|
||||
|
||||
|
||||
def to_python(obj):
|
||||
"""递归地把 numpy / torch 等类型转成纯 Python，可 JSON 序列化。"""
|
||||
try:
|
||||
|
|
@ -374,10 +409,17 @@ async def ws_serve(websocket, path=None):
|
|||
|
||||
websocket.status_dict_asr = {} # hotword 等
|
||||
websocket.status_dict_asr_online = {"cache": {}, "is_final": False}
|
||||
websocket.status_dict_vad = {"cache": {}, "is_final": False}
|
||||
# Pass the test knob as an explicit FunASR argument so it uses a fixed threshold.
|
||||
websocket.status_dict_vad = {
|
||||
"cache": {},
|
||||
"is_final": False,
|
||||
"max_end_silence_time": VAD_MAX_END_SILENCE_MS,
|
||||
"speech_noise_thres": VAD_SPEECH_NOISE_THRESHOLD,
|
||||
}
|
||||
websocket.status_dict_punc = {"cache": {}}
|
||||
|
||||
websocket.chunk_interval = 10
|
||||
websocket.sentence_strategy = 0
|
||||
websocket.vad_pre_idx = 0
|
||||
speech_start = False
|
||||
speech_end_i = -1
|
||||
|
|
@ -504,6 +546,16 @@ async def ws_serve(websocket, path=None):
|
|||
messagejson["chunk_interval"], websocket.chunk_interval
|
||||
)
|
||||
|
||||
if "sentence_strategy" in messagejson:
|
||||
# Map the Tencent selector to FunASR's VAD endpoint duration.
|
||||
strategy = _safe_int(messagejson["sentence_strategy"], 0)
|
||||
websocket.sentence_strategy = strategy if strategy in (0, 1) else 0
|
||||
websocket.status_dict_vad["max_end_silence_time"] = (
|
||||
VAD_PARAGRAPH_MAX_END_SILENCE_MS
|
||||
if websocket.sentence_strategy == 1
|
||||
else VAD_MAX_END_SILENCE_MS
|
||||
)
|
||||
|
||||
if "wav_name" in messagejson:
|
||||
websocket.wav_name = messagejson.get("wav_name") or websocket.wav_name
|
||||
|
||||
|
|
|
|||
|
|
@ -146,6 +146,11 @@ class BrowserSession:
|
|||
else None
|
||||
)
|
||||
self.speed_factor = max(0.5, min(3.0, float(start.get("speed_factor") or 1.0)))
|
||||
try:
|
||||
sentence_strategy = int(start.get("sentence_strategy", 0))
|
||||
except (TypeError, ValueError):
|
||||
sentence_strategy = 0
|
||||
self.sentence_strategy = sentence_strategy if sentence_strategy in (0, 1) else 0
|
||||
self.pending_pcm = bytearray()
|
||||
self.turn_audio = bytearray()
|
||||
self.total_audio_ms = 0.0
|
||||
|
|
@ -241,6 +246,21 @@ class BrowserSession:
|
|||
start_ms = self.turn_start_ms
|
||||
end_ms = self.total_audio_ms
|
||||
if final_text:
|
||||
# Punctuation runs on finalized turns; optional-model failures
|
||||
# leave the raw ASR text usable.
|
||||
try:
|
||||
punctuation = await self.auxiliary.punctuate(final_text)
|
||||
if not punctuation.get("available", False):
|
||||
LOGGER.warning(
|
||||
"FunASR punctuation is unavailable: %s",
|
||||
punctuation.get("error", "model is not configured"),
|
||||
)
|
||||
else:
|
||||
punctuated_text = str(punctuation.get("text") or "").strip()
|
||||
if punctuated_text:
|
||||
final_text = punctuated_text
|
||||
except Exception:
|
||||
LOGGER.exception("FunASR punctuation request failed; keeping raw text")
|
||||
await self.emit_sentence(
|
||||
final_text, final=True, start_time_ms=start_ms, end_time_ms=end_ms
|
||||
)
|
||||
|
|
@ -386,6 +406,7 @@ async def websocket_handler(request: web.Request) -> web.WebSocketResponse:
|
|||
"decoder_chunk_look_back": int(
|
||||
os.getenv("FUNASR_DECODER_LOOK_BACK", "1")
|
||||
),
|
||||
"sentence_strategy": session.sentence_strategy,
|
||||
"audio_fs": SAMPLE_RATE,
|
||||
"wav_name": voice_id,
|
||||
"is_speaking": True,
|
||||
|
|
|
|||
|
|
@ -1,7 +0,0 @@
|
|||
aiohttp==3.11.11
|
||||
python-dotenv>=1.0
|
||||
funasr==1.4.16
|
||||
modelscope[framework]==1.34.0
|
||||
soundfile==0.13.1
|
||||
librosa==0.11.0
|
||||
websockets>=12,<14
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
aiohttp==3.11.11
|
||||
python-dotenv>=1.0
|
||||
funasr==1.4.16
|
||||
modelscope[framework]==1.34.0
|
||||
soundfile==0.13.1
|
||||
librosa==0.11.0
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
# Install a torch/torchaudio build matching the host CUDA or CPU environment first.
|
||||
aiohttp==3.11.11
|
||||
python-dotenv>=1.0
|
||||
funasr==1.4.16
|
||||
modelscope[framework]==1.34.0
|
||||
soundfile==0.13.1
|
||||
librosa==0.11.0
|
||||
numpy>=1.24
|
||||
websockets>=12,<14
|
||||
|
|
@ -1,7 +0,0 @@
|
|||
# 使用新版 VLLM 原生支持 Qwen3-ASR，避免 qwen-asr[vllm] 将 VLLM 锁定到 0.14.0。
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130
|
||||
torch==2.13.0
|
||||
torchvision==0.28.0
|
||||
torchaudio==2.11.0
|
||||
vllm==0.28.0
|
||||
python-dotenv>=1.0
|
||||
|
|
@ -16,10 +16,11 @@ from urllib.request import urlopen
|
|||
from dotenv import load_dotenv
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
# Direct execution from backend/ puts that directory first; prepend the repository
|
||||
# root so backend modules and the local model manifest resolve consistently.
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from scripts.model_manifest import (
|
||||
from backend.model_manifest import (
|
||||
auxiliary_models,
|
||||
load_manifest,
|
||||
model_directory,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@ import sys
|
|||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from backend.realtime_websocket.funasr_server import main
|
||||
|
|
|
|||
|
|
@ -19,8 +19,6 @@ from dotenv import load_dotenv
|
|||
|
||||
# 启动器自动读取 demo/.env；系统环境变量仍然优先，便于部署平台临时覆盖配置。
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
# Resolve the shared model manifest for direct script startup.
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
load_dotenv(PROJECT_ROOT / ".env")
|
||||
|
||||
|
|
@ -28,7 +26,7 @@ load_dotenv(PROJECT_ROOT / ".env")
|
|||
# VLLM 子进程命令和就绪提示都使用同一个端口，避免配置不一致导致误判。
|
||||
SERVER_PORT = int(os.getenv("VLLM_PORT", "9950"))
|
||||
|
||||
from scripts.model_manifest import load_manifest, model_directory, resolve_model_id
|
||||
from backend.model_manifest import load_manifest, model_directory, resolve_model_id
|
||||
|
||||
|
||||
def has_model_weights(model_path: Path) -> bool:
|
||||
|
|
@ -125,7 +123,7 @@ def build_server_command(args: argparse.Namespace, model_id: str, model_path: Pa
|
|||
executable_name = os.getenv("VLLM_EXECUTABLE", "vllm")
|
||||
executable = shutil.which(executable_name)
|
||||
if executable is None:
|
||||
raise RuntimeError(f"{executable_name} was not found; install backend/requirements-qwen-legacy.txt first")
|
||||
raise RuntimeError(f"{executable_name} was not found; install requirements.txt first")
|
||||
|
||||
served_model_name = args.served_model_name or model_id
|
||||
command = [
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ import unittest
|
|||
from pathlib import Path
|
||||
|
||||
from scripts.download_models import fix_camplusplus_config
|
||||
from scripts.model_manifest import auxiliary_models, load_manifest, model_directory, resolve_model_id
|
||||
from backend.model_manifest import auxiliary_models, load_manifest, model_directory, resolve_model_id
|
||||
|
||||
|
||||
class ModelManifestTests(unittest.TestCase):
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
# Dependencies for serving the unchanged Tencent demo UI.
|
||||
aiohttp==3.11.11
|
||||
python-dotenv>=1.0
|
||||
|
|
@ -94,8 +94,6 @@ const SPEAKER_COLORS = ['#4a7dff', '#52c41a', '#faad14', '#ff4d4f', '#9254de', '
|
|||
let sentenceMap = {};
|
||||
let speakerOrderMap = {};
|
||||
let speakerOrderCounter = 0;
|
||||
let lastConfirmedBubbleEl = null;
|
||||
let pendingSpanMap = {};
|
||||
|
||||
// ===== Input Mode Tabs =====
|
||||
function switchMode(mode) {
|
||||
|
|
@ -342,31 +340,11 @@ function renderBubble(sentence) {
|
|||
|
||||
if (isUnknown) {
|
||||
const pendingText = sentence.sentence + (isInterim ? ' ...' : '');
|
||||
let pending = pendingSpanMap[id];
|
||||
if (pending) {
|
||||
pending.spanEl.textContent = pendingText;
|
||||
// Keep unclassified text separate so it is not attributed to the prior speaker.
|
||||
renderFallbackPendingBubble(id, pendingText, isInterim, sentence);
|
||||
elResultArea.scrollTop = elResultArea.scrollHeight;
|
||||
return;
|
||||
}
|
||||
if (lastConfirmedBubbleEl) {
|
||||
const body = lastConfirmedBubbleEl.querySelector('.bubble-body');
|
||||
const span = document.createElement('span');
|
||||
span.className = 'pending-text';
|
||||
span.dataset.sentenceId = id;
|
||||
span.textContent = pendingText;
|
||||
body.appendChild(span);
|
||||
pendingSpanMap[id] = { hostEl: lastConfirmedBubbleEl, spanEl: span };
|
||||
} else {
|
||||
renderFallbackPendingBubble(id, pendingText, isInterim);
|
||||
}
|
||||
elResultArea.scrollTop = elResultArea.scrollHeight;
|
||||
return;
|
||||
}
|
||||
|
||||
if (pendingSpanMap[id]) {
|
||||
pendingSpanMap[id].spanEl.remove();
|
||||
delete pendingSpanMap[id];
|
||||
}
|
||||
|
||||
let entry = sentenceMap[id];
|
||||
let insertBefore = null;
|
||||
|
|
@ -418,17 +396,15 @@ function renderBubble(sentence) {
|
|||
const timeSpan = el.querySelector('.bubble-time');
|
||||
const body = el.querySelector('.bubble-body');
|
||||
timeSpan.textContent = formatTimeRange(sentence.start_time, sentence.end_time);
|
||||
body.querySelectorAll('.pending-text').forEach(s => s.remove());
|
||||
Array.from(body.childNodes).filter(n => n.nodeType === Node.TEXT_NODE).forEach(n => n.remove());
|
||||
const textNode = document.createTextNode(sentence.sentence + (isInterim ? ' ...' : ''));
|
||||
body.insertBefore(textNode, body.firstChild);
|
||||
body.className = 'bubble-body' + (isInterim ? ' interim' : '');
|
||||
entry.speakerId = speakerId;
|
||||
lastConfirmedBubbleEl = el;
|
||||
elResultArea.scrollTop = elResultArea.scrollHeight;
|
||||
}
|
||||
|
||||
function renderFallbackPendingBubble(id, text, isInterim) {
|
||||
function renderFallbackPendingBubble(id, text, isInterim, sentence) {
|
||||
let entry = sentenceMap[id];
|
||||
if (!entry) {
|
||||
const el = document.createElement('div');
|
||||
|
|
@ -457,6 +433,9 @@ function renderFallbackPendingBubble(id, text, isInterim) {
|
|||
entry = { el: el, speakerId: -1 };
|
||||
sentenceMap[id] = entry;
|
||||
}
|
||||
entry.el.querySelector('.bubble-time').textContent = formatTimeRange(
|
||||
sentence.start_time, sentence.end_time
|
||||
);
|
||||
const body = entry.el.querySelector('.bubble-body');
|
||||
body.textContent = text;
|
||||
body.className = 'bubble-body' + (isInterim ? ' interim' : '');
|
||||
|
|
@ -482,8 +461,6 @@ async function startRecognition() {
|
|||
sentenceMap = {};
|
||||
speakerOrderMap = {};
|
||||
speakerOrderCounter = 0;
|
||||
lastConfirmedBubbleEl = null;
|
||||
pendingSpanMap = {};
|
||||
audioChunks = [];
|
||||
elBtnExportWav.disabled = true;
|
||||
elResultPlaceholder?.remove();
|
||||
|
|
|
|||
|
|
@ -45,6 +45,22 @@
|
|||
"min_total_size_bytes": 1000000,
|
||||
"funasr_alias": "fsmn-vad"
|
||||
},
|
||||
"iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch": {
|
||||
"alias": "ct-punc",
|
||||
"directory": "iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch",
|
||||
"kind": "punctuation",
|
||||
"description": "Optional FunASR CT-Transformer punctuation restoration",
|
||||
"required_files": [
|
||||
"configuration.json",
|
||||
"config.yaml"
|
||||
],
|
||||
"any_files": [
|
||||
"*.pt",
|
||||
"*.bin",
|
||||
"*.onnx"
|
||||
],
|
||||
"min_total_size_bytes": 1000000
|
||||
},
|
||||
"iic/speech_campplus_speaker-diarization_common": {
|
||||
"alias": "diarization",
|
||||
"directory": "iic/speech_campplus_speaker-diarization_common",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
# Unified dependencies for the frontend, FunASR, CAM++, and Qwen3-ASR vLLM.
|
||||
# The PyTorch and vLLM pins below target the GB10 CUDA 13 deployment.
|
||||
# Update the CUDA index and torch-family pins for a different host.
|
||||
aiohttp==3.11.11
|
||||
python-dotenv>=1.0
|
||||
numpy>=1.24
|
||||
funasr==1.4.16
|
||||
modelscope[framework]==1.34.0
|
||||
soundfile==0.13.1
|
||||
librosa==0.11.0
|
||||
websockets>=12,<14
|
||||
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130
|
||||
torch==2.13.0
|
||||
torchvision==0.28.0
|
||||
torchaudio==2.11.0
|
||||
vllm==0.28.0
|
||||
|
|
@ -73,7 +73,7 @@ def download_model(
|
|||
from modelscope.hub.snapshot_download import snapshot_download
|
||||
except ImportError as exc:
|
||||
raise RuntimeError(
|
||||
"ModelScope is required for downloading; install backend/requirements-funasr.txt first"
|
||||
"ModelScope is required for downloading; install requirements.txt first"
|
||||
) from exc
|
||||
|
||||
model_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
|
@ -174,12 +174,12 @@ def main() -> int:
|
|||
auxiliary_group.add_argument(
|
||||
"--auxiliary-only",
|
||||
action="store_true",
|
||||
help="Only download/check VAD, speaker, diarization, and aligner assets",
|
||||
help="Only download/check VAD, speaker, punctuation, diarization, and aligner assets",
|
||||
)
|
||||
auxiliary_group.add_argument(
|
||||
"--funasr-runtime",
|
||||
action="store_true",
|
||||
help="Download/check only streaming Paraformer, FSMN-VAD, and CAM++",
|
||||
help="Download/check streaming Paraformer, FSMN-VAD, CAM++, and punctuation",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
|
|
@ -191,7 +191,7 @@ def main() -> int:
|
|||
cache_dir = args.cache_dir.resolve() if args.cache_dir else None
|
||||
selected_assets: list[tuple[str, dict[str, object]]] = []
|
||||
if args.funasr_runtime:
|
||||
# The realtime branch needs the streaming ASR plus its VAD and CAM++ only.
|
||||
# Punctuation is optional at runtime, but included for complete local output.
|
||||
asr_id = resolve_model_id("paraformer-zh-streaming", manifest)
|
||||
selected_assets.append((asr_id, manifest["models"][asr_id]))
|
||||
assets = auxiliary_models(manifest)
|
||||
|
|
@ -204,7 +204,14 @@ def main() -> int:
|
|||
if config.get("kind") == "speaker_verification"
|
||||
and model_id.startswith("iic/")
|
||||
)
|
||||
selected_assets.extend((model_id, assets[model_id]) for model_id in (vad_id, cam_id))
|
||||
punctuation_id = next(
|
||||
model_id for model_id, config in assets.items()
|
||||
if config.get("kind") == "punctuation"
|
||||
)
|
||||
selected_assets.extend(
|
||||
(model_id, assets[model_id])
|
||||
for model_id in (vad_id, cam_id, punctuation_id)
|
||||
)
|
||||
else:
|
||||
if not args.auxiliary_only:
|
||||
model_id = resolve_model_id(args.model, manifest)
|
||||
|
|
|
|||
|
|
@ -1,79 +1,29 @@
|
|||
"""独立 VLLM 部署项目的模型清单与路径解析辅助函数。"""
|
||||
"""Compatibility exports for shared model download scripts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import sys
|
||||
|
||||
# Direct script execution puts scripts/ first on sys.path; add the repository root
|
||||
# so the shared backend manifest remains importable in both launch modes.
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
MANIFEST_PATH = PROJECT_ROOT / "model_manifest.json"
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
|
||||
def load_manifest(path: Path = MANIFEST_PATH) -> dict[str, Any]:
|
||||
"""读取本项目自己的模型清单，整个过程不导入原项目代码。"""
|
||||
with path.open("r", encoding="utf-8") as manifest_file:
|
||||
manifest = json.load(manifest_file)
|
||||
if not isinstance(manifest.get("models"), dict) or not manifest["models"]:
|
||||
raise ValueError("model_manifest.json 必须包含非空的 models 对象")
|
||||
return manifest
|
||||
|
||||
|
||||
def resolve_model_id(model: str | None, manifest: dict[str, Any]) -> str:
|
||||
"""将默认值、短别名或完整模型 ID 解析为一个 ASR 模型。"""
|
||||
models = manifest["models"]
|
||||
requested = (model or "default").strip()
|
||||
if requested == "default":
|
||||
requested = str(manifest["default_model"])
|
||||
|
||||
if requested in models:
|
||||
return requested
|
||||
|
||||
for model_id, config in models.items():
|
||||
if requested.lower() == str(config.get("alias", "")).lower():
|
||||
return model_id
|
||||
raise ValueError(f"不支持的 ASR 模型 '{model}'，可选模型：{', '.join(models)}")
|
||||
|
||||
|
||||
def model_directory(model_id: str, manifest: dict[str, Any], models_dir: Path) -> Path:
|
||||
"""根据清单返回 ASR 或辅助模型实际使用的本地目录。"""
|
||||
config = manifest.get("models", {}).get(model_id)
|
||||
if config is None:
|
||||
config = manifest.get("auxiliary_models", {}).get(model_id)
|
||||
if not isinstance(config, dict) or not config.get("directory"):
|
||||
raise ValueError(f"模型 '{model_id}' 在清单中没有配置本地目录")
|
||||
return models_dir / str(config["directory"])
|
||||
|
||||
|
||||
def auxiliary_models(manifest: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||
"""返回可独立部署的 VAD、说话人和对齐模型资产。"""
|
||||
models = manifest.get("auxiliary_models", {})
|
||||
if not isinstance(models, dict):
|
||||
raise ValueError("model_manifest.json 的 auxiliary_models 必须是对象")
|
||||
return {str(model_id): config for model_id, config in models.items() if isinstance(config, dict)}
|
||||
|
||||
def resolve_auxiliary_model_id(
|
||||
model: str | None,
|
||||
manifest: dict[str, Any],
|
||||
kind: str | None = None,
|
||||
) -> str:
|
||||
"""Resolve a configured auxiliary asset by ID, alias, or FunASR alias."""
|
||||
assets = auxiliary_models(manifest)
|
||||
requested = (model or "").strip()
|
||||
if requested in assets and (kind is None or assets[requested].get("kind") == kind):
|
||||
return requested
|
||||
|
||||
for model_id, config in assets.items():
|
||||
if kind is not None and config.get("kind") != kind:
|
||||
continue
|
||||
aliases = [config.get("alias"), config.get("funasr_alias")]
|
||||
if requested.lower() in {
|
||||
str(alias).lower() for alias in aliases if isinstance(alias, str)
|
||||
}:
|
||||
return model_id
|
||||
expected = ", ".join(
|
||||
model_id for model_id, config in assets.items()
|
||||
if kind is None or config.get("kind") == kind
|
||||
from backend.model_manifest import (
|
||||
MANIFEST_PATH,
|
||||
auxiliary_models,
|
||||
load_manifest,
|
||||
model_directory,
|
||||
resolve_auxiliary_model_id,
|
||||
resolve_model_id,
|
||||
)
|
||||
raise ValueError(f"Unsupported auxiliary model '{model}'; configured assets: {expected}")
|
||||
|
||||
__all__ = [
|
||||
"MANIFEST_PATH",
|
||||
"PROJECT_ROOT",
|
||||
"auxiliary_models",
|
||||
"load_manifest",
|
||||
"model_directory",
|
||||
"resolve_auxiliary_model_id",
|
||||
"resolve_model_id",
|
||||
]
|
||||
|
|
|
|||
Loading…
Reference in New Issue