"""任务执行后端:Celery(生产)与 inline 线程池(本地/离线自检)。""" from __future__ import annotations import logging import threading import time from concurrent.futures import ThreadPoolExecutor from app.config import settings from app.models import Task log = logging.getLogger("nex.execution") _celery = None _pool: ThreadPoolExecutor | None = None _pool_lock = threading.Lock() _futures: set = set() CELERY_TASK_NAMES = { "generate_shot": "app.jobs.generate_shot", "voiceover": "app.jobs.voiceover", "qc_shot": "app.jobs.qc_shot", "compose": "app.jobs.compose", "import": "app.jobs.import_package", "asset": "app.jobs.asset", } QUEUES = {"generate_shot": "generate", "voiceover": "audio", "qc_shot": "qc", "compose": "compose", "import": "io", "asset": "io"} def celery_app(): global _celery if _celery is None: from app.worker import celery_app as app_obj _celery = app_obj return _celery def inline_pool() -> ThreadPoolExecutor: global _pool with _pool_lock: if _pool is None: _pool = ThreadPoolExecutor(max_workers=max(1, settings.inline_workers), thread_name_prefix="nex-inline") return _pool def backend_name() -> str: return (settings.task_executor or "celery").lower() def submit(db_task: Task) -> str: """把 DB 任务交给执行后端,返回句柄(celery id / inline future id)。""" kind = db_task.kind if backend_name() == "inline": from app.jobs import handlers def _run(task_id: int = db_task.id, task_kind: str = kind) -> None: try: handlers.run(task_kind, task_id) except Exception: # handler 内部已记录错误与重试策略 log.exception("inline 任务执行失败 kind=%s task=%s", task_kind, task_id) future = inline_pool().submit(_run) _futures.add(future) future.add_done_callback(_futures.discard) return f"inline-{id(future)}" name = CELERY_TASK_NAMES.get(kind) if not name: raise RuntimeError(f"未知的任务类型:{kind}") result = celery_app().send_task(name, args=[db_task.id], queue=QUEUES.get(kind, "default")) return result.id def pending() -> int: """inline 模式下仍在执行的任务数(celery 模式返回 0,队列状态以数据库为准)。""" return sum(1 for f in list(_futures) if not f.done()) def wait_idle(timeout: float | None = None, *, poll: float = 0.05) -> bool: """等待 inline 线程池空闲;超时返回 False。""" deadline = None if timeout is None else time.monotonic() + timeout while pending(): if deadline is not None and time.monotonic() > deadline: return False time.sleep(poll) return True def shutdown() -> None: global _pool if _pool is not None: _pool.shutdown(wait=False, cancel_futures=False) _pool = None