WeKnora/internal/middleware
wizardchen 3ae3ea97c5 fix(knowledge): prevent documents from getting stuck in "processing"
Several failure modes left Knowledge.parse_status pinned at "processing"
forever, with no signal to users beyond a permanent spinner. This commit
addresses the root causes and adds a safety net.

- Asynq worker pool: explicit Concurrency (default 16, env-tunable via
  WEKNORA_ASYNQ_CONCURRENCY) so batch uploads don't queue behind a
  CPU-count-sized worker pool. Redis op timeouts raised to 500ms/1000ms
  (WEKNORA_REDIS_OP_TIMEOUT_MS) to absorb bursty multimodal counter ops.

- DocReader RPC: cap each call with WEKNORA_DOCREADER_CALL_TIMEOUT
  (default 30m). Without this, a hung docreader pinned a worker for the
  full DocumentProcessTimeout window.

- ImageMultimodal: finalize-on-last-attempt semantics. A permanently
  failing single image no longer strands the parent — the asynq retry
  is allowed to run, but on the final attempt we count the image
  regardless of outcome. Redis DECR errors fall back to enqueuing the
  post-process task instead of returning silently.

- Dead-letter callback: when DocumentProcess / KnowledgePostProcess /
  ManualProcess exhausts retries, immediately mark the corresponding
  Knowledge as failed with the last error. This surfaces the failure
  in the UI without waiting for the housekeeping sweep.

- HousekeepingService: 5-minute cron that flips knowledge rows stuck
  in "processing" past DocumentProcessTimeout + 10m to failed, plus
  summary rows stuck > 1h. Catches anything the other safety nets
  miss (worker SIGKILL mid-handler, etc.). Disable with
  WEKNORA_HOUSEKEEPING_ENABLED=false.

- Distributed startup recovery: previously the post-restart sweep was
  skipped whenever REDIS_ADDR was set, even though Asynq does not
  reschedule the task that was actively running on the dead instance.
  Now the sweep runs in distributed mode too, but only against rows
  older than 30 minutes to avoid racing peer instances.
2026-05-28 15:14:45 +08:00
..
asynqdl fix(knowledge): prevent documents from getting stuck in "processing" 2026-05-28 15:14:45 +08:00
access.go refactor(tenant_member): simplify tenant ID resolution in handlers 2026-05-18 17:28:58 +08:00
access_test.go refactor(tenant_member): simplify tenant ID resolution in handlers 2026-05-18 17:28:58 +08:00
audit_provider.go feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303) 2026-05-18 17:28:58 +08:00
auth.go fix(im): make presigned URL flow diagnosable end-to-end 2026-05-28 08:03:57 +08:00
auth_public_ratelimit.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
auth_role_test.go chore(rbac): update default behavior for tenant RBAC configuration 2026-05-18 21:24:28 +08:00
error_handler.go init commit 2025-08-05 15:08:07 +08:00
kb_access.go refactor(rbac): address kb-access guard review findings 2026-05-18 17:28:58 +08:00
kb_access_test.go refactor(rbac): address kb-access guard review findings 2026-05-18 17:28:58 +08:00
language.go fix(i18n): use WEKNORA_LANGUAGE env for prompt language fallback instead of hardcoded zh-CN 2026-04-02 12:01:25 +08:00
logger.go fix(middleware): mask camelCase secret fields in request logs 2026-05-12 19:27:29 +08:00
logger_test.go fix(middleware): mask camelCase secret fields in request logs 2026-05-12 19:27:29 +08:00
rbac.go feat(system-admin): implement revocation of system admin privileges with safeguards 2026-05-26 21:13:56 +08:00
rbac_audit_test.go feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303) 2026-05-18 17:28:58 +08:00
rbac_test.go chore(rbac): update default behavior for tenant RBAC configuration 2026-05-18 21:24:28 +08:00
recovery.go fix: replace standard log with structured logger in main and recovery middleware 2026-02-06 17:59:15 +08:00
trace.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00