WeKnora/internal
wizardchen c8a4fd5ba3 fix(wiki): prevent silent work loss and cross-batch doc races
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:

- Slug-lock timeout / reduce failure no longer silently drops work. When a
  per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
  an error, the page's read-modify-write never lands. Previously the owning
  document was still counted as successful and its pending row was deleted at
  trim time, permanently losing the contribution (finalize only rebuilds the
  index / cross-links, it does not re-run reduce). Such documents are now
  folded into failedOps: their rows stay queued, their finalizing slot is
  held, and requeueFailedOps' fail_count budget retries them (dead-lettering
  a permanently hot/broken slug).

- ClaimBatch now claims by dedup_key affinity so a document is never split
  across concurrent batches. limit counts distinct dedup_keys (documents),
  and ALL eligible rows sharing a key are claimed together. On Postgres the
  per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
  concurrent claimers disjoint key sets. This restores in-document ordering
  (e.g. ingest-then-retract collapses correctly) that the per-batch dedup
  could not guarantee once the exclusive per-KB lock was removed, preventing
  a stale ingest from resurrecting a retracted document.

Adds repository tests for same-key affinity and in-flight sibling rows.
2026-07-09 17:41:07 +08:00
..
agent Simplify KB duplicate API and improve naming/docs coverage. 2026-07-08 20:55:54 +08:00
application fix(wiki): prevent silent work loss and cross-batch doc races 2026-07-09 17:41:07 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
common fix(redis): harden TLS tests and surface TLS config at startup 2026-07-06 23:04:52 +08:00
config test: add document process task options tests 2026-05-28 20:27:53 +08:00
container feat(wiki): dedicated worker pool with concurrent claim-based batching 2026-07-09 17:41:07 +08:00
database feat(system-info): surface DB migration errors with troubleshooting links 2026-05-14 16:34:50 +08:00
datasource fix(security): close remaining GHSA gaps from security triage 2026-07-02 15:03:53 +08:00
errors feat(knowledge): track per-stage parsing progress with /stages API 2026-05-28 15:14:45 +08:00
event feat(mcp): prompt and resume OAuth authorization mid-conversation 2026-06-29 23:22:28 +08:00
handler Simplify KB duplicate API and improve naming/docs coverage. 2026-07-08 20:55:54 +08:00
im fix(im): harden DingTalk file download with SSRF checks and pictureDownloadCode fallback 2026-07-09 15:38:13 +08:00
infrastructure fix(web-search): polish Keenable provider integration after #1946 2026-07-07 17:14:39 +08:00
logger fix: harden document pipeline concurrency and diagnostics 2026-07-09 13:26:40 +08:00
mcp feat(im): include per-service MCP OAuth URLs in IM auth notices 2026-07-01 17:53:54 +08:00
middleware Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
models fix(security): harden model client redirects 2026-07-02 19:15:46 +08:00
ratelimit feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
router feat(wiki): dedicated worker pool with concurrent claim-based batching 2026-07-09 17:41:07 +08:00
runtime fix(redis): harden TLS tests and surface TLS config at startup 2026-07-06 23:04:52 +08:00
sandbox perf(sandbox): compile command-substitution regexps once 2026-06-16 19:51:32 +08:00
searchutil feat(backend): improve parent-child chunk merge and imageinfo matching 2026-06-16 01:09:07 +08:00
stream feat(redis): support TLS for Redis connections (#1930) 2026-07-06 23:02:02 +08:00
tracing/langfuse feat(observability): enhance Langfuse retrieve and rerank tracing 2026-06-16 01:09:23 +08:00
types fix(wiki): prevent silent work loss and cross-batch doc races 2026-07-09 17:41:07 +08:00
utils fix(security): harden outbound requests and file previews 2026-07-02 19:15:46 +08:00