WeKnora/internal/container
wizardchen 70e2e85fa6 feat(wiki): dedicated worker pool with concurrent claim-based batching
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.

- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
  own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
  concurrency, default 16), split from the parse pool (NewParseAsynqServer).
  Both run the same mux over disjoint queue sets. Replaces the old
  weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
  and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
  without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
  dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
  batches per KB; Map/Reduce fan-out sized within a batch. New config knob
  + system setting + swagger docs.
2026-07-09 17:41:07 +08:00
..
audit_sink.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
audit_sink_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
container.go feat(wiki): dedicated worker pool with concurrent claim-based batching 2026-07-09 17:41:07 +08:00
engine_factory.go feat: support Milvus database selection for vector stores 2026-06-05 15:11:37 +08:00
engine_factory_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
engine_factory_test.go feat: support Milvus database selection for vector stores 2026-06-05 15:11:37 +08:00
reset_pending_tasks.go fix(container): correct resetPendingTasks SQL on startup 2026-05-29 15:46:11 +08:00
reset_pending_tasks_test.go fix(container): correct resetPendingTasks SQL on startup 2026-05-29 15:46:11 +08:00