* fix(tenant): stop wiki tasks of soft-deleted tenants from issuing model requests
Tenant deletion only soft-deletes the tenant and its memberships; its
knowledge bases, model configs, and durable task_pending_ops rows survive.
Wiki ingest/finalize tasks restored from those ops only checked KB state,
so a deleted tenant kept accruing new LLM requests (observed as ~11h of
continued synthesis calls in #3593).
Gate every task entry point on tenant liveness:
- EnqueueIfKnowledgeBaseActive now also requires tenants.deleted_at IS NULL
inside the same transaction (fail closed).
- Startup recovery purges pending rows whose tenant is soft-deleted (the
predicate is wrapped in parentheses: NOT EXISTS(...) AND EXISTS(...) would
parse as (NOT EXISTS(...)) AND EXISTS(...) and silently match nothing).
- Wiki ingest and finalize handlers check tenant liveness from the payload
before any model call and drop the KB's queue when the tenant is gone.
Liveness lookup errors fail open; the guarded enqueue and recovery paths
enforce the same invariant on their own DB access.
Adds TaskPendingOpsTenantLiveness as an optional extension interface,
regression tests for the enqueue guard, startup recovery, and the task
entry guards.
Fixes#3593
* style(container): wrap long log line over 120 chars for golangci-lint (lll)
---------
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
- Abort check: a read failure now stops the pipeline instead of carrying
on. Continuing let a later full-row Save overwrite a cancel or delete
it never saw. processChunks now returns an error, and the handlers
return it, so asynq retries an unknown state instead of acking it.
- Resolve the vector store before deleting the old chunks, and retry
when the lookup itself failed. A failed persist or an interrupted
context no longer acks with the row still in flight.
- Drain and release the wiki slots in one transaction, so a failed
release rolls the delete back for the retry. Skip any document that
still has a fresh claim in the lane: its live batch releases it.
- Re-armed and restart-recovered wiki triggers carry the language the
ops were queued with, rather than the server default.
Housekeeping skips any row that still has a durable wiki ingest op, with
no age limit, so a wiki that can never run left its documents in
"finalizing" forever:
- A KB whose wiki was disabled, has no synthesis model, or whose model
was deleted failed the trigger before claiming anything; after the
retries the trigger was archived and the ops stayed. Treat these as
terminal: drain the unclaimed ingest ops and release each document's
wiki slot. Retract ops stay queued; rows a live batch holds are left
to it.
- A lost or archived trigger left the ops with nothing to wake their
consumer until a restart. Housekeeping now re-arms one trigger per KB
for rows held only by durable ops, at most once per stale threshold.
* fix(queue): order ClaimBatch by fail_count so retries cannot starve new work
ClaimBatch picks work with ORDER BY id ASC (Postgres) / MIN(id) ASC (GORM),
i.e. oldest-first by primary key.
Retry does not move a row: requeueFailedOps calls IncrFailCount and then
ReleaseByIDs, which deliberately keeps the row in place so the fail_count
budget keeps counting down.
Together those two produce a starvation mode. A document that fails
repeatedly keeps its original, lowest id, so it is always selected first,
while every never-attempted document - enqueued later, hence a higher id -
waits behind it until the stuck op finally exhausts wikiMaxFailRetries and
dead-letters. Observed on a real 87-document KB: 42 ops pending, the four
head ops were all re-runs of documents that already had a root page, and 38
never-attempted documents were never claimed. With 8 concurrent workers and
a drained asynq queue (pending=0, retry=0, active=4), the bottleneck was the
claim ordering, not throughput.
Order by least-failed first, oldest-first within the same fail_count. When
every row has fail_count = 0 this degenerates exactly to the previous
id-ordering, so the existing FIFO tests still pass.
Test: TestTaskPendingOps_ClaimBatch_PrefersLeastFailed replays the real retry
path (claim -> IncrFailCount -> ReleaseByIDs) and asserts that the
never-attempted document is claimed first and the retried one is still
eventually picked up rather than stranded.
* fix(queue): apply the same least-failed PeekBatch order in Lite mode
Lite wiki ingest still peeks rather than claims. Ordering that path by
id alone would keep the same head-of-line starvation this PR already
fixed for ClaimBatch. Align comments with the new selection rule.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
- Renamed TestTaskPendingOps_ClaimBatch_NewSiblingRowNotDoubleClaimed to TestTaskPendingOps_ClaimBatch_LateSiblingBlockedByFreshClaim for clarity on behavior.
- Added new test TestTaskPendingOps_ClaimBatch_LateSiblingClaimableAfterStale to verify claim behavior when a holder crashes.
- Updated ClaimBatch logic to ensure that fresh claims block late siblings entirely, maintaining serialization of same-document operations.
- Improved comments in task_queue.go to clarify the handling of concurrent claims and the implications of the claimed_at field.
- Added crash safety mechanism in ProcessWikiIngest to release claims on abnormal exits, ensuring timely re-claims of pending operations.
Two correctness fixes on the Phase 3 concurrent claim-based ingest path:
- Slug-lock timeout / reduce failure no longer silently drops work. When a
per-slug lock can't be acquired within wikiSlugLockWait, or reduce returns
an error, the page's read-modify-write never lands. Previously the owning
document was still counted as successful and its pending row was deleted at
trim time, permanently losing the contribution (finalize only rebuilds the
index / cross-links, it does not re-run reduce). Such documents are now
folded into failedOps: their rows stay queued, their finalizing slot is
held, and requeueFailedOps' fail_count budget retries them (dead-lettering
a permanently hot/broken slug).
- ClaimBatch now claims by dedup_key affinity so a document is never split
across concurrent batches. limit counts distinct dedup_keys (documents),
and ALL eligible rows sharing a key are claimed together. On Postgres the
per-key anchor row is locked with FOR UPDATE SKIP LOCKED, handing
concurrent claimers disjoint key sets. This restores in-document ordering
(e.g. ingest-then-retract collapses correctly) that the per-batch dedup
could not guarantee once the exclusive per-KB lock was removed, preventing
a stale ingest from resurrecting a retracted document.
Adds repository tests for same-key affinity and in-flight sibling rows.
Isolate the wiki generation pipeline from the document parse pipeline so
the two can no longer starve each other during concurrent uploads.
- Dedicated asynq worker pool for QueueWiki (NewWikiAsynqServer) with its
own concurrency budget (WEKNORA_WIKI_ASYNQ_CONCURRENCY / asynq.wiki_
concurrency, default 16), split from the parse pool (NewParseAsynqServer).
Both run the same mux over disjoint queue sets. Replaces the old
weighted-lottery approach where wiki sat on the low queue (weight 1).
- Task queue: add ClaimBatch (SELECT ... FOR UPDATE SKIP LOCKED + claimed_at)
and ReleaseByIDs so multiple concurrent consumers pull disjoint rows
without an external per-KB lock; PeekBatch retained for Lite mode.
- New TypeWikiFinalize debounced KB-global finalize task (index rebuild /
dead-link cleanup / cross-linking), registered on both Redis and Lite paths.
- Two-level wiki ingest concurrency: IngestMaxInflight caps concurrent
batches per KB; Map/Reduce fan-out sized within a batch. New config knob
+ system setting + swagger docs.
The wiki ingest post-process pipeline OOM'd and ran for hours on KBs
with ~40k documents. The dominant tail was a per-batch ListAllPages
that pulled every page (multi-MB content blobs) into Go memory, plus a
24h-TTL Redis pending list whose data could be evicted before a long
serialized backlog drained. dedup did O(P × N) Jaccard scoring in Go
on every batch. Lint loaded the full graph. The index page kept the
entire wiki directory in its content column and rewrote the TOAST
chunk on every ingest. None of these survive at 4w docs.
This change reworks the write path end to end and pulls the durable-
queue + dead-letter primitives out of wiki and into shared
infrastructure that every asynq task type now benefits from.
- migration 000041: task_pending_ops + task_dead_letters tables, plus
three GIN indexes on wiki_pages (source_refs jsonb_path_ops,
source_refs text fallback, lower(title) trgm).
- TaskPendingOpsRepository + TaskDeadLetterRepository in
internal/application/repository/task_queue.go: cursor-paginated
list, atomic IncrFailCount via UPDATE…RETURNING, dedup-key scoped
delete that refuses empty keys so a buggy caller can't wipe a KB's
queue.
- internal/middleware/asynqdl: writes a task_dead_letters row when an
asynq task exhausts its retry budget. Payload-agnostic — a small
probe struct extracts TenantID + scope hints across every existing
payload type, so summary:generation / image:multimodal /
faq_import / kb:clone / etc. all dead-letter without per-handler
code. Best-effort: an insert failure never masks the underlying
task error. Installed in router/task.go before the langfuse mw so
it sees raw errors.
- Pending queue moved Redis → PG (no TTL, restart-durable). Redis
keeps just the active-batch lock and the delete tombstone.
- In-batch retry budget (wikiMaxFailRetries=5) tracked via
pendingRepo.IncrFailCount; over the cap the row is moved to
task_dead_letters with task_type=wiki:ingest, related_id=knowledge
id. Asynq retries (10) are handled by the new middleware.
- Removed the per-batch ListAllPages. WikiBatchContext now carries
lazy fetcher closures (SlugTitleMany, SummaryByKnowledgeID) with
mutex-protected caches; reduce reaches for titles / summaries on
demand instead of pre-loading the whole KB.
- getExistingPageSlugsForKnowledge now uses ListSlugsBySourceRef,
which the new GIN index on source_refs serves as a Bitmap Index
Scan instead of a sequential text LIKE.
- Dedup pre-filter uses pg_trgm via FindSimilarPages
(idx_wiki_pages_title_trgm). For each new entity/concept (and each
of its aliases) we ask the DB for the top-K trigram-similar
existing pages and union the results — bounded prompt size, no Go-
side O(P × M) loop. Small KBs (≤25 entities) bypass the pre-filter.
- cleanDeadLinks / injectCrossLinks are scoped to the batch's
affected slugs (a few dozen) instead of every page in the KB. Both
use the new lite ListBySlugs / ExistsSlugs repo methods so they
pull only slug + title + outlinks, not full content.
- Slug fuzzy resolve (slug_fuzzy.go): when the LLM emits
[[bad-slug|display]] the cleanup path now tries display-text
reverse lookup → hyphen/case normalized equality → char-bigram
Jaccard ≥ 0.8 before stripping. Recovers the common pinyin-word-
break drift case ("shang-hai-tower" vs "shanghai-tower") in place
instead of replacing the link with plain text.
- rebuildIndexPage uses ListByTypeRecent(200) for the first-time
intro and drops the full DocumentSummaries blob from the
incremental update prompt, so its context stays bounded regardless
of KB size.
- Concurrency tunables surfaced in WikiConfig: IngestBatchSize /
IngestMapParallel / IngestReduceParallel, with sensible defaults
via OrDefault helpers. scheduleFollowUp drops to ProcessIn(0) so
follow-ups don't waste asynq retry slots bouncing on the active
lock.
- RunLint walks pages via the new ListPagesCursor in 200-page
windows and computes the live-slug set with a one-column
ListAllSlugs Pluck instead of a Limit:0 GetGraph that materialized
every node + edge. Memory is now bounded; 40k pages walks in
constant ~4MB.
- 14 GORM tests for both repos against an in-memory SQLite mirror of
the production DDL (task_queue_test.go).
- 6 tests for the dead-letter middleware covering retry budget
detection, payload-agnostic scope inference, error truncation, and
repo-failure isolation.
- 7 tests for the slug fuzzy resolve helper covering the three
resolution stages, display-text priority over normalized equality,
bigram fallback acceptance, and rejection of unrelated slugs.
- Existing wiki_ingest / wiki_lint / wiki_page tests updated to the
new fetcher / cursor APIs.