* feat(knowledge): flag documents that stop making progress
A document stuck in processing looked the same as one still working: a
spinner that polled forever, no hint of where it stopped, and, hours
later, a generic housekeeping failure.
- The batch and spans endpoints return last_activity_at for in-flight
rows: the later of updated_at and the newest span write. updated_at
only moves at stage transitions; span writes keep moving while a long
stage still works.
- After 20 minutes with no progress, the list and card views show
"may be stuck" with a tooltip. The timeline shows the stage it
stopped at and a stop-parsing button, plus a "last progress" stat.
Both pollers back off to 15s while everything in flight is stalled.
- When housekeeping fails a row, the error names the stalled stage and
the time of the last progress. It closes the row's open spans with
TASK_STALLED (the running stage fails, the rest are cancelled), so
the timeline shows where the run stopped instead of a spinner.
* fix(knowledge): locate stalls in finalizing and tell backlog from stuck
- Housekeeping reads the stall site from the latest attempt. It takes
the running stages, or, when none runs (post-process closes its stage
once enrichment is fanned out), the innermost running spans, mapped
back to their stage. Those spans fail with TASK_STALLED and are named
in the message; every stalled stage is listed in pipeline order. The
other open spans are cancelled. Every closed span gets a duration, and
the message leaves out the time when the heartbeat query failed.
- The spans tree prefers a TASK_STALLED failure for last_error over a
later subtask failure. current_stage falls back to the stage owning a
running subspan.
- Batch and spans responses flag rows quiet past the stall hint whose
work is still queued (waiting_in_queue), using the sweep's own probes.
At most 20 rows are probed per request, cached for a minute. The UI
shows these as queued, with no stop button in the banner.
* fix(knowledge): answer stall verdicts from one shared queue scan
- Probing a quiet document for a backlog rescanned the whole queue once
per document. When the document was absent, which is exactly the
stuck case, the scan ran to the end, and it read a Redis error as
"nothing queued". QueuedKnowledgeIDs now collects every referenced
knowledge ID in one pass and returns list errors. HousekeepingService
keeps that snapshot for a minute, shared across requests and
single-flight. A failed scan is not cached.
- With the per-request cap of 20 gone, every quiet row on a page gets a
verdict. The response carries it as stall_state (queued or stalled),
replacing waiting_in_queue. The UI marks a row as stuck only on a
"stalled" verdict. A row with no verdict, because the probe failed,
shows as ordinary processing instead of inviting a stop.
Second slice of #3338 (tracked as #3345).
Count alignment: CountKnowledgeByKnowledgeBaseID now excludes rows
mid-deletion, mirroring the document list's existing filter — a stranded
delete no longer shows as '4 documents, 3 listed' on the KB card. Failed
rows stay counted: they are visible, actionable cards.
Sweep C: housekeeping now recovers rows stuck at 'deleting' whose delete
task no longer exists. The delete dead-letter callback only flips rows
that are still 'deleting' when retries exhaust, so a worker death
mid-delete or a lost queue left the row hidden from the list forever.
Same recovery contract as the parse sweep: flip to failed with a clear
error so the row becomes visible and actionable; the delete executor is
idempotent and plans by row ID, so a late-arriving task still finishes
cleanly.
Liveness gate: a new TaskInspector probe (HasQueuedDeleteTasksForKnowledge)
matches knowledge:list_delete batch payloads — the per-parse matcher
cannot see them. A live queued/active delete defers the row (backlog,
not stranded); a probe error defers that row; a nil inspector defers the
whole sweep. Lite mode's noop returns false by design: inline deletes
never queue, so a stale deleting row there genuinely has no task.
Tests: sweep matrix (stranded recovered / backlogged kept / fresh
untouched / probe-error deferred / nil-inspector deferred), count
alignment, and the payload matcher table.
Refs #3345, #3338
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
Operators can clear all finally-failed tasks in one queue from the runtime
drawer, with audit logging and scoped API access. Also localize the
chat_attachment queue label in the runtime queue details table.
Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
Replace offset-based page numbers with opaque Redis-backed cursors so
scheduled/retry tasks sort by next run time and live lists stay stable
when queue state changes between requests.
Added new tests for the ImageMultimodalService to verify the handling of orphan tasks and the finalization of pending counters. Improved runtime task inspection with better error handling and task cancellation logic, ensuring robust management of orphan tasks in the system.
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.