* feat(knowledge): flag documents that stop making progress
A document stuck in processing looked the same as one still working: a
spinner that polled forever, no hint of where it stopped, and, hours
later, a generic housekeeping failure.
- The batch and spans endpoints return last_activity_at for in-flight
rows: the later of updated_at and the newest span write. updated_at
only moves at stage transitions; span writes keep moving while a long
stage still works.
- After 20 minutes with no progress, the list and card views show
"may be stuck" with a tooltip. The timeline shows the stage it
stopped at and a stop-parsing button, plus a "last progress" stat.
Both pollers back off to 15s while everything in flight is stalled.
- When housekeeping fails a row, the error names the stalled stage and
the time of the last progress. It closes the row's open spans with
TASK_STALLED (the running stage fails, the rest are cancelled), so
the timeline shows where the run stopped instead of a spinner.
* fix(knowledge): locate stalls in finalizing and tell backlog from stuck
- Housekeeping reads the stall site from the latest attempt. It takes
the running stages, or, when none runs (post-process closes its stage
once enrichment is fanned out), the innermost running spans, mapped
back to their stage. Those spans fail with TASK_STALLED and are named
in the message; every stalled stage is listed in pipeline order. The
other open spans are cancelled. Every closed span gets a duration, and
the message leaves out the time when the heartbeat query failed.
- The spans tree prefers a TASK_STALLED failure for last_error over a
later subtask failure. current_stage falls back to the stage owning a
running subspan.
- Batch and spans responses flag rows quiet past the stall hint whose
work is still queued (waiting_in_queue), using the sweep's own probes.
At most 20 rows are probed per request, cached for a minute. The UI
shows these as queued, with no stop button in the banner.
* fix(knowledge): answer stall verdicts from one shared queue scan
- Probing a quiet document for a backlog rescanned the whole queue once
per document. When the document was absent, which is exactly the
stuck case, the scan ran to the end, and it read a Redis error as
"nothing queued". QueuedKnowledgeIDs now collects every referenced
knowledge ID in one pass and returns list errors. HousekeepingService
keeps that snapshot for a minute, shared across requests and
single-flight. A failed scan is not cached.
- With the per-request cap of 20 gone, every quiet row on a page gets a
verdict. The response carries it as stall_state (queued or stalled),
replacing waiting_in_queue. The UI marks a row as stuck only on a
"stalled" verdict. A row with no verdict, because the probe failed,
shows as ordinary processing instead of inviting a stop.