WeKnora/internal
toy0116 bf12868923 fix(wiki): cap requeueFailedOps retry count to prevent queue pile-up
Root cause: requeueFailedOps unconditionally RPush'd every failed op back to
wiki:pending, causing unbounded growth when a document consistently triggers
LLM timeouts. Observed: 553 entries in wiki:pending:08134644 (one KB), mostly
duplicates for ~5 unique documents that each timed out on every batch cycle.

Fix:
- Add wikiFailCountKeyPrefix ("wiki:failcount:") and wikiMaxFailRetries (5)
  constants in wiki_ingest.go
- requeueFailedOps now atomically Incr(wiki:failcount:{kbID}:{knowledgeID})
  before each RPush; skips re-queue (drops op permanently) if count > 5
- Expire the fail-count key at wikiPendingTTL (24h) on every update
- wiki_ingest_batch.go: Del the fail-count key at ingestSucceeded++ so
  transient errors don't permanently burn through a document's retry budget

Evidence of problem severity: 5 persistently-failing docs × ~110 requeue
cycles each = 553 pending entries; had to use Lua atomic dedup to recover.

Behaviour after this fix:
- First 5 failures → normal retry (re-queued, follow-up batch scheduled)
- 6th failure → logged as WARN "dropping op … after 6 failures (limit 5)",
  not re-queued; prevents the feedback loop
- If document is fixed upstream and re-ingested via EnqueueWikiIngest, the
  op is a fresh entry with no existing fail-count key → full 5-retry budget

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:46:36 +08:00
..
agent chore(agent): log natural-stop candidate when no tool calls 2026-05-07 18:28:28 +08:00
application fix(wiki): cap requeueFailedOps retry count to prevent queue pile-up 2026-05-08 21:46:36 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
common feat: Implement deadlock retry mechanism for chunk creation 2026-04-22 21:17:21 +08:00
config feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
container fix(im): rewrite private storage URLs to HTTP in IM channel replies 2026-04-30 11:40:30 +08:00
database fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
datasource feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
errors feat: Add agent configuration and cleanup scripts for database migrations 2025-11-05 23:18:44 +08:00
event fix: improve tenant access token logging and update test cases 2026-04-22 21:17:30 +08:00
handler perf(wiki): subgraph API and interactive exploration for large graphs 2026-05-08 13:32:54 +08:00
im fix(im): clamp reconnect backoff to avoid int64 overflow → busy loop 2026-05-05 00:36:18 +08:00
infrastructure fix(chunker): recognise CN chapter titles and multi-level numeric headings 2026-05-06 22:14:16 +08:00
logger feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
mcp feat(mcp): implement reconnection logic for MCP tool calls and tool listing 2026-03-31 11:57:15 +08:00
middleware fix(im): rewrite private storage URLs to HTTP in IM channel replies 2026-04-30 11:40:30 +08:00
models chore(llm): add fire-once OpenAI-layer stream diagnostic logs 2026-05-07 18:28:28 +08:00
router fix: 为前端静态资源设置正确的 Cache-Control 响应头 2026-05-07 12:08:02 +08:00
runtime init commit 2025-08-05 15:08:07 +08:00
sandbox feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
searchutil feat(prompt): enhance question rewriting and intent classification in prompt templates 2026-04-16 23:15:02 +08:00
stream feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
tracing feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
types perf(wiki): subgraph API and interactive exploration for large graphs 2026-05-08 13:32:54 +08:00
utils fix(crypto): fail loudly when encrypted DB fields cannot be decrypted 2026-04-30 15:17:56 +08:00