WeKnora/internal/handler
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
..
session fix(sessions): review fixes for keyword search / pinning / IM titles 2026-04-30 16:23:00 +08:00
auth.go feat: implement auto-setup for Lite edition, enhancing user experience by automating user and tenant creation on first launch 2026-04-11 18:51:16 +08:00
chunk.go feat(api): enhance organization shared knowledge base and agent functionalities 2026-02-09 11:40:25 +08:00
chunker_debug.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
chunker_debug_test.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
custom_agent.go fix(agent): improve tenant context handling and error reporting 2026-04-24 22:40:57 +08:00
datasource.go feat(datasource): enforce tenant isolation and knowledge base ownership 2026-03-30 20:46:07 +08:00
evaluation.go feat: 新增API Key认证支持,优化Swagger文档安全配置 2025-12-22 15:01:09 +08:00
faq.go feat(api): enhance organization shared knowledge base and agent functionalities 2026-02-09 11:40:25 +08:00
im.go feat(im): tenant-wide IM channels overview under the user menu 2026-04-30 15:29:14 +08:00
initialization.go fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness 2026-05-06 17:17:07 +08:00
knowledge.go refactor(knowledge-base): address review feedback 2026-04-29 12:01:38 +08:00
knowledgebase.go Merge branch 'main' into storage-allow-list 2026-04-29 19:35:13 +08:00
mcp_service.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
message.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
model.go feat(asr): add ASR model connection check and update related components 2026-04-02 21:27:27 +08:00
organization.go fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
skill_handler.go feat: update sandbox service configuration in Docker Compose files 2026-02-04 21:34:07 +08:00
storage_allowlist.go feat(ks3): add KS3 object storage support 2026-05-05 00:22:27 +08:00
system.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
tag.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
tenant.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
vectorstore.go feat: add VectorStore CRUD API endpoints and documentation 2026-04-14 17:34:57 +08:00
web_search.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
web_search_provider.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
wechat_qrcode.go feat(im): add WeChat integration with QR code login and long-polling support 2026-04-13 14:17:57 +08:00
weknoracloud.go feat(weknoracloud): implement credential management and status checks 2026-04-14 17:36:01 +08:00
wiki_page.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00