WeKnora/internal/agent/tools
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
..
capabilities.go feat: Enhance agent tool compatibility and knowledge base filtering 2026-04-23 17:28:00 +08:00
data_analysis.go fix(data-analysis): capture LOCAL_STORAGE_BASE_DIR at init time (#1040) 2026-05-06 11:19:56 +08:00
data_analysis_duckdb_test.go test(agent): add end-to-end DuckDB excel tests and fix sheet enumeration (#1007) 2026-04-24 19:57:56 +08:00
data_analysis_materialize_test.go fix(agent): materialize knowledge files to temp path for DuckDB (#1007) 2026-04-24 19:57:56 +08:00
data_analysis_test.go feat: data_analysis sql validation & type processing 2026-04-28 23:47:34 +08:00
data_schema.go feat: add organization management features 2026-02-09 11:40:25 +08:00
database_query.go feat(database): enhance DatabaseQueryTool with search scope filtering 2026-03-31 10:52:01 +08:00
definitions.go feat: Introduce Wiki Fixer agent and related functionalities 2026-04-22 21:18:08 +08:00
final_answer.go fix(agent): terminate ReAct loop on final_answer regardless of arg parse success 2026-04-23 19:36:28 +08:00
final_answer_test.go fix(agent): terminate ReAct loop on final_answer regardless of arg parse success 2026-04-23 19:36:28 +08:00
get_document_info.go feat: Update agent type presets and prompt templates for improved knowledge base handling 2026-04-22 21:18:34 +08:00
grep_chunks.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
json_repair.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
json_repair_test.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
knowledge_search.go fix(search): align rerank priority between Execute and rerankResults 2026-04-28 20:23:57 +08:00
list_knowledge_chunks.go feat: Update agent type presets and prompt templates for improved knowledge base handling 2026-04-22 21:18:34 +08:00
mcp_tool.go feat(mcp): implement reconnection logic for MCP tool calls and tool listing 2026-03-31 11:57:15 +08:00
mcp_tool_image_test.go feat: auto-describe MCP tool result images via VLM 2026-03-25 21:38:15 +08:00
mcp_tool_test.go feat(tests): add comprehensive tests for MCPTool functionality 2026-03-25 22:08:29 +08:00
normalize_id.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
normalize_id_test.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
param_cast.go refactor(prompt): streamline context handling in templates 2026-03-25 22:08:29 +08:00
param_cast_test.go refactor(prompt): streamline context handling in templates 2026-03-25 22:08:29 +08:00
param_validate.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
param_validate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
query_knowledge_graph.go fix(agent): correct graph config reporting 2026-05-06 11:04:55 +08:00
query_knowledge_graph_test.go test(agent): expand graph query regression fixture 2026-05-06 11:04:55 +08:00
registry.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
sanitize_messages.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
sanitize_messages_test.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
sequentialthinking.go refactor(prompt): streamline context handling in templates 2026-03-25 22:08:29 +08:00
skill_execute.go feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
skill_read.go feat: support agent skills 2026-02-04 20:08:49 +08:00
strip_think.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
strip_think_test.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
todo_write.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
tool.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
truncate.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
truncate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
web_fetch.go fix(security): support IPv6 in SSRF validation via whitelist mechanism 2026-03-31 20:45:15 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
wiki_delete_page.go feat: Implement wiki editing functionalities and UI components 2026-04-22 21:18:08 +08:00
wiki_flag_issue.go feat: Introduce `wiki_flag_issue` tool for reporting wiki page issues 2026-04-22 21:17:38 +08:00
wiki_index_overview_test.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_read_issue.go feat: Introduce Wiki Fixer agent and related functionalities 2026-04-22 21:18:08 +08:00
wiki_read_source_doc.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
wiki_rename_page.go feat: Implement wiki editing functionalities and UI components 2026-04-22 21:18:08 +08:00
wiki_replace_text.go feat: Implement wiki editing functionalities and UI components 2026-04-22 21:18:08 +08:00
wiki_tools.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_tools_test.go feat: Add Wiki Researcher agent and system prompt for enhanced knowledge retrieval 2026-04-22 21:17:35 +08:00
wiki_update_issue.go feat: Introduce Wiki Fixer agent and related functionalities 2026-04-22 21:18:08 +08:00
wiki_write_page.go feat: Implement wiki editing functionalities and UI components 2026-04-22 21:18:08 +08:00