WeKnora/internal/agent/tools
hujiaxin 8c0e634ee9 perf(agent): make knowledge_search MMR selection incremental
applyMMR recomputed every candidate's redundancy against every
already-selected result on each round, re-tokenizing the selected passages
each time. With C candidates and k selections that is O(k^2) calls to
getEnrichedPassage + tokenizeSimple over full chunk bodies, and the
diagnostic average-redundancy loop afterwards re-tokenizes another O(k^2)
times. Since k defaults to len(candidates), this is quadratic in the number
of results that survive dedup/rerank.

On a knowledge base where ~250 candidates reach the MMR stage this burns
tens of seconds of pure CPU inside every knowledge_search call, and no log
line attributes the time to MMR.

Cache each candidate's running max jaccard against the selected set and fold
in only the newly selected result per round, and reuse the already-computed
token sets for the average-redundancy diagnostic. Tokenization now happens
once per candidate. Selection output is unchanged, including tie-breaking;
the added test pins the new implementation against the previous one over a
deterministic corpus, and a benchmark contrasts the two.
2026-08-20 15:44:32 +08:00
..
capabilities.go refactor(agent): remove final_answer tool references and update related logic 2026-06-04 15:48:04 +08:00
data_analysis.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
data_analysis_duckdb_test.go test(agent): add end-to-end DuckDB excel tests and fix sheet enumeration (#1007) 2026-04-24 19:57:56 +08:00
data_analysis_materialize_test.go fix: deep-copy stored files and images when cloning a knowledge base 2026-06-03 14:45:59 +08:00
data_analysis_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
data_schema.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
database_query.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
database_query_test.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
definitions.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
exec_context.go feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
faq_snippet.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
faq_snippet_test.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
get_document_info.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
grep_chunks.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
grep_chunks_aggregate_test.go feat(agent): add tool result persistence trimming for large outputs 2026-06-16 01:09:23 +08:00
grep_chunks_scope_test.go fix(agent): avoid expanding tag scopes for grep chunks 2026-06-29 19:17:42 +08:00
grep_chunks_snippet_test.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
image_markdown_output_test.go fix(chat): use Markdown for LLM image context and normalize legacy XML 2026-07-15 18:10:31 +08:00
json_repair.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
json_repair_test.go feat: Enhance agent retrieval strategies and regex handling 2026-04-22 21:18:21 +08:00
knowledge_search.go perf(agent): make knowledge_search MMR selection incremental 2026-08-20 15:44:32 +08:00
knowledge_search_metadata_test.go refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
knowledge_search_mmr_test.go perf(agent): make knowledge_search MMR selection incremental 2026-08-20 15:44:32 +08:00
knowledge_search_rerank_test.go feat(agent): enhance suggested questions API with tag scopes 2026-07-17 10:19:43 +08:00
list_knowledge_chunks.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
mcp_oauth.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_tool.go feat(agent): add configurable MCP OAuth wait timeout per agent 2026-07-01 14:37:37 +08:00
mcp_tool_image_test.go feat: auto-describe MCP tool result images via VLM 2026-03-25 21:38:15 +08:00
mcp_tool_test.go feat(tests): add comprehensive tests for MCPTool functionality 2026-03-25 22:08:29 +08:00
normalize_id.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
normalize_id_test.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
output_budget.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
output_budget_test.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
param_cast.go fix(agent): harden tool parameter parsing against LLM type mismatches (#1505) 2026-05-28 07:50:26 +08:00
param_cast_test.go refactor(prompt): streamline context handling in templates 2026-03-25 22:08:29 +08:00
param_validate.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
param_validate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
persist.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
persist_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
query_knowledge_graph.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
query_knowledge_graph_test.go Simplify KB duplicate API and improve naming/docs coverage. 2026-07-08 20:55:54 +08:00
registry.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
registry_test.go fix(agent/tools): sort function definitions for deterministic ordering 2026-05-25 16:47:14 +08:00
sandbox_ls.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
sandbox_read.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
sandbox_read_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
sanitize_messages.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
sanitize_messages_test.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
scope_authorization.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
scope_authorization_test.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
search_conversations.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
search_memory.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
search_memory_test.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
sequentialthinking.go refactor(agent): remove final_answer tool references and update related logic 2026-06-04 15:48:04 +08:00
shell_exec.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
shell_exec_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
skill_execute.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
skill_read.go feat: support agent skills 2026-02-04 20:08:49 +08:00
strip_think.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
strip_think_test.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
think_stream.go feat(agent): enhance event routing for reasoning and answers in streaming 2026-06-04 15:48:04 +08:00
think_stream_test.go feat(agent): enhance event routing for reasoning and answers in streaming 2026-06-04 15:48:04 +08:00
todo_write.go fix(agent): harden tool parameter parsing against LLM type mismatches (#1505) 2026-05-28 07:50:26 +08:00
tool.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
tool_policy_coverage_test.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
truncate.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
truncate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
web_fetch.go fix(web_fetch): address review follow-ups for structured fetch results 2026-07-28 22:17:31 +08:00
web_fetch_test.go fix(web_fetch): address review follow-ups for structured fetch results 2026-07-28 22:17:31 +08:00
web_search.go fix(agent): preserve web evidence when page fetches fail 2026-07-28 22:17:31 +08:00
wiki_delete_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_flag_issue.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
wiki_index_overview_test.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_link_mutation.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_link_mutation_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_read_issue.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_read_page_budget_test.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
wiki_read_source_doc.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wiki_rename_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_replace_text.go fix: replace all exact wiki text matches 2026-08-05 19:30:40 +08:00
wiki_replace_text_test.go fix: replace all exact wiki text matches 2026-08-05 19:30:40 +08:00
wiki_route_resolver.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_tools.go Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
wiki_tools_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_update_issue.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_write_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_write_page_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00