WeKnora/internal/agent/tools
lyingbug 182f67bd86
refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372)
* refactor(wiki): consolidate source_refs parsing and resolution

source_refs entries ("uuid" or "uuid|title") were parsed by ten
independent implementations that disagreed on whitespace handling, on
whether a model-supplied title could be trusted, and on how a leading
pipe was treated. The scope-enforced and legacy agent resolvers also
duplicated the lookup and the routing step looked every document up a
second time.

- types.ParseWikiSourceRef / FormatWikiSourceRef / WikiSourceRefMatchesKnowledge
  are now the only readers and writer of the format.
- resolveWikiSourceDocuments replaces resolveAuthorizedSourceRefs,
  resolveSourceRefs and wikiKnowledgeBasesForSourceRefs: one lookup per
  document, titles always rebuilt from server data, dedupe by canonical ID,
  unknown documents fail closed in every mode.
- removeSourceRef now matches padded or titled refs of a deleted document.

* refactor(data_analysis): expose the selected document as a fixed table name

The tool required the document ID inside the SQL text and blindly
string-replaced it with the physical DuckDB table name, which forced the
model-context codec to rewrite handles inside quoted SQL and let the
UUID-derived k_<uuid> table name reach the model through stored schema
summaries.

The document is now selected only by knowledge_id. Every query runs on a
dedicated connection where "dataset" is a temporary view over the
physical table (DuckDB temporary objects are connection-local, so
concurrent sessions stay isolated). The physical name remains accepted
for internal callers; any other table fails validation with an error
naming the table the model should use. Schema summaries, the schema
tool, the analyst prompt and the tool schema all name the same table, so
data_analysis.sql no longer carries source handles and its text codec
policy is removed.

* feat(modelcontext): report raw identifiers that survive encoding

Text-level compaction exists because raw UUIDs reach model context through
producers the registry never sees: error strings, slugs, results of
unknown shape. Retiring it requires knowing where those producers are.

Registry.LeakedIdentifiers scans an encoded message list for UUID-shaped
values in every application-composed field (user messages are skipped),
classifies each as registered (codec missed a rewrite) or unregistered
(producer emitted a raw ID), and SummarizeLeaks renders one line per
field. The agent engine and the three RAG pipeline call sites log the
report as a warning after EncodeMessages.

* fix(chat_pipeline): align RAG data analysis with the fixed table contract

The RAG data-analysis plugin still asked the planning model to fill in
knowledge_id and to write SQL from a prompt that exposed the raw
knowledge UUID, then passed the model's JSON straight to Execute. Under
the new data_analysis contract a query that names anything but the
dataset table is rejected, and the plugin logged that rejection at error
level and moved on, so the analysis vanished without any signal. A
"no analysis needed" plan took the same error path.

The pipeline now owns document selection: the model returns only
needs_analysis and sql, the prompt names the dataset table and never
shows the knowledge ID or physical table, and the plugin builds the tool
input itself. A rejected query is fed back to the model once for repair;
a query that still fails is reported through pipelineWarn with the SQL
and the tool error, and "not needed" is logged as such instead of as a
failure.

Covered by unit tests with a scripted model and executor: not-needed,
pipeline-owned knowledge_id, one-shot repair, persistent failure
surfaced, unparseable plan rejected.
2026-09-17 22:08:16 +08:00
..
browserskill.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
browserskill_image.go fix(browser): complete task cancellation, screenshots, and result display (#3206) 2026-09-12 09:32:26 +08:00
browserskill_image_test.go fix(browser): complete task cancellation, screenshots, and result display (#3206) 2026-09-12 09:32:26 +08:00
browserskill_prompt.go fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
browserskill_result.go fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
browserskill_result_test.go fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
browserskill_schema.go fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
browserskill_schema_test.go fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
browserskill_test.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
capabilities.go refactor(agent): remove final_answer tool references and update related logic 2026-06-04 15:48:04 +08:00
data_analysis.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
data_analysis_duckdb_test.go test(agent): add end-to-end DuckDB excel tests and fix sheet enumeration (#1007) 2026-04-24 19:57:56 +08:00
data_analysis_materialize_test.go fix: deep-copy stored files and images when cloning a knowledge base 2026-06-03 14:45:59 +08:00
data_analysis_query_test.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
data_analysis_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
data_schema.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
database_query.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
database_query_test.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
definitions.go refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
exec_context.go feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
execution_policy.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
faq_snippet.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
faq_snippet_test.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
file_mutation_queue.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
file_mutation_queue_test.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
get_document_info.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
grep_chunks.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
grep_chunks_aggregate_test.go feat(agent): add tool result persistence trimming for large outputs 2026-06-16 01:09:23 +08:00
grep_chunks_scope_test.go fix(agent): avoid expanding tag scopes for grep chunks 2026-06-29 19:17:42 +08:00
grep_chunks_snippet_test.go feat(faq): enhance FAQ handling and localization support 2026-06-04 21:43:16 +08:00
image_markdown_output_test.go fix(chat): use Markdown for LLM image context and normalize legacy XML 2026-07-15 18:10:31 +08:00
json_repair.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
json_repair_test.go fix(agent): recover tool arguments with a stray trailing fragment 2026-09-02 10:12:55 +08:00
knowledge_search.go fix(agent): stop using the chat model to rerank knowledge search 2026-08-27 11:44:30 +08:00
knowledge_search_metadata_test.go refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
knowledge_search_mmr_test.go perf(agent): make knowledge_search MMR selection incremental 2026-08-20 15:44:32 +08:00
knowledge_search_rerank_test.go fix(agent): stop using the chat model to rerank knowledge search 2026-08-27 11:44:30 +08:00
list_knowledge_chunks.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
mcp_catalog.go fix(chat): preserve steering continuity and composer focus 2026-09-09 20:15:23 +08:00
mcp_catalog_integration_test.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_catalog_regression_test.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_catalog_test.go fix(mcp): normalize JSON-encoded argument envelopes 2026-09-09 16:11:52 +08:00
mcp_directory_test.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_exposure.go feat(mcp)!: streamline setup and use short routing handles 2026-09-09 16:11:52 +08:00
mcp_exposure_test.go feat(mcp)!: streamline setup and use short routing handles 2026-09-09 16:11:52 +08:00
mcp_oauth.go refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
mcp_schema.go refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
mcp_schema_test.go refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
mcp_tool.go fix(agent): honor MCP service advanced_config.timeout for CallTool window 2026-09-11 17:47:04 +08:00
mcp_tool_enabled_test.go feat(mcp): add per-tool enable toggle (#2989) 2026-09-05 17:19:20 +08:00
mcp_tool_image_test.go feat: auto-describe MCP tool result images via VLM 2026-03-25 21:38:15 +08:00
mcp_tool_test.go feat(tests): add comprehensive tests for MCPTool functionality 2026-03-25 22:08:29 +08:00
mcp_tool_timeout_test.go fix(agent): honor MCP service advanced_config.timeout for CallTool window 2026-09-11 17:47:04 +08:00
normalize_id.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
normalize_id_test.go feat(agent): enhance tool execution and message sanitization 2026-03-25 22:08:29 +08:00
output_budget.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
output_budget_test.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
output_links.go perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
output_links_test.go perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
param_cast.go fix(agent): harden tool parameter parsing against LLM type mismatches (#1505) 2026-05-28 07:50:26 +08:00
param_cast_test.go refactor(prompt): streamline context handling in templates 2026-03-25 22:08:29 +08:00
param_validate.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
param_validate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
persist.go fix(browser): complete task cancellation, screenshots, and result display (#3206) 2026-09-12 09:32:26 +08:00
persist_test.go fix(browser): complete task cancellation, screenshots, and result display (#3206) 2026-09-12 09:32:26 +08:00
python_syntax.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
python_syntax_test.go refactor(skills): introduce write_skill_file and edit_skill_file tools for skill directory management 2026-09-02 15:08:05 +08:00
query_knowledge_graph.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
query_knowledge_graph_test.go Simplify KB duplicate API and improve naming/docs coverage. 2026-07-08 20:55:54 +08:00
read_file.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
read_file_test.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
read_web_page.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
registry.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
registry_outcome_test.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
registry_test.go fix(agent/tools): sort function definitions for deterministic ordering 2026-05-25 16:47:14 +08:00
sandbox_diff.go feat(agent): stream live +/- progress for sandbox write and edit 2026-09-02 23:36:58 +08:00
sandbox_diff_test.go feat(agent): stream live +/- progress for sandbox write and edit 2026-09-02 23:36:58 +08:00
sandbox_edit.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
sandbox_edit_test.go feat(agent): stream live +/- progress for sandbox write and edit 2026-09-02 23:36:58 +08:00
sandbox_ls.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
sandbox_write.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
sandbox_write_test.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
sanitize_messages.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
sanitize_messages_test.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
scope_authorization.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
scope_authorization_test.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
search_conversations.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
search_memory.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
search_memory_test.go feat(memory): introduce long-term memory feature with i18n updates 2026-08-13 18:00:02 +08:00
sequentialthinking.go refactor(agent): remove final_answer tool references and update related logic 2026-06-04 15:48:04 +08:00
shell_command_output.go feat(sandbox): stream live shell and install command output 2026-09-12 10:16:39 +08:00
shell_command_output_test.go feat(sandbox): stream live shell and install command output 2026-09-12 10:16:39 +08:00
shell_env_extract.go fix(sandbox): make skill env capture fill blanks only 2026-08-27 14:32:00 +08:00
shell_env_extract_test.go fix(sandbox): make skill env capture fill blanks only 2026-08-27 14:32:00 +08:00
shell_exec.go perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
shell_exec_test.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00
shell_runtime_integration_test.go feat(sandbox): run sandboxed execs as root by default (#3067) 2026-09-07 13:03:35 +08:00
shell_stdin_test.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
skill_file.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
skill_file_test.go refactor(skills): introduce write_skill_file and edit_skill_file tools for skill directory management 2026-09-02 15:08:05 +08:00
skill_file_tree_test.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
skill_resources.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
skill_runtime_guard.go feat(sandbox): run sandboxed execs as root by default (#3067) 2026-09-07 13:03:35 +08:00
skill_runtime_guard_test.go feat(sandbox): run sandboxed execs as root by default (#3067) 2026-09-07 13:03:35 +08:00
strip_think.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
strip_think_test.go refactor(agent): remove query display from DatabaseQuery component and update tool result structure 2026-03-25 22:08:29 +08:00
think_stream.go feat(agent): enhance event routing for reasoning and answers in streaming 2026-06-04 15:48:04 +08:00
think_stream_test.go feat(agent): enhance event routing for reasoning and answers in streaming 2026-06-04 15:48:04 +08:00
todo_write.go refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
tool.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
tool_policy_coverage_test.go refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
truncate.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
truncate_test.go feat(agent): implement memory consolidation and context management 2026-03-25 22:08:29 +08:00
web_fetch.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_fetch_test.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_pages_test.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_search.go feat(web-search): add Serply provider 2026-09-17 12:25:09 +08:00
web_search_test.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
wiki_delete_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_flag_issue.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_index_overview_test.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_link_mutation.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_link_mutation_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_read_issue.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_read_page_budget_test.go fix(agent): keep batched wiki_read_page results within output budget 2026-07-27 13:22:16 +08:00
wiki_read_source_doc.go fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wiki_rename_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_replace_text.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_replace_text_test.go fix: replace all exact wiki text matches 2026-08-05 19:30:40 +08:00
wiki_route_resolver.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_source_refs.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_source_refs_test.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_tools.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_tools_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_update_issue.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
wiki_write_page.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
wiki_write_page_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
workspace_reader.go perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
workspace_reader_test.go fix(sandbox): separate file access from artifact publishing (#3356) 2026-09-17 16:18:01 +08:00