Commit Graph

2 Commits (014f95f1394a7bfd1c7bbbdcdc19057a80dcaca5)

Author SHA1 Message Date
lyingbug 92e7c0ab37
refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381)
* refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents

The agent's retrieval surface had grown to five overlapping tools whose
behaviour was scattered across long descriptions, interleaved "tips" in
tool output (most of which modelcontext discarded before the model saw
them), and a grep_chunks implementation that ran an unindexed regex scan
over the chunks table ordered by created_at, so large knowledge bases lost
recall and paid a full-table scan on every call.

Tool surface
- search_knowledge replaces knowledge_search and grep_chunks: one query,
  mode = hybrid | semantic | keyword, optional knowledge_base_ids, limit.
  Keyword mode runs through the retrieval engine's keyword index instead of
  the DB regex scan. Modes unavailable for the scope are refused with a
  reason. Rerank, MMR and de-duplication are kept; already_seen markers,
  retrieval_statistics and the zero-result "CRITICAL" block are gone.
- read_document replaces list_knowledge_chunks, get_document_info and
  wiki_read_source_doc: id accepts a dN or cN handle, always returns the
  document header, pages by exact offset (fixing the old floor-to-page
  bug), finds passages with query (literal by default, regex opt-in) and
  reads neighbours around a chunk with context.
- list_documents pages through one knowledge base, honouring @file/@tag
  scope.
- wiki_search takes a single query plus regex, knowledge_base_ids, limit;
  literal queries are escaped so the model never reasons about JSON
  escaping. Legacy queries / knowledge_base_id are still accepted.
- query_knowledge_graph is registered only when a graph-enabled KB is in
  scope and its description/tips were trimmed.

Compatibility
- Retired names in stored allowlists, presets and API calls are mapped to
  their successors by NormalizeAllowedTools at registration time; no data
  migration. Stored histories with old names still decode, render and
  redact. Calls to retired names return a replacement hint.
- modelcontext gains an "id" key space that decodes against both the chunk
  and document handle tables (decode-only), plus rendering for the
  document header, chunk roles, in-document matches and next_offset.

Prompts and surfaces
- The RAG, wiki researcher and hybrid templates now name the tools and the
  search -> read -> cite order; tool descriptions are short and contain no
  workflow instructions.
- Engine display names, history redaction, grounding prompt, IM rendering,
  the built-in MCP server (search_knowledge gains mode/limit, grep_chunks
  becomes keyword search, read_document gains query, wiki_search gains
  regex), frontend editor/renderers/i18n, presets, website docs and the
  changelog are updated accordingly.

Tests cover name mapping, mode validation and refusal, output shape,
read_document paging/context/query/scope, list_documents, wiki_search
escaping and legacy arguments, modelcontext decoding and rendering, and
registration-time aliasing and graph gating.

* fix(agent): address review findings on read_document, list_documents and model rendering

- read_document: fetch the second page whenever the first page did not
  reach the end of the document, so an unaligned window near the end no
  longer drops the trailing chunks (also fixes context windows around a
  chunk near the end); a query passed together with a cN handle now
  searches the owning document instead of being ignored; query mode stops
  collecting once the output budget is nearly spent and reports truncated;
  the document row is authorized in place instead of being fetched twice;
  the context read takes the total from the window fetch instead of a
  separate count query.
- list_documents: resolve the page's document scope in one batched tag
  lookup instead of one call per row.
- modelcontext: render read_document output once and read next_offset
  through the existing intValue helper.
- search_knowledge: point title/file-name lookups at list_documents.
- Tests for the boundary window, chunk context near the end, cN + query,
  and budget-bound query results.

* fix(agent): close compatibility gaps in the consolidated retrieval tools

- Wiki-only scopes: read_document and list_documents are registered
  whenever a wiki KB is in scope, since every KB stores chunks. Their
  capability requirement includes wiki but is marked auxiliary, so they do
  not widen a RAG agent's derived "all knowledge bases" filter to wiki-only
  bases (Go and frontend derivations both honour it).
- search_knowledge resolves the retrieval mode per knowledge base instead
  of refusing the call: FAQ and vector-only bases answer keyword requests
  semantically, keyword-only bases answer semantic requests by keyword.
  Results carry requested_mode and mode_fallbacks, the empty-result text
  names the fallback, and the call is refused only when no base in scope
  has a chunk index.
- The model rendering uses the mode the search actually ran with (it was
  always labelled semantic) and emits <mode_fallback> per degraded base.
- list_documents applies the @file/@tag scope before paging: tags are
  pushed into the database filter, pinned documents are loaded by ID, and
  total/next_page count only readable documents.
- read_document resolves chunk context by chunk_index through a new
  ChunkRepository.ListChunkNeighbors, so parent, summary and image chunks
  sharing the index sequence no longer shift the neighbourhood; offset is
  documented as a reading-order position, not a chunk index.
- wiki_search (agent and MCP) keeps its regular-expression default and
  only matches literally when the text is not a valid expression;
  regex=false forces a literal match and regex=true rejects invalid input.
- MCP grep_chunks keeps grep semantics: literal terms from the pattern
  fetch candidates from the keyword index (semantic for bases without
  one), and every returned chunk is verified against the pattern.

* docs(changelog): align the consolidation entry with regex wiki_search and grep_chunks semantics

* docs(mcp): note that grep_chunks recall is bounded by index candidates
2026-09-18 14:12:22 +08:00
lyingbug 76a19f14c4
feat(mcp): add built-in MCP server with per-workspace endpoints (#3371)
* feat(mcp): add built-in MCP server with per-workspace endpoints

WeKnora previously exposed itself to MCP clients only through the Python
package under mcp-server/, which binds one process to one API key and
therefore to one workspace, and mirrors the REST API tool-for-tool.

This adds a native MCP server surface to the Go backend:

- New mcp_endpoints table (PostgreSQL 000100, SQLite 000020). A workspace
  can publish any number of endpoints, each with its own bearer token
  (SHA-256 stored, rotatable, shown once), knowledge-base allowlist, tool
  allowlist, default agent and per-minute rate limit.
- Management API under /api/v1/mcp-endpoints (Viewer reads, Admin writes,
  API keys need manage_channels) plus a tool catalog endpoint.
- internal/mcpserver hosts one mcp-go Streamable HTTP server in stateless
  mode at /mcp/:endpoint_id. tools/list is filtered per endpoint and
  tools/call re-checks the allowlist, applies rate limiting and touches
  last_used_at. MCPEndpointAuth resolves the token and injects the
  workspace, an mcp_endpoint principal and an API-key style scope derived
  from the endpoint, so existing knowledge-base and capability checks
  apply unchanged.
- Redesigned tool set (12 tools in four groups): retrieval and reading
  reuse the agent's native tools; ask runs the default or a named agent
  synchronously and returns a cited answer with a session_id for
  follow-ups; wiki tools cover search, read and index; write tools are
  off by default. Knowledge bases accept ids or names.
- Frontend: new "MCP Server" integrations tab (tab id mcpserver, to
  avoid clashing with the existing MCP-client settings section) with
  endpoint cards, a create/edit drawer, a one-time token dialog and
  ready-to-paste snippets for Cursor/VS Code/Claude Desktop, Claude Code
  and mcp-remote. Locales updated in all five languages.
- Deployment capability integrations.mcpserver wired on both sides.
- Docs: website-docs MCP page now leads with the built-in server; the
  Python mcp-server README carries a deprecation notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): wire the MCP server into DI/router and tighten authorization

The previous commit lost eight staged files to the repo pre-commit hook,
which resolves ROOT to the main checkout and re-adds the main-checkout
copies of tracked files into the worktree index. This commit restores them
(container providers, RouterParams and route registration, context key,
principal, clone table entry, deployment capability key, SQLite parity
test) and addresses the review findings:

- Knowledge-base scope is authorized, not trusted: endpoint allowlists and
  every tool call resolve each knowledge base through access.ResolveKB for
  the calling principal (ownership or organization share). Foreign IDs are
  rejected on create/update and dropped at call time.
- Write tools mint explicit editor grants via access.KBAccess.WithGrant
  before calling the knowledge service, so add/update/delete go through the
  same RequireKBWrite path as the REST API. add_document also checks the
  grant itself.
- ask no longer accepts agent_id. It runs only the endpoint's configured
  agent; tenant-owned agents and user-facing builtins qualify, internal
  builtins (wiki fixer, skill installer) are refused both on save and at
  call time. Web search is forced off for MCP calls.
- Tool annotations match behaviour: ask is not read-only, add_document is
  a mutation.
- CORS allows MCP-Protocol-Version / Mcp-Session-Id / Last-Event-ID and
  exposes Mcp-Session-Id for browser-based MCP clients.
- Rate limiter runs its local cleanup goroutine; SQLite migration gains the
  deleted_at index for parity with PostgreSQL; non-ASCII endpoint names get
  unique mcpServers keys derived from the endpoint id.
- Swagger annotations on the management handler.
- Tests: KB isolation (service and server), write grants, endpoint-agent
  restriction, tool annotations, and a router test that mounts the public
  /mcp/:endpoint_id surface and checks the deployment capability.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(mcp): renumber migrations after rebase onto main

main took 000100 / 000020 for chunks_index_diet, so the mcp_endpoints
migrations move to 000101 (PostgreSQL) and 000021 (SQLite); the SQLite
parity test and docs follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): run document tools under the knowledge base owner tenant

Shared knowledge bases resolved through access.ResolveKB were only half
usable: documents of an organization-shared knowledge base are stored under
the owner tenant, so the document-level tools that queried by the calling
tenant found nothing, and add_document would have created rows under the
caller's tenant inside someone else's knowledge base.

- knowledgeInScope loads documents tenant-agnostically and then requires
  the document to belong to an allowed knowledge base and to that base's
  tenant.
- scopedKBContext replaces grantContext: it resolves the grant, switches
  the execution tenant to the knowledge base owner (KBAccess.Context) and
  swaps TenantInfo to the owner so storage/model settings resolve there.
  list_documents, read_document and every write tool run under it.
- Locale text and docs no longer claim clients can pass agent_id.
- Go SDK gains MCP endpoint methods (client/mcp_endpoint.go) with docs, and
  the Swagger artifacts are regenerated.
- Tests: shared knowledge base read/write under the owner tenant, spoofed
  document tenant rejected, viewer share cannot write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(mcp): renumber migrations after rebase onto main

main now carries 000101 / 000021 (knowledge_profiles), so the
mcp_endpoints migrations move to 000102 (PostgreSQL) and 000022 (SQLite);
the SQLite parity test expects version 22 and the docs follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): guard last_used touch without a repository and wrap SDK line

The transport tests build the server without a repository; the background
last_used_at write then dereferenced a nil interface and, on the slower CI
runner, took the test binary down. touchLastUsed now returns early when no
repository is wired, and the tests inject a recording repository and assert
the touch instead. Also wraps a 123-column SDK signature for the client
module's lll check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 22:40:05 +08:00