Commit Graph

343 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
Perfecto 336f84c3be
fix(data-analysis): skip style-only worksheets before DuckDB loading (#3660)
* fix(data-analysis): skip style-only worksheets before DuckDB loading

* chore: retrigger CI for XLSX sheet fix

* fix(data-analysis): keep all sheets when the worksheet preflight fails

The style-only sheet preflight is best-effort: if it cannot read the
workbook parts, log a warning and load the sheets DuckDB enumerated
instead of failing the whole workbook. Context cancellation still
aborts the load.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:29 +08:00
Captain 172d0a85ab
fix(mcp): pass embedded resources in tool results to the agent (#3766)
CallTool kept only text and image content from a tool result. A tool
can also return audio, a link to a resource, or a resource embedded in
the result, and GitHub's MCP server returns files that way:
get_file_contents answers with a short message and the file as an
embedded resource. The agent only ever saw the message.

Convert audio, embedded resources (text or blob) and resource links,
recording the resource URI. The agent now follows the reference to a
text resource with its text, and shows links and binary resources as
references. Base64 data of any kind, not only images, is redacted from
the logged content items.
2026-10-08 01:25:04 +08:00
linus-liu-web e394c7adec
fix(agent): 工具自起 goroutine 的 panic 自行兜底,MCP 预热不再带走进程 (#3846)
recover 只对调用它的那个 goroutine 生效。executeRecovered(registry.go)覆盖的是
tool.Execute 这次调用本身,act.go 的 errgroup 回调同理:工具自己起的 goroutine 上
发生 panic,两处都接不到,整个进程退出,正在进行的对话、上传、同步全部中断。

修复站点:
- mcp_exposure.go 的 catalog 预热 goroutine 与最多 8 个 worker 各装一道屏障;worker
  内按服务再兜一层,panic 时沿用文件里既有的 store/status 机制把该服务标记为
  "error"(与加载报错同一个状态)、清掉 partial snapshot,close(preloadDone) 照常
  执行,prepareMCPToolsWithMode 的 select 仍能返回。
- search_knowledge.go 检索扇出的三处 goroutine(embedding 分组、合并 KB 检索、单 KB
  检索):panic 记日志后按既有的 fail() 机制把对应知识库报成失败,工具调用降级为
  失败结果而不是进程退出。
- think.go 的 stream stall 看门狗改用同一屏障(stallTimeout 非正时 NewTicker 会 panic)。

新增 internal/agent/tools/goroutine.go:
- GoRecovered(ctx, fn):起 goroutine 并兜住 panic,供工具内部统一使用;
- RecoverGoroutine(ctx, label, onPanic):defer 形式,onPanic 在同一 goroutine 上记录
  失败单元,panic 值不外传。
registry.go 的 executeRecovered 注释补上"recover 只覆盖本 goroutine,工具自起的
goroutine 必须自己兜底"。

回归测试:
- TestMCPPreloadWorkerPanicIsContained:loader panic 后进程存活、PrepareMCPTools 正常
  返回、该服务 status 为 error 且不保留缓存工具;
- TestMCPPreloadFailureMarkingKeepsReadySnapshots:协调者屏障只标记未完成的加载,
  ready 快照不动;
- TestToolChildGoroutinePanicDegradesToFailedResult:走真实 executeToolCallsParallel,
  子 goroutine panic 降级为失败结果;
- TestGoRecoveredContainsPanicAndRunsDefers / TestRecoverGoroutineReportsPanicToCaller。

去掉屏障后上述用例的测试进程 exit=1,装回后全部通过。

Fixes #3837

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:24:36 +08:00
诉说晚秋 fa4f85d3dd
fix(agent): 拒绝将断流摘要保存为记忆检查点 (#3994) 2026-10-08 01:24:30 +08:00
澍 52893006ad
fix(agent): validate tool string lengths by Unicode code points (#3937)
Fixes Tencent/WeKnora#3935
2026-10-08 01:22:18 +08:00
诉说晚秋 f6119ae5fd
fix(agent): 保留数据分析 SQL 字符串字面量 (#3812) 2026-10-08 01:22:11 +08:00
Perfecto a8c291c6ff
fix: keep failed data-analysis table loads retryable (#3745) 2026-10-08 01:20:48 +08:00
knqiufan e29301115a
fix(graph): preserve relationship direction and document provenance (#3950)
* fix(graph): preserve direction and document provenance in graph retrieval

* test(graph): verify live query bounds and seed evidence ordering

* fix(graph): deduplicate expansion using bounded seed identities

* ci: retry upstream scan after missing CodeCC report
2026-10-08 00:48:51 +08:00
linus-liu-web 9490035f15
fix(agent): 图谱查询的三处截断必须可见,模型不再把子图当全集 (#3885)
query_knowledge_graph 有三处静默上限:返回关系 30 条、每个知识库抓取的
证据 chunk 10 条、送进图谱存储的实体词 8 个。命中上限后既不计数也不提示,
输出还按截断后的长度报告 "Found N relations",读起来就是"该实体的关系就
这么多"。模型据此把被砍过的子图当作实体的完整邻域作答,漏答或得出与图谱
事实不符的结论,且没有任何信号提示它应当收窄实体。同一仓库的图谱存储层
(neo4j/repository.go 的 graphSearchMaxRows 告警)已经把"截断必须可见"
写成明确要求,工具层没有遵守。

改动:
- 关系去重后继续计数,Data 增加 relations_total / relations_omitted;
- 证据 chunk 边取边计数,Data 增加 graph_chunks_total / graph_chunks_omitted
  (未接 chunk 仓库、抓取失败都不计入,避免把错误报成截断);
- 被上限丢弃的查询词计数进入 Data(query_terms_total / query_terms_omitted);
- 命中上限时 Output 追加 "=== ⚠️ Truncated ===" 段并给出收窄建议,主行改为
  "Found 30 of 42 relations";未命中上限不输出任何内容,沉默只表示没有丢弃;
- modelcontext 的 <retrieval> 视图补 <graph_truncated .../>:模型在有 chunk
  行时读的是重建视图,标记只写 Output 到不了模型。

上限数值、去重语义与检索行为不变,只让截断可见。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-07 23:59:50 +08:00
linus-liu-web 436d02fdf3 fix(compaction): 流因取消而关闭时不得当成空摘要成功返回
streamSummary 的 select 里 ctx.Done() 分支与 stream 关闭分支会同时就绪,
select 在就绪分支之间随机选择:已经取消的调用有一半概率走关闭分支返回 nil,
被报成"成功返回空摘要",另一半才返回 context.Canceled。CI 上因此偶发:

--- FAIL: TestSummarizationIsNotRetriedOnceTheTurnIsCancelled (0.00s)
    stream_test.go:122: Error: Should be true
FAIL	github.com/Tencent/WeKnora/internal/agent/compaction

这不只是测试问题。流因取消而关闭被当成成功,会让调用方把中途被截断的摘要
当有效 checkpoint 落盘(成功路径会走到 saveContextCheckpoint),而
validateSummary 只拦得住空摘要和 finish_reason 为 length 的截断。

修复:在流关闭分支返回成功之前先查 ctx.Err()。既纠正语义(取消就是取消,
不是空摘要),也消除两个分支同时就绪时的随机性。streamErr 的优先级仍在
ctx.Err() 之前,因为供应商报的错更具体;ctx 未取消、流自然关闭的正常完成
仍返回内容加 nil,stall 与重试逻辑未动。

回归用例 TestSummarizationIsCancelledWhenTheStreamCloses:假模型同步返回
一个已经关闭的 channel,传入已取消的 ctx,两个分支必然同时就绪。修复前
-count=50 有 20 次 FAIL,修复后 -count=50 全 PASS;既有 flaky 用例
TestSummarizationIsNotRetriedOnceTheTurnIsCancelled -count=100 全 PASS。
2026-09-29 01:30:01 +08:00
Victor Maznyuk cf079b8dcd fix(models): keep Retry-After on HTTPError and honour it in agent LLM retries
Endpoint.Do turns a non-2xx reply into &HTTPError{StatusCode, Body} and
drops the response headers. A caller that retries a 429 or 503 therefore
cannot see the provider's Retry-After and falls back to its own guess: the
agent retry loop waits 1s, 2s, 3s, which against a provider asking for, say,
20s spends every attempt inside the rate-limit window and then fails the
turn.

- HTTPError carries the reply's Header, and RetryAfter() parses the header
  in both forms (delay-seconds and HTTP-date); absent, malformed or past
  values give zero.
- The agent retry loop waits max(linear backoff, Retry-After), with the
  vendor's value capped at 30s so a large Retry-After cannot hold a turn
  indefinitely.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:22:15 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug a829d7cc9c
feat(plugin): agent tools served by plugins, with result views (P2 batch 8) (#3733)
* feat(plugin): agent tools served by plugins, with result views
* fix(plugin): expand plugin tool views in place in the agent stream
2026-09-26 14:48:37 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
MercuialC c7601b2d2a
feat(wiki): 增加跨知识库 Wiki 搜索 POST /wiki-search (#3535)
* feat(wiki): add cross-KB wiki search POST /wiki-search

Give external callers a knowledge-search counterpart for wiki pages: one
SQL IN query ranked by title/slug/summary/content, slim hits with
match_snippet, and Viewer retrieve auth including shared KBs.

* fix(wiki): satisfy golangci-lint on cross-KB wiki search

Check c.Error returns, wrap lines over 120 columns, and use wikiDialect
instead of the embedded Dialector selector.

* fix(wiki): address review on cross-KB search helpers

Share KB viewer auth and snippet extraction, keep ID limits in the
service layer, and escape SQLite LIKE wildcards.

* fix(lint): wrap knowledge span fixtures over 120 columns

Go Lint still diffs from the original PR base, so later main tests
in knowledge_spans_test.go were reported as new line-length issues.

* fix(wiki): authorize KB access before checking wiki enablement in cross-KB search

Checking IsWikiEnabled before AuthorizeKBAccess let a caller tell apart a
foreign KB with wiki disabled (400) from one it cannot see (404), leaking
the KB's existence and settings. Run the access check first.

---------

Co-authored-by: root <root@localhost.localdomain>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:22:58 +08:00
linus-liu-web 6eb3cce164
fix(agent): 合并相邻 assistant 消息时保留 tool_calls,避免孤儿 tool 消息导致整轮 400 (#3574)
* fix(agent): 合并相邻 assistant 消息时保留 tool_calls

SanitizeMessages 合并相邻同 role 消息时只拼接 Content、丢掉了后一条的
ToolCalls;而紧随其后的 tool 消息是否保留,判断的是原始切片,于是保留下来的
tool 消息在请求体里找不到宿主 → OpenAI 兼容端点 400。400 不是 transient,管线
会降级到不带 tools 的总结调用,那一轮模型只能把工具调用写成正文(现象是
"模型输出了调用但页面没变")。

合并时把 ToolCalls 一并合并,并复制而非原地 append,避免与调用方的切片共享
底层数组。

Fixes #3569

* style(agent): 折行以符合 lll 的 120 字符上限

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-25 18:47:55 +08:00
wizardchen 8356742dd5 fix(llm): treat cut-off streams as broken and tighten LLM retries
A stream body that ran out without a finish reason (a proxy cutting the
connection) was closed as a clean stop on the Completions, Responses and
Gemini paths, so a half-written answer was stored as complete. It is now
marked incomplete, as the Anthropic path already did, and both the chat
pipeline and the agent treat an incomplete answer as a stream error.

Retry classification checks typed HTTP/transport errors first and matches
status codes as whole numbers, so "max_tokens 5000" no longer reads as a
500. The retry backoff returns as soon as the turn is cancelled.
2026-09-25 18:22:14 +08:00
wizardchen ef6f28c1ca fix(agent): recover tool panics instead of crashing the server
Parallel tool calls run on errgroup goroutines, and errgroup does not carry
a panic back to Wait, so one panicking tool (MCP client, sandbox, SDK) took
down the whole process. The registry now turns a panic into a failed tool
result, and the parallel path guards its own bookkeeping as a backstop.
2026-09-25 18:22:14 +08:00
lyingbug 04adc13272
fix(retrieval): fix recall correctness, score scales, cost and agent tools (#3687)
* fix(retrieval): keep index writes and shared-KB lookups correct

Index writes
- Image OCR/caption chunks were indexed without IsEnabled, so every engine
  but Postgres stored them disabled and retrieval never returned them.
  The SQLite engine re-enables the affected rows at startup.
- FAQ edits, similar-question additions and import merges re-index under
  the old SourceID and relied on the engine overwriting it. Postgres and
  SQLite kept the stale row (ON CONFLICT DO NOTHING); ES v8 and Qdrant
  mint a new document ID and kept both. FAQ re-indexing now deletes the
  old rows first, and Postgres/SQLite BatchSave upserts by
  (source_id, source_type).
- Postgres: drop the GORM default tag on is_enabled, which replaced false
  with true on insert and indexed disabled chunks as enabled.
- SQLite: BatchSave no longer attaches vectors and FTS rows to the wrong
  rows when part of a batch already exists, and vecTables is guarded
  against concurrent map writes.
- CopyIndices: Postgres filters to the copied knowledge, pages by id and
  keeps is_enabled; SQLite copies every row of a chunk (generated
  questions included) with stable source IDs and enabled state.

Shared knowledge bases
- Chat merge: FAQ answer population and neighbor expansion look chunks up
  without the caller-tenant filter, like parent resolution (#3342).
- Neighbor expansion never cuts the retrieval hit; the length budget goes
  to the text adjacent to it.
- FAQ negative questions are applied whenever an FAQ KB is in scope and
  to chunks of shared FAQ KBs.
- Search results get image_info for shared-KB chunks.
- search_knowledge reads FAQ metadata from the search result instead of
  one tenant-scoped query per hit, so shared FAQs render their answers.

* fix(retrieval): put every retrieval score on one scale

Scores from different retrieval paths were ranked together while living on
different scales, so ordering depended on which path produced a hit.

Fusion
- RRF ranked a chunk by its position in the concatenation of every
  vector (or keyword) list, appended in goroutine completion order. The
  second list's best hit ranked behind every hit of the first, and the
  order changed between identical requests. Each list is now ranked on
  its own by score and a chunk takes its best rank.
- RRF is divided by its maximum, so hybrid scores land in [0, 1] like
  vector-only (cosine) and keyword-only (rescaled BM25) results. Raw RRF
  topped out near 0.016 and lost to any single-retriever hit it was ranked
  against across searches (document targets, model groups, FAQ calls);
  rerank's base-score term and MMR's relevance term had nothing to use.
- Keyword-only BM25 is rescaled per list; engines' BM25 is not comparable.
- FAQ iterative retrieval fuses each round like the first pass instead of
  sorting raw BM25 and cosine together, starts at twice the depth already
  searched, triggers when any single vector list is full (the old summed
  count never matched with more than one list) and replaces results per
  round.

Rerank and chat pipeline
- The FAQ boost is no longer capped at 1; capping made strong FAQs tie at
  1.0 and fall to the tie-breaker. The composite score before boosts is
  kept as composite_score.
- The FAQ direct-answer threshold is compared against that pre-boost
  score, and the FAQ that cleared it is the one marked match="exact"
  rather than the first FAQ.
- Graph hits no longer carry a fabricated retrieval score of 1.0, which
  ranked them above every scored hit when rerank did not run; the
  composite score uses the model score in its place.

Engines
- OpenSearch and Weaviate return (1 + cos) / 2; their drivers now report
  cosine similarity and map thresholds onto the engine scale, so a
  threshold means the same on every engine.
- Milvus L2 returned squared distance, ranked farthest-first downstream,
  and took the threshold as a radius; scores and radius are converted.
- The multi-store normalizer no longer maps Milvus through (s + 1) / 2,
  which put it on a different scale from every other engine and from its
  own single-store results.

Docs updated for fusion, normalization, FAQ boost and iterative recall.

* perf(retrieval): bound rerank, agent and pipeline cost; stop failing on slow or missing models

Rerank
- Rerankers report the longest document a request can carry
  (max_document_chars / max_request_chars minus the query) and the rerank
  stage fits its own passages to it, trimming appended captions, OCR and
  generated questions first. One chunk with a large screenshot's OCR used
  to exceed the vendor limit and fail the whole batch, so every candidate
  lost its score and the threshold stopped applying.
- Rerank HTTP requests default to a 60 s timeout; chat and agent contexts
  carry no deadline, so a stalled self-hosted rerank server held the turn
  open.
- OCR and caption chunks no longer repeat their own text from image_info in
  the passage.
- The chat pipeline and the agent tool rerank at most 200 candidates, best
  retrieval score first.
- A rerank model that cannot be loaded degrades to retrieval order instead
  of failing every chat turn, as the search APIs already do; without rerank
  results the merge stage keeps only RerankTopK hits instead of enriching
  every one.

Retrieval APIs
- knowledge-search match_count and rerank.top_k are capped at 200
  (unbounded values multiplied into tens of thousands of billed rerank
  passages); normalizedMatchCount clamps to the pool size so the
  over-retrieval arithmetic cannot overflow.
- hybrid-search with rerank recalls at least rerank.top_k deep.

Agent
- search_knowledge fits its rendered output to four fifths of the tool
  output budget, dropping the lowest-ranked rows and telling the model how
  many it omitted; the model's view is rebuilt from Data, which registry
  truncation never reached.
- Agent searches skip neighbor/parent/related enrichment rows, which were
  never retrieved yet tripled the rerank input.
- ParallelToolCalls is enabled. The model was always told it may call tools
  in parallel while the engine ran them one by one; only read-only tools
  overlap, anything else stays a barrier.

Chat pipeline
- Query expansion searches keywords only (its variants are local rewrites
  whose embeddings repeat the main search) and is skipped when keyword
  matching is off.
- Entity extraction, a second LLM call, is skipped for intents that need no
  KB retrieval and uses the rewritten query.
- Partial-overlap dedup tokenizes each text once instead of per pair, and
  only compares token overlap between texts within 3x of each other's size.

Storage
- Migration 000112 adds btree indexes on embeddings(knowledge_id) and
  (chunk_id); document deletes, chunk re-index and enable/tag updates
  scanned the whole table.
- Shared-KB knowledge rows are fetched in one query instead of one per ID.
- Drop GORM Debug() from the ParadeDB keyword query and the knowledge batch
  lookup, which logged SQL and user queries on every search.

Docs updated for the API limits, rerank timeout and passage fitting, agent
parallel tools and output budget, and pipeline changes.

* fix(retrieval): make agent tools and pipeline stages report what they did

Agent tools
- query_knowledge_graph now queries the graph store: entity names from the
  query are matched in each KB's graph, relations are returned (and shown to
  the model), and the entities' source chunks lead the results, with text
  search as a supplement. It used to run text search only, so a graph-only
  KB always answered "no relevant graph information".
- search_knowledge reports failures instead of "no matching chunks": when
  every search fails the tool fails with the reason; partial failures are
  listed. A failed query embedding falls back to keyword search for hybrid
  KBs, as the chat pipeline does.
- Keyword mode uses rerank to order hits, not to reject them (identifiers
  score low), and the empty-result hint no longer sends keyword mode back to
  keyword mode.
- Dedup keeps siblings under one parent, no longer merges every image chunk
  of a document through chunk_index 0, and keeps the better-scored copy.
- Search rows omit a <match> excerpt that only repeats the content; snippet
  terms are segmented for Chinese queries and skip stopwords, and FAQ
  matched questions are chosen by match strength.
- read_document query mode returns next_offset when truncated and resumes
  from offset.
- wiki_search over-fetches under document/tag scopes before filtering and
  says when every match was out of scope.
- Prompts point to read_document(id=cN, context=k) for surrounding text.
- The agent editor hides the retrieval-strategy section in agent mode,
  where search_knowledge uses the global retrieval settings.

Chat pipeline
- Intent labels are normalized; an unknown label retrieves instead of
  silently skipping the knowledge base, and web_search without web search
  enabled searches the knowledge bases.
- Image turns get a 2048-token rewrite budget, and a reply cut off by the
  cap is salvaged field by field instead of being discarded.
- History references are matched by how much of the query they cover;
  Jaccard against chunk-sized text never reached its threshold.
- Entity search resolves chunks of org-shared KBs within the turn's KB
  scope, skips chunks whose document is gone (it used to panic) and caps
  graph chunks at 30.
- Empty history is not fetched twice; fetched pages are cut on a rune
  boundary.

Retrieval
- A multi-KB search whose primary KB has no vectors embeds the query with a
  vector KB's model instead of failing on an empty model ID.
- Rerank passages include the chunk's heading breadcrumb, which is part of
  what was embedded.
- SQLite FTS searches the content column only and turns a lone Han
  character into a prefix query; TencentVectorDB ef follows TopK.

Docs updated for the agent tools and pipeline stages.

* fix(retrieval): address review findings on the retrieval fixes

- Restore NeedsRetrieval for web_search without web search enabled: the
  intent prompt "web_search" is how the user learns web search is off, and
  running KB retrieval instead bypassed it.
- query_knowledge_graph keeps the case of entity terms; Neo4j matches node
  names with case-sensitive CONTAINS, so lowercased terms never matched
  English entities.
- An empty search_knowledge or query_knowledge_graph result states partial
  failures in its text, which is all the model sees for an empty result.
- SQLite BatchSave replaces a source in one transaction, so a failed insert
  keeps the old rows instead of leaving the source unindexed.
- Reuse stringSliceValue and an exported searchutil.ContainsChinese instead
  of new copies; drop an unused named return and the FAQ obsolete-source
  delete that the full per-chunk delete now covers.

* fix(retrieval): address second review of the retrieval fixes

- query_knowledge_graph: under a document/tag scope, keep only relations
  whose entities come from an in-scope evidence chunk (the graph namespace
  is the whole KB); drop chunks whose document no longer exists; report a
  graph or text search failure even when the other source returned hits.
- search_knowledge: record the hybrid -> keyword fallback taken when the
  query embedding fails in mode_fallbacks, so an empty result is not read
  as "no semantic match".
- query understanding salvage: take a cut-off value only for the image
  description; a truncated rewrite_query or intent is dropped instead of
  replacing the user's query.
- Graph hits (no retrieval score) are kept past the no-rerank RerankTopK
  cut and the rerank candidate cap instead of always falling below them.
- FAQ boost is capped at 1 again so scores stay on the [0, 1] scale; FAQs
  tied at the cap are ordered by their pre-boost score.
- Negative-question filtering loads only candidates of FAQ knowledge bases.
- read_document query continuation starts one chunk before offset so the
  first match keeps its preceding context at a page boundary.
- graphSearchTerms and searchQueryTokens share one queryTerms splitter.
2026-09-25 10:48:18 +08:00
nullptr-error e2433b1293
feat(lite): install and run skills on the host (#3661)
* fix(desktop): keep Lite signed in with a stable JWT secret

Auto-setup fell through to the login form, and a newly generated signing key replaced the HMAC already used for presigned links. Reuse an existing HMAC, fill only a placeholder JWT secret, and clear a sticky auto-setup failure so the desktop window can enter the local account.

* fix(desktop): make wails dev start locally and store uploads at home

The dev app has no copied config, and the repo env asks for Redis and the Docker path /data/files. Stay on the repo config, use the in-memory stream manager when Redis is unset, write uploads to ~/.weknora/data/files, and allow dragging the top of the window.

* feat(lite): install and run skills on the host instead of a remote sandbox

Lite has no sandbox image, so skills are stored under ~/.weknora/skills as version directories with a live symlink, and chats execute them in the host Seatbelt. The settings UI no longer asks which sandbox to install into. The base profile opens /tmp, chat commands may use the network, and a failed ready-write rolls the symlink back to the previous version.

* fix(ci): align Lite skill checks with the JWT split and lint limits

The desktop signing test still expected the session secret to equal the AES key, and the skill settings source test still looked for the old i18n keys. Wrap the lines golangci-lint rejects so the PR checks can pass.

* fix(ci): separate multi-line test helpers for gofumpt

gofumpt requires a blank line after a multi-line function. The host skill fakes were still packed together, so lint failed on the first one.

* fix(lite): address review findings on host skill install

- Ask mode denies network again, and /tmp writes open only with a
  writable root, so nothing runs unapproved.
- Lite still reports and accepts the workspace script switch; the
  Skills page offers to re-enable it when it is off.
- The host process refuses symlinks the installer planted before it
  clears the install cache or reads the env declaration.
- An existing signing or AES key stays the signer and is never copied
  into signing.key; sessions then get their own jwt.key.
- Skill env capture writes under the host target on Lite.
- Sweep keeps the previous version and skips skills another process
  holds the lock for.
- Terminal and desktop panels never resolve remote sandboxes on Lite.
- A detached install stream resubscribes instead of freezing.
- Upload migration follows the bundle name and the old relative path.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-24 21:30:34 +08:00
lyingbug 9fc08c2079
feat(retrieval): unify rerank across search APIs and expose diagnostics (#3675)
* feat(retrieval): unify rerank across search APIs and expose diagnostics

The two retrieval APIs each missed half of the job: hybrid-search could
tune recall but never reranked, knowledge-search always reranked with the
tenant config and exposed no recall parameters. Rerank itself lived in
four places with drifting thresholds, passages and composite scores.

- internal/reranking: one rerank stage (passage = title + cleaned body +
  captions/OCR + generated questions, threshold with degradation and
  top-1 fallback, composite score, MMR) used by the chat pipeline, the
  agent search_knowledge tool and both APIs.
- hybrid-search: optional `rerank` object; absent keeps the old behavior.
- knowledge-search: vector/keyword thresholds, match_count, recall-path
  switches and a `rerank` object that can override or disable rerank.
- Rerank model resolution: request > tenant RetrievalConfig > first
  rerank model; an unknown requested model is a 400. Load or call
  failures degrade to retrieval order instead of failing the search.
- Responses carry meta.rerank (outcome, model, thresholds, top score) so
  an empty result explains itself (#2391).
- Go SDK and website-docs updated, including a "which API" section.

Refs #2976

* docs(retrieval): lead with which search API to use and when

knowledge-search is the default for callers; hybrid-search is the
low-level recall interface. Add a scenario table so the choice does not
hinge on reading the whole capability comparison.
2026-09-24 20:41:49 +08:00
wizardchen e6284f4eac fix(browser): let shared-space agents drive the caller's local browser
A shared agent runs in its owner's workspace, so the execution tenant in
the turn context is the owner's. The local_browser tool built its device
scope from that tenant, while the browser is paired under the asking
member's own workspace, so every call from a shared agent found no device
and told the user to pair again.

Resolve the browser scope from the authenticated caller instead, both
when registering the tool and in its owner check. Agents in the caller's
own workspace are unaffected: without a tenant switch the caller and the
execution tenant are the same.
2026-09-24 18:19:17 +08:00
wizardchen 1fa915d712 chore: update BrowserSkill extension version to 0.3.1 and enhance localization support
- Updated the BROWSERSKILL_EXTENSION_PATH in .env.example and Dockerfile.app to reference the new extension version 0.3.1.
- Modified localization files for English, Japanese, Korean, Russian, and Chinese to reflect the support for both Chrome and Edge browsers.
- Added checks for extension version compatibility in the BrowserConnectionSettings component to improve user experience.

This commit ensures that the application is aligned with the latest BrowserSkill extension and enhances the localization for better user guidance.
2026-09-23 17:12:19 +08:00
lyingbug e179f44445
refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589)
* refactor(models): separate providers, catalog and runtime configuration

* refactor(models): isolate provider definitions and runtime catalogs

* docs: exclude refactoring analysis notes from the pull request
2026-09-23 14:20:08 +08:00
linus-liu-web 653307b5c7
fix(wiki): 整页重写被补全预算截断时续写,续写用尽则拒绝写回(长名单 146→119) (#3582)
* fix(wiki): 整页重写被补全预算截断时续写,续写用尽则拒绝写回

wiki ingest 路径对 finish_reason 完全无感知:generateWithTemplate 只返回
response.Content,被截断的半页会被原样写回(长名单 146 行 → 119 行,没有提示,
revision history 里看起来就是一次普通编辑)。对照 agent 路径早已有
isLengthFinishReason 处理,两条路径对同一个 provider 信号的处理不对称。

- prompts_wiki.go:新增 WikiPageModifyContinuationPrompt(重放半页,从断点续写,
  禁止重写已写内容)
- wiki_ingest.go:新增 wikiTemplateResult{Content, FinishReason}、
  errWikiPageRewriteTruncated、isLengthStopFinishReason;generateWithTemplate 变薄壳,
  新 generateWithTemplateResult 承担原逻辑并保留既有重试语义;仅对
  WikiPageModifyUserPrompt 在 finish_reason=length 时续写(≤3 轮,无缝拼接,
  "(complete)" 视为已完),仍截断则返回错误 → 走既有分支:warning + 不写页 +
  additionFailed
- 新增 wiki_page_rewrite_continuation_test.go(5 个用例)

Fixes #3581

* style(wiki): 折行以符合 lll 的 120 字符上限

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-23 12:17:07 +08:00
wizardchen 5279972eec fix(agent): give local_browser one text cap and per-method signatures
The flat schema advertised both max_tokens and max_text_chars, so models
sent the log field to observe. Keep a single character budget, map page
reads onto the extension token cap, and list each method's arguments.
2026-09-23 11:49:39 +08:00
lyingbug a89a5a2c43
fix(browser): use unmodified BrowserSkill upstream and remove downstream patches (#3563)
* fix(browser): move BrowserSkill baseline to upstream main with merged patches

Upstream merged the navigation response grace (#291), the preview/focus UI
channel (#296) and popup attribution (#297). Pin both ends to commit c61eb7e4
and drop patches 02, 04 and 06. WeKnora now calls the official ui.task_preview
and ui.task_focus methods.

Patch 05 (last-tab lifecycle) is replaced host-side: before tool.tab_close the
gateway lists the Agent Window's tabs and, when the target is the only one,
creates an agent-owned about:blank tab through tool.tab_create so Chrome does
not remove the window and report a user interruption. Patch 01 (bounded CDP
reads and screenshot fence) is rebased onto the new baseline and remains the
only downstream extension change.

The real-extension test follows the upstream borrow semantics: an unowned tab
inside the Agent Window is rejected in place and borrowed after being moved to
a regular window.

* fix(browser): narrow the BrowserSkill read-reliability patch

Keep isAbortError strict and add isCaptureTerminalError for the capture
pipeline, so a renderer read timeout stops fallback reads but tool handlers
report it as cdp_failed instead of cancelled; the gateway no longer pauses the
task on a read timeout.

Drop the CDP-level screenshot fence and the 3 s screenshot deadline: upstream
PR #296 already bounds preview captures and fences them per attachment, and
tool screenshots are serialized by the daemon queue. Stop routing the focus
emulation toggle through the deadline so upstream's raw-command serialization
keeps holding. Remove the duplicate background rendering browser test; the
upstream background-execution and background-screenshot tests cover it.

* docs(browser): link the read-reliability patch to upstream PR #318

* fix(browser): keep BrowserSkill frame discovery consistent after a read timeout

Sync patch 01 with the follow-up pushed to upstream PR #318: a timed-out child
Target.setAutoAttach keeps its session and is retried from getFrameGraph, and
timed-out frame-tree, frame-owner and root auto-attach reads reject the frame
graph instead of letting the coordinator queue a snapshot behind them.

* style(browser): gofmt focus.go

* fix(browser): roll back the last-tab placeholder and surface renderer read timeouts

The placeholder about:blank tab created before tool.tab_close stayed behind
when the extension refused the close (unauthorized or borrowed tab) or the
call was interrupted. After a failed close the gateway lists the Agent Window
again and removes the placeholder while the target tab still exists; when the
extension already closed the target before the reply was lost, the placeholder
is what keeps the window open and stays.

Sync patch 01 with upstream PR #318: renderer reads on a tab are refused while
a timed-out read is still running in Chrome, and observation errors carry
data.reason = renderer_read_timeout. Map that reason to a model hint that
stops repeated reads on the unresponsive tab instead of the generic
inspect-and-retry advice for cdp_failed.

* fix(browser): remove remaining BrowserSkill patches

* fix(browser): add official extension link and setup guidance
2026-09-23 00:22:34 +08:00
nullptr-error 22cbcdfa57
feat(sandbox): run unpinned Lite sessions on the host OS sandbox (#3549)
* feat(sandbox): resolve workspace paths through WorkspaceLayout

Keep remote sessions on the /workspace contract, while file, shell, and
artifact code follow the session layout instead of hardcoded paths.

* fix(sandbox): fail closed on host layout lookup and restore the shell blacklist

Treat host vs remote as an explicit origin, refuse inspect outside ReadRoots,
and skip collection when the output tree is missing or equals the project root.

* fix(sandbox): close leftover fail-open holes in the layout path

Keep host copy, collection, and prompt text from falling back to /workspace when a lookup fails or an adapter returns unclean roots.

* fix(sandbox): satisfy golangci-lint on the layout helpers

Wrap new copy and test lines to 120 columns, and add the exported comments revive requires.

* fix(sandbox): gofmt the read_file layout copy wrap

* feat(sandbox): add host as an OS-enforced backend type

Host is not a named/pinnable config: each command is a fresh local process, unlike Docker, Cube, and E2B.

* feat(session): persist Lite host workspace directory

Empty means an auto-allocated folder. Standard edition leaves the column unused so the shared session model still loads. SQLite also gains the tenant skill tables Lite shell_exec already expects.

* feat(localsandbox): adapt the host OS sandbox to session interfaces

File tools stay outside Seatbelt and are bounded by PathGuard. A revoked project directory fails closed instead of allocating a replacement workspace.

* feat(sandbox): run unpinned Lite sessions on the host backend

Lite opts into host explicitly; web stays disabled when no named remote config exists. Host workspaces are not destroyed or auto-committed.

* feat(desktop): pick and persist host project directories

The native folder dialog is the authorization path; typed paths are rejected. settings.sandbox.host tells the client when this edition can use it.

* fix(sandbox): fail closed on Lite host layout lookup misses

Unpinned host sessions were described as /workspace when pin read or
sandbox resolve failed. Prompts and attachment staging now share the
same fallback so Lite never pretends the remote tree exists.

* fix(frontend): keep settings.sandbox.host in sync with the backend keys

TestDeploymentCapabilityKeysMatchFrontend requires the lists to match. Host stays fail-closed like docker so a missed probe does not show Lite-only UI.

* fix(lint): satisfy golangci-lint on the host sandbox wiring

Check c.Error, wrap lines over 120 columns, put context first on versionsWorkspace, and document exported adapter methods.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
2026-09-22 19:35:03 +08:00
hailongzhao e55e342387 fix(skill): count create-time sandbox env vars in the required check
A required skill key filled only on the sandbox config was already in the
container, but the gate still treated it as unset.
2026-09-22 16:45:58 +08:00
nullptr-error 7d688cb8d3
feat(localsandbox): add host OS sandbox core and macOS Seatbelt backend (#3472)
* feat(localsandbox): add host OS sandbox core and macOS Seatbelt backend

Introduce a standalone localsandbox package so Lite can run agent
commands on the user's machine under Seatbelt, without linking this
into cmd/server or changing remote sandbox behavior.

* fix(localsandbox): stop post-wait Kill, deny credential files, and refuse swapped roots

Seatbelt was SIGKILL-ing the process group after every successful Wait,
credential write denies omitted literal matches for files, PathGuard
followed a replaced workspace root, and the login-shell test never ran -lc.

* fix(localsandbox): tighten app-data deny, env inherit, and post-wait Kill

Keep ask/full from silently widening to auto, stop inheriting host secrets,
and make session workspaces survive midnight so the Seatbelt kernel stays
fail-closed before adapter wiring.

* fix(localsandbox): pass gofmt, errcheck, and revive on the new package

CI now lints from the merge base, so the host sandbox needs formatted tests, checked Close() results, and exported comments.

* fix(localsandbox): refuse broad workspaces and stop leaking host secrets

Keep the host sandbox fail-closed before adapter wiring: home/Users/Volumes
cannot be the workspace, login shells no longer re-import rc secrets, and
Service applies the env allowlist on every Run. Also close the Lstat TOCTOU
and post-wait Kill window.

* fix(localsandbox): check Close() in seatbelt spawn tests

* fix(localsandbox): deny host temp dirs and refuse leftover wide workspaces

file-read* still left /tmp and /var/folders readable, and Build/Relax
accepted /private/var, /System/Volumes/Data, and ~/Library/** as a
workspace, which re-opened the private deny. Also re-open only ~/.local/bin.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-21 13:24:05 +08:00
lyingbug c40a9dd194
refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470)
把集中式的 OpenAI 方言客户端重建为三层:协议层 internal/models/api/<protocol>(OpenAI Chat Completions / OpenAI Responses / Anthropic Messages / Google GenAI),目录层 internal/models/catalog,厂商层 internal/models/vendors/<id>(27 家,go:embed)。27 家厂商逐条对照官方文档核验,厂商事实以数据声明,协议层不出现厂商名。

既有行的运行时行为会变的四处:
1. extra_config.api 变为保留的协议选择器,非法取值被拒绝;
2. Azure 未填 api_version 的行改走 /openai/v1 GA 数据面;
3. api.openai.com 的一方流量改走 Responses 协议(中转/网关不受影响);
4. 7 家厂商的输出上限字段按文档纠正。

不需要数据库迁移。详见 PR 描述与 website-docs/03-features/06-models.md。
2026-09-20 21:28:23 +08:00
lyingbug 093c16cbb3
fix(agent): end the turn when the completion cap truncates an answer (#3449)
* fix(agent): end the turn when the completion cap truncates an answer

A round that came back with finish_reason=length and no tool calls used to
fall through analyzeResponse as non-terminal. With no tool calls to run,
appendToolResults pushed the half-written answer back into the history as a
plain assistant message with nothing telling the model to continue, so the
next round rewrote the answer from the top, hit the same cap, and repeated
until the round budget ran out or the user cancelled.

Three guards, each independently load-bearing:

- analyzeResponse gains a truncation case: deliver the partial text and end
  the turn. A cap hit that produced no text at all has nothing to hand over
  and defers to the existing bounded empty-content retry. No continuation
  nudge is sent — it only grows the prompt with the discarded fragment, and
  assistant prefill, the reliable form, is unavailable on most
  OpenAI-compatible endpoints.
- A consecutive-length breaker stops the loop when truncation keeps landing
  inside tool-call arguments, the one shape that still legitimately loops.
- The stuck-loop detector now counts an identical lack of content. A model
  burning its whole budget on reasoning returns empty every round, which used
  to land in the else branch and reset the counter, so the guard could never
  fire on the case that needed it most.

The answer event and its Done marker carry a truncated flag through the SSE
metadata, and the chat UI shows the same style of notice it already uses for
fallback answers.

Fixes #3446

* fix(agent): treat unterminated reasoning as no answer and keep the truncation marker

Review follow-up on three ways the truncation path could still mislead.

An inline <think> block that the cap cut off never gets its closing tag, so
the pair regex left it whole and the round looked like it had produced an
answer. The truncation case then skipped the empty-content retry it exists
for and emitted the chain of thought as the final answer. StripThinkBlocks
now drops a dangling <think> through end of input, which is what the
streaming splitter already does with the same text, so the two agree on what
counts as an answer.

The truncation marker only rode on the SSE metadata, and a finished
conversation is rebuilt from agent_steps rather than replayed from stream
events. A reloaded, shared or re-opened turn therefore lost the notice and
its half-written answer looked complete. AgentStep carries the flag now — it
is a jsonb column, so no migration — and the frontend marks the rebuilt
answer event and message from it.

The consecutive-length guard ran before the round was recorded, so a final
round of "let me look that up" plus a truncated tool call handed that
preamble over as the final answer, with its tool calls never refused and the
preamble never retracted from the answer area. The guard now records the
round and fails its tool calls the way every other truncated round does, and
a response carrying tool calls never donates its text as the answer.
2026-09-20 13:17:50 +08:00
autumnseasonism 1e4389bb21
fix(wiki): clarify alias matching rules in deduplication prompt (#3418)
Co-authored-by: autumnseasonism <275002454+autumnseasonism@users.noreply.github.com>
2026-09-20 11:24:45 +08:00
lyingbug b82fd6c0f9
fix(security): close the low-severity shared-space findings (#3416)
* fix(security): close the low-severity shared-space findings

Verified and fixed from the shared-space permission review:

- Embed channels kept knowledge_ids, tag_ids, mentioned_items, skill_names
  and summary_model_id from anonymous visitors. Mentions are only scoped
  for shared agents, so a visitor could query any KB (and pick any model)
  of the channel workspace by ID. The payload now drops them; the widget
  never sends them.
- PUT /knowledgebase/:kb_id/wiki/issues/:issue_id/status updated any issue
  by ID. The update is now scoped to the path's KB (404 otherwise).
- A direct org invite let the inviter attach any user of the target
  workspace (or any user by ID) as its representative, whose username and
  email the roster then showed to all members. Direct adds attach no
  representative, and the roster returns emails only for the caller's own
  workspace.
- COS and KS3 dropped the bucket (and COS the region) segment when turning
  a path into an object key, while the tenant check scans every segment:
  cos://42/<region>/prefix/7/... passed as tenant 42 and read tenant 7's
  object from a shared bucket. Both must now match the service's config.
- Shared agent listings returned the full agent to receivers. Prompts and
  the creator's user ID are withheld; capabilities and scope stay visible.
- Temporary document uploads let the caller pick the parser engine (and
  its credentials) of a shared agent's workspace; only the agent's rules
  choose one there now.
- Suggested questions used request / message-snapshot knowledge_ids
  without a permission check. Each document is now resolved to its KB and
  checked like KB IDs, including a shared agent's scope.
- IM channels accepted any knowledge_base_id (and any agent ID on create).
  Both must belong to the channel's workspace, the KB also to a restricted
  API key's allow-list.
- KB-restricted API keys could list org-shared KBs outside their allow-list
  in the organization views; those listings are now filtered.
- With enable_rbac=false, RequireKBAccess let cross-workspace denials
  through. That switch is for roles inside a workspace; the guard now
  always denies another workspace's KB, since handlers behind it load by
  ID without a tenant filter.
- The org member limit was counted before and enforced apart from the
  insert; it is now checked in the insert's transaction with the
  organization row locked (SQLite serializes writers instead).
- The UpdateOrganization route comment and API doc said owner-only; the
  service and UI allow org admins. Comment and doc corrected.

Documented, not changed: issued /r/ and presigned links stay valid until
they expire after a share is revoked.

* fix(security): address review of the low-severity shared-space fixes

- Embed payloads also drop question_origin. It selects a KB like an
  @mention for agents that retrieve only when mentioned, and the widget
  never sends it; the selection stays inside the agent's scope, but an
  anonymous visitor should not pick among its KBs. suggestion_attribution is
  kept (the widget sends it and it is validated).
- The receiver view of a shared agent also withholds the follow-up
  suggestion instruction, a prompt like the others. Starter questions are
  shown to receivers anyway, and the sharer's name is displayed by design.
- Follow-up suggestion generation moves into the agent's workspace with
  types.WithExecutionTenant instead of a bare tenant rewrite, so the
  caller is pinned before the document permission check runs. The auth
  middleware and the shared run already pinned it; this removes the
  dependency on that.
- KS3 CopyFile copies only from the service's own bucket (as COS and S3
  do); another bucket's path is a cross-backend copy.

Tests cover the dropped question_origin, the follow-up instruction and the
/shared-agents listing, a follow-up context with the tenant switched, a
repeated wiki issue status update, and the KS3 copy source.
2026-09-18 22:58:46 +08:00
lyingbug 03d9d66a1f
feat(agent): size agent history by the window and calibrate token estimates (#3413)
* feat(agent): size agent history by the window and calibrate token estimates

Agent history was capped at history_turns (default 5) because compaction
could not survive a turn. With checkpoints persisted, load history by
tokens instead, and let compaction bound the session.

Budget. The loader drops the oldest turns that do not fit, and what it
drops is lost: neither replayed nor summarized. A budget equal to the
compaction threshold (the earlier attempt, since reverted) let the loader
trim first, so whenever one turn was larger than the system prompt plus
the new question the request never crossed the threshold; nothing was
summarized and the session became a sliding window with no checkpoint.
agent.HistoryTokenBudget is now the whole window: overflow reaches the
first round's compaction, which summarizes it into a new checkpoint.
A turn-by-turn regression test drives the real loader, budget and
compactor and fails as soon as a stored turn after the checkpoint is
missing from the loaded history; restoring the old budget fails it.

Summarizer input. History can now reach the window, so the part being
summarized can outgrow one request. An overflowing request degrades to a
raw archive that is never persisted, so every later turn would fail the
same way. Prepare keeps the newest messages that fit, the prompt states
how many were left out, and file paths still come from all of them.

Calibration. cl100k counts Chinese at about a token a character where
Qwen and DeepSeek spend about 0.6. The estimator now carries a scale
(clamped to 0.5-1.5, text only), learned from consecutive requests in a
turn: they share tools, system prompt and earlier messages, so the
provider's growth between them prices exactly the appended messages,
without mixing in how a provider renders tool schemas. The turn's scale
is stored on its usage (context_token_scale); the next turn's loader
prices history with it and hands it to the engine. Loading, the first
compaction check and compaction itself share one estimator, so they
agree; calibrating only the trigger would let the loader drop turns
again, which the regression test covers.

Also: history is priced as the engine sends it (redacted KB/Wiki results
cost one line); a checkpoint older than the rows read is matched by the
assistant rows' (created_at, id) order; a read ending on a turn's
question keeps that turn; a compaction discarded for freeing under 5%
still persists its checkpoint; a checkpoint write matching no row is an
error; a partial index backs the newest-checkpoint lookup. Agent mode no
longer reads history_turns: the field leaves AgentConfig, the editor
hides it in smart-reasoning mode, and quick-answer keeps its turn cap.

* fix(agent): address review of window-sized history loading

Summarization. The summarizer call had a fixed 60s total timeout. With
history loaded up to the window, a request could carry ~175k tokens and be
cut off while prefilling or writing normally; the degraded result left no
checkpoint, so the next turn reloaded the same history and failed the same
way. Summarization now streams and is cancelled only after no output for the
engine's stall timeout (120s by default, LLMCallTimeout overrides), with the
overall ceiling left to the transport, as for the engine's own rounds and as
in Codex and Pi. A stopped turn is not retried. When the summarizer still
fails, the raw archive is persisted as a checkpoint marked degraded so the
loop is broken, and the next compaction refines it as the previous summary.
The archive is held to the summary budget, newest messages first, so a
degraded compaction always frees room.

Loading. The loader priced turns as sent but returned and held them as
stored, so wiki pages stored in full and sent as one line could pile up in
memory. It now returns the as-sent history and releases each turn's stored
rows once replayed, keeping about a page of rows at a time.

Index. The partial checkpoint index is replaced by a composite
(session_id, created_at DESC, id DESC) index serving the backwards page,
the newest-checkpoint lookup and the recent-messages reads. It is built with
CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the
migration holds that single statement, since a multi-statement query runs as
an implicit transaction. A PostgreSQL integration test, skipped without
WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and
up through golang-migrate, and checks both queries use the index without a
sort; the SQLite migration test checks the same plans.

Calibration. It no longer waits for a total token count, and a usage with
only a scale is persisted. A turn that measures nothing carries forward the
scale it started from, so the newest turn always has the latest one.
2026-09-18 22:51:31 +08:00
lyingbug 62f7da5eea
feat(agent): persist compaction checkpoints across turns (#3407)
* feat(agent): persist compaction checkpoints across turns

Agent compaction only lived inside the turn that ran it: the summary was
discarded at turn end and LoadAgentHistory rebuilt raw history from the
messages table, so every later turn past the threshold paid for another
summarization of nearly the same history.

A compaction whose history summary ends exactly on a stored turn is now
written onto that turn's assistant message (messages.context_checkpoint).
LoadAgentHistory starts from the newest checkpoint: the summary replaces
that turn and everything before it, and later turns are replayed verbatim.
The compactor already treats a leading summary as the previous summary, so
checkpoints fold forward incrementally.

History messages carry their turn's assistant message ID (chat.Message
TurnID, engine-internal) so compaction can tell whether its cut lands on a
stored-turn boundary. Nothing is persisted when the cut ends inside a stored
turn, covers only the live turn, or the history summary degraded to a raw
archive; a split turn's prefix summary is never persisted because that turn
is replayed whole next time. Writes are column-scoped and best-effort, and a
failed lookup falls back to history without a checkpoint.

Storing the checkpoint on the covered turn keeps the boundary implicit, so
session forks copy it with the turn and deleting the turn drops it. The
HistoryTurns cap still applies, now to the turns after the checkpoint.

* test(skill): wait for the background install in the name-conflict test

InstallSkill queues the install on a goroutine, and the fixture row already
starts at installing, so reading the status right after the call raced the
mock install and failed whenever it landed on ready first. Wait for the run
to finish on the reused row instead, as the other install tests do.

* feat(agent): size agent history by tokens instead of turn count

Agent history was capped at history_turns (default 5) because every turn
used to rebuild raw history and compaction could not persist. With
compaction checkpoints persisted, the cap is no longer what keeps a session
inside the window, and it threw away context that would have fit.

LoadAgentHistory now loads the newest checkpoint summary plus the newest
turns after it that fit a token budget: the run's compaction threshold
(agent.HistoryTokenBudget, the window less the reply reserve). History
that large already triggers the first round's compaction, which persists a
new checkpoint, so compaction rather than a turn count bounds the session.
Loading more would only hand that compaction a summarizer input larger
than the model can read.

Rows are paged backwards by (created_at, id) and the read stops at the
checkpoint or once the budget is full, so a long session is never read
whole. Kept turns stay contiguous, the newest turn is kept even when it
alone exceeds the budget (compaction splits it), and a turn cut by the read
is left out rather than replayed without its original question.

Agent mode no longer reads history_turns: AgentConfig.HistoryTurns is
removed, the editor hides the input in smart-reasoning mode and explains
the automatic behavior, and the smart-reasoning builtins drop the setting.
Quick-answer (KnowledgeQA) keeps its turn cap since it has no compaction.

* Revert "feat(agent): size agent history by tokens instead of turn count"

This reverts commit d05fb6e5f1f161813a88e9127609f7a059f38f12.

Loading history up to the compaction threshold lets the loader drop the
oldest turns before compaction can trigger: whenever a single turn's
replay is larger than the system prompt plus the new question, the
trimmed history stays under the threshold, nothing is summarized, and the
session degrades into a large sliding window that never gets a
checkpoint. A turn-by-turn simulation of LoadAgentHistory with the real
compactor showed zero compactions over 60 turns in those cases.

Keep the persisted checkpoints with the HistoryTurns cap for now; token
budgeted history returns in a separate change with a budget that leaves
the overflow to compaction.
2026-09-18 20:38:56 +08:00
lyingbug 2be7bd4063
fix(agent): search bound knowledge bases before answering, and pass suggested-question origins (#3400)
* fix(agent): search bound knowledge bases before answering, and pass suggested-question origins

A user picked the suggested question "为什么在比较图形时需要谨慎处理?"
and the agent answered in round 1 with no tool call, claiming it had
searched the knowledge base and found nothing, then explained misleading
statistical charts. The question had been generated from "Corners Don't
Look Like That: Regarding Screenspace Ambient Occlusion" in a bound base,
where "comparing graphs" means SSAO falloff curves.

Two things let that happen: the grounding prompt said "Stable general
explanations need no lookup", and nothing told the agent where the
question came from - the base in question had no profile, only two
recent titles in the directory, and the click sent the text alone.

Prompt
- With knowledge tools available, search bound bases before answering a
  question about a topic, even one that reads like general knowledge;
  directory profiles and recent titles cannot prove a base lacks it.
- The general-explanation exemption applies only without bound bases.
- Say you searched or found nothing only when a search tool ran.

Question origin
- Suggested questions from FAQ/document chunks also return knowledge_id.
- Picking one on the home or chat page sends question_origin
  {knowledge_base_id, knowledge_id} with that question only.
- The service keeps the origin only when its base is inside the turn's
  resolved scope and the document belongs to that base; it never
  changes the scope (unlike knowledge_base_ids, which narrows it).
- The engine renders it in runtime_context as <question_origin> with the
  base and document handles and a note to search it before answering.

* fix(agent): keep question origins across file/tag scopes and bind them to the send

Review follow-up for the suggested-question origin.

- Scope check: the origin was kept only when its base was in the resolved
  KB list, which is empty for file-only and tag-only scopes even though
  their bases are searched. It is now checked against SearchTargets after
  they are built; the document is kept only when a target for that base
  covers it (unfiltered base, or a document scope listing it).
  resolveQuestionOriginInfo re-checks SearchTargets, looks up the name of
  a base that is not in kbInfos, and the renderer omits an unknown name.
- Frontend: the origin travelled through module-level "pending" state, so
  a send that returned early (agent not ready, upload in progress,
  steering) left it behind and a second pick overwrote it. It now rides
  with the one send: triggerSend(text, options) -> createSession ->
  send-msg -> sendMsg / changeFirstQuery.
- RetrieveKBOnlyWhenMentioned: suggestions are still generated from the
  agent's bases, but picking one retrieved nothing. The origin now
  selects its base, as an @mention would, when the agent may read it.
- API sync: question_origin in docs/api/chat.md and swagger;
  QuestionOrigin on the Go SDK's KnowledgeQARequest/AgentQARequest and
  knowledge_id on SuggestedQuestion.
2026-09-18 17:00:47 +08:00
wizardchen 2a6e5dcbb2 fix(agent): allow keyword retry after search_knowledge rerank rejection
Repeating the same query in another mode is usually wasted, but keyword
mode can still recall identifiers that hybrid never surfaced. Stop telling
the model that switching mode cannot help.
2026-09-18 16:06:52 +08:00
wizardchen 62bb9da270 fix(agent): score search_knowledge reranks with the document title
The agent searched seven times for "Echo 开放权重模型 降低成本". Hybrid
retrieval ranked the right chunk first (RRF rank 0) every time, but the
rerank model saw only the chunk text, which never names Echo or cost;
it scored 0.0025 and all 18 candidates fell below the 0.15 fallback
floor. The tool reported "No matching chunks", and the model cycled
hybrid -> semantic -> keyword, which cannot help since every mode shares
the same rerank check. Scoring the same chunk with its document title
("Show HN: Echo - ... at 1/3 the cost using open-weight models") gives
0.45, and 0.55 for the query written as a question.

- Rerank passages are title + enriched chunk (rerankPassage); FAQ chunks
  keep their own question, and the pattern filter still sees chunk text
  only.
- When retrieval found candidates but the rerank model rejected all of
  them, the empty statement says so and that switching mode will not
  help, instead of "No matching chunks".
- The tool description, query schema and agent system prompt ask for a
  complete natural-language question in hybrid/semantic mode rather than
  a keyword list, and drop "switch mode" as the retry advice. Every
  keyword engine matches any token (OR + BM25), so a question costs the
  keyword leg no recall; on the same 20 bases the Echo chunks rank the
  same for both query styles.
2026-09-18 15:33:56 +08:00
wizardchen ad5d6a8fc2 fix(agent): make read_document queries keyword-based and bound its pages
read_document's in-document query was matched as one literal string, so a
keyword query such as "sunflower f(n,k)" found nothing in a 708-chunk
document even when both words were there. The zero-match result gave the
model no next step, and the UI rendered it as "0 / 708 chunks loaded",
which reads like a failed load.

Paging mode also had no size bound: the registry truncates
ToolResult.Output, but the model sees the result rendered from Data, so a
limit=100 page reached ~68k characters in the model context.

- A plain query is split on whitespace; a chunk matches when it contains
  every word, in any order. regex=true still matches one expression.
- A zero-match result carries a <hint> suggesting fewer/different words
  or reading a chunk from search_knowledge results.
- A page stops adding chunks at 80% of the output budget (counting
  per-chunk rendering overhead); next_offset resumes at the first chunk
  not returned, and an oversized first chunk is still returned.
- The UI summary shows in-document search results ("no matches" /
  "N matches") and, for offset paging, the chunk range instead of a page
  number that no longer holds.
2026-09-18 15:09:25 +08:00
lyingbug 92e7c0ab37
refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381)
* refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents

The agent's retrieval surface had grown to five overlapping tools whose
behaviour was scattered across long descriptions, interleaved "tips" in
tool output (most of which modelcontext discarded before the model saw
them), and a grep_chunks implementation that ran an unindexed regex scan
over the chunks table ordered by created_at, so large knowledge bases lost
recall and paid a full-table scan on every call.

Tool surface
- search_knowledge replaces knowledge_search and grep_chunks: one query,
  mode = hybrid | semantic | keyword, optional knowledge_base_ids, limit.
  Keyword mode runs through the retrieval engine's keyword index instead of
  the DB regex scan. Modes unavailable for the scope are refused with a
  reason. Rerank, MMR and de-duplication are kept; already_seen markers,
  retrieval_statistics and the zero-result "CRITICAL" block are gone.
- read_document replaces list_knowledge_chunks, get_document_info and
  wiki_read_source_doc: id accepts a dN or cN handle, always returns the
  document header, pages by exact offset (fixing the old floor-to-page
  bug), finds passages with query (literal by default, regex opt-in) and
  reads neighbours around a chunk with context.
- list_documents pages through one knowledge base, honouring @file/@tag
  scope.
- wiki_search takes a single query plus regex, knowledge_base_ids, limit;
  literal queries are escaped so the model never reasons about JSON
  escaping. Legacy queries / knowledge_base_id are still accepted.
- query_knowledge_graph is registered only when a graph-enabled KB is in
  scope and its description/tips were trimmed.

Compatibility
- Retired names in stored allowlists, presets and API calls are mapped to
  their successors by NormalizeAllowedTools at registration time; no data
  migration. Stored histories with old names still decode, render and
  redact. Calls to retired names return a replacement hint.
- modelcontext gains an "id" key space that decodes against both the chunk
  and document handle tables (decode-only), plus rendering for the
  document header, chunk roles, in-document matches and next_offset.

Prompts and surfaces
- The RAG, wiki researcher and hybrid templates now name the tools and the
  search -> read -> cite order; tool descriptions are short and contain no
  workflow instructions.
- Engine display names, history redaction, grounding prompt, IM rendering,
  the built-in MCP server (search_knowledge gains mode/limit, grep_chunks
  becomes keyword search, read_document gains query, wiki_search gains
  regex), frontend editor/renderers/i18n, presets, website docs and the
  changelog are updated accordingly.

Tests cover name mapping, mode validation and refusal, output shape,
read_document paging/context/query/scope, list_documents, wiki_search
escaping and legacy arguments, modelcontext decoding and rendering, and
registration-time aliasing and graph gating.

* fix(agent): address review findings on read_document, list_documents and model rendering

- read_document: fetch the second page whenever the first page did not
  reach the end of the document, so an unaligned window near the end no
  longer drops the trailing chunks (also fixes context windows around a
  chunk near the end); a query passed together with a cN handle now
  searches the owning document instead of being ignored; query mode stops
  collecting once the output budget is nearly spent and reports truncated;
  the document row is authorized in place instead of being fetched twice;
  the context read takes the total from the window fetch instead of a
  separate count query.
- list_documents: resolve the page's document scope in one batched tag
  lookup instead of one call per row.
- modelcontext: render read_document output once and read next_offset
  through the existing intValue helper.
- search_knowledge: point title/file-name lookups at list_documents.
- Tests for the boundary window, chunk context near the end, cN + query,
  and budget-bound query results.

* fix(agent): close compatibility gaps in the consolidated retrieval tools

- Wiki-only scopes: read_document and list_documents are registered
  whenever a wiki KB is in scope, since every KB stores chunks. Their
  capability requirement includes wiki but is marked auxiliary, so they do
  not widen a RAG agent's derived "all knowledge bases" filter to wiki-only
  bases (Go and frontend derivations both honour it).
- search_knowledge resolves the retrieval mode per knowledge base instead
  of refusing the call: FAQ and vector-only bases answer keyword requests
  semantically, keyword-only bases answer semantic requests by keyword.
  Results carry requested_mode and mode_fallbacks, the empty-result text
  names the fallback, and the call is refused only when no base in scope
  has a chunk index.
- The model rendering uses the mode the search actually ran with (it was
  always labelled semantic) and emits <mode_fallback> per degraded base.
- list_documents applies the @file/@tag scope before paging: tags are
  pushed into the database filter, pinned documents are loaded by ID, and
  total/next_page count only readable documents.
- read_document resolves chunk context by chunk_index through a new
  ChunkRepository.ListChunkNeighbors, so parent, summary and image chunks
  sharing the index sequence no longer shift the neighbourhood; offset is
  documented as a reading-order position, not a chunk index.
- wiki_search (agent and MCP) keeps its regular-expression default and
  only matches literally when the text is not a valid expression;
  regex=false forces a literal match and regex=true rejects invalid input.
- MCP grep_chunks keeps grep semantics: literal terms from the pattern
  fetch candidates from the keyword index (semantic for bases without
  one), and every returned chunk is verified against the pattern.

* docs(changelog): align the consolidation entry with regex wiki_search and grep_chunks semantics

* docs(mcp): note that grep_chunks recall is bounded by index candidates
2026-09-18 14:12:22 +08:00
lyingbug 5574ebc011
feat(knowledge): derive knowledge base descriptions from document profiles (#3370)
* feat(knowledge): derive knowledge base descriptions from document profiles

Knowledge bases only had a hand-written description, which most users
leave empty, so agents bound to several knowledge bases had nothing to
route on and searched all of them. This adds a generated description
that is derived from per-document data instead of maintained by a model.

Document profile: the summary call now returns JSON with a 2-3 sentence
summary plus a gist, 3-5 topic keywords, a document type and one
typical question (knowledges.profile). Input is capped at 8k chars and
output at 1024 tokens; plain-text replies from custom templates still
work and simply carry no profile. The summary chunk embeds the gist and
short summary.

Knowledge base aggregate: a database-only fold over those rows (counts
per file type, tag, normalized topic and doc type; stratified samples of
titles and questions; a hash of the inputs). Deleting, moving or
re-parsing a document changes nothing but the rows, so the next rebuild
is exact without any model judgement.

Description text: one small model call over the aggregate yields a
gist, merged topics and typical questions (knowledge_bases.
generated_profile). The call is skipped when the aggregate hash is
unchanged; failures keep the previous text and record the error; an
empty knowledge base clears it. The manual description is never
touched, and the agent runtime context renders both so the model can
pass knowledge_base_ids instead of fanning out.

Refreshes are requested from summary task exit, post-process completion,
deletion, cross-KB moves and enabling the option, debounced per
knowledge base in 30 s windows via kb:profile on the summary queue.
POST /knowledge-bases/:id/profile/generate regenerates synchronously.
profile_config (enabled, model_id, custom_instructions) opts a document
knowledge base in; the settings dialog shows the generated card with
regenerate and adopt-as-description actions in five locales. Search
results now log hits per chunk type so the value of indexing summary
chunks can be measured.

Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the
SQLite schema test expects the new version and columns.

* fix(knowledge): keep generated descriptions out of duplicates and refresh clones

DuplicateKnowledgeBase copies the source row through JSON, which carried
generated_profile onto a knowledge base that has no documents yet, so
an empty duplicate advertised the source's topics. The duplicate now
starts without a generated description and keeps profile_config so it
derives its own once documents arrive.

CopyKnowledgeBase builds the target field by field and skipped
profile_config, and the clone task never asked for a description after
copying the documents, so a clone stayed undescribed until its next
upload. The target now carries a copy of profile_config and both clone
paths request a refresh on completion.

* fix(knowledge): refresh the loaded clone target and carry document profiles

ProcessKBClone requested the description refresh with the reservation
stub it builds before CopyKnowledgeBase runs. For the main create path
that stub has neither a type nor profile_config, so the refresh helper
rejected it and nothing was enqueued. The refresh now uses the loaded
target returned by CopyKnowledgeBase.

cloneKnowledge copied the summary text but not the structured profile,
so even a scheduled refresh would have aggregated titles without topics
until every clone was re-summarized. Cloned documents now carry an
independent copy of the profile.
2026-09-17 22:21:55 +08:00
lyingbug 182f67bd86
refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372)
* refactor(wiki): consolidate source_refs parsing and resolution

source_refs entries ("uuid" or "uuid|title") were parsed by ten
independent implementations that disagreed on whitespace handling, on
whether a model-supplied title could be trusted, and on how a leading
pipe was treated. The scope-enforced and legacy agent resolvers also
duplicated the lookup and the routing step looked every document up a
second time.

- types.ParseWikiSourceRef / FormatWikiSourceRef / WikiSourceRefMatchesKnowledge
  are now the only readers and writer of the format.
- resolveWikiSourceDocuments replaces resolveAuthorizedSourceRefs,
  resolveSourceRefs and wikiKnowledgeBasesForSourceRefs: one lookup per
  document, titles always rebuilt from server data, dedupe by canonical ID,
  unknown documents fail closed in every mode.
- removeSourceRef now matches padded or titled refs of a deleted document.

* refactor(data_analysis): expose the selected document as a fixed table name

The tool required the document ID inside the SQL text and blindly
string-replaced it with the physical DuckDB table name, which forced the
model-context codec to rewrite handles inside quoted SQL and let the
UUID-derived k_<uuid> table name reach the model through stored schema
summaries.

The document is now selected only by knowledge_id. Every query runs on a
dedicated connection where "dataset" is a temporary view over the
physical table (DuckDB temporary objects are connection-local, so
concurrent sessions stay isolated). The physical name remains accepted
for internal callers; any other table fails validation with an error
naming the table the model should use. Schema summaries, the schema
tool, the analyst prompt and the tool schema all name the same table, so
data_analysis.sql no longer carries source handles and its text codec
policy is removed.

* feat(modelcontext): report raw identifiers that survive encoding

Text-level compaction exists because raw UUIDs reach model context through
producers the registry never sees: error strings, slugs, results of
unknown shape. Retiring it requires knowing where those producers are.

Registry.LeakedIdentifiers scans an encoded message list for UUID-shaped
values in every application-composed field (user messages are skipped),
classifies each as registered (codec missed a rewrite) or unregistered
(producer emitted a raw ID), and SummarizeLeaks renders one line per
field. The agent engine and the three RAG pipeline call sites log the
report as a warning after EncodeMessages.

* fix(chat_pipeline): align RAG data analysis with the fixed table contract

The RAG data-analysis plugin still asked the planning model to fill in
knowledge_id and to write SQL from a prompt that exposed the raw
knowledge UUID, then passed the model's JSON straight to Execute. Under
the new data_analysis contract a query that names anything but the
dataset table is rejected, and the plugin logged that rejection at error
level and moved on, so the analysis vanished without any signal. A
"no analysis needed" plan took the same error path.

The pipeline now owns document selection: the model returns only
needs_analysis and sql, the prompt names the dataset table and never
shows the knowledge ID or physical table, and the plugin builds the tool
input itself. A rejected query is fed back to the model once for repair;
a query that still fails is reported through pipelineWarn with the SQL
and the tool error, and "not needed" is logged as such instead of as a
failure.

Covered by unit tests with a scripted model and executor: not-needed,
pipeline-owned knowledge_id, one-shot repair, persistent failure
surfaced, unparseable plan rejected.
2026-09-17 22:08:16 +08:00
wizardchen 1c43934e53 perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
lyingbug 5780cb0a71
fix(sandbox): separate file access from artifact publishing (#3356)
* fix(sandbox): separate file access from artifact publishing

* test(sandbox): align service coverage and pre-push checks with CI
2026-09-17 16:18:01 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
serply e80f5dfbc7 feat(web-search): add Serply provider
Serply returns Google results through a keyed JSON API, which fills the
gap between Google Custom Search (needs an engine ID and a restricted
scope) and the scrapers. The adapter follows brave.go: hardcoded
endpoint, redirects refused so the key never leaves the official host,
4MB response cap, and per-call country and freshness filters. country
maps to Google's gl parameter and pd/pw/pm/py map to tbs qdr:d/w/m/y.
Serply has no date-range filter, so that form returns an error instead
of being silently dropped. num is clamped to 10 because the API serves
one results page per call.

Wiring matches the Bocha provider: type constant and metadata, registry
factory, service validation, frontend union, swagger enum entries (edited
by hand; a full swag regen drags in unrelated drift), and the web search
docs table.
2026-09-17 12:25:09 +08:00
wizardchen 84c9871d16 fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
lyingbug d7ccd5ba51
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling

* fix: preserve trusted Doris redirects and prepare browser tests

* fix: align proxy address checks and integration test coverage
2026-09-16 16:20:57 +08:00
vicfei 780f2beb5a fix(agent): stop emitting terminal answer events for retryable empty stops
analyzeResponse emitted EventAgentFinalAnswer with Done=true before
returning emptyContent=true to the retry loop, so every empty natural
stop produced a premature terminal answer event while the agent was
still running. Consumers that finalize on Done=true (e.g. the IM
channel, which closes its answer-done channel and then only waits a
bounded time for EventAgentComplete) can lock in a fallback and cancel
the still-running agent when the nudged retry is slower than that
window (#2906).

An empty natural stop now emits nothing — the verdict alone tells the
caller the round is retryable. When retries are exhausted, the loop
emits the selected fallback as the turn's sole terminal answer, so a
turn always ends with exactly one terminal answer event.

Regression tests cover all three shapes: the empty verdict emits no
answer events; empty-then-answer ends with exactly one terminal event
carrying the real answer; all-empty runs end with exactly one terminal
event carrying the fallback (each red on the pre-fix code).
2026-09-14 21:02:15 +08:00