Commit Graph

1810 Commits (22cbcdfa5789b33dbb8d47532c21b24da2fdf5aa)

Author SHA1 Message Date
nullptr-error 22cbcdfa57
feat(sandbox): run unpinned Lite sessions on the host OS sandbox (#3549)
* feat(sandbox): resolve workspace paths through WorkspaceLayout

Keep remote sessions on the /workspace contract, while file, shell, and
artifact code follow the session layout instead of hardcoded paths.

* fix(sandbox): fail closed on host layout lookup and restore the shell blacklist

Treat host vs remote as an explicit origin, refuse inspect outside ReadRoots,
and skip collection when the output tree is missing or equals the project root.

* fix(sandbox): close leftover fail-open holes in the layout path

Keep host copy, collection, and prompt text from falling back to /workspace when a lookup fails or an adapter returns unclean roots.

* fix(sandbox): satisfy golangci-lint on the layout helpers

Wrap new copy and test lines to 120 columns, and add the exported comments revive requires.

* fix(sandbox): gofmt the read_file layout copy wrap

* feat(sandbox): add host as an OS-enforced backend type

Host is not a named/pinnable config: each command is a fresh local process, unlike Docker, Cube, and E2B.

* feat(session): persist Lite host workspace directory

Empty means an auto-allocated folder. Standard edition leaves the column unused so the shared session model still loads. SQLite also gains the tenant skill tables Lite shell_exec already expects.

* feat(localsandbox): adapt the host OS sandbox to session interfaces

File tools stay outside Seatbelt and are bounded by PathGuard. A revoked project directory fails closed instead of allocating a replacement workspace.

* feat(sandbox): run unpinned Lite sessions on the host backend

Lite opts into host explicitly; web stays disabled when no named remote config exists. Host workspaces are not destroyed or auto-committed.

* feat(desktop): pick and persist host project directories

The native folder dialog is the authorization path; typed paths are rejected. settings.sandbox.host tells the client when this edition can use it.

* fix(sandbox): fail closed on Lite host layout lookup misses

Unpinned host sessions were described as /workspace when pin read or
sandbox resolve failed. Prompts and attachment staging now share the
same fallback so Lite never pretends the remote tree exists.

* fix(frontend): keep settings.sandbox.host in sync with the backend keys

TestDeploymentCapabilityKeysMatchFrontend requires the lists to match. Host stays fail-closed like docker so a missed probe does not show Lite-only UI.

* fix(lint): satisfy golangci-lint on the host sandbox wiring

Check c.Error, wrap lines over 120 columns, put context first on versionsWorkspace, and document exported adapter methods.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
2026-09-22 19:35:03 +08:00
Ankit Jha 5f1b7a2b3b style: trim the comments added in this branch to the repo's cap 2026-09-22 18:04:32 +08:00
Ankit Jha 0b4cd706c2 fix(knowledge): stop synthesizing never-run stages as failed
Closes #3452.

buildSpanTree fills every canonical stage that has no span row using a
status inferred from knowledge.parse_status, so a failed parse painted
all of them "failed". That status carries information it cannot know:
it cannot tell "this stage ran and failed" from "this stage never ran".

The frontend picks the focused stage with the first failed one in
canonical order, so whenever the real failure left an earlier stage
rowless, the UI named the earlier stage. On main today a text passage is
chunked without ever opening a docreader stage, so an embedding failure
reports 文档解析 / docreader as the broken step for content that has no
document to parse, and four of five segments render as failures.

Once a stage has actually recorded a failure its position is known, so
missing stages are now resolved against it:

- downstream of the failure, per StageDependencies or canonical order →
  cancelled, which is what the status is documented to mean
- before it → skipped; the stage either did not apply or its row was
  never written, and neither of those is a failure

With no failed row anywhere the parse_status fallback stands unchanged,
which is the case it was written for: legacy knowledge parsed before
span tracking existed still renders done/failed rather than five pending
segments.

Also maps parse_status "cancelled" to a cancelled placeholder; it fell
through the switch to pending, so a cancelled document showed spinners
on the stages it never reached.

The DAG walk is redundant with canonical order for today's stage list,
where every dependency points at an earlier stage. It is there so the
answer stays right if the list is reordered or a stage gains an upstream
that is not its predecessor.

No frontend change: skipped and cancelled already render, with localized
labels in every locale.
2026-09-22 18:04:32 +08:00
vicfei 4b0dc3b687 fix: satisfy revive unused-parameter and gofumpt in #3493 review findings 2026-09-22 17:52:49 +08:00
vicfei 2dd36212d6 fix(knowledge): bind extracted images to their document and unscope shared-KB parent lookups
Org-shared KB viewers got 403 for extracted images on current main even
though the retrieving chunk text carried the resource:// handle and the
share grant was valid (#3342, second report after #3022/#3069).

Root cause is a mismatch between ingestion and authorization: document
parses store extracted images through SaveBytes, which registers the
resource but never claims it, while both file proxies (ResolveKBFile and
the message fallback via resourceAccessibleViaSharedKB) authorize through
IsReferencedByKnowledgeBase — which accepts explicit resource_bindings
rows only. The binding-only hardening landed without adding the missing
bindings, so every extracted image became invisible cross-workspace.

Ingestion: bindStoredImages claims each stored image for its knowledge
(ResourceRelationExtractedImage), mirroring bindContentResources — best
effort, same-workspace resources only. Wired into both the file-parse
path (knowledge_process) and manual knowledge (triggerManualProcessing).

Merge: resolveParentChunks fetched parents, grandparents, and image_info
children with the caller's tenant, so shared-KB hits from the sharing
workspace resolved none of them and lost parent context and image_info
(the second defect in the #3342 audit). Those lookups now use the
existing *Only unscoped variants, matching their shared-KB precedent.

Historical documents parsed before this change still carry no bindings;
a backfill is tracked as the follow-up slice.

Verified end-to-end on a fresh stack: publishing a manual document with
an embedded image now records kind=image plus an extracted_image binding
alongside the content-derived attachment binding.
2026-09-22 17:52:49 +08:00
Easton Gu d1691cbc26 fix(upload): clean multipart temporary files 2026-09-22 17:40:56 +08:00
kerthans afeee71e48 fix(wiki): release inflight slot on task cancellation 2026-09-22 17:30:29 +08:00
linus-liu-web 7feac9c570 fix(mcp): resolve MCP endpoint sessions by their storage ID
resolveAskSession stores the session owner as
MCPEndpointPrincipal(...).StorageID() -> "mcp_endpoint:<tenant>:<endpoint>",
but SessionOwnerIDFromContext has no PrincipalMCPEndpoint case and falls
back to the raw user id set by the MCP auth middleware ("mcp-<endpoint>"),
so every ask call failed with
"failed to create user message: session not found".

Handle PrincipalMCPEndpoint the same way as PrincipalEmbedSession.
2026-09-22 16:53:10 +08:00
hailongzhao e55e342387 fix(skill): count create-time sandbox env vars in the required check
A required skill key filled only on the sandbox config was already in the
container, but the gate still treated it as unset.
2026-09-22 16:45:58 +08:00
lyingbug 33c0333ec9
fix(chat): restore shared workspace images after stream completion (#3523)
* fix(chat): restore shared workspace images after stream completion

* test(chat): wrap shared image completion cases for lint

* fix(chat): wait for persisted completion and preserve streamed replies
2026-09-22 00:16:58 +08:00
wizardchen 58260d192c fix(upload): accept XMind files before parser engine discovery 2026-09-21 23:18:11 +08:00
wizardchen da049f04e0 fix(browserskill): release connection capacity when members disconnect
BROWSERSKILL_MAX_CONNECTIONS counted every member a node had served in the
last 90 days: disconnected devices stayed in the device map until their
authorization expired, and devices created by a refused lease claim or a
pre-pairing select had a zero expiry and were never evicted. Once 32 distinct
members had connected, the next one was refused even with one member online.

Capacity now counts live and handshaking members only. Idle devices of other
members are evicted on the next acquisition; interrupted tasks are durable and
reload paused on reconnection. A handshake reserves its device so it cannot be
evicted mid-connect, and a full node reports the capacity error explicitly.

The daemon dial and WebSocket upgrade no longer run under the device lock,
which every other member's device acquisition waits on while holding the
manager lock. The upgrade gains a handshake timeout, and a connection whose
device was revoked, re-paired or evicted while dialing is discarded.
2026-09-21 23:03:11 +08:00
lyingbug 4640a1ea0c
test(rerank): pin every vendor's request, and correct three facts the table surfaced (#3513) 2026-09-21 20:01:09 +08:00
Lvcy_cd cc82e75cb5 fix(parser): keep PDFs on builtin instead of defaulting to anydoc
anydoc has no document model for PDF: Convert force-disables asset
extraction, and AnydocReader falls back to builtin only when the whole
file yields no text. A born-digital PDF with figures and tables that
still yields a few characters is therefore reported as parsed, with its
images, tables and layout silently dropped.

35275d56 widened the type-level preference from ppt/pptx to every type
anydoc claims, and pdf is on that list. The predicate asks whether
anydoc can convert a type, not whether it is the better parser for it.
60207997, one commit earlier, recorded the original intent: "Types
without a fallback (pdf, docx, ...) never consult this".

The builtin PDF parser classifies pages individually and routes scanned
ones through OCR/VLM, so it stays the default for PDF. Every other
anydoc format is unchanged, and an explicit pdf -> anydoc engine rule
still wins.
2026-09-21 16:28:02 +08:00
Leo Dawn 56d075e213
fix(sandbox): resolve a session's pinned sandbox in the workspace that owns it (#3471)
* fix(sandbox): resolve a session's pinned sandbox in the workspace that owns it

tenant_sandbox_configs is keyed by (tenant_id, id), so a config id is not an
address on its own. sessions.sandbox_config_id stored only the id and every
reader took the workspace from the ambient request context.

That holds inside a chat turn, which already runs under the agent's workspace
(WithExecutionTenant in the QA handler). It breaks for a shared agent
everywhere else, because a shared agent runs on ITS OWNER's sandbox config
while the session belongs to the borrower, and these paths run from plain
requests whose only tenant is the session owner's:

  * DeleteSession resolved nothing and returned after a warning. Cube and E2B
    create session sandboxes with onTimeout=pause, so the MicroVM was not
    reclaimed later either: it stayed paused, billing, with nobody holding its
    id. ReapOrphanSandboxes has no production caller, so nothing recovered it.
  * The terminal and desktop panels failed on both branches - attaching to the
    pinned sandbox and provisioning a new one - with ErrSandboxConfigNotFound,
    which terminalErrorFrame maps to a bare INTERNAL close reason. The panel is
    offered unconditionally and already forwards agent_source_tenant_id, so
    this was a dead end the user could reach in one click.
  * Fork read the pin through PinnedSessionSandbox, got no manager, and
    silently degraded to SNAPSHOT_UNSUPPORTED, losing the sandbox state.
  * The fork snapshot lease recorded the session's workspace next to the
    lending workspace's config id, so the reaper could not resolve it either.

Record the owning workspace beside the pin and make the pair travel together
as SandboxPin. Readers now take the workspace from the pin and fall back to the
session's own, which is what every non-borrowed pin (and every row written
before this column) resolves to - so behaviour for own agents is unchanged.

The provision path has no pin yet, so the WebSocket handler passes the agent's
workspace along with its config: terminalProvisionPin already resolved the
shared agent, it just discarded the tenant resolveAgent handed back.

Backfill in the migration is limited to sessions whose pinned config does not
exist in their own workspace - the only ones that can have borrowed it - using
the lending workspace recorded on their newest shared-agent message. Lite has
no shared spaces, so the SQLite mirror adds the column without a backfill.

* fix(sandbox): backfill the pin owner from the config row and self-heal legacy pins

Three gaps in the session sandbox pin owner:

Backfilling from messages.agent_tenant_id names the wrong workspace. The
pin is sticky to the session's FIRST sandbox, while the newest message
carrying agent_tenant_id may belong to a later shared agent from another
workspace. Writing that tenant leaves teardown resolving a config the
workspace does not own - the very failure the column exists to fix.
tenant_sandbox_configs.id is a global UUID primary key, so joining the
config row itself names the workspace that actually owns the pin.

The column was INTEGER while every other tenant id is BIGINT/uint64.
Postgres INTEGER is 32-bit signed, so a large tenant id would not fit.

Rows the SQL backfill cannot reach keep sandbox_config_tenant_id = 0
forever: the reader falls back to the caller's workspace but never
records what it learned. The next chat turn already runs under the
workspace that can resolve the config, so persisting it there is what
lets a later DELETE, panel open or fork find the sandbox. recordOwner
only writes when the lookup actually succeeded and the stored owner is
still unset - stamping the caller's tenant onto a failed lookup would
brand a sticky pin left by a previous shared agent with the wrong
workspace.

Also renames the lentSandboxConfg typo and corrects the fork degrade
reason in a comment: a lent config that resolves to nil fails
BoundSandboxID, so the fork degrades to SANDBOX_GONE, not
SNAPSHOT_UNSUPPORTED.

* fix(sandbox): require a named backend before recording a pin's owner

The self-heal wants to persist an owner "that actually resolved this config",
but a nil error does not prove that. resolveTenantSandboxForConfig returns a
disabled manager with no error twice before it ever reads the config row: for
the workspace kill switch, and for an empty or deployment-default config id.

That reopens the leak this column exists to close. The SQL backfill correctly
leaves sandbox_config_tenant_id at 0 whenever the pinned config belongs to the
session's own workspace, which is most existing rows. One turn of a shared
agent lent by a workspace whose admin has script execution turned off then
short-circuits on the kill switch, returns nil, and stamps that lending
workspace onto a config it does not own. The next DELETE resolves the pin there,
finds nothing, and strands the paused MicroVM.

The pin branch runs before agentConfigID is read, so the shared agent does not
even need a sandbox config of its own to trigger it.

A named backend (Cube/E2B/Docker) is the proof, and it is the same condition
that is already allowed to CREATE a pin further down: what may write a pin may
update whose it is. A config that exists but resolves to type "disabled" is
skipped as well - ownership is genuine there, but such a config never creates a
sandbox, so no teardown can go missing.

The deployment-default pin was inert either way (both workspaces short-circuit
to a manager with no DestroySession), but it is covered too so a later
relaxation of the proof cannot quietly reopen the kill-switch case.

Both tests fail without the guard.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
2026-09-21 15:38:39 +08:00
lyingbug 41dde3b05f
refactor(models): delete the internal/models/provider façade (#3506) 2026-09-21 14:41:48 +08:00
wizardchen 2d37f8fd7d fix(container): wire shared agent lookup for MCP handler 2026-09-21 14:38:54 +08:00
YG52052051 cab16dcacb
fix(storage): strip storage-backend scope in local provider path normalization (#2979)
KB deletion runs on the process-wide default file service, which is not
wrapped by BackendScopedFileService. Storage-scoped physical paths
("storage://<backend-id>/local://<tenant>/...") therefore reach the raw
local provider, fall through to the relative-path branch, and get joined
under baseDir — producing a nonexistent path so every os.Remove fails with
ENOENT and the physical files silently leak on every KB/document delete.

normalizePathForBase now strips the scope wrapper up front (both the
canonical "storage://" form and the "storage:/" form produced when a path
has been through filepath.Clean), and tolerates the cleaned "local:/"
provider scheme. Unscoped paths are unaffected.

Co-authored-by: YG52052051 <YG52052051@users.noreply.github.com>
2026-09-21 14:21:25 +08:00
lyingbug 4a9aa06274
refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
nullptr-error 7d688cb8d3
feat(localsandbox): add host OS sandbox core and macOS Seatbelt backend (#3472)
* feat(localsandbox): add host OS sandbox core and macOS Seatbelt backend

Introduce a standalone localsandbox package so Lite can run agent
commands on the user's machine under Seatbelt, without linking this
into cmd/server or changing remote sandbox behavior.

* fix(localsandbox): stop post-wait Kill, deny credential files, and refuse swapped roots

Seatbelt was SIGKILL-ing the process group after every successful Wait,
credential write denies omitted literal matches for files, PathGuard
followed a replaced workspace root, and the login-shell test never ran -lc.

* fix(localsandbox): tighten app-data deny, env inherit, and post-wait Kill

Keep ask/full from silently widening to auto, stop inheriting host secrets,
and make session workspaces survive midnight so the Seatbelt kernel stays
fail-closed before adapter wiring.

* fix(localsandbox): pass gofmt, errcheck, and revive on the new package

CI now lints from the merge base, so the host sandbox needs formatted tests, checked Close() results, and exported comments.

* fix(localsandbox): refuse broad workspaces and stop leaking host secrets

Keep the host sandbox fail-closed before adapter wiring: home/Users/Volumes
cannot be the workspace, login shells no longer re-import rc secrets, and
Service applies the env allowlist on every Run. Also close the Lstat TOCTOU
and post-wait Kill window.

* fix(localsandbox): check Close() in seatbelt spawn tests

* fix(localsandbox): deny host temp dirs and refuse leftover wide workspaces

file-read* still left /tmp and /var/folders readable, and Build/Relax
accepted /private/var, /System/Volumes/Data, and ~/Library/** as a
workspace, which re-opened the private deny. Also re-open only ~/.local/bin.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-21 13:24:05 +08:00
tayadwx 911a1490ea
fix(wiki): enrich cited chunk content with image captions in Reduce phase 实体/摘要页面引用子块时同样使用富文本内容 (#3292)
* fix(wiki): enrich cited chunk content with image captions in Reduce phase

The citation-pass comment claims image/ocr chunks are "already merged
into the text content via reconstructEnrichedContent", but that function
has exactly one call site — the per-document summary page. The Reduce /
page-edit phase resolves cited chunks straight from the repository and
feeds the LLM bare Markdown image links with no caption or OCR context.

This mismatch is observable: on concept/entity pages generated from an
image-heavy knowledge base, the page-editing model receives
![filename](wkimg:NNN) with no way to judge the image, and empirically
either copies the image links verbatim or drops them wholesale —
including real content images. A controlled experiment where the same
image reached two pages byte-identically produced opposite keep/drop
decisions, and <image_caption> appeared in 0 of 9 outgoing page-edit
requests.

resolveCitedChunks now collects image_info for the cited text chunks
(CollectImageInfoByChunkIDs, one query per knowledge batch, outside the
per-page loop) and inlines it with EnrichContentWithImageInfo — the same
enrichment the summary path uses. Image info collection skips disabled
child chunks, so downstream decorative-image cleanup composes with this
path automatically.

* test(wiki): cover cited-chunk image enrichment in resolveCitedChunks

Add unit tests asserting that cited text chunks reach the Reduce phase
enriched with caption/OCR info from their enabled image child chunks,
that disabled children are skipped (decorative-image cleanup composes),
and that unknown chunk IDs still fall through gracefully.

---------

Co-authored-by: tayadwx <tayadwx@users.noreply.github.com>
2026-09-21 13:13:48 +08:00
JYao-Chen 07a5af673e
fix(graph): increase entity extraction output budget (#2859) 2026-09-21 13:08:49 +08:00
JYao-Chen e3d52e8353
fix(wiki): skip ambiguous single-Han auto-links (#2860) 2026-09-21 13:08:04 +08:00
HJX|翻斗花园二蛋 2f2bcc7aa7
fix(suggestion): avoid echoing the current query (#2582)
* fix(suggestion): avoid echoing the current query

* test(suggestion): use generic follow-up examples
2026-09-21 13:06:01 +08:00
Leo Dawn a3c7be10ca
fix(agent): scope the @Skill and @MCP pickers to the agent that will run (#3475)
* fix(agent): scope the @Skill and @MCP pickers to the agent that will run

Both pickers listed the caller's own workspace, but a shared agent invokes its
OWNER's skills and MCP services. The two never line up, so for every shared
agent:

  * @Skill was always empty, in every skills mode. GET /skills resolves the
    sandbox config with (tenant, id), and the agent's config id does not exist
    in the borrower's workspace - effectiveTenantSkills short-circuits before
    it reads a single skill row. The response still reported
    skills_available: true, so the category appeared with nothing under it.
  * @MCP in "selected" mode was empty for the same reason: the preset holds the
    owner's service ids, which match nothing in the borrower's catalog.
  * @MCP in "all" mode was worse than empty. isMCPAllowedByAgent returns true
    for every enabled service, so the picker offered the BORROWER's own
    services; resolvePerRequestMCPScope then intersected the mention against
    the agent's preset, found nothing, and dropped it with only a warning.

Resolve the agent when the request carries agent_id + agent_source_tenant_id
and read its resources where they live. Only an explicit source workspace takes
the share path, so an own-agent request is byte-for-byte unchanged; built-in
agent ids exist in every workspace, and falling back to a share lookup without
one would let an org member take over the caller's default agent, exactly as
session.Handler.resolveAgent already documents.

Both endpoints serve only what the agent can actually invoke:

  * skills: the sandbox config comes from the agent, never from the query, so a
    borrower cannot name another config in the owner's workspace and enumerate
    it; "selected" narrows to the configured names and a disabled picker never
    reaches the owner's inventory at all.
  * MCP: the agent's explicit preset and nothing else, because that is exactly
    what resolvePerRequestMCPScope accepts for a shared agent. "all" therefore
    offers nothing rather than offering services every mention would drop.
    Whether "all" should become mentionable is a separate decision about that
    guard, not about the picker.

The cross-workspace MCP rows use a narrowed response built field by field.
NewMCPServiceResponse reveals endpoint URL, headers, env vars and stdio command
to an Admin, and a borrower can be an Admin of their own workspace - that shape
must be narrow by construction, not by role.

On the client the two lists now follow the selected agent: their cache keys
carry it, and the chat input refetches on a switch behind a request token, so a
slow response for the agent the user just left cannot overwrite the picker with
another workspace's services.

* fix(agent): check the c.Error return on the shared-agent picker paths

golangci-lint judges pull requests on the lines they introduce, so the
repository's older bare c.Error calls are grandfathered while these three are
not. Match the _ = c.Error form already used in mcp_service.go.
2026-09-21 13:02:34 +08:00
Fuzzycorrect 13d0b9f94a
fix(queue): order ClaimBatch by fail_count so retries cannot starve new work (#3444)
* fix(queue): order ClaimBatch by fail_count so retries cannot starve new work

ClaimBatch picks work with ORDER BY id ASC (Postgres) / MIN(id) ASC (GORM),
i.e. oldest-first by primary key.

Retry does not move a row: requeueFailedOps calls IncrFailCount and then
ReleaseByIDs, which deliberately keeps the row in place so the fail_count
budget keeps counting down.

Together those two produce a starvation mode. A document that fails
repeatedly keeps its original, lowest id, so it is always selected first,
while every never-attempted document - enqueued later, hence a higher id -
waits behind it until the stuck op finally exhausts wikiMaxFailRetries and
dead-letters. Observed on a real 87-document KB: 42 ops pending, the four
head ops were all re-runs of documents that already had a root page, and 38
never-attempted documents were never claimed. With 8 concurrent workers and
a drained asynq queue (pending=0, retry=0, active=4), the bottleneck was the
claim ordering, not throughput.

Order by least-failed first, oldest-first within the same fail_count. When
every row has fail_count = 0 this degenerates exactly to the previous
id-ordering, so the existing FIFO tests still pass.

Test: TestTaskPendingOps_ClaimBatch_PrefersLeastFailed replays the real retry
path (claim -> IncrFailCount -> ReleaseByIDs) and asserts that the
never-attempted document is claimed first and the retried one is still
eventually picked up rather than stranded.

* fix(queue): apply the same least-failed PeekBatch order in Lite mode

Lite wiki ingest still peeks rather than claims. Ordering that path by
id alone would keep the same head-of-line starvation this PR already
fixed for ClaimBatch. Align comments with the new selection rule.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-21 13:00:29 +08:00
lyingbug 9030f556f2
refactor(embedding): move the wire formats into protocol packages, and correct the vendor facts (#3500) 2026-09-21 11:55:38 +08:00
lyingbug d6c7665cb5
refactor(rerank): move the wire formats into protocol packages, and correct the vendor facts (#3488)
把 8 个手写 rerank client 收敛成 3 个协议包(internal/models/api/{cohererank,dashscoperank,nimrerank}),每个带一份从厂商文档逐字粘来的 golden fixture。LKEAP 和火山保留官方 SDK,但只实现 api.Reranker。rerank.NewReranker 改成 catalog.Resolve + 按协议分发,函数体里没有厂商名。生产代码净 -147 行。

三件原来每个 client 各管各、现在由共用层统一的事:分批(按厂商声明的上限,智谱 128 / NIM 512 等此前无人执行)、分数标度(NIM 与 GPUStack 返回 logit,其余为 0~1 概率,统一换算后再与 RerankThreshold 比较)、NIM 的 truncate(默认 NONE 会让超长 passage 整请求失败,此前从未发送)。

对照官方文档订正四处厂商事实:GPUStack 分数标度、千帆三项上限、阿里 500 条上限、qwen3-rerank 是第二种协议(隐藏并在构造期拒绝)。Novita / OpenRouter / LiteLLM 三家官方有 rerank 但未开类型,本次开放,零协议代码。

运行时行为变更:NVIDIA 起发 truncate: END;GPUStack 分数按 logit 换算(可按行覆盖);truncate_prompt_tokens 收窄为厂商声明制;qwen3-rerank 存量行从静默打错变为构造期报错。详见 PR 描述。

关联 #3486。
2026-09-21 10:36:26 +08:00
lyingbug 9aa63c866f
fix(models): correct four vendor request shapes that contradict the docs (#3489)
PR #3470 逐条核对了 27 家厂商,但有四处(五个 entry)的声明与厂商文档对不上,
其中一处在默认路径上就是错的。

1. DeepSeek:参考文档写明 "required and named tool choices are not supported
   in thinking mode; the API returns a 400 error",而 vendor.go 的事实注释写成
   「plain OpenAI set」并因此没有配 tool_choice_modes,默认继承全集。现在只发
   none / auto,和 Moonshot 同类限制的写法一致。compat 无法表达「仅思考模式下
   受限」,而思考是 DeepSeek 的默认状态,所以不做条件区分。

2. 智谱 glm-5.2:官方对 GLM-5.2 的折叠规则是 none|minimal 放弃思考、low|medium
   折成 high。models.json 却把 minimal 改写成 "low",发出去被平台折成 high,
   用户选最弱档实际跑最高强度。删掉该覆盖,回落到厂商级的 minimal。

3-4. NVIDIA NIM:z-ai/glm-5.3 与 moonshotai/kimi-k3 配了 thinking_levels,却
   沿用厂商默认的 chat-template-kwargs 编码——该分支只发 enable_thinking 布尔,
   档位被整个丢弃,而前端仍按 capabilities 渲染出一个不起作用的档位选择器。
   两个 entry 改为顶层 reasoning_effort。kimi-k3 的 NIM 卡片只写了
   "configurable low, high, or max reasoning effort" 而未展示参数,按其第一方
   API 的字段推定,已记入 Unverified。

5. 阿里云 qwen3.8:官方参数表写明「不支持 reasoning_effort 与 thinking_budget
   同时设置,同时设置会报错」,vendor.go 的注释也记了这条,但 applyThinking 在
   ThinkingBudgetTokens > 0 时无条件两个都写。新增 compat 键
   thinking_budget_excludes_effort,冲突时档位优先、预算让路。

3 与 4 在当前默认路径上已经影响到档位;1 和 5 目前够不到(全仓库只有
agent/finalize.go 发 tool_choice="none",且没有任何地方给 ThinkingBudgetTokens
赋值),属于修掉的埋雷。2 是默认路径上的实际错误。

新断言做过反向验证:数据回滚到修改前,7 条 vendor 断言与新增的 golden 用例全部
报错。
2026-09-21 00:05:11 +08:00
lyingbug c40a9dd194
refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470)
把集中式的 OpenAI 方言客户端重建为三层:协议层 internal/models/api/<protocol>(OpenAI Chat Completions / OpenAI Responses / Anthropic Messages / Google GenAI),目录层 internal/models/catalog,厂商层 internal/models/vendors/<id>(27 家,go:embed)。27 家厂商逐条对照官方文档核验,厂商事实以数据声明,协议层不出现厂商名。

既有行的运行时行为会变的四处:
1. extra_config.api 变为保留的协议选择器,非法取值被拒绝;
2. Azure 未填 api_version 的行改走 /openai/v1 GA 数据面;
3. api.openai.com 的一方流量改走 Responses 协议(中转/网关不受影响);
4. 7 家厂商的输出上限字段按文档纠正。

不需要数据库迁移。详见 PR 描述与 website-docs/03-features/06-models.md。
2026-09-20 21:28:23 +08:00
lyingbug 50bbf9ad5f
fix(embedding): two wire-level bugs in the remote embedders (#3484)
* fix(embedding): stop four embedders from returning (nil, nil) on a dead upstream

doRequestWithRetry keeps resp and err in the enclosing scope and returns
(nil, err) once the retries run out. Writing `req, err := ...` inside the
loop introduces a loop-local err, so `resp, err = client.Do(req)` updates
that copy and the enclosing err stays nil. Every retry failing therefore
returned (nil, nil): BatchEmbed's `if err != nil` did not fire and the next
line dereferenced resp.Body, a nil-pointer panic that took the process down
instead of surfacing "connection refused". Stopping the embedding upstream
and asking any RAG question reproduced it.

The bug was found, documented and fixed in openai.go, then re-fixed in
zhipu.go and again, differently, in azure_openai.go; jina.go, nvidia.go,
aliyun.go and volcengine.go kept the original spelling because the retry
loop exists in eight near-identical copies and a fix to one reaches none of
the others.

TestEmbeddersReportUnreachableUpstream now drives all eight against a
refused port with the retry budget zeroed, so the next copy of this loop is
covered the moment it appears.

* fix(embedding): read the documented index field of DashScope multimodal vectors

The multimodal embedding response carries each vector's position in `index`:

  {"output": {"embeddings": [{"index": 0, "embedding": [...], "type": "text"},
                             {"index": 1, "embedding": [...], "type": "image"}]}}

The decoder read `text_index`, which is the field name of DashScope's *text*
embedding API one page over. A missing field decodes as 0, so every vector in
a batch was written to slot 0 and the rest of the slice stayed empty: with the
default BATCH_EMBED_SIZE of 5, four of every five chunks were indexed with an
empty vector. Only models routed to this client are affected — an Aliyun
embedding model whose name contains "vision" or "multimodal"; the text-only
models go through the OpenAI-compatible path.

An index outside the batch, or a response that covers fewer inputs than were
sent, is now an error. Handing back an empty vector stores it and poisons
retrieval without anything failing.

The regression test replays the response example from the reference page, so
the decoder is pinned against the vendor's own document rather than against
our structs.

https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference
2026-09-20 21:12:33 +08:00
nullptr-error d4b9fd0c72
feat(session): in-place rewind with workspace checkpoint reset (#3469)
* refactor(session): extract shared workspace git reset helper

Fork bootstrap and in-place rewind both need the same /workspace reset
script. Move it out of fork_bootstrapper so rewind can reuse it without
copying ExecShellCommand plumbing.

* feat(message): add DeleteMessagesFrom for rewind truncation

Rewind needs a composite created_at+id cursor so same-millisecond
messages after the fork point are not left behind. Delete inside a
transaction and return the rows so callers can log the count.

* feat(session): add in-place rewind service

Truncate the current session at a user or assistant message and git-reset
the live sandbox when a reachable checkpoint exists. Reset failure is
fail-closed so messages are never deleted ahead of the workspace.

* feat(session): expose POST /sessions/:id/rewind

Wire SessionRewindService through the session handler and container so
the client can rewind in place. Busy sources map to 409; git reset
failures stay 500.

* feat(chat): add rewind API client and i18n copy

Expose rewindSession with the same 180s timeout as fork, and add
chat.rewind strings in all five locales so the confirm dialog is not
hardcoded Chinese.

* feat(chat): add rewind button with confirm dialog

Offer in-place rewind next to fork on user and assistant messages.
Confirm before truncating the thread, then reset locally and prefill
the original question when the cut point is a user message.

* fix(chat): confirm rewind with a compact popover

Replace the full-page confirm dialog with a small popconfirm above the
rewind button, and shorten the swagger description to satisfy line length.

* fix(chat): skip markdown images with an empty destination

Empty ![]() was rendered as <img src="">, which shows as a broken image in the reply.

* fix(chat): clip sandbox panel slide-in and rename open tooltip

Keep the enter animation inside the viewport so it cannot flash a document
scrollbar and jump the page. Rename the header toggle from sandbox terminal
to expand sidebar.

* fix(session): 回滚时完成截断、互斥发送并重拉历史

客户端中断不再让 git reset 成功后留下未删的消息;reset/删消息前复查 busy,并锁住当前会话输入。成功后重拉历史,避免切会话预填串台和 steer 分段截错。

* fix(session): 工作区 git 外置,并让未开箱 fork 的回滚对齐 SHA

checkpoint、回滚和 fork 共用 --git-dir=/var/lib/weknora/workspace.git,避免 rm -rf /workspace 丢掉历史。
未开箱分叉会话回滚时改写或清除 ForkBootstrap,避免第一次开箱仍 reset 到旧切点。

* fix(session): 回滚加会话锁,并堵住发送中回滚与脏 SHA

同一会话并发回滚互斥(进程内 + Redis),冲突返回 409。发送/流式中禁用回滚按钮;历史重拉成功后再改界面。未开箱 fork 的 checkpoint SHA 必须是 40 位 hex。工作区 git 只认外置 GIT_DIR,不再 mv/rm 根上的 .git。

* fix(session): 未开箱 fork 回滚成功时不要提示没有沙箱

ForkBootstrap 已对齐切点后返回 workspace_reset,前端走「已回滚」;清掉引导时改报 NO_CHECKPOINT。纯聊天仍用 NO_SANDBOX。

* fix(chat): stub composerLocked in steer interaction tests

handleSteerMsg now bails when rewind locks the composer. The VM harness
did not provide that ref, so CI threw ReferenceError on every steer case.

* fix(session): allow git-layout mkdir in pinned checkpoint test

SkipWorkspacePrep still omits the input/output bootstrap exec. The
checkpoint script now mkdirs WORK_TREE and GIT_DIR, so the old
"command must not contain mkdir" assertion was stale.

* fix(session): block send while rewind holds the session lock

BeginTurn now fails if the rewind key is held, so AgentQA cannot start
during git reset. Knowledge-chat and executeQA check the same lock
before persisting messages. composerLocked is still page-local.

* fix(session): close send/rewind races and fail closed without a checkpoint

Hold the turn lease before persist so rewind cannot truncate an in-flight
send. TryLockRewind fails while a turn is live; emptying the conversation
resets the workspace through the same git path as a SHA rewind.

* fix(session): wrap rewind reset call to satisfy line length lint

* fix(session): fail closed on a replaced sandbox and close leftover rewind holes

Do not truncate history when the live workspace cannot follow the checkpoint, retry deletes after a successful git reset, and keep send and rewind exclusive even without a sandbox manager.

* fix(session): address rewind review findings

Correctness:

- Adopt a pre-move /workspace/.git into SessionGitDir on first use. Sandboxes
  provisioned before the git dir moved out of the work tree still hold their
  checkpoints there, so every SHA recorded on those turns stopped resolving:
  rewind 500'd instead of truncating and fork's rollback failed on a bad
  object. Adoption is guarded on the new dir being absent and on the work-tree
  repo carrying the checkpointer's committer identity, so an agent's own
  `git init` in /workspace is left alone.
- Only fail a send closed when BeginSessionTurn reports a rewind lock. Every
  other lease error degrades to running without a lease, as before rewind
  existed; a Redis blip on the lease script was rejecting the user's message.
- Release an abandoned fork snapshot after the truncate lands, not before.
  Clearing the bootstrap column is a DB write a retry can redo; deleting the
  snapshot is not, and a delete that exhausted its retries left the fork with
  no workspace to boot from under a full transcript.
- Re-acquire the rewind lock when a renew call errors, not just when it
  reports the lease lost. Sitting the tick out let a short Redis outage expire
  the 2m key mid-reset and hand the session to a second rewind.
- Keep the history cursor when a post-rewind reload fails. Clearing it sent
  the next scroll-up back to the newest page, which the kept prefix already
  holds, instead of to older messages.

Efficiency:

- Add ListAssistantCheckpointsUpTo and use it to answer "which checkpoint does
  kept history still reach". Rewind was reading every message body and artifact
  row in the session to find one SHA.
- Take the send-side turn lease once per request. The handler already holds it
  before persisting; QARequest.TurnLeaseHeld lets the QA services skip their
  own, so IM/MCP still take one and HTTP send stops doubling every rewind check
  and lease round trip.
- Batch the suggestion delete behind DeleteByMessageIDs instead of one query
  per deleted message while the rewind lock is held.
- Resolve the rewind affordance for the whole transcript in one pass rather
  than two list scans per rendered row on every streamed token.

Cleanup:

- Drop the SANDBOX_REPLACED skip reason, which the service never produces (it
  fail-closes with a 409), along with its copy in all five locales.
- Drop GetMessagesBySession from rewindMessageStore; rewind never calls it.
- Drop the unused chat.rewind.confirmTitle copy from all five locales.
- Restore the chatHeader.toggleSandboxPanel wording this branch had changed
  unrelated to rewind.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-20 21:01:00 +08:00
lyingbug 093c16cbb3
fix(agent): end the turn when the completion cap truncates an answer (#3449)
* fix(agent): end the turn when the completion cap truncates an answer

A round that came back with finish_reason=length and no tool calls used to
fall through analyzeResponse as non-terminal. With no tool calls to run,
appendToolResults pushed the half-written answer back into the history as a
plain assistant message with nothing telling the model to continue, so the
next round rewrote the answer from the top, hit the same cap, and repeated
until the round budget ran out or the user cancelled.

Three guards, each independently load-bearing:

- analyzeResponse gains a truncation case: deliver the partial text and end
  the turn. A cap hit that produced no text at all has nothing to hand over
  and defers to the existing bounded empty-content retry. No continuation
  nudge is sent — it only grows the prompt with the discarded fragment, and
  assistant prefill, the reliable form, is unavailable on most
  OpenAI-compatible endpoints.
- A consecutive-length breaker stops the loop when truncation keeps landing
  inside tool-call arguments, the one shape that still legitimately loops.
- The stuck-loop detector now counts an identical lack of content. A model
  burning its whole budget on reasoning returns empty every round, which used
  to land in the else branch and reset the counter, so the guard could never
  fire on the case that needed it most.

The answer event and its Done marker carry a truncated flag through the SSE
metadata, and the chat UI shows the same style of notice it already uses for
fallback answers.

Fixes #3446

* fix(agent): treat unterminated reasoning as no answer and keep the truncation marker

Review follow-up on three ways the truncation path could still mislead.

An inline <think> block that the cap cut off never gets its closing tag, so
the pair regex left it whole and the round looked like it had produced an
answer. The truncation case then skipped the empty-content retry it exists
for and emitted the chain of thought as the final answer. StripThinkBlocks
now drops a dangling <think> through end of input, which is what the
streaming splitter already does with the same text, so the two agree on what
counts as an answer.

The truncation marker only rode on the SSE metadata, and a finished
conversation is rebuilt from agent_steps rather than replayed from stream
events. A reloaded, shared or re-opened turn therefore lost the notice and
its half-written answer looked complete. AgentStep carries the flag now — it
is a jsonb column, so no migration — and the frontend marks the rebuilt
answer event and message from it.

The consecutive-length guard ran before the round was recorded, so a final
round of "let me look that up" plus a truncated tool call handed that
preamble over as the final answer, with its tool calls never refused and the
preamble never retracted from the answer area. The guard now records the
round and fails its tool calls the way every other truncated round does, and
a response carrying tool calls never donates its text as the answer.
2026-09-20 13:17:50 +08:00
lyingbug eaca7364cd
fix(knowledge): stop two parse paths from stranding a row in processing (#3445)
* fix(knowledge): stop two parse paths from stranding a row in processing

Both of these left parse_status on "processing" with nothing left to move
it, so the document sat there until the housekeeping sweep failed it an
hour later with "task stuck in processing, recovered by housekeeping" —
reaching the user as a slow, unexplained parse failure.

processChunks returned silently when the embedding model could not be
resolved (deleted model, expired credentials, unreachable provider). It
reports nothing to its callers, so the attempt ended with the row still
marked processing. Persist the failure with the underlying cause and
record the embedding stage as failed, matching how the sibling
GetASRModel branch already handles a model it cannot resolve.

enqueueImageMultimodalTasks seeded the multimodal fan-in counter with
len(images) before the fan-out, then only warned when a payload failed
to marshal or asynq refused the write. Those slots have no task behind
them to drain them, so the counter never reached zero and post-process
was never triggered. Release the shortfall after the fan-out, and drive
post-process directly when the release itself brings the counter to zero
or when nothing reached the queue at all — the same reconciliation
KnowledgePostProcess already performs for its enrichment subtasks.

Seeding the counter before enqueueing is kept deliberately: an image
task that finishes early would otherwise decrement a key the fan-out is
about to overwrite. The release is what settles the resulting gap.

Also collapse the Redis key into multimodalPendingKey so the fan-out and
fan-in cannot drift, and share one enqueueKnowledgePostProcessTask
between the no-multimodal path and the slot release.

* fix(knowledge): keep the embedding-model guard from clobbering abort state

Review follow-up on the two stranded-row fixes.

The embedding-model guard wrote parse_status=failed unconditionally, which
made two cases worse than the silent return it replaced. Resolving a model
reads the database, so the entry guard's verdict is already stale by the
time the error comes back:

  - A user cancel signals asynq CancelProcessing, which cancels the
    handler context. GetEmbeddingModel then fails with context.Canceled
    and the guard overwrote "cancelled" with "failed: ... context
    canceled" — mislabelling an interrupted run as a broken model. Worker
    preemption and graceful shutdown look the same.
  - ReplaceKnowledgeFile keeps the knowledge ID and swaps file_path.
    UpdateKnowledge is a full-row Save and file_path is not among
    omitFieldsOnUpdate, so the stale row rolled the path back over the
    replacement and marked the replacement's live attempt failed.

So the guard now returns without writing when the context is gone or the
error is a context error, re-checks isKnowledgeAborted before writing, and
persists through updateKnowledgeUnlessSourceReplaced like the completion
path already does.

The multimodal slot release gave up on the first Redis error. Since the
release is the only thing that can drain a slot with no task behind it,
that put the row straight back into the stranded state this fix exists to
remove. It now retries a bounded number of times and logs at error level
when it still cannot settle the counter. DECRBY is not idempotent, so a
retry after a lost reply can over-decrement; that finalizes early rather
than stranding, which is the direction to fail in, and post-process
already tolerates repeat deliveries. The nothing-enqueued branch's Del
also moves onto the detached context, so a cancelled parent no longer
leaves the seeded key to expire on its own 24h TTL.

Two reporting corrections while here: the span now carries the same
message as the document list instead of a fixed generic string, because
the timeline renders error_message and withholds error_detail; and the
comment no longer claims expired credentials or an unreachable provider
surface here — resolving a model only reads the row and constructs a
client, so those fail later on the first BatchIndex round-trip.

Adds 6 tests for the paths above. Two of them initially passed against
the unfixed code because the entry guard caught them before the model
call, so the repository fake now serves the cancelled / replaced row only
after the guard's own reads, which is the window that actually exists.
2026-09-20 13:02:05 +08:00
lyingbug 9e0c28fbc9
feat(artifacts): let users delete generated files (#3447)
* feat(artifacts): let users delete generated files

技能生成的文件此前只能列出、预览和下载,没有任何删除入口,后端
handler/service/repository 三层都没有实现。DeleteSession 里那句注释
("a dedicated GC job or explicit hard-delete API is the right place to
reclaim storage") 留的就是这个位置。

Tombstone, not a row delete. message_artifacts 新增 deleted_at,删除只
打标记不删行。position 是下载接口寻址用的下标(msg.Artifacts[index]),
物理删掉中间一行会让它后面的文件整体前移,已有链接下到错的文件;而
ArtifactCollector 的去重集就是从这张表读的,行没了之后沙箱里 mtime 未变
的同名文件下一轮就会被重新采集回来,删了又出现。墓碑同时挡住这两条。
loadArtifactsByMessage 因此刻意保留墓碑,展示侧用 MessageArtifacts.Live
过滤;NewMessageArtifactRecords 带上 DeletedAt,任何一次消息重写都能把
墓碑原样写回去。

Bytes. 字节是真回收的,走 catalog.Release 引用计数:一份文件可能同时被
知识库条目和后续回答持有,只有所有持有者都放手才 DeleteFile。删多个版本
时同一个对象可能来自多条消息,每条消息的绑定都要释放,字节只删一次。
回收发生在打标记之后,且只针对本次调用真正认领到的行——并发的第二次删除
认领不到任何行,也就不会去动别人正在回收的对象。释放失败一律保留字节:
留个孤儿 blob 以后还能靠墓碑上的 url 回收,删错了则是别人的文档里的图片
凭空消失。

Scope. 会话内面板每个版本单独一行,删的就是点的那一版;产物库一行代表
一个文件(带版本数),默认连同该文件在这个会话里的所有版本一起删,否则
删完这一行还在、只是显示上一版。权限比下载严一档:下载认共享智能体的
只读访问,删除只认会话归属人。

Inline references. 正文里的 resource:// 引用渲染成置灰加删除线的卡片,
不可点击,和「模型引用了不存在的文件」的 --pending 态区分开。图片类产物
也降级成卡片:走 renderImage 只会拉取失败,留下一张永远加载不出来的图。
墓碑必须留在传给渲染器的产物列表里——句柄形式与知识库图片同形,过滤掉
之后句柄对不上本条回答,会被当成外部受保护图片去拉一个已经不存在的
blob,结果是裂图。组件里因此拆出 liveArtifacts 只给计数和文件列表用。

Also: 产物库是 offset 分页的无限列表,删一行会让后续页整体前移漏掉一条,
改成按已显示行数取下一页并按 key 去重;access.ResolveMessageArtifact 与
MessageReferencesFile 不再让墓碑授权访问;SessionArtifacts 过滤墓碑,
避免新回答把引用绑到已删文件上;版本澄清文案不再把已删版本算进版本数。

docs/swagger.* 未重新生成:仓库里那份已经落后若干功能,重生成会带进约
1000 行无关漂移,另开 PR 同步。

* fix(artifacts): close the delete holes found in review

Regeneration. loadKnownSet 把墓碑的 ContentHash 也放进了去重集,于是
knownSameContent 会把「删除后技能重新生成、字节恰好相同」当成 fork
checkout 那种 restore:不上传、不挂回本轮回答,文件再也回不来。墓碑现在
只登记 (path, mtime),不进 hash 也不进 byPath —— 前者才是「沙箱文件没动
过就别复活」需要的键,后者是「同内容恢复」的短路,两件事不该共用。
RecordRestoredArtifactMtime 也不再改墓碑:它的 mtime 就是那把锁。

Message file proxy. /sessions/:id/messages/:message_id/files 走
MessageReferencesFile,而正文检查在最前面且无条件为真。删除不会重写回答,
resource:// 句柄还留在正文里,于是这条代理会继续把已删文件交出去——包括
交给共享智能体的只读访客,也包括「字节因为知识库还持有而没被回收」的情形。
现在先看 artifacts:命中本条消息的墓碑直接 false,命中 live 直接 true,
其余(知识库图、附件)才回落到正文。同一文件挂在多个位置时只要还有一个
live 就仍然可取。

Cross-session bytes. CreateForked 复制产物行时连同存储 URL 一起复制,但
不会给副本重新 Bind,所以 catalog 的引用计数看不到分叉。在父会话里删除会
把分叉还在用的字节删掉。无 catalog 部署、以及存的是裸 provider 路径的行,
同样不在计数覆盖内。回收前新增一道 CountLiveArtifactsByURL:只要还有任何
未删除的行指向同一对象就保留字节,查不出来也保留。配套加了 url 的部分索引。

Atomicity. SoftDeleteSessionArtifacts 原来逐条 UPDATE 不带事务,删多版本
时中途失败会留下「一部分已隐藏、字节没回收、接口 500、用户重试 404」。
现在整批放进一个事务。writeMessageArtifacts 是整表替换,拿着删除前快照的
并发写会让墓碑复活而字节已在回收,现在按 position 把已存在的 deleted_at
合并回去——删除是唯一不允许被过期写覆盖的属性。

all_versions 两个路由改用同一个解析器(strconv.ParseBool),此前一个是
== "true"、另一个是 != "false",显式传 1/TRUE 在两边语义不同;缺省值仍
按路由区分。

测试:收集器三条(未动不复活 / 重新生成要回来 / live 的同内容恢复仍短路)、
access 三条(正文不能复活已删授权、多副本只要有 live 就可取、非产物引用
仍走正文)、service 七条(归属人校验、他人会话拒绝、并发认领为空、live
行仍在则保留字节、guard 失败保留字节、多版本按对象归并 owner、无
source_path 不扫全会话)、repository 三条(墓碑抗过期写、事务回滚、跨会话
URL 计数)、handler 一条(两个路由的 all_versions 解析矩阵)。其中
「重新生成要回来」在改回旧实现时确认会失败。

client/ 没有 artifacts 相关 API(只有一个 SSE 事件类型常量),无需同步。
2026-09-20 13:01:46 +08:00
autumnseasonism 1e4389bb21
fix(wiki): clarify alias matching rules in deduplication prompt (#3418)
Co-authored-by: autumnseasonism <275002454+autumnseasonism@users.noreply.github.com>
2026-09-20 11:24:45 +08:00
Ankit Jha 110134f187
test(im): close the lifecycle test database and keep its name unique (#3433)
* test(im): give each lifecycle test its own sqlite database

newLifecycleTestDB derived its in-memory SQLite DSN from
time.Now().UnixNano() alone. Two tests whose helper calls land in the
same wall-clock tick get the same DSN, and since the database is
cache=shared, the second test's CREATE TABLE collides with the
first's ("table im_channels already exists"). A shared database also
lets one test observe rows created by another.

Every other in-memory test database in this repo names itself from
t.Name() instead (tenant_api_key_test.go, memory/service_test.go,
and others); this helper was the only one still using the wall clock.
Switch to the same convention: t.Name() is unique per test function,
which is all that's needed here since none of the lifecycle tests
call the helper from subtests.

Added TestNewLifecycleTestDBIsolatesConcurrentTests, which runs the
seed and the isolation check as actual parallel subtests rather than
sequentially, closer to the real collision scenario than a plain
sequential pair. Confirmed it fails with the old DSN scheme (fixed
name, forcing the collision) and passes with the fix; ran the full
package 8 times as separate `go test` invocations, matching how the
race is actually triggered, with -race and no failures.

Fixes #3198

Signed-off-by: Ankit Jha <jhaankit373@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(im): close the lifecycle test database and keep its name unique

Review on #3415: switching the DSN to t.Name() fixed the collision between
tests started in the same tick but broke repeated runs. t.Name() is stable
across -count iterations in one process, and newLifecycleTestDB never closed
the sql.DB, so a shared-cache in-memory database outlived the iteration that
created it and the next CREATE TABLE failed with "table im_channels already
exists".

Reproduced with the reported command:

    go test ./internal/im -run '^TestNewLifecycleTestDBIsolatesConcurrentTests$' -count=2

Two changes, both needed:

- t.Cleanup closes the sql.DB immediately after opening, so the database is
  released with the test. Registered before the caller's own cleanups, which
  therefore still run first and can use the database while stopping a service.
- A process-wide sequence number in the name, so a single test calling the
  helper twice gets two databases; those overlap before any cleanup runs, so
  closing alone is not sufficient.

Added the repeated-run check that was asked for, plus one asserting the
connection is actually closed. Verified each half is load-bearing: dropping the
cleanup fails TestNewLifecycleTestDBClosesItsConnection, dropping the sequence
fails TestNewLifecycleTestDBIsUsableTwiceInOneProcess. Package is green at
-count=1 and -count=3 under -race.

---------

Signed-off-by: Ankit Jha <jhaankit373@gmail.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-20 11:24:28 +08:00
Captain bcfb6dfc12
fix(storage): keep the SSRF redirect policy on the KS3 client (#3440)
ks3s3.New merges its default config into the one it is given and then
installs the SDK's own redirect policy on the http.Client it was handed:
up to ten hops, no check of the target, and the previous hop's
Authorization header copied onto every redirect, cross-host included.
That silently replaced the SSRF-safe policy the client was built with,
so a redirect from the KS3 endpoint was followed without the target
check every other request gets, and the request signature travelled to
whatever host the redirect named. Go's own client drops Authorization
on a cross-domain redirect precisely to avoid that.

Save the policy before New and put it back on the client afterwards.
The per-hop transport guard was never affected; this restores the
redirect-time check and keeps the signature with the endpoint.
2026-09-20 11:12:52 +08:00
Leon 562b131e6c
fix(qdrant): treat a missing dimension collection as an empty delete target (#3419)
* fix(qdrant): treat a missing dimension collection as an empty delete target

Ingest and re-index re-index a chunk by deleting its old vectors first and
writing the new ones second (`updateChunkVector`, `syncChunkIndex`). Vectors
are stored in a per-dimension collection that is created lazily by that first
write, so when the delete is the first touch of a dimension, Qdrant answers
`Collection weknora_embeddings_<dim> doesn't exist!` and the delete fails --
taking down the write that would have created the collection (#3337).

`VectorRetrieve` already treats a missing collection as "no results". Apply the
same rule to the three delete paths: probe once, skip the RPC when the
collection is absent, and keep failing loudly when the probe itself fails.
Dimensions this process already created stay cache-short-circuited, so the
steady-state delete is still a single round-trip.

* fix(qdrant): treat a missing collection on Delete as a no-op

A stale initializedCollections cache still issued DeletePoints after a
Qdrant wipe, which revived Collection ... doesn't exist! and blocked the
write that would recreate the collection. Drop the cache on that error
so the following ensureCollection can run.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-20 11:12:18 +08:00
wizardchen 640e356e31 fix(browser): rebase BrowserSkill patches and use native session lifecycle 2026-09-20 00:09:50 +08:00
tayadwx 7ea5bf4349
fix(image): stop appending KB custom instructions to the OCR prompt OCR提示词不再错误的附加图片解析用户自定义提示词 (#3291)
* fix(image): stop appending KB custom instructions to the OCR prompt

CustomInstructions is documented as adding image interpretation guidance
"without replacing the system-owned OCR and Markdown output contract"
(types.VLMConfig.CustomInstructions), and the UI states the same. But
image_multimodal.go appended the same custom-instruction string to both
the caption prompt and the OCR prompt, so free-form business rules
(e.g. output-marker rules intended for descriptions) competed with the
OCR prompt's own contract that a text-free image must reply exactly
"No text content".

On a production knowledge base this poisoned 116/255 (45%) of image_ocr
chunks: the 8B VLM leaned on the nearest output-format rule and emitted
the KB's fixed marker phrase instead of "No text content", which passed
the sanitizer (it only recognizes known empty replies), created enabled
image_ocr child chunks with vectors, and made real photos retrievable as
decorative-image hits. Section names from the custom instructions
appeared verbatim in OCR output, confirming prompt cross-contamination.

Caption prompts keep their custom instructions; only the OCR append is
removed.

* test(image): pin the OCR prompt contract against custom instructions

Extract buildVLMOCRPrompt so the OCR prompt assembly is unit-testable and
add a regression test asserting that KB custom instructions never reach
the OCR prompt (while the caption prompt keeps them), and that the
No text content contract stays intact for both source types.

---------

Co-authored-by: tayadwx <tayadwx@users.noreply.github.com>
2026-09-19 23:37:28 +08:00
yushangcan fca1cd06f8 fix(ima): traverse subfolders using list response media IDs 2026-09-19 22:42:57 +08:00
po-et 1b36030502 perf(chat): replay a stored stream a segment at a time
continue-stream replays every stored event as its own SSE frame, and
events are stored at the granularity the model emits them: one chunk per
AppendEvent, no accumulation. A long answer with a dozen length-limited
continuations is tens of thousands of chunks, and every reconnect during
it (a refresh, a re-entered page) replays them all, one write, one flush
and one client-side pass each; the report on #3362 counts 47,196 events
for a single replay, with the page frozen while they arrive.

Coalesce them on the way out. Each run of consecutive, unfinished
thinking or answer chunks of one segment (same type and event ID) is
replayed as one event carrying the run's concatenated content, its last
timestamp and the union of its data. A done chunk, a chunk of another
type or of another segment ends the run and passes through untouched:
the client marks a segment finished on its done chunk without appending
that chunk's content, and it accumulates content by event_id, so what it
assembles is the same text in the same order, in a few frames per
segment instead of one per token. The live path, the stored events and
the completion marker are unchanged.
2026-09-19 21:43:57 +08:00
Captain a5bdf82a02
fix(storage): let object storage transfers outlive 30 seconds (#3368)
* fix(storage): let object storage transfers outlive 30 seconds

The S3, COS, KS3, OBS and OSS file services build their SDK client from
DefaultSSRFSafeHTTPClientConfig, whose 30-second Timeout is Go's
whole-request timeout, body included. So every request had to finish
within 30 seconds whatever it carried: a 40 MB PDF needs about
11 Mbit/s of uplink to make it as one COS Object.Put (OSS uploads in
10 MB parts, so there the cap was per part), and the same file coming
back from a distant region was cut the same way, while the local backend
took any size (#3306). MinIO and TOS, which only hand the SDK a
transport, never had the cap.

Give the object storage clients a config without the whole-request
timeout, and bound connection setup on their transport instead (TLS
handshake, response headers, idle connections), so a peer that accepts
the connection and then goes silent still fails within about a minute;
the header wait starts after the request body is written, so it does
not cap an upload. A transfer is then bounded by its request context.
The bucket probe every constructor runs, which had relied on the client
timeout, gets a 30-second context of its own, and the OSS and KS3
connectivity checks get the 10-second deadline the other backends
already set. KS3 called its SDK without a context at all; it now passes
the caller's, so the same bounds apply there.

* fix(storage): bound object transfers and split bucket probes

Dropping the 30s client timeout left asynq/API transfers unbounded, and Head+Create shared one setup context so a slow miss aborted create. Add a 30-minute fallback deadline, give each probe its own timeout, and only create a KS3 bucket on a real 404.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-19 21:30:55 +08:00
wizardchen ca88fe74ae fix(embedding): preserve batching and concurrency limits for WeKnoraCloud 2026-09-19 20:26:46 +08:00
po-et aea51ecb62 fix(wiki): batch taxonomy embeddings instead of one request per set
Taxonomy selection asked the embedding provider for every folder in one
request, and for every item in another, by calling BatchEmbed directly.
BatchEmbedWithPool is what reads BATCH_EMBED_SIZE and splits the inputs,
so these two calls sat outside the only control an operator has over
request width: a provider that caps inputs per request rejected them
however low that setting was, and the planner fell back to feeding all
folders (#3390).

Send both sets through the pool, the way every other embedding caller in
the service does. Nothing else changes: the pool reassembles the vectors
in input order, which is what the similarity ranking below depends on.

The regression test carries the real pooling implementation behind a
provider that records the width of every request it receives, so a call
that goes back to the provider directly shows up as one oversized
request rather than as a passing test.
2026-09-19 20:19:06 +08:00
lyingbug b82fd6c0f9
fix(security): close the low-severity shared-space findings (#3416)
* fix(security): close the low-severity shared-space findings

Verified and fixed from the shared-space permission review:

- Embed channels kept knowledge_ids, tag_ids, mentioned_items, skill_names
  and summary_model_id from anonymous visitors. Mentions are only scoped
  for shared agents, so a visitor could query any KB (and pick any model)
  of the channel workspace by ID. The payload now drops them; the widget
  never sends them.
- PUT /knowledgebase/:kb_id/wiki/issues/:issue_id/status updated any issue
  by ID. The update is now scoped to the path's KB (404 otherwise).
- A direct org invite let the inviter attach any user of the target
  workspace (or any user by ID) as its representative, whose username and
  email the roster then showed to all members. Direct adds attach no
  representative, and the roster returns emails only for the caller's own
  workspace.
- COS and KS3 dropped the bucket (and COS the region) segment when turning
  a path into an object key, while the tenant check scans every segment:
  cos://42/<region>/prefix/7/... passed as tenant 42 and read tenant 7's
  object from a shared bucket. Both must now match the service's config.
- Shared agent listings returned the full agent to receivers. Prompts and
  the creator's user ID are withheld; capabilities and scope stay visible.
- Temporary document uploads let the caller pick the parser engine (and
  its credentials) of a shared agent's workspace; only the agent's rules
  choose one there now.
- Suggested questions used request / message-snapshot knowledge_ids
  without a permission check. Each document is now resolved to its KB and
  checked like KB IDs, including a shared agent's scope.
- IM channels accepted any knowledge_base_id (and any agent ID on create).
  Both must belong to the channel's workspace, the KB also to a restricted
  API key's allow-list.
- KB-restricted API keys could list org-shared KBs outside their allow-list
  in the organization views; those listings are now filtered.
- With enable_rbac=false, RequireKBAccess let cross-workspace denials
  through. That switch is for roles inside a workspace; the guard now
  always denies another workspace's KB, since handlers behind it load by
  ID without a tenant filter.
- The org member limit was counted before and enforced apart from the
  insert; it is now checked in the insert's transaction with the
  organization row locked (SQLite serializes writers instead).
- The UpdateOrganization route comment and API doc said owner-only; the
  service and UI allow org admins. Comment and doc corrected.

Documented, not changed: issued /r/ and presigned links stay valid until
they expire after a share is revoked.

* fix(security): address review of the low-severity shared-space fixes

- Embed payloads also drop question_origin. It selects a KB like an
  @mention for agents that retrieve only when mentioned, and the widget
  never sends it; the selection stays inside the agent's scope, but an
  anonymous visitor should not pick among its KBs. suggestion_attribution is
  kept (the widget sends it and it is validated).
- The receiver view of a shared agent also withholds the follow-up
  suggestion instruction, a prompt like the others. Starter questions are
  shown to receivers anyway, and the sharer's name is displayed by design.
- Follow-up suggestion generation moves into the agent's workspace with
  types.WithExecutionTenant instead of a bare tenant rewrite, so the
  caller is pinned before the document permission check runs. The auth
  middleware and the shared run already pinned it; this removes the
  dependency on that.
- KS3 CopyFile copies only from the service's own bucket (as COS and S3
  do); another bucket's path is a cross-backend copy.

Tests cover the dropped question_origin, the follow-up instruction and the
/shared-agents listing, a follow-up context with the tenant switched, a
repeated wiki issue status update, and the KS3 copy source.
2026-09-18 22:58:46 +08:00
lyingbug 03d9d66a1f
feat(agent): size agent history by the window and calibrate token estimates (#3413)
* feat(agent): size agent history by the window and calibrate token estimates

Agent history was capped at history_turns (default 5) because compaction
could not survive a turn. With checkpoints persisted, load history by
tokens instead, and let compaction bound the session.

Budget. The loader drops the oldest turns that do not fit, and what it
drops is lost: neither replayed nor summarized. A budget equal to the
compaction threshold (the earlier attempt, since reverted) let the loader
trim first, so whenever one turn was larger than the system prompt plus
the new question the request never crossed the threshold; nothing was
summarized and the session became a sliding window with no checkpoint.
agent.HistoryTokenBudget is now the whole window: overflow reaches the
first round's compaction, which summarizes it into a new checkpoint.
A turn-by-turn regression test drives the real loader, budget and
compactor and fails as soon as a stored turn after the checkpoint is
missing from the loaded history; restoring the old budget fails it.

Summarizer input. History can now reach the window, so the part being
summarized can outgrow one request. An overflowing request degrades to a
raw archive that is never persisted, so every later turn would fail the
same way. Prepare keeps the newest messages that fit, the prompt states
how many were left out, and file paths still come from all of them.

Calibration. cl100k counts Chinese at about a token a character where
Qwen and DeepSeek spend about 0.6. The estimator now carries a scale
(clamped to 0.5-1.5, text only), learned from consecutive requests in a
turn: they share tools, system prompt and earlier messages, so the
provider's growth between them prices exactly the appended messages,
without mixing in how a provider renders tool schemas. The turn's scale
is stored on its usage (context_token_scale); the next turn's loader
prices history with it and hands it to the engine. Loading, the first
compaction check and compaction itself share one estimator, so they
agree; calibrating only the trigger would let the loader drop turns
again, which the regression test covers.

Also: history is priced as the engine sends it (redacted KB/Wiki results
cost one line); a checkpoint older than the rows read is matched by the
assistant rows' (created_at, id) order; a read ending on a turn's
question keeps that turn; a compaction discarded for freeing under 5%
still persists its checkpoint; a checkpoint write matching no row is an
error; a partial index backs the newest-checkpoint lookup. Agent mode no
longer reads history_turns: the field leaves AgentConfig, the editor
hides it in smart-reasoning mode, and quick-answer keeps its turn cap.

* fix(agent): address review of window-sized history loading

Summarization. The summarizer call had a fixed 60s total timeout. With
history loaded up to the window, a request could carry ~175k tokens and be
cut off while prefilling or writing normally; the degraded result left no
checkpoint, so the next turn reloaded the same history and failed the same
way. Summarization now streams and is cancelled only after no output for the
engine's stall timeout (120s by default, LLMCallTimeout overrides), with the
overall ceiling left to the transport, as for the engine's own rounds and as
in Codex and Pi. A stopped turn is not retried. When the summarizer still
fails, the raw archive is persisted as a checkpoint marked degraded so the
loop is broken, and the next compaction refines it as the previous summary.
The archive is held to the summary budget, newest messages first, so a
degraded compaction always frees room.

Loading. The loader priced turns as sent but returned and held them as
stored, so wiki pages stored in full and sent as one line could pile up in
memory. It now returns the as-sent history and releases each turn's stored
rows once replayed, keeping about a page of rows at a time.

Index. The partial checkpoint index is replaced by a composite
(session_id, created_at DESC, id DESC) index serving the backwards page,
the newest-checkpoint lookup and the recent-messages reads. It is built with
CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the
migration holds that single statement, since a multi-statement query runs as
an implicit transaction. A PostgreSQL integration test, skipped without
WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and
up through golang-migrate, and checks both queries use the index without a
sort; the SQLite migration test checks the same plans.

Calibration. It no longer waits for a total token count, and a usage with
only a scale is persisted. A turn that measures nothing carries forward the
scale it started from, so the newest turn always has the latest one.
2026-09-18 22:51:31 +08:00
lyingbug db8e7611be
fix(security): close cross-workspace access paths in shared spaces (#3408)
* fix(security): close cross-workspace access paths in shared spaces

A review of how shared-space (organization) members reach each other's
resources found six paths that bypass the org-share permission model:

- presigned-preview signed any resource handle or storage path, turning a
  known resource:// handle into an anonymous /r/ URL that outlives share
  revocation. It now requires the path to belong to the caller's tenant,
  like /files.
- POST /initialization/initialize let a shared-KB editor rewrite the owner
  workspace's models (base URL, API key) in place. Only the KB's own
  workspace may call it, and rewriting a stored model needs the same
  authority as PUT /models/:id.
- KB JSON carried the legacy inline credentials (storage_config
  secret_id/secret_key, which initialization fills with the MinIO env
  keys, and vlm_config api_key) to every share receiver. They are withheld
  from JSON; their DB columns are unchanged.
- A chat without agent_source_tenant_id resolved shares before the
  caller's own agent, so a member sharing a same-ID built-in agent took
  over other workspaces' default agent. Own agents now win, built-in
  agents can no longer be shared (existing such shares no longer resolve),
  and embed channels drop a visitor-supplied source workspace.
- Any org-admin tenant (even a tenant Viewer inside it) could raise another
  workspace's KB share to editor/admin, and the route guard was bypassable
  via an unrelated :id. Share management is now: the sharer from the source
  tenant (Contributor+), source-tenant Admin+, or an org-admin tenant's
  Admin+ that may only remove or lower. share_id must belong to :id.
- Wiki write tools were registered for every wiki KB in the search targets,
  which only need read access, so a viewer share (e.g. via the built-in
  wiki fixer) could edit or delete another workspace's wiki pages. They now
  act only on KBs the caller may edit.

* fix(security): close the KB settings path and owner details for share receivers

Follow-up to the shared-space review:

- PUT /initialization/config shares the route guard that admits share
  editors, so an editor could still rebind the owner's KB models, chunking
  and storage, and storage backends resolved against the caller's workspace
  let it bind the owner's KB to its own backend. Another workspace now needs
  an admin share (the only share level the frontend offers settings to) and
  may not change the storage binding at all.
- GET /initialization/config showed model base URLs and bucket locations to
  any admin of the caller's own workspace, including share receivers. They
  now only see whether credentials are configured.
- Wiki write tools on the caller's own KBs had no role floor, so a tenant
  Viewer (and the IM, embed and MCP endpoint principals that run as Viewer)
  could write through an agent what the HTTP routes deny. Writes now need
  Contributor+, or the ingest capability for a scoped API key, following
  the RBAC rollout switch.
- resolveAgent logs the share lookup error instead of a nil own-agent error
  when a shared selector fails.

* fix(security): close the medium shared-space access findings

Follow-up to the high-risk fixes, from the same review:

- Wiki fixer moved into a shared KB's workspace ran with the caller's own
  config there: an unset MCP mode registered all of the owner's MCP
  services, and an "all" KB scope reached the owner's other KBs. The scoped
  run now uses the built-in config pinned to that one KB, without MCP,
  skills, sandbox or web search, and falls back to the KB's own model.
- An unset MCP mode means "all" at runtime but is shown as none in the share
  scope; shared runs now treat it as none.
- Quick-answer mode used the request's web_search_enabled alone, so a
  receiver could turn on search the agent disables and spend the owner's
  provider quota. It now requires the agent setting too, as agent mode does.
- summary_model_id picked any model of the workspace a run executes in. It
  is ignored for shared agents and the relocated wiki fixer.
- A workspace that left or was removed from an organization kept sharing its
  KBs and agents. Its shares are revoked on departure, and share queries and
  permission checks require the source workspace's membership and a live
  organization, which also covers rows left behind earlier.
- Sharing an agent exposes its KB scope to all members, so a Contributor
  could share colleagues' KBs (or "all") through an agent. The sharer must
  now be able to share each KB directly (creator or Admin+; "all" needs
  Admin+), also when a shared agent's scope is widened later.
- A scoped API key with manage_channels could mint an MCP endpoint token
  wider than itself. Endpoints it creates, updates or rotates must stay
  within its KB allow-list and capabilities.
- A shared agent's run indexed the receiver's Q&A into the owner's chat
  history KB; indexing now uses the session owner's configuration.
- Sharing an agent with skills keeps injecting the owner's skill
  environment values (product decision); the share panel now warns that
  members can read them. Preview of originals through read-only shares is
  kept as designed and documented.
2026-09-18 22:00:02 +08:00
wizardchen 2a40cc37c0 fix(migrations): renumber the context checkpoint migration to 000105
#3406 (000104_skill_served_version) and #3407
(000104_message_context_checkpoint) each took version 104 and were merged
eleven minutes apart. golang-migrate refuses to load a directory with a
duplicate version ("duplicate migration file"), so every PostgreSQL
deployment of main fails to migrate.

Renumber the later of the two. It is safe for every deployment that could
exist: one that migrated between the two merges applied the skill
migration as 104 and now applies this one as 105, and after the second
merge main could not load its migrations at all, so no deployment applied
this file as 104. The migration is ADD COLUMN IF NOT EXISTS either way.

Add a test that loads both migration directories. Pull request CI runs on
the merge with main, so a second PR taking an already merged number now
fails before it lands.
2026-09-18 21:26:26 +08:00