Commit Graph

1574 Commits (ebfd5ef0fba44b5ed13b4e5ef71b2fa255fd3b30)

Author SHA1 Message Date
wizardchen ebfd5ef0fb feat(sandbox): browse installed skill files from settings
Read the stored skill bundle without booting a sandbox, and open a
resizable secondary drawer with a file tree and markdown preview.
2026-08-25 20:27:26 +08:00
wizardchen 19bcc9b532 fix(sandbox): hide Cube skill snapshots from the template catalog
Cube stores snapshots in the same template store as GET /templates, so
skill-image generations leaked into the settings picker. Subtract
ListSnapshots (and snap- IDs) so the step only offers base templates.
2026-08-25 19:11:41 +08:00
wizardchen 625f9db80a fix(sandbox): advertise only image skills and unstick installs
The host skills/preloaded tree is not what execute_skill_script finds
inside the sandbox, so stop merging it into the agent's advertised set.
Seed install files as a tar, publish transcript locators before the copy,
and wait in the UI instead of 404-polling.
2026-08-25 18:03:53 +08:00
wizardchen 94a2faf391 feat(chat): render shell_exec as a command and stream card
Give sandbox shell calls a display_type and keep stdout/stderr on the
live payload so the timeline can show the command, exit code, and
streams instead of the generic raw-output fallback.
2026-08-25 17:56:13 +08:00
wizardchen b6cfb34be6 docs(sandbox): clarify skill source requirements in installation process
Updated documentation across multiple files to specify that the `source` for skill installation must be explicitly defined as `@owner/slug` for ClawHub or a full URL for GitHub/SkillHub. Bare `owner/slug` is now rejected as ambiguous. This change enhances clarity for users regarding valid input formats and improves error handling in the installation process.
2026-08-25 15:47:48 +08:00
wizardchen 92fd75f54a refactor(sandbox): remove token requirement for skill installation from source
Updated the skill installation process to eliminate the need for a token when installing skills from ClawHub, SkillHub, or Git URLs. The source must now be readable anonymously, ensuring that no credentials are sent during the fetch process. Adjusted related documentation and UI components to reflect this change, enhancing the user experience by simplifying the installation workflow.
2026-08-25 15:47:48 +08:00
wizardchen 260057b5f4 feat(sandbox): install skills from ClawHub, SkillHub, and git URLs
Paste a registry or GitHub link in the skill sandbox instead of only
uploading a zip. skillhub.cn pages resolve through api.skillhub.cn.
2026-08-25 15:47:48 +08:00
lyingbug b4128552b4
fix(sandbox): skip identical skill installs and prune old snapshots (#2791)
* fix(sandbox): skip identical skill installs and prune old snapshots

Re-uploading the same archive of a ready skill was still booting a billed sandbox and growing a new generation. Superceded snapshots also accumulated on the provider because the ledger never deleted them.

Skip the rebuild when the digest already matches a skill still in the live image, and let the reaper delete superseded provider snapshots after 24h while leaving extras not in the ledger untouched.

* fix(sandbox): retry stale same-zip installs and prune active leftovers

A re-upload of the same archive skipped every installing row, so a dead process could not recover until the reaper. Skip only recent in-flight runs, still refresh the stored bundle, and prune non-live active snapshots after retention.

* fix(sandbox): beat install liveness and guard the snapshot prune

A fixed window on the submission time cannot tell a slow install from a
dead one: a single agent command may take installCommandTimeout and an
install runs several of them, so a two-minute window called live runs
dead, reset their progress and pushed the reaper's own deadline out.
Stamp InstallingSince while the run works and read that instead, which
also stops the reaper from failing installs that are merely long.

Answer the skip question from the image only for a ready row. The ledger
records which skill a snapshot carries, not which archive, so a row
installing a new bundle would look finished from the previous bundle's
snapshot and report a success that never happened.

Prune now refuses to delete once the credentials no longer address the
account that built the image: the delete would come back not-found, and
the ledger would record another account's still-billed snapshots as
gone. It also waits for an eligible row before building a provider
client, since most configs have nothing to prune on most sweeps.
2026-08-25 15:46:29 +08:00
lyingbug 01ec6c0509
ui(sandbox): rename settings to skill sandbox and fix skill install (#2792)
* ui(sandbox): rename settings to code sandbox and simplify cards

"Sandbox backend" leaked implementation jargon into the settings nav.
Call the page what users configure — an isolated runtime for skill
scripts — and only surface incomplete-config warnings on the cards.

* fix(sandbox): type the card overflow menu options

vue-tsc rejected pushing a themed delete item onto an untyped options
array; name the option shape so the menu still marks delete as danger.

* fix(sandbox): keep the editor wizard moving into skills

Saving on the runtime step used to close the drawer when editing, so
step 4 was only reachable from the rail. Continue after save, move
full verification off the runtime form, and stop treating "using the
base template" as the empty value for every image field.

* fix(sandbox): store skill bundles on the workspace default backend

Skill install passed a stub tenant with no default storage, so saving
the zip failed with empty provider and the 500 hid the real cause.

* ui(sandbox): regroup skill install and rename to skill sandbox

Show install vs installed groups, fold long descriptions in the list,
and call the settings surface a skill sandbox.

* ui(sandbox): show skill install transcript in a compact popup

Keep the card from growing; the run opens beside the history button
with a capped height and its own scroll.

* ui(sandbox): show installed skills on config cards

Cube/E2B tiles now list skill names so the list reads as skill sandboxes; a click opens the editor on the skills step.

* ui(sandbox): show finished skill installs as static history

A completed run already has durable messages; replaying the event log would animate the whole install again. Hide chat answer actions in this transcript.

* style(frontend): enhance compact skill timeline display with improved font sizes

Updated the CSS for the compact skill timeline to adjust font sizes for various elements, ensuring better readability in the popup interface. This includes specific adjustments for embedded agent streams and markdown content.

* ui(sandbox): refactor header structure in SandboxConfigEditorDrawer and SettingDrawer

Updated the header structure in both SandboxConfigEditorDrawer and SettingDrawer components to improve layout and organization. Added a new slot for extra header content in SettingDrawer and adjusted CSS for better alignment and spacing in both components.

* ui(sandbox): enhance skill loading feedback and improve deep check logic

Updated the SandboxConfigEditorDrawer to refine the logic for deep checks based on the current step. Enhanced the SkillInstallTimeline component to manage in-flight live loops and provide better feedback for skill loading failures. Adjusted the SandboxSettings view to display error messages for failed skill loads, ensuring users are informed of any issues during the skill installation process.
2026-08-25 13:24:58 +08:00
nullptr-error 18d22b014b
feat(sandbox): expose tenant skill management over HTTP and in the console (#2789)
* feat(sandbox): enhance skill management in sandbox configurations

Updated the skill management system to allow listing, installing, and removing skills within sandbox configurations. The ListSkills function now accepts a sandboxConfigID to filter skills based on the selected configuration. Introduced a new SandboxSkillsDrawer component for managing skills through the UI, along with necessary updates to the API and localization files to support these features.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: keep skill maintenance sessions out of the console session list

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: store and expose the installer transcript locators

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: record the skill installer agent's conversation

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: stream and persist the installer transcript during a skill install

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: expose install transcript locators on the skill type

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add the floating skill install transcript window

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: open the install transcript from the sandbox skills drawer

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): 重构技能安装体验,支持内联时间线、编辑第4步及保留镜像指针

- 替换右下角浮窗为内联安装时间线 (SkillInstallTimeline.vue)
  - 新增 GET /api/v1/sandbox-configs/:id/skills/:skillId/transcript 接口,支持回放与追尾
  - 移除 /sessions/continue-stream 依赖,避免 404 问题
  - 事件日志以 install_prompt 开头,不再将工具调用 preamble 拼入最终答案
  - 删除 SkillInstallTranscript.vue,技能条目下可展开/折叠时间线

- 编辑向导新增第 4 步「管理技能」
  - Cube/E2B 环境:连接 → 模板 → 运行配置 → 技能(第4步)
  - Docker/Local 环境:无第4步(无快照镜像)
  - 首次创建必须按 1→2→3 顺序,第3步保存成功后自动进入第4步
  - 已有配置支持步骤条数字跳转,可直接访问任意步骤
  - 卡片菜单「管理技能」打开同一抽屉并定位至第4步
  - SandboxSkillsDrawer.vue 重命名为 SandboxSkillsPanel.vue,去除独立抽屉壳

- 修复保存运行配置时冲掉 skill_image 的问题
  - 顶部展示信息(基础模板/快照ID/版本/构建时间)读取 config.skill_image
  - 编辑器保存 payload 不再包含 skill_image,合并时保留已有 SkillImage 和 VolumeMount
  - MergeSandboxConfigForUpdate 增加对 SkillImage 和 VolumeMount 的保护
  - 已冲掉的配置需重新安装技能才能恢复镜像指针

- 「查看安装过程」在安装开始时即显示
  - 技能列表中出现「正在安装」状态即展示按钮
  - 若记录尚未写入,展示等待提示并自动重试,避免误报「没有留下记录」

* 沙盒内技能目录从 UUID 改成 SKILL.md 的 name。

* fix(sandbox): heal stuck skill rows and block planted maintenance sessions

The reaper left a mid-upgrade install spinning forever when the old snapshot was still live, and restored removals to ready after the pointer had already moved. Client session create/update can no longer hide chats behind the skill_maintenance: prefix.

* style: gofumpt the skill reaper test stubs

* style: satisfy golangci on the tenant skill admin API

* feat(sandbox): rebuild skill images at turn boundaries, not mid-exec

Hold a session turn lease through AgentQA so an install cannot tear down
the live sandbox between tool calls; the next turn's first resolve picks
up the new image. Configs can instead keep existing sandboxes and boot
the new snapshot only for new sessions.

* fix(sandbox): judge stuck skill runs from the snapshot chain

InstalledSnapshotID only records the generation that installed a skill; later
installs of other skills advance the live pointer without rewriting older rows.
Comparing those IDs made the reaper fail healthy installs and delete live
skills whose removal had stalled.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-25 11:28:12 +08:00
wizardchen aa30ced85f fix(sandbox): apply gofumpt spacing in skill-install tests 2026-08-24 17:51:13 +08:00
wizardchen ad593652e8 fix(sandbox): satisfy pre-push lint on the skill install surface
Break long lines and add the missing constructor comment so the
maintainer push hook can reach the fork.
2026-08-24 17:51:13 +08:00
wizardchen 82ce97ba72 fix(sandbox): keep skill install from clobbering a newer owner
A queued remove or a failed older install could stamp or delete a row a
re-upload already claimed, and an empty MCPSelectionMode handed the root
installer every tenant MCP tool. Fail closed on those races, pin the
installer surface, and refuse a missing bundle or leftover zip files.
2026-08-24 17:51:13 +08:00
wizardchen 03755f11f7 fix(sandbox): harden tenant skill install against races and credential rotation
Abort a queued install if the skill row was removed or superseded, refuse to
restamp a snapshot with a rotated provider fingerprint, and honor the workspace
sandbox kill switch before creating a maintenance session.
2026-08-24 17:51:13 +08:00
hailongzhao 67cdf26216 feat(sandbox): add tenant skill install and remove service layer
Introduce the hidden skill-installer agent and TenantSkillService so a
sandbox config can install or remove skills into snapshot images under a
per-config lock, without yet exposing HTTP or UI entry points.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 17:51:13 +08:00
wizardchen e683008188 fix(chat): align live timestamps with persisted created_at
Echo the stored user/assistant times on the first stream event so the
UI stops showing a client-clock fallback that would jump after reload.
2026-08-24 17:04:48 +08:00
wizardchen 7f11b7b5b9 fix(sandbox): keep skill image pointer out of sandbox config PUT
The settings form rebuilds the payload without skill_image, so merge
must ignore client values or a save would drop the snapshot (and a
crafted PUT could plant one). Also reject skill names that escape
the image root and require SupportsSnapshots on the snapshot manager.
2026-08-24 16:00:23 +08:00
hailongzhao 00fe035668 feat(sandbox): add tenant skill snapshot infrastructure
Give Cube/E2B a shared snapshot API and persist skill image pointers so later session sandboxes can inherit installed skills from a config snapshot instead of the base template.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 16:00:23 +08:00
wizardchen 0dbac4ce86 fix(im): wrap applyRichText signature for lll 2026-08-24 14:51:04 +08:00
wizardchen 81b4ab9e98 fix(im): tighten DingTalk rich-text follow-up handling
Keep image/file attachments out of the empty-query guard, parse audio
recognition, and surface a hint when extra rich-text pictures are dropped.
2026-08-24 14:51:04 +08:00
waterbang bdaa4618c8 fix(im): handle DingTalk rich text messages 2026-08-24 14:51:04 +08:00
PP1 c5b6c10971 fix(handler): validate knowledge search pagination 2026-08-24 14:16:35 +08:00
wizardchen adcb9fc10d fix(sandbox): 产物目录 bootstrap 降权,堵住 chown 跟随符号链接的容器内提权
会话级容器让「产物目录 bootstrap 以 root 执行」变成一条可复现的提权链:
沙箱账号拥有 /workspace,可以把产物目录换成指向 /etc 的符号链接;下一次
执行时 root 跑的 chown/chmod 会跟随链接,把 /etc 的属主交给沙箱账号,随后
删除重建 passwd 即可让该账号在下一次 exec 时解析成 uid 0——本 PR 刚做的
shell_exec 降权和文件操作降权会一并失效。一次性 docker run --rm 下两次执行
不共享文件系统,这条链走不通。真机验证过完整链路。

修法是让 bootstrap 和其它 exec 一样以沙箱账号运行:chown 对自己刚建的目录
是 no-op,对不属于自己的目标被内核拒绝。同时把 dockerExecUser 的空值兜底
从 root 改成 DefaultSandboxExecUser,漏传账号只会失去权限而非拿到权限,
也让三个后端对「空 User」的语义一致。

顺带订正若干过强的声明:find 只对路径最后一段不跟随符号链接,中间层的链接
仍由内核展开,因此 read_sandbox_file 的「拒绝符号链接」只覆盖路径本身指向
链接的情况;对应单测原先 mock 了一个真实 find 不会产生的返回值,改为断言
它实际覆盖的场景。文件头注释仍写着文件操作走 archive 接口,也一并更新。
2026-08-24 12:22:58 +08:00
wizardchen d6ddad5430 fix(sandbox): Docker 文件操作改走沙箱账号,堵住符号链接越权并回收僵尸进程
archive 接口(CopyToContainer / CopyFromContainer / ContainerStatPath)由 daemon
执行,它忽略 exec user 一律以 root 操作,并在解析路径时跟随符号链接。而沙箱账号对
/workspace 有写权限,调用侧的路径守卫(isUnderRoot、cleanSessionInputPath)又都是
字符串前缀比较,于是 `ln -s /root /workspace/output/esc` 之后,
/workspace/output/esc/secret.txt 既能通过守卫,又会被 daemon 以 root 读出来。真机
验证过:非特权账号 cat 该路径是 Permission denied,docker cp 同一路径拿到明文。

WriteFile / ReadFile / Stat 改为以 DefaultSandboxExecUser 走 exec,能不能读写交给
内核判定,符号链接指向哪里都不再重要,也不存在「先校验后使用」之间被换掉链接的窗口
(这正是 realpath 预校验方案挡不住的)。Stat 用 find,它不跟随符号链接,因此链接如实
报告为 other 类型,read_sandbox_file 在尝试读取之前就会拒绝。两个 copy 端点从
dockerEngineAPI 接口里移除——留着未使用的入口等于把这一类问题继续敞着。

容器入口是 sleep,从不调用 wait(),后台进程活得比启动它的 exec 久就会变成没人回收的
僵尸,堆到 pids_limit 之后所有 exec 都会失败。补 HostConfig.Init 交给 tini。实测:
三个孤儿进程,不开 Init 留下三个 Z,开了之后为零。

同步订正文档与注释中过时的 root 表述:上一次提交已把 shell_exec 降权到沙箱账号,但
docs 仍写着「exec 默认以 root 运行」「shell_exec 默认就是 root」,并据此把活跃标记可被
无限刷新归因于「容器内 root 信任边界」——标记是 chmod 666,普通技能脚本就能刷新。
2026-08-24 12:22:58 +08:00
wizardchen c920a00918 fix(sandbox): shell_exec 降权到沙箱账号并修复 exec 取消时的数据竞争
shell_exec 承载的是模型自己写的命令行,此前未指定 User。Docker 适配器把空
User 解析为 root,envd 系后端解析为沙箱账号,同一个工具在不同后端拿到的权限
并不一致。改为显式指定 DefaultSandboxExecUser,并把空 User 的语义收窄为仅供
manager 自身的 bootstrap 使用——chown artifact 目录必须以 root 执行,不能以
被授权的那个账号执行。

取消 exec 时原先直接读取输出缓冲区,而 StdCopy 协程仍在向同一块内存写入。改
为先关闭流让 StdCopy 解除阻塞,等它收敛后再读;超出宽限期则丢弃部分输出,而
不是从仍在写入的协程手里读走。

Docker 守护进程地址此前只校验管理员填写的字符串,而空值会在其后由 DOCKER_HOST
或 docker context 填充,真正拨号的是填充后的值。这让 DOCKER_HOST 指向明文
tcp:// 守护进程的部署能存下一份必然失败的配置。校验改到解析之后、
RequireCompleteConfig 之后,缺镜像这类表单里看得见的问题仍然优先报告。
2026-08-24 12:22:58 +08:00
wizardchen 9c5c52f0a4 feat(sandbox): Docker 后端支持会话级长驻容器与远程守护进程
将 Docker 沙箱后端从一次性 exec 容器改为会话级长驻容器,使其会话与文件
系统语义与 E2B / Cube 后端对齐:

- 新增 docker_engine / docker_remote_client / docker_rpc_timeout,直连
  Docker Engine API,支持 TCP+TLS 远程守护进程,并为拉镜像等慢操作单独
  设置超时;daemon 地址留空时跟随本机 docker context。
- 新增空闲清理器 docker_idle_sweeper,按 exec 刷新的活跃标记回收长驻
  容器;标记文件对沙箱非 root 账号可写。
- 会话容器补 PID1 以回收僵尸进程;标准镜像补入 curl。
- 镜像即模板:docker_template_catalog 只上报可识别的沙箱镜像,标准模板
  的拉取在后台进行并报告 building。
- 设置页与租户配置支持守护进程地址、TLS 与 CPU/内存限额,四种语言文案
  同步;沙箱健康检查覆盖 Docker 后端。
- 补充 docs/sandbox-docker-backend.md 与 POC,说明能力边界与快照语义;
  git 钩子的变更包检测跳过嵌套 module,避免拿 POC 的包去跑主 module 测试。
2026-08-24 12:22:58 +08:00
ochan.kwon 894cbec48a feat(agent): report and attribute per-turn LLM token usage
Accumulate token usage across every LLM round of an agent turn and
expose the aggregate everywhere a turn is observable:

- AgentState.TurnUsage accumulates each round's reported usage,
  including the final-answer synthesis call; TokenUsage.Accumulate
  keeps the cache counters as prompt subsets and recomputes the
  combined cache status (unsupported survives only while every call
  was itself unsupported).
- The agent completion event carries the aggregate, and the complete
  stream event exposes it both as data.usage and as a typed
  StreamEvent.Usage that buildStreamResponse promotes into the
  previously-unused StreamResponse.Usage wire field.
- The assistant message persists it as a nullable usage column on
  both migration targets (postgres 000085 jsonb, sqlite 000012 text),
  for web and IM channels alike, so history reads still carry the
  turn's cost after the live stream is gone.
- "[LLM Usage]" log lines gain ", session_id=..., principal=type:id"
  when the context carries them; calls outside sessions stay
  byte-identical.

Usage from failed retry attempts is not accumulated: the per-call
usage log remains the ground truth, while the turn aggregate serves
as the operational signal.
2026-08-24 10:36:19 +08:00
wizardchen 412dcc41c6 fix(datasource): restore sync_deletions on the in-memory object after create
GORM hydrates default:true back onto the struct during Create, so the HTTP
201 body (and a later Update on the same pointer) could disagree with the
row just written.
2026-08-21 16:37:05 +08:00
liguowei 7a910aa05a fix(datasource): persist disabled deletion sync setting 2026-08-21 16:37:05 +08:00
MidoriKurage ba03be775f
feat(system-admin): add endpoint for creating users (#2722)
* feat(types): add AdminCreateUserRequest for system-admin user creation

Password un-tagged so an empty value reaches the service and triggers
OIDC-style random-password generation.

* feat(system-admin): scaffold POST /system/admin/users/create endpoint

- Handler skeleton under the SystemAdmin()-guarded group
- Binding and validation live (400); happy path 501 until the
  service logic lands
- Response type declares the once-only generated_password contract

* test(system-admin): contract tests for user creation

- Create/audit/generated-password cases assert the final contract
  and are intentionally red against the 501 stub
- Binding and trim validation cases already pass
- Adds the system.user_created audit action as the audit contract

* feat(frontend): add createSystemUser API wrapper

Mirrors promote/revoke/reset-password conventions, including the
once-only generated_password contract on the response type.

* fix(system-admin): never rewrite password; re-check trimmed username

- Handler no longer trims the password: whitespace is part of the
  credential, hashed byte-for-byte (mirrors auth.go Register)
- Trimmed username re-checked to 2-50 runes (binding ran on the
  raw JSON)

* feat(service): implement AdminCreateUser with random-password fallback

- Empty password: OIDC-style crypto/rand generation, returned once
- Explicit password: must satisfy the 8-32 letter-and-number policy
- Delegates to Register: duplicate checks, tenant provisioning and
  Owner membership bootstrap match public registration

* feat(system-admin): implement create-user handler

- Resolves tenant mode from the shared auth.default_tenant_mode
  policy (single source for registration and admin creation)
- Emits system.user_created audit without the plaintext password
- 201 + once-only generated password; 400 for policy/duplicate,
  500 for internal failures

* test(service): whitespace-only password must be rejected

- All-whitespace password: ErrPasswordPolicy, nothing persisted
  (matches AdminResetPassword); only truly empty triggers generation
- Red against the current TrimSpace-empty check
- Handler test drops the whitespace-only 201 expectation, keeps
  byte-for-byte pass-through for spacey valid passwords

* fix(service): treat only a truly empty password as generate-request

Generation now triggers only on the empty string; whitespace-only
input is rejected by the 8-32 policy like AdminResetPassword.

* fix(service): regenerate until the generated password passes policy

A 32-char base64url draw misses digits ~0.4% of the time and could
400 a creation nobody picked a password for; regenerate until the
policy passes (expected ~1.004 draws).

* refactor(system-admin): drop TenantProvisioning from create payload

The value lived both on the request and as the service argument;
keep it only as the argument, mirroring LoginWithOIDC.

* test(system-admin): pin password_generated audit value; cover dupes

- Audit assertions check the exact password_generated boolean
  (true generated, false caller-supplied)
- Duplicate mapping table-driven over email and username errors

* feat(system-admin): explicit provide-and-validate password semantics

- Password pointer-typed: absence (or null) triggers random
  generation; any provided value, empty string included, is
  policy-checked and rejected otherwise
- Service: generation on nil only, all provided values hashed
  byte-for-byte
- Tests: empty string -> 400, absent -> generate, nil pass-through
  at the handler boundary
- Frontend: document that any provided value is policy-checked

* test(system-admin): merge duplicate rejection tests

Whitespace-only and weak-password rejection tests were structurally
identical, differing only in inputs. Merged into one table-driven
test covering weak, empty, whitespace and whitespace-only values.

* feat(system-admin): idempotent create on duplicate identity

- Register returns typed sentinels ErrUser{Email,Username}Exists
- AdminCreateUser re-queries the existing row, lookup targeted at the
  sentinel-named identity (email or username)
- Handler: duplicate sentinels -> 200 + UserInfo with idempotent=true
  audit, policy errors stay 400; drops the substring-matched branch

* fix(system-admin): reject partial identity conflicts on user create

Idempotent 200 now requires both email and username to match the existing
row; partial collisions return 409. Align handler identity sanitization and
password docs with nil-or-null generation semantics.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-20 19:51:10 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
hujiaxin 95c3a10ecc fix(wiki): stop housekeeping from force-failing documents with durable Wiki backlog
The stuck-row sweep asks the asynq inspector whether a candidate still has
work queued, and treats "nothing queued" as proof the row is orphaned. For
Wiki that question cannot be answered by asynq at all:

  - Wiki ingest work is durable per document in task_pending_ops, keyed by
    dedup_key = knowledge ID (newWikiIngestPendingOp).
  - asynq only receives a per-KB wake-up trigger, whose payload carries
    KnowledgeBaseID and no knowledge ID (enqueueWikiIngestTrigger).
  - matchesKnowledge() gates on taskTypesForKnowledgeCancel, which
    deliberately excludes TypeWikiIngest.

So HasQueuedTasksForKnowledge structurally returns false for a document
whose only outstanding work is a queued Wiki ingest. The sweep then flips a
perfectly healthy 'finalizing' row to 'failed' with "task stuck in
processing", zeroes pending_subtasks_count, and the durable op later runs
anyway — which is how a document ends up failed while present in neither the
dead-letter table nor any visible queue.

Probe task_pending_ops directly before consulting the inspector. A durable
probe error defers every candidate to the next sweep instead of failing them:
wrongly failing a live document is not user-recoverable, waiting one interval
is. The inspector's own fail-safe direction is unchanged.

A nil inspector no longer disables the whole gate, only the transient half.

TestHousekeeping_NoFalseKill_DurableWikiIngestPending reproduces the bug (it
fails on the unpatched sweep); TestHousekeeping_StillRecoversWhenNoDurableOp
guards against the gate becoming a blanket amnesty.
2026-08-20 19:36:15 +08:00
hujiaxin 8c0e634ee9 perf(agent): make knowledge_search MMR selection incremental
applyMMR recomputed every candidate's redundancy against every
already-selected result on each round, re-tokenizing the selected passages
each time. With C candidates and k selections that is O(k^2) calls to
getEnrichedPassage + tokenizeSimple over full chunk bodies, and the
diagnostic average-redundancy loop afterwards re-tokenizes another O(k^2)
times. Since k defaults to len(candidates), this is quadratic in the number
of results that survive dedup/rerank.

On a knowledge base where ~250 candidates reach the MMR stage this burns
tens of seconds of pure CPU inside every knowledge_search call, and no log
line attributes the time to MMR.

Cache each candidate's running max jaccard against the selected set and fold
in only the newly selected result per round, and reuse the already-computed
token sets for the average-redundancy diagnostic. Tokenization now happens
once per candidate. Selection output is unchanged, including tie-breaking;
the added test pins the new implementation against the previous one over a
deterministic corpus, and a benchmark contrasts the two.
2026-08-20 15:44:32 +08:00
hujiaxin ead70dec02 fix(session): bound the auto-generated session title to fit the column
GenerateTitle writes the model completion straight into sessions.title,
which every shipped migration declares as VARCHAR(255). When the model
ignores the "short title" instruction and returns an explanation along with
the title, the follow-up Update fails with SQLSTATE 22001 (value too long
for type character varying(255)) and the session silently ends up with no
title. The error only reaches the log and the answer itself is unaffected,
so the failure is easy to miss.

Trim the reasoning prefix and surrounding whitespace, then truncate to 100
runes - by rune rather than byte, so a multi-byte character is never cut in
half - and log a warning when truncation happens so the prompt can be
revisited.
2026-08-20 15:35:34 +08:00
Anky 0eb79c25e4
feat: support editing document summaries (#2752)
* feat: support editing document summaries

* fix: address review feedback for manual summary editing

Reset summary edit state when the drawer closes or generation is in
flight, sync summary_status on manual description updates, and add
service tests plus API docs for partial PUT behavior.

* chore: gofmt client/initialization.go for pre-push hook

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-20 14:54:53 +08:00
wizardchen 71fe7f31ee fix(migrations): 补齐 SQLite 迁移与 versioned 的索引与数据迁移缺口
对齐 000005/010/011 的邀请索引、tag_id 迁入与 MCP principal 唯一索引,
并扩展测试覆盖分享链接、OAuth upsert 与 v4 标签迁移路径。
2026-08-20 14:35:49 +08:00
icatw b888172dce fix(migrations): 补齐 Lite 模式缺失的 SQLite 版本化迁移
migrations/sqlite 落后于 migrations/versioned,导致 Lite 模式全新数据库
首次建号/建空间失败(tenants.api_principal_config、system_settings、
task_pending_ops 等缺失)。新增 000005-000011 七个 SQLite 迁移,
按 versioned 迁移主题一一补齐:

- 000005  <- 000034/000054: messages.attachments、tenant_invitations token/accepted_count
- 000006  <- 000041: task_pending_ops / task_dead_letters(与 #2433 同一 DDL)
- 000007  <- 000053: system_settings、users.is_system_admin
- 000008  <- 000055/000056: knowledge_processing_spans、knowledges.pending_subtasks_count
- 000009  <- 000060: embed_channels.allow_memory
- 000010  <- 000063: knowledge_tag_relations
- 000011  <- 000064: tenants.api_principal_config、mcp_oauth_tokens principal 列

测试:新增 internal/database 迁移测试,覆盖全新库迁移到 v11 与
v4->v11 升级保数据两条路径;go test ./internal/database 全绿。

关联 #2158;与 #2433 范围重叠(任务表部分 DDL 保持一致)。
2026-08-20 14:35:49 +08:00
lawrence 28dd15ec30 fix(retrieval): 分段中删除的图片不再被检索到
从分段里删掉一张图片后,这张图仍会回到检索结果和模型上下文里。

三个问题串在一起:

1. 删图时 syncEditedChunkImages 会把 image_ocr / image_caption 子块置为
   禁用,但 CollectImageInfoByChunkIDs 查子块时不看 is_enabled,又把它们的
   image_info 挂回了父块。knowledge_search 工具随后把每条 image_info 无条件
   渲染成 markdown 图片写进 LLM 上下文,图片就这样回来了。这里按 is_enabled
   过滤,两级遍历都跳过禁用子块。过滤放在 collector 而不是工具输出层:纯图片
   分块的正文本身就是 OCR 文本、没有 markdown 图片语法,按正文过滤会误杀。
   chat_pipeline 里既有的 fake 仓储 fixture 同步补上 IsEnabled: true:它们不
   走 DB 默认值,零值 false 会被新的过滤当成已禁用子块。

2. UpdateImageInfo 补建 image_ocr / image_caption 子块时没有设置 IsEnabled。
   CreateChunks 用 Select("*") 插入,gorm 的 default:true 不生效,零值 false
   直接落库,这些子块一出生就是禁用的。补上 IsEnabled: true —— 否则第 1 点
   的过滤会把它们永久丢弃。其余 CreateChunks 调用方均已显式设置该字段。

3. 摘要刷新每次都新建一条 summary 分块,旧的留着且仍然启用。原因是它复用了
   ListChunksByKnowledgeID 的结果,而那个查询硬编码 chunk_type = 'text',
   "查找已有 summary 分块"的分支永远匹配不上。残留的旧摘要描述的是编辑前的
   文档,会让用户已经删掉的内容继续被检索到。新增按 chunk_type 过滤的仓储
   方法,刷新时原地更新已有 summary 分块。
2026-08-20 12:01:28 +08:00
LHMQ878 35820ebb16 fix: localize builtin agent name/description from YAML when reading DB rows
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 11:54:03 +08:00
Zhang GH aac4f1586b
Fix/hybrid search match count normalization (#2750)
* fix(hybrid-search): normalize MatchCount once instead of at the truncation site

An omitted match_count deserializes to Go's zero value, which reached the
truncation step in HybridSearch and sliced the deduplicated chunk list to
[:0] — a successful retrieval returned an empty result set. A negative
value panicked on that same slice bound and surfaced as a 500.

Resolve the value once at the top of HybridSearch via normalizedMatchCount
so all three consumers agree, rather than patching the truncation site
alone: the over-retrieval floor, the FAQ iterative-retrieval trigger
(which never fired on this path because len(chunks) < 0 is false), and the
truncation itself.

Bound the FAQ iterative path while here, since normalization is what makes
it reachable with a large MatchCount: its TopK seeds from MatchCount*3 and
doubles five times, previously without limit. It now stops at the pool cap
instead of re-issuing an identical query.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(retrieval): share the default retrieval depth as one constant

The value 50 was written three times — the over-retrieval floor in
HybridSearch, the MatchCount fallback added alongside it, and
RetrievalConfig.GetEffectiveEmbeddingTopK — with nothing recording that the
three are meant to agree. Anyone tuning one would silently desync the
others.

Promote it to types.DefaultRetrievalTopK. No behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: drakezhang <drakezhang@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 11:36:38 +08:00
Zhang GH 478a6521ea
Merge pull request #1156 from toy0116/pr/hybrid-search-matchcount-fix
fix(hybrid-search): guard MatchCount=0 truncating all results
2026-08-19 20:16:23 +08:00
Zhang GH c42778556c
Merge pull request #2701 from oscarlius/codex/fix-chunk-context-header-nul
fix(chunk): sanitize context headers before persistence
2026-08-19 15:17:31 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
lyingbug 7f0e1a91e9
fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界 (#2720)
* build(anydoc): 构建脚本从 Cargo.toml 读取待打补丁的 crate 版本

原先 0.1.8 硬编码在脚本的 6 处,升级要同时改动它们;现在统一从
`anydoc = "=X.Y.Z"` 这一处 pin 推导。

另外补两道拦截:patched-anydoc/ 的复用标记记录已打补丁的版本,避免
版本变更后沿用上一版的副本、最终在 cargo 的 patch 解析里报出难以定位
的错误;构建前校验 version.go 与该 pin 一致,因为这个常量会作为
anydoc_version 写进每一篇已解析文档的元数据。

* fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界

anydoc 0.1.9 只把 pdf-inspector 从 0.1.8 提到 1.14.2,但那一版含 9 个
修复,其中 7 个是给此前完全无上界的解析工作加限额:每页 Form XObject
展开、CID /W 区间、Encoding cidrange 与 ToUnicode bfrange 展开、
分配操作符前的内容流解码、detector 的 Tj/TJ 操作数回溯、以及不相交
矩形的表格聚类。它们全都落在 process_pdf_mem 这条路径上,而这是
anydoc 转换 PDF 的唯一入口,且只由上传文件的字节驱动。

这正是 guarded() 拦不住的那一类问题:耗尽 CPU 不会 panic,分配失败直接
abort。以 detector 回溯为例,缺少操作数的 TJ 会重扫整个内容流,因此
一串裸 `] TJ` 的开销是长度的平方:977 KB 的 PDF 要 26.7 秒,1.9 MB 要
111.9 秒;升级后分别是 5.8 毫秒和 11.0 毫秒。
TestDetectorLookbackStaysLinear 用 15 秒预算把这个上界钉住——它在
0.1.8 上会超时失败,在 0.1.9 上 0.01 秒通过。

另外 2 个修复改善提取质量,直接影响入库内容:Form XObject 内的文字
现在跟踪 text line matrix 并处理 T*/TL/'/"/Tc/Tw,嵌套表单文字不再
丢换行和字间距;小型大写字母(small-caps)的文字段会合并,不再被当成
额外的表格列而产出错位的 markdown 表格。

升级本身不需要改绑定代码:anydoc 0.1.8 与 0.1.9 的 src/ 字节级一致,
C ABI、文档模型和 GFM 序列化器都没变,6 项本地修改原样保留。依赖树也
只动了 3 行版本和 1 个 checksum,lopdf 仍是 0.42,cargo audit 结果与
升级前相同(仅 ttf-parser 未维护这一条已允许的警告)。

* docs(anydoc): 文档与注释中的耗时数字对齐到实测记录

表格与注释里的数字改为与随 PR 附上的实测日志同一次运行的结果,便于核对。
2026-08-16 12:30:55 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
shangfeng-liu b6b1148bbb fix(chunk): sanitize context headers before persistence 2026-08-14 00:33:09 +08:00
wizardchen 49010d2526 fix(chat): 兜底回答不再把原始命中数报成检索结果数
检索阶段以 ErrSearchNothing 短路时会走兜底回答,而兜底 prompt 只拿到知识库的
文档标题清单,一条检索片段都没进模型上下文,引用集合同样为空。但进度条的计数
在 MergeResult / RerankResult 都为空时回退到原始 SearchResult,于是时间轴上写着
「找到 10 个结果」——既不是模型看到的上下文数,也没有引用可以点开。

现在兜底路径下 count / doc_count / web_count 归零,原始命中数放进新的
candidate_count,前端相应显示「命中 N 条候选,相关性不足,未用于回答」,与完全
没有候选的「未检索到相关内容」区分开:这个区别指向的是相关性阈值,而不是知识库。
2026-08-13 21:34:21 +08:00
wizardchen 9ed6266264 fix(chat): 持久化快速问答的检索步骤,刷新后不再丢失时间轴
快速问答的检索步骤只存在于 SSE 事件流里,刷新后前端只能靠 knowledge_references
反推时间轴,于是「检索了但没命中」的那一轮回到历史时步骤全部消失,只剩一条
孤立的记忆行。

改为在快速问答模式下订阅 pipeline 的工具事件,把 query_understand /
knowledge_search / attachment_parsing / image_analysis 落到 agent_steps。只在收到
tool_result 时写入,避免被中断的步骤刷新后显示成已完成。

这些调用来自 pipeline 而非模型,ID 加 ragpipe- 前缀并在 Agent 历史回放里过滤掉,
否则会把模型没发起过、甚至没被提供的工具调用塞进请求协议。
2026-08-13 21:34:21 +08:00
ttommybot 1d68b4dcd7
fix: filter menus by deployment capabilities (#2674)
* fix: filter menus by deployment capabilities

* fix: address deployment capabilities review feedback

Restore Lite organizations hiding when the capabilities probe fails,
align embed capability detection with route registration, move the
handler into SystemHandler for Swagger/docs/client sync, and add
cross-layer capability key contract tests.

* style: gofmt deployment capabilities files

Fix App CI formatting check so the vet/test step can run.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-13 20:18:48 +08:00
ChenRussell 2a60a7f30a fix(chunk-merge): stop dropping content when contiguous chunks repeat text
AppendWithOverlap deduplicates chunker-prepended table headers by
  matching a suffix of acc against a head window of next. When chunks are
  strictly contiguous (positionOverlap == 0) there is no overlap to
  deduplicate, but the loop still ran. If a sentence at the tail of acc
  reappeared as real content inside next's head window (same sentence
  written multiple times in the document), the head of next was mistaken
  for a prepended header and deleted — silently losing up to hundreds of
  bytes from the merged result.

  This affects three callers:
    - chat_pipeline.mergeOverlappingChunks: RAG search results returned
      to the LLM with truncated content
    - wiki_ingest.reconstructContent: distillation text fed to the LLM
      is truncated, so wiki search returns 0 hits for present content
    - frontend doc-content.vue port: the "merged" document view drops
      the same segment

  Fix: skip text matching when positionOverlap <= 0 and concatenate
  verbatim. The chunker's prepended-header case only occurs inside the
  overlap region (positionOverlap > 0), which still runs the loop.

  Add a regression test covering a compliance-doc scenario where a
  logging-requirement sentence recurs in a later Compliance section.
2026-08-13 19:58:57 +08:00