Commit Graph

550 Commits (ebfd5ef0fba44b5ed13b4e5ef71b2fa255fd3b30)

Author SHA1 Message Date
lyingbug b4128552b4
fix(sandbox): skip identical skill installs and prune old snapshots (#2791)
* fix(sandbox): skip identical skill installs and prune old snapshots

Re-uploading the same archive of a ready skill was still booting a billed sandbox and growing a new generation. Superceded snapshots also accumulated on the provider because the ledger never deleted them.

Skip the rebuild when the digest already matches a skill still in the live image, and let the reaper delete superseded provider snapshots after 24h while leaving extras not in the ledger untouched.

* fix(sandbox): retry stale same-zip installs and prune active leftovers

A re-upload of the same archive skipped every installing row, so a dead process could not recover until the reaper. Skip only recent in-flight runs, still refresh the stored bundle, and prune non-live active snapshots after retention.

* fix(sandbox): beat install liveness and guard the snapshot prune

A fixed window on the submission time cannot tell a slow install from a
dead one: a single agent command may take installCommandTimeout and an
install runs several of them, so a two-minute window called live runs
dead, reset their progress and pushed the reaper's own deadline out.
Stamp InstallingSince while the run works and read that instead, which
also stops the reaper from failing installs that are merely long.

Answer the skip question from the image only for a ready row. The ledger
records which skill a snapshot carries, not which archive, so a row
installing a new bundle would look finished from the previous bundle's
snapshot and report a success that never happened.

Prune now refuses to delete once the credentials no longer address the
account that built the image: the delete would come back not-found, and
the ledger would record another account's still-billed snapshots as
gone. It also waits for an eligible row before building a provider
client, since most configs have nothing to prune on most sweeps.
2026-08-25 15:46:29 +08:00
nullptr-error 18d22b014b
feat(sandbox): expose tenant skill management over HTTP and in the console (#2789)
* feat(sandbox): enhance skill management in sandbox configurations

Updated the skill management system to allow listing, installing, and removing skills within sandbox configurations. The ListSkills function now accepts a sandboxConfigID to filter skills based on the selected configuration. Introduced a new SandboxSkillsDrawer component for managing skills through the UI, along with necessary updates to the API and localization files to support these features.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: keep skill maintenance sessions out of the console session list

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: store and expose the installer transcript locators

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: record the skill installer agent's conversation

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: stream and persist the installer transcript during a skill install

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: expose install transcript locators on the skill type

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add the floating skill install transcript window

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: open the install transcript from the sandbox skills drawer

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): 重构技能安装体验,支持内联时间线、编辑第4步及保留镜像指针

- 替换右下角浮窗为内联安装时间线 (SkillInstallTimeline.vue)
  - 新增 GET /api/v1/sandbox-configs/:id/skills/:skillId/transcript 接口,支持回放与追尾
  - 移除 /sessions/continue-stream 依赖,避免 404 问题
  - 事件日志以 install_prompt 开头,不再将工具调用 preamble 拼入最终答案
  - 删除 SkillInstallTranscript.vue,技能条目下可展开/折叠时间线

- 编辑向导新增第 4 步「管理技能」
  - Cube/E2B 环境:连接 → 模板 → 运行配置 → 技能(第4步)
  - Docker/Local 环境:无第4步(无快照镜像)
  - 首次创建必须按 1→2→3 顺序,第3步保存成功后自动进入第4步
  - 已有配置支持步骤条数字跳转,可直接访问任意步骤
  - 卡片菜单「管理技能」打开同一抽屉并定位至第4步
  - SandboxSkillsDrawer.vue 重命名为 SandboxSkillsPanel.vue,去除独立抽屉壳

- 修复保存运行配置时冲掉 skill_image 的问题
  - 顶部展示信息(基础模板/快照ID/版本/构建时间)读取 config.skill_image
  - 编辑器保存 payload 不再包含 skill_image,合并时保留已有 SkillImage 和 VolumeMount
  - MergeSandboxConfigForUpdate 增加对 SkillImage 和 VolumeMount 的保护
  - 已冲掉的配置需重新安装技能才能恢复镜像指针

- 「查看安装过程」在安装开始时即显示
  - 技能列表中出现「正在安装」状态即展示按钮
  - 若记录尚未写入,展示等待提示并自动重试,避免误报「没有留下记录」

* 沙盒内技能目录从 UUID 改成 SKILL.md 的 name。

* fix(sandbox): heal stuck skill rows and block planted maintenance sessions

The reaper left a mid-upgrade install spinning forever when the old snapshot was still live, and restored removals to ready after the pointer had already moved. Client session create/update can no longer hide chats behind the skill_maintenance: prefix.

* style: gofumpt the skill reaper test stubs

* style: satisfy golangci on the tenant skill admin API

* feat(sandbox): rebuild skill images at turn boundaries, not mid-exec

Hold a session turn lease through AgentQA so an install cannot tear down
the live sandbox between tool calls; the next turn's first resolve picks
up the new image. Configs can instead keep existing sandboxes and boot
the new snapshot only for new sessions.

* fix(sandbox): judge stuck skill runs from the snapshot chain

InstalledSnapshotID only records the generation that installed a skill; later
installs of other skills advance the live pointer without rewriting older rows.
Comparing those IDs made the reaper fail healthy installs and delete live
skills whose removal had stalled.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-25 11:28:12 +08:00
hailongzhao 67cdf26216 feat(sandbox): add tenant skill install and remove service layer
Introduce the hidden skill-installer agent and TenantSkillService so a
sandbox config can install or remove skills into snapshot images under a
per-config lock, without yet exposing HTTP or UI entry points.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 17:51:13 +08:00
wizardchen 7f11b7b5b9 fix(sandbox): keep skill image pointer out of sandbox config PUT
The settings form rebuilds the payload without skill_image, so merge
must ignore client values or a save would drop the snapshot (and a
crafted PUT could plant one). Also reject skill names that escape
the image root and require SupportsSnapshots on the snapshot manager.
2026-08-24 16:00:23 +08:00
hailongzhao 00fe035668 feat(sandbox): add tenant skill snapshot infrastructure
Give Cube/E2B a shared snapshot API and persist skill image pointers so later session sandboxes can inherit installed skills from a config snapshot instead of the base template.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 16:00:23 +08:00
wizardchen 9c5c52f0a4 feat(sandbox): Docker 后端支持会话级长驻容器与远程守护进程
将 Docker 沙箱后端从一次性 exec 容器改为会话级长驻容器,使其会话与文件
系统语义与 E2B / Cube 后端对齐:

- 新增 docker_engine / docker_remote_client / docker_rpc_timeout,直连
  Docker Engine API,支持 TCP+TLS 远程守护进程,并为拉镜像等慢操作单独
  设置超时;daemon 地址留空时跟随本机 docker context。
- 新增空闲清理器 docker_idle_sweeper,按 exec 刷新的活跃标记回收长驻
  容器;标记文件对沙箱非 root 账号可写。
- 会话容器补 PID1 以回收僵尸进程;标准镜像补入 curl。
- 镜像即模板:docker_template_catalog 只上报可识别的沙箱镜像,标准模板
  的拉取在后台进行并报告 building。
- 设置页与租户配置支持守护进程地址、TLS 与 CPU/内存限额,四种语言文案
  同步;沙箱健康检查覆盖 Docker 后端。
- 补充 docs/sandbox-docker-backend.md 与 POC,说明能力边界与快照语义;
  git 钩子的变更包检测跳过嵌套 module,避免拿 POC 的包去跑主 module 测试。
2026-08-24 12:22:58 +08:00
ochan.kwon 894cbec48a feat(agent): report and attribute per-turn LLM token usage
Accumulate token usage across every LLM round of an agent turn and
expose the aggregate everywhere a turn is observable:

- AgentState.TurnUsage accumulates each round's reported usage,
  including the final-answer synthesis call; TokenUsage.Accumulate
  keeps the cache counters as prompt subsets and recomputes the
  combined cache status (unsupported survives only while every call
  was itself unsupported).
- The agent completion event carries the aggregate, and the complete
  stream event exposes it both as data.usage and as a typed
  StreamEvent.Usage that buildStreamResponse promotes into the
  previously-unused StreamResponse.Usage wire field.
- The assistant message persists it as a nullable usage column on
  both migration targets (postgres 000085 jsonb, sqlite 000012 text),
  for web and IM channels alike, so history reads still carry the
  turn's cost after the live stream is gone.
- "[LLM Usage]" log lines gain ", session_id=..., principal=type:id"
  when the context carries them; calls outside sessions stay
  byte-identical.

Usage from failed retry attempts is not accumulated: the per-call
usage log remains the ground truth, while the turn aggregate serves
as the operational signal.
2026-08-24 10:36:19 +08:00
MidoriKurage ba03be775f
feat(system-admin): add endpoint for creating users (#2722)
* feat(types): add AdminCreateUserRequest for system-admin user creation

Password un-tagged so an empty value reaches the service and triggers
OIDC-style random-password generation.

* feat(system-admin): scaffold POST /system/admin/users/create endpoint

- Handler skeleton under the SystemAdmin()-guarded group
- Binding and validation live (400); happy path 501 until the
  service logic lands
- Response type declares the once-only generated_password contract

* test(system-admin): contract tests for user creation

- Create/audit/generated-password cases assert the final contract
  and are intentionally red against the 501 stub
- Binding and trim validation cases already pass
- Adds the system.user_created audit action as the audit contract

* feat(frontend): add createSystemUser API wrapper

Mirrors promote/revoke/reset-password conventions, including the
once-only generated_password contract on the response type.

* fix(system-admin): never rewrite password; re-check trimmed username

- Handler no longer trims the password: whitespace is part of the
  credential, hashed byte-for-byte (mirrors auth.go Register)
- Trimmed username re-checked to 2-50 runes (binding ran on the
  raw JSON)

* feat(service): implement AdminCreateUser with random-password fallback

- Empty password: OIDC-style crypto/rand generation, returned once
- Explicit password: must satisfy the 8-32 letter-and-number policy
- Delegates to Register: duplicate checks, tenant provisioning and
  Owner membership bootstrap match public registration

* feat(system-admin): implement create-user handler

- Resolves tenant mode from the shared auth.default_tenant_mode
  policy (single source for registration and admin creation)
- Emits system.user_created audit without the plaintext password
- 201 + once-only generated password; 400 for policy/duplicate,
  500 for internal failures

* test(service): whitespace-only password must be rejected

- All-whitespace password: ErrPasswordPolicy, nothing persisted
  (matches AdminResetPassword); only truly empty triggers generation
- Red against the current TrimSpace-empty check
- Handler test drops the whitespace-only 201 expectation, keeps
  byte-for-byte pass-through for spacey valid passwords

* fix(service): treat only a truly empty password as generate-request

Generation now triggers only on the empty string; whitespace-only
input is rejected by the 8-32 policy like AdminResetPassword.

* fix(service): regenerate until the generated password passes policy

A 32-char base64url draw misses digits ~0.4% of the time and could
400 a creation nobody picked a password for; regenerate until the
policy passes (expected ~1.004 draws).

* refactor(system-admin): drop TenantProvisioning from create payload

The value lived both on the request and as the service argument;
keep it only as the argument, mirroring LoginWithOIDC.

* test(system-admin): pin password_generated audit value; cover dupes

- Audit assertions check the exact password_generated boolean
  (true generated, false caller-supplied)
- Duplicate mapping table-driven over email and username errors

* feat(system-admin): explicit provide-and-validate password semantics

- Password pointer-typed: absence (or null) triggers random
  generation; any provided value, empty string included, is
  policy-checked and rejected otherwise
- Service: generation on nil only, all provided values hashed
  byte-for-byte
- Tests: empty string -> 400, absent -> generate, nil pass-through
  at the handler boundary
- Frontend: document that any provided value is policy-checked

* test(system-admin): merge duplicate rejection tests

Whitespace-only and weak-password rejection tests were structurally
identical, differing only in inputs. Merged into one table-driven
test covering weak, empty, whitespace and whitespace-only values.

* feat(system-admin): idempotent create on duplicate identity

- Register returns typed sentinels ErrUser{Email,Username}Exists
- AdminCreateUser re-queries the existing row, lookup targeted at the
  sentinel-named identity (email or username)
- Handler: duplicate sentinels -> 200 + UserInfo with idempotent=true
  audit, policy errors stay 400; drops the substring-matched branch

* fix(system-admin): reject partial identity conflicts on user create

Idempotent 200 now requires both email and username to match the existing
row; partial collisions return 409. Align handler identity sanitization and
password docs with nil-or-null generation semantics.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-20 19:51:10 +08:00
Anky 0eb79c25e4
feat: support editing document summaries (#2752)
* feat: support editing document summaries

* fix: address review feedback for manual summary editing

Reset summary edit state when the drawer closes or generation is in
flight, sync summary_status on manual description updates, and add
service tests plus API docs for partial PUT behavior.

* chore: gofmt client/initialization.go for pre-push hook

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-20 14:54:53 +08:00
lawrence 28dd15ec30 fix(retrieval): 分段中删除的图片不再被检索到
从分段里删掉一张图片后,这张图仍会回到检索结果和模型上下文里。

三个问题串在一起:

1. 删图时 syncEditedChunkImages 会把 image_ocr / image_caption 子块置为
   禁用,但 CollectImageInfoByChunkIDs 查子块时不看 is_enabled,又把它们的
   image_info 挂回了父块。knowledge_search 工具随后把每条 image_info 无条件
   渲染成 markdown 图片写进 LLM 上下文,图片就这样回来了。这里按 is_enabled
   过滤,两级遍历都跳过禁用子块。过滤放在 collector 而不是工具输出层:纯图片
   分块的正文本身就是 OCR 文本、没有 markdown 图片语法,按正文过滤会误杀。
   chat_pipeline 里既有的 fake 仓储 fixture 同步补上 IsEnabled: true:它们不
   走 DB 默认值,零值 false 会被新的过滤当成已禁用子块。

2. UpdateImageInfo 补建 image_ocr / image_caption 子块时没有设置 IsEnabled。
   CreateChunks 用 Select("*") 插入,gorm 的 default:true 不生效,零值 false
   直接落库,这些子块一出生就是禁用的。补上 IsEnabled: true —— 否则第 1 点
   的过滤会把它们永久丢弃。其余 CreateChunks 调用方均已显式设置该字段。

3. 摘要刷新每次都新建一条 summary 分块,旧的留着且仍然启用。原因是它复用了
   ListChunksByKnowledgeID 的结果,而那个查询硬编码 chunk_type = 'text',
   "查找已有 summary 分块"的分支永远匹配不上。残留的旧摘要描述的是编辑前的
   文档,会让用户已经删掉的内容继续被检索到。新增按 chunk_type 过滤的仓储
   方法,刷新时原地更新已有 summary 分块。
2026-08-20 12:01:28 +08:00
LHMQ878 35820ebb16 fix: localize builtin agent name/description from YAML when reading DB rows
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 11:54:03 +08:00
Zhang GH aac4f1586b
Fix/hybrid search match count normalization (#2750)
* fix(hybrid-search): normalize MatchCount once instead of at the truncation site

An omitted match_count deserializes to Go's zero value, which reached the
truncation step in HybridSearch and sliced the deduplicated chunk list to
[:0] — a successful retrieval returned an empty result set. A negative
value panicked on that same slice bound and surfaced as a 500.

Resolve the value once at the top of HybridSearch via normalizedMatchCount
so all three consumers agree, rather than patching the truncation site
alone: the over-retrieval floor, the FAQ iterative-retrieval trigger
(which never fired on this path because len(chunks) < 0 is false), and the
truncation itself.

Bound the FAQ iterative path while here, since normalization is what makes
it reachable with a large MatchCount: its TopK seeds from MatchCount*3 and
doubles five times, previously without limit. It now stops at the pool cap
instead of re-issuing an identical query.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(retrieval): share the default retrieval depth as one constant

The value 50 was written three times — the over-retrieval floor in
HybridSearch, the MatchCount fallback added alongside it, and
RetrievalConfig.GetEffectiveEmbeddingTopK — with nothing recording that the
three are meant to agree. Anyone tuning one would silently desync the
others.

Promote it to types.DefaultRetrievalTopK. No behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: drakezhang <drakezhang@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 11:36:38 +08:00
wizardchen 9ed6266264 fix(chat): 持久化快速问答的检索步骤,刷新后不再丢失时间轴
快速问答的检索步骤只存在于 SSE 事件流里,刷新后前端只能靠 knowledge_references
反推时间轴,于是「检索了但没命中」的那一轮回到历史时步骤全部消失,只剩一条
孤立的记忆行。

改为在快速问答模式下订阅 pipeline 的工具事件,把 query_understand /
knowledge_search / attachment_parsing / image_analysis 落到 agent_steps。只在收到
tool_result 时写入,避免被中断的步骤刷新后显示成已完成。

这些调用来自 pipeline 而非模型,ID 加 ragpipe- 前缀并在 Agent 历史回放里过滤掉,
否则会把模型没发起过、甚至没被提供的工具调用塞进请求协议。
2026-08-13 21:34:21 +08:00
MidoriKurage 236d8c098f
fix(datasource): perform real sync deletions scoped per data source (#2690)
* test(datasource): cover scoped sync deletions

- deleted connector items remove the matching KB knowledge, scoped per data
  source via metadata datasource_id + external_id
- SyncDeletions=false neither deletes nor counts
- stream handler classifies deleted/failed items like the batch loop

* fix(datasource): perform real sync deletions scoped per data source

- applyFetchedItem now performs the KB deletion instead of only counting
- lookup is scoped to datasource_id + external_id so identical external IDs
  from two data sources cannot collide or overwrite each other
- ingestItem update-path lookup uses the same scoped query

* fix(datasource): note failed deletions retry only on full sync

- SyncResult.DeletionFailed tracks deletion failures (subset of Failed)
- streaming partial message warns that failed deletions only retry on a full
  sync, since the connector cursor is checkpointed past the deleted item

* fix(datasource): humanize deletion errors, keep raw detail in logs

- raw lookup/delete errors go to server logs only, with full context
- UI samples carry a humanized "see server logs" fallback
- add datasource.syncError.* i18n keys

Note: ko-KR / ru-RU syncError translations were generated without a
native-speaker review.

* fix(datasource): hard-delete synced rows and tighten deletion scoping

Sync-internal deletions now call HardDeleteKnowledge after the soft-delete
cascade so tombstones cannot block re-sync; subtree sweeps and URL metadata
attach failures are handled with datasource scoping and explicit errors.

* chore: add git hooks aligned with CI and PR checks

Install via scripts/install-git-hooks.sh (sets core.hooksPath). Pre-commit
runs whitespace/gofmt/golangci-lint on staged files; pre-push mirrors
app/frontend/cli workflows on the diff since origin/main.

* fix: gofmt test file; tighten pre-push to match CI app workflow

Pre-push now gofmts the full PR diff and runs go vet on all app packages
(like app.yml), not only Go files in the latest commit. Previous pushes
skipped checks via --no-verify and because hook-only commits had no .go files.

* fix(hooks): align pre-push diff with GitHub PR base via gh

Use three-dot diff against the open PR baseRefOid (same as CI) instead of
a two-dot merge-base range; run full go vet on every push with Go PR changes.

* style: gofumpt datasource_service_test.go

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-13 19:52:44 +08:00
wizardchen 98239de0d2 refactor(context): replace middleware.EmbedChannelContextKey with types.EmbedChannelContextKey
- Updated test files to use types.EmbedChannelContextKey instead of middleware.EmbedChannelContextKey for consistency.
- Introduced context cloning decisions for EmbedChannelContextKey to ensure proper context management during background tasks.
- Added tests to verify the correct behavior of context key declarations and their cloning decisions.
2026-08-13 18:00:02 +08:00
wizardchen 10cc694fed feat(memory): introduce long-term memory feature with i18n updates
- Added a new configuration option for agents to enable or disable long-term memory.
- Implemented UI elements in the Agent Editor for managing long-term memory settings.
- Updated internationalization files for English, Korean, Russian, and Chinese to include new labels and descriptions related to long-term memory.
- Enhanced backend logic to support memory retrieval and management based on user and agent preferences.
- Introduced a new tool for searching long-term memory, ensuring it respects the agent's memory settings.
- Added tests to verify the functionality of the new memory features and their integration with existing systems.
2026-08-13 18:00:02 +08:00
wizardchen fda037e7f3 feat(memory): add forced consolidation feature and related i18n updates
- Introduced a new method to mark memories as forced consolidated, allowing users to request immediate consolidation.
- Added a new message for users indicating that a tidy-up just ran and to try again shortly.
- Updated internationalization files for English, Korean, Russian, and Chinese to include the new message.
- Enhanced memory repository and service logic to support the new forced consolidation functionality.
- Updated database schema to include a new timestamp for forced consolidation.
2026-08-13 18:00:02 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00
jacentsao 64d17a0211 feat(im): process file attachments in QA 2026-08-12 16:08:51 +08:00
joeyczheng 664d193ca7 feat(datasource): add Tencent IMA datasource
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-08-12 14:56:25 +08:00
Easton Gu d4a16cc8f3 feat: support updating API key configuration
Signed-off-by: Easton Gu <xcgu59545@gmail.com>
2026-08-12 11:43:03 +08:00
lyingbug 517942c846
feat(sandbox): unify remote sandbox on E2B protocol with self-hosted support (#2655)
* feat(sandbox): unify remote sandbox on E2B protocol with self-hosted support

- Add e2b.proxy_url gateway config and generic gateway transport layer
- Add envd compat layer for Basic auth and multipart /files uploads
- Support Dockerfile.sandbox runtime and e2b-backend build targets in CI
- Rename sandbox image non-root user to align with E2B template convention
- Add sandbox-protocol.md and extend cluster deployment / local repro docs

Enables connecting to any E2B-compatible control plane (E2B Cloud,
CubeSandbox, Agent-Sandbox, etc.) without per-backend client code.

* style(sandbox): gofmt tenant_resolver struct alignment
2026-08-11 20:14:19 +08:00
Syt3s 5eaabf6bbb
feat: add Exa WebSearch (#2617)
* feat: add Exa WebSearch

* Fix test for Exa config options validation

* feat: add exa connection test
2026-08-11 17:05:52 +08:00
wizardchen 738fe1ec53 feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断
沙箱的后端类型、凭据、模板、超时和私网访问策略此前分散在部署级
WEKNORA_SANDBOX_* 环境变量里,一个实例只能有一套,租户之间无法各用
自己的 E2B 账号或自建 Cube 集群。现在这些配置全部落到空间维度的具名
沙箱配置上:部署环境不再提供覆盖项,未选择配置的智能体直接不启用脚本
执行,而不是静默回落到某个部署默认值——后者会让「这个智能体到底跑在
哪」无从判断。

同时把连接检测从「能不能连上控制面」扩展成可解释的诊断:
- E2B 标准模板显式带上 default 标签构建。创建沙箱时裸模板名/ID 只解析
  default 标签,构建带了别的标签就永远拿不到,表现为一个与「模板不存在」
  无法区分的 404。
- 模板状态不再从历史构建或 spawn 次数反推就绪,只认当前构建;构建完成
  却没有 default 标签的模板单独报成 untagged,因为它在列表里和「仍在构建」
  长得一样,但等待永远不会有结果。
- 创建失败带上 HTTP 状态,404 走模板解释路径,给出具体到「重建模板」还是
  「Key 属于其他团队」的结论。
- 探针脚本改在进程工作目录下暂存。系统临时目录在 macOS 上位于
  /var/folders,Docker 虚拟机并不共享,挂载进容器后是空目录,真实技能执行
  (脚本在 skills/ 下)却不会踩到,检测因此报假失败。
- 检测失败不再只说「命令输出与预期不符」,带上退出码与 stderr 首行。

前端把沙箱设置页与配置抽屉重做:凭据按「是否已配置」展示而非回显掩码,
删除与停用改为就地确认,占用情况走右侧抽屉,列表卡片展示模板状态、凭据、
超时等可判断健康度的事实。
2026-08-11 15:42:02 +08:00
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
jay3cx 8fc36d0265 fix(mcp): preserve omitted service update fields
Clarify partial update contract in tests.
2026-08-10 19:14:21 +08:00
wizardchen afd1ef8cf3 fix(invitation): harden auto-accept follow-ups from code review
- Reconcile stale pending invites after auto-accept AddMember
- Adopt invited workspace as home tenant when invitee is tenantless
- Await refreshFromAuthMe on SPA boot; sync auto_accept in hydrateSessionFromToken
- Document capabilities.auto_accept_invitation; extend client SDK and OpenAPI
2026-08-10 15:47:29 +08:00
Binks Riverton 355d161d64 feat: add Metaso web search provider 2026-08-10 13:33:55 +08:00
Xi(Darcy) Chen 090612da54
fix: pptx/ppt attachment parsing fails — missing parser engine + markitdown extras (#2621) 2026-08-09 16:23:36 +08:00
wizardchen 5494e669bc fix(auth): harden self-service change password after #2591
Address post-merge review: verify old password before policy checks,
reject same-password rotations, mark OIDC-provisioned users in preferences
and hide the self-service form with guidance, map API error details to
i18n, sync docs/api, Swagger, and the Go client SDK.
2026-08-07 15:49:58 +08:00
wizardchen f1ef364820 fix(knowledge): harden auto-tagging and let it respect manual tags
Automatic tagging shipped with a few paths that quietly disabled it and a
prompt that spent most of its budget on data the model never needed.

Robustness:
- Treat an omitted confidence as a deliberate selection instead of zero.
  Models frequently drop the field, and the 0.75 floor then filtered out
  every match, silently turning the feature off. Percentage-scale values
  are rescaled rather than accepted as out-of-range certainty.
- Classify the first 500 tags when a knowledge base has more, logging a
  warning, instead of dropping the task entirely. Large knowledge bases
  previously got no auto-tagging at all with no user-visible signal.
- Reuse common.ParseLLMJsonResponse for the model reply, and give that
  helper a balanced-scan fallback. The local extraction cut the payload at
  the last "}" or "]", so a trailing citation such as "see [1]" produced
  invalid JSON and burned the task's retries.

Prompt:
- Address candidates by ordinal rather than UUID. Tag names are short while
  a UUID is 36 characters, so this cuts candidate tokens roughly tenfold,
  keeps the model from having to reproduce (and corrupt) an id, and is what
  makes the raised 500 ceiling affordable. Ordinals are range-checked and
  mapped back to ids server-side.
- Drop the rationale field. Nothing read it, and at max_tags=10 it competed
  with the 1024-token output budget and risked truncating the JSON.
- Fence document text in <document> so file content cannot be read as
  instructions.

Behaviour:
- Add auto_tag_config.skip_if_tagged, defaulting to true, so documents the
  user already tagged at upload time keep their classification. Existing
  tags are now loaded before the model call, so a skipped document costs no
  LLM round-trip. Configs stored without the field adopt the default.

Also promote AddKnowledgeTagRelations to KnowledgeRepository instead of
reaching for it through a type assertion that failed at runtime, and route
the max_tags bounds through AutoTagConfig.Normalize so the prompt, the match
filter and the persisted config cannot drift apart.

Adds handler-level coverage for scope, cancellation, idempotency, ordinal
validation and the new toggle, plus SDK, Swagger and API doc updates.
2026-08-07 15:35:58 +08:00
lkx 926f1ac582 fix(knowledge): address auto-tag review findings 2026-08-07 15:01:01 +08:00
lkx 9426ee1470 feat: 增加自动添加标签的功能 2026-08-07 15:01:01 +08:00
MidoriKurage 0a3f6b0f21
fix(chunks): 流水线重写后的检索结果不用位置匹配 (#2558)
* chore(types): add internal ContentRewritten to types.SearchResult

* test(chunks): pin rewritten results out of the position path

Pipeline-replaced content (parent or neighbor expansion) must not trust
stale coordinates even when the replacement satisfies the length
invariant, and must not extend the merged group's range.

* fix(chunks): keep pipeline-rewritten results off the position path

Parent and neighbor expansion replace Content after retrieval. Without a
marker their stale coordinates could be incorrectly trusted.

Ensure ContentRewritten is false in chunkTrusted.
2026-08-06 14:32:24 +08:00
wizardchen c9dd5a574e chore(types): pin the gorm column for SearchResult.ContentRevision
The merge pipeline treats revision 0 as "never edited", so the field fails
open to the position path wherever it is not carried over. Make the column
explicit for direct row scans and document the remaining JSON gap.
2026-08-06 12:26:15 +08:00
MidoriKurage e2a75db67f chore(types): add internal only ContentRevision to types.SearchResult 2026-08-06 12:14:25 +08:00
PP1 f16a4ede93 fix(wiki): exclude archived pages from stats queries 2026-08-06 11:41:32 +08:00
Greenplumwine 439af21f9b
feat(feishu-driver): 新增飞书云盘数据源 (#2466)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-05 19:39:49 +08:00
ttommybot 0827cd6cf0 fix: retry document summaries before fallback 2026-08-05 19:29:32 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
qilifan 3507156929
feat(storage): support AWS S3 NoAk default credentials (#2008)
* feat(storage): support AWS S3 default credentials

* fix(storage): allow clearing S3 credentials

* fix static checks and trim locale diff

---------

Co-authored-by: qilifan <4359902+qilifan@users.noreply.github.com>
2026-08-05 11:59:02 +08:00
lyingbug 5fea7a1695 feat(knowledge): browse folders like a file manager and allow re-filing
The scope switch is gone. It existed because "select a folder" was ambiguous
in a list that could not show folders, and turning that ambiguity into a
control just moved the problem onto the user.

The list now shows what a folder contains: its sub-folders as entries first,
then the documents directly inside it. So the mode follows what the user is
already doing instead of a switch - browsing shows one level, and any active
filter searches the whole subtree flat, with a breadcrumb note saying so. A
document uploaded on its own needs no special row either: it sits at the top
level next to the folders, exactly where it is.

Folders can also be adjusted now, which they could not before - a mis-filed
upload previously had to be deleted and re-uploaded. Documents move via the
row menu or the batch bar into an existing or newly typed folder, and a
folder can be renamed in place, carrying its subtree. Both are plain
folder_path updates: nothing is re-parsed, re-chunked or re-embedded.
2026-08-04 20:45:29 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
shangfeng-liu 583b87b1b9 fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00
wizardchen ab9f705843 fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
wizardchen 123f2d6349 fix(session): include external users in API session group 2026-07-30 11:56:05 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
ochan.kwon c15c3e972f fix(retriever): rebuild a missing vector store engine on demand
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:

  - Startup skips a store whose engine fails to construct and never
    retries it, so that instance cannot serve the store again -- not
    even after a restart, if the backend is slow to come up.
  - A store registered on one instance is absent on every other until
    they restart, because registration is not broadcast.
  - Registration can fail on the instance handling the create, leaving
    the store missing even there.

Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.

Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.

Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.

Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.

Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.

For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.

A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.

Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.

Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
2026-07-29 12:25:18 +08:00