Commit Graph

204 Commits (da4ef68d9d5327dfd236299d882f092943a1a165)

Author SHA1 Message Date
lyingbug da4ef68d9d
feat(plugin): standalone plugin hosts with Redis discovery (P2 batch 4) (#3725)
* feat(plugin): standalone plugin hosts with Redis discovery
2026-09-26 14:46:26 +08:00
lyingbug 728007ada9
feat(plugin): remote runtime, Python SDK and python host plugins (P2 batch 3) (#3724)
* feat(plugin): remote runtime
* feat(plugin): Python SDK and python host plugins
* fix(pluginsdk): skip reverse DNS when a Python plugin binds
2026-09-26 14:46:03 +08:00
lyingbug ff49f9491d
feat(plugin): Host API with a key-value store, and document parsers from plugins (#3721)
* feat(plugin): Host API with plugin tokens and a key-value store
* feat(plugin): document parsers from code plugins
* docs(pluginsdk): parsers and the Host API
2026-09-26 14:45:39 +08:00
lyingbug 4328ab26a9
feat(plugin): code plugins — extension protocol, Go SDK, plugin host, web search and data source adapters (#3720)
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
2026-09-26 14:45:15 +08:00
lyingbug ed63e35cc7
feat(plugin): install declarative plugins — MCP servers, skills and model vendors (#3716)
* feat(plugin): install declarative plugin packages across nodes
* feat(plugin): serve installed plugins' MCP servers in each workspace
* feat(plugin): let installed plugins contribute skills and model vendors
* feat(plugin): register plugin skills into the workspace skill catalog
* feat(frontend): manage installed plugins and their configuration
* test(weaviate): read the class state before the probe barrier
* fix(plugin): keep MCP tool directories reachable behind the plugin wrapper
* test(plugin): wrap a long line
2026-09-26 14:44:34 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
wizardchen 8a75bece6d fix(mcp): close MCP connections on shutdown without blocking exit
MCPManager.Shutdown was never registered with the resource cleaner, so
remote sessions were left to time out. CloseAll also disconnected clients
one by one while holding the lock; it now detaches them, disconnects them
concurrently and gives up after a bound.
2026-09-25 18:22:14 +08:00
nullptr-error e2433b1293
feat(lite): install and run skills on the host (#3661)
* fix(desktop): keep Lite signed in with a stable JWT secret

Auto-setup fell through to the login form, and a newly generated signing key replaced the HMAC already used for presigned links. Reuse an existing HMAC, fill only a placeholder JWT secret, and clear a sticky auto-setup failure so the desktop window can enter the local account.

* fix(desktop): make wails dev start locally and store uploads at home

The dev app has no copied config, and the repo env asks for Redis and the Docker path /data/files. Stay on the repo config, use the in-memory stream manager when Redis is unset, write uploads to ~/.weknora/data/files, and allow dragging the top of the window.

* feat(lite): install and run skills on the host instead of a remote sandbox

Lite has no sandbox image, so skills are stored under ~/.weknora/skills as version directories with a live symlink, and chats execute them in the host Seatbelt. The settings UI no longer asks which sandbox to install into. The base profile opens /tmp, chat commands may use the network, and a failed ready-write rolls the symlink back to the previous version.

* fix(ci): align Lite skill checks with the JWT split and lint limits

The desktop signing test still expected the session secret to equal the AES key, and the skill settings source test still looked for the old i18n keys. Wrap the lines golangci-lint rejects so the PR checks can pass.

* fix(ci): separate multi-line test helpers for gofumpt

gofumpt requires a blank line after a multi-line function. The host skill fakes were still packed together, so lint failed on the first one.

* fix(lite): address review findings on host skill install

- Ask mode denies network again, and /tmp writes open only with a
  writable root, so nothing runs unapproved.
- Lite still reports and accepts the workspace script switch; the
  Skills page offers to re-enable it when it is off.
- The host process refuses symlinks the installer planted before it
  clears the install cache or reads the env declaration.
- An existing signing or AES key stays the signer and is never copied
  into signing.key; sessions then get their own jwt.key.
- Skill env capture writes under the host target on Lite.
- Sweep keeps the previous version and skips skills another process
  holds the lock for.
- Terminal and desktop panels never resolve remote sandboxes on Lite.
- A detached install stream resubscribes instead of freezing.
- Upload migration follows the bundle name and the old relative path.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-24 21:30:34 +08:00
lyingbug 967ed09706
feat(models): add system admin model catalog console (#3662)
* feat(models): add system admin model catalog console

System administrators can now maintain the model catalog from
Settings → System administration → Model catalog instead of editing
models.json on every replica. Administrator changes are stored as a
versioned overlay in the database (new model_catalog_config table),
layered between the deployment file and explicit per-model settings,
and synchronised to other instances within about 5 seconds.

Backend
- GET/POST preview/PUT /system/admin/model-catalog with optimistic
  version + deployment baseline checks (409 on conflict), 20-version
  history and audit logging.
- Console overlays reject credentials, headers, env interpolation and
  non-inline icons; those stay deployment-owned.
- Catalog views include each chat model's resolved thinking levels and
  the vendor defaults, so the console never re-implements level
  inheritance.

Frontend
- Model list in the shared data-table style with provider/type filters,
  "modified only" filter and source tags (built-in / deployment / admin).
- Edit drawer saves and applies immediately: display name, context
  window, max output, input modalities, reasoning, thinking levels,
  embedding dimension and picker visibility, each with its inherited
  default and a layer comparison table.
- Add-model drawer, JSON editor with server-side change check before
  publishing, and version history with one-step restore.
- Model settings gets a system-admin shortcut to the catalog; model
  editor reloads providers when opened so published catalogs apply.

* fix(models): harden the admin model catalog console

- Reject console base_url, base_urls, url_patterns and auth: moving a
  vendor's default URL would send its deployment api_key (and keys of rows
  without their own base URL) to another host. With every interpolated
  field now deployment-owned, '$' in names and descriptions is plain text.
- Drop deployment base URLs from catalog views; they may carry
  interpolated tokens.
- Remember a stored version this replica cannot compile and skip it until
  the version changes, instead of re-reading and recompiling every poll.
- Resolve vendor default thinking levels with reasoning forced on, like
  per-model levels.
- Compute builtin/deployment views once; preview returns only the
  candidate catalog and normalized overlay.
- Share deployment file lookup via runtime.ReadDeploymentOverlay.
- The model editor force-refreshes its own type instead of resetting the
  provider store for every type; the refreshed list wins in the id index.
2026-09-24 16:31:45 +08:00
lyingbug e179f44445
refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589)
* refactor(models): separate providers, catalog and runtime configuration

* refactor(models): isolate provider definitions and runtime catalogs

* docs: exclude refactoring analysis notes from the pull request
2026-09-23 14:20:08 +08:00
Captain 171097f06e
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS

A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).

SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.

Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.

Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.

Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.

docreader enforces the same switch in its own validator, and compose
passes the variable to both services.

Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.

The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.

Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.

The switch defaults to off, so nothing changes for deployments that do
not set it.

* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs

docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.

The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 12:02:06 +08:00
lyingbug 63ef76dd80
docs: 迁移有效文档至 website-docs 并清理旧文档 (#3580)
* docs: consolidate maintained documentation in website-docs

* docs: remove migrated legacy documentation and update remaining references
2026-09-22 23:25:40 +08:00
nullptr-error 22cbcdfa57
feat(sandbox): run unpinned Lite sessions on the host OS sandbox (#3549)
* feat(sandbox): resolve workspace paths through WorkspaceLayout

Keep remote sessions on the /workspace contract, while file, shell, and
artifact code follow the session layout instead of hardcoded paths.

* fix(sandbox): fail closed on host layout lookup and restore the shell blacklist

Treat host vs remote as an explicit origin, refuse inspect outside ReadRoots,
and skip collection when the output tree is missing or equals the project root.

* fix(sandbox): close leftover fail-open holes in the layout path

Keep host copy, collection, and prompt text from falling back to /workspace when a lookup fails or an adapter returns unclean roots.

* fix(sandbox): satisfy golangci-lint on the layout helpers

Wrap new copy and test lines to 120 columns, and add the exported comments revive requires.

* fix(sandbox): gofmt the read_file layout copy wrap

* feat(sandbox): add host as an OS-enforced backend type

Host is not a named/pinnable config: each command is a fresh local process, unlike Docker, Cube, and E2B.

* feat(session): persist Lite host workspace directory

Empty means an auto-allocated folder. Standard edition leaves the column unused so the shared session model still loads. SQLite also gains the tenant skill tables Lite shell_exec already expects.

* feat(localsandbox): adapt the host OS sandbox to session interfaces

File tools stay outside Seatbelt and are bounded by PathGuard. A revoked project directory fails closed instead of allocating a replacement workspace.

* feat(sandbox): run unpinned Lite sessions on the host backend

Lite opts into host explicitly; web stays disabled when no named remote config exists. Host workspaces are not destroyed or auto-committed.

* feat(desktop): pick and persist host project directories

The native folder dialog is the authorization path; typed paths are rejected. settings.sandbox.host tells the client when this edition can use it.

* fix(sandbox): fail closed on Lite host layout lookup misses

Unpinned host sessions were described as /workspace when pin read or
sandbox resolve failed. Prompts and attachment staging now share the
same fallback so Lite never pretends the remote tree exists.

* fix(frontend): keep settings.sandbox.host in sync with the backend keys

TestDeploymentCapabilityKeysMatchFrontend requires the lists to match. Host stays fail-closed like docker so a missed probe does not show Lite-only UI.

* fix(lint): satisfy golangci-lint on the host sandbox wiring

Check c.Error, wrap lines over 120 columns, put context first on versionsWorkspace, and document exported adapter methods.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
2026-09-22 19:35:03 +08:00
wizardchen 2d37f8fd7d fix(container): wire shared agent lookup for MCP handler 2026-09-21 14:38:54 +08:00
Leo Dawn a3c7be10ca
fix(agent): scope the @Skill and @MCP pickers to the agent that will run (#3475)
* fix(agent): scope the @Skill and @MCP pickers to the agent that will run

Both pickers listed the caller's own workspace, but a shared agent invokes its
OWNER's skills and MCP services. The two never line up, so for every shared
agent:

  * @Skill was always empty, in every skills mode. GET /skills resolves the
    sandbox config with (tenant, id), and the agent's config id does not exist
    in the borrower's workspace - effectiveTenantSkills short-circuits before
    it reads a single skill row. The response still reported
    skills_available: true, so the category appeared with nothing under it.
  * @MCP in "selected" mode was empty for the same reason: the preset holds the
    owner's service ids, which match nothing in the borrower's catalog.
  * @MCP in "all" mode was worse than empty. isMCPAllowedByAgent returns true
    for every enabled service, so the picker offered the BORROWER's own
    services; resolvePerRequestMCPScope then intersected the mention against
    the agent's preset, found nothing, and dropped it with only a warning.

Resolve the agent when the request carries agent_id + agent_source_tenant_id
and read its resources where they live. Only an explicit source workspace takes
the share path, so an own-agent request is byte-for-byte unchanged; built-in
agent ids exist in every workspace, and falling back to a share lookup without
one would let an org member take over the caller's default agent, exactly as
session.Handler.resolveAgent already documents.

Both endpoints serve only what the agent can actually invoke:

  * skills: the sandbox config comes from the agent, never from the query, so a
    borrower cannot name another config in the owner's workspace and enumerate
    it; "selected" narrows to the configured names and a disabled picker never
    reaches the owner's inventory at all.
  * MCP: the agent's explicit preset and nothing else, because that is exactly
    what resolvePerRequestMCPScope accepts for a shared agent. "all" therefore
    offers nothing rather than offering services every mention would drop.
    Whether "all" should become mentionable is a separate decision about that
    guard, not about the picker.

The cross-workspace MCP rows use a narrowed response built field by field.
NewMCPServiceResponse reveals endpoint URL, headers, env vars and stdio command
to an Admin, and a borrower can be an Admin of their own workspace - that shape
must be narrow by construction, not by role.

On the client the two lists now follow the selected agent: their cache keys
carry it, and the chat input refetches on a switch behind a request token, so a
slow response for the agent the user just left cannot overwrite the picker with
another workspace's services.

* fix(agent): check the c.Error return on the shared-agent picker paths

golangci-lint judges pull requests on the lines they introduce, so the
repository's older bare c.Error calls are grandfathered while these three are
not. Match the _ = c.Error form already used in mcp_service.go.
2026-09-21 13:02:34 +08:00
lyingbug c40a9dd194
refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470)
把集中式的 OpenAI 方言客户端重建为三层:协议层 internal/models/api/<protocol>(OpenAI Chat Completions / OpenAI Responses / Anthropic Messages / Google GenAI),目录层 internal/models/catalog,厂商层 internal/models/vendors/<id>(27 家,go:embed)。27 家厂商逐条对照官方文档核验,厂商事实以数据声明,协议层不出现厂商名。

既有行的运行时行为会变的四处:
1. extra_config.api 变为保留的协议选择器,非法取值被拒绝;
2. Azure 未填 api_version 的行改走 /openai/v1 GA 数据面;
3. api.openai.com 的一方流量改走 Responses 协议(中转/网关不受影响);
4. 7 家厂商的输出上限字段按文档纠正。

不需要数据库迁移。详见 PR 描述与 website-docs/03-features/06-models.md。
2026-09-20 21:28:23 +08:00
nullptr-error d4b9fd0c72
feat(session): in-place rewind with workspace checkpoint reset (#3469)
* refactor(session): extract shared workspace git reset helper

Fork bootstrap and in-place rewind both need the same /workspace reset
script. Move it out of fork_bootstrapper so rewind can reuse it without
copying ExecShellCommand plumbing.

* feat(message): add DeleteMessagesFrom for rewind truncation

Rewind needs a composite created_at+id cursor so same-millisecond
messages after the fork point are not left behind. Delete inside a
transaction and return the rows so callers can log the count.

* feat(session): add in-place rewind service

Truncate the current session at a user or assistant message and git-reset
the live sandbox when a reachable checkpoint exists. Reset failure is
fail-closed so messages are never deleted ahead of the workspace.

* feat(session): expose POST /sessions/:id/rewind

Wire SessionRewindService through the session handler and container so
the client can rewind in place. Busy sources map to 409; git reset
failures stay 500.

* feat(chat): add rewind API client and i18n copy

Expose rewindSession with the same 180s timeout as fork, and add
chat.rewind strings in all five locales so the confirm dialog is not
hardcoded Chinese.

* feat(chat): add rewind button with confirm dialog

Offer in-place rewind next to fork on user and assistant messages.
Confirm before truncating the thread, then reset locally and prefill
the original question when the cut point is a user message.

* fix(chat): confirm rewind with a compact popover

Replace the full-page confirm dialog with a small popconfirm above the
rewind button, and shorten the swagger description to satisfy line length.

* fix(chat): skip markdown images with an empty destination

Empty ![]() was rendered as <img src="">, which shows as a broken image in the reply.

* fix(chat): clip sandbox panel slide-in and rename open tooltip

Keep the enter animation inside the viewport so it cannot flash a document
scrollbar and jump the page. Rename the header toggle from sandbox terminal
to expand sidebar.

* fix(session): 回滚时完成截断、互斥发送并重拉历史

客户端中断不再让 git reset 成功后留下未删的消息;reset/删消息前复查 busy,并锁住当前会话输入。成功后重拉历史,避免切会话预填串台和 steer 分段截错。

* fix(session): 工作区 git 外置,并让未开箱 fork 的回滚对齐 SHA

checkpoint、回滚和 fork 共用 --git-dir=/var/lib/weknora/workspace.git,避免 rm -rf /workspace 丢掉历史。
未开箱分叉会话回滚时改写或清除 ForkBootstrap,避免第一次开箱仍 reset 到旧切点。

* fix(session): 回滚加会话锁,并堵住发送中回滚与脏 SHA

同一会话并发回滚互斥(进程内 + Redis),冲突返回 409。发送/流式中禁用回滚按钮;历史重拉成功后再改界面。未开箱 fork 的 checkpoint SHA 必须是 40 位 hex。工作区 git 只认外置 GIT_DIR,不再 mv/rm 根上的 .git。

* fix(session): 未开箱 fork 回滚成功时不要提示没有沙箱

ForkBootstrap 已对齐切点后返回 workspace_reset,前端走「已回滚」;清掉引导时改报 NO_CHECKPOINT。纯聊天仍用 NO_SANDBOX。

* fix(chat): stub composerLocked in steer interaction tests

handleSteerMsg now bails when rewind locks the composer. The VM harness
did not provide that ref, so CI threw ReferenceError on every steer case.

* fix(session): allow git-layout mkdir in pinned checkpoint test

SkipWorkspacePrep still omits the input/output bootstrap exec. The
checkpoint script now mkdirs WORK_TREE and GIT_DIR, so the old
"command must not contain mkdir" assertion was stale.

* fix(session): block send while rewind holds the session lock

BeginTurn now fails if the rewind key is held, so AgentQA cannot start
during git reset. Knowledge-chat and executeQA check the same lock
before persisting messages. composerLocked is still page-local.

* fix(session): close send/rewind races and fail closed without a checkpoint

Hold the turn lease before persist so rewind cannot truncate an in-flight
send. TryLockRewind fails while a turn is live; emptying the conversation
resets the workspace through the same git path as a SHA rewind.

* fix(session): wrap rewind reset call to satisfy line length lint

* fix(session): fail closed on a replaced sandbox and close leftover rewind holes

Do not truncate history when the live workspace cannot follow the checkpoint, retry deletes after a successful git reset, and keep send and rewind exclusive even without a sandbox manager.

* fix(session): address rewind review findings

Correctness:

- Adopt a pre-move /workspace/.git into SessionGitDir on first use. Sandboxes
  provisioned before the git dir moved out of the work tree still hold their
  checkpoints there, so every SHA recorded on those turns stopped resolving:
  rewind 500'd instead of truncating and fork's rollback failed on a bad
  object. Adoption is guarded on the new dir being absent and on the work-tree
  repo carrying the checkpointer's committer identity, so an agent's own
  `git init` in /workspace is left alone.
- Only fail a send closed when BeginSessionTurn reports a rewind lock. Every
  other lease error degrades to running without a lease, as before rewind
  existed; a Redis blip on the lease script was rejecting the user's message.
- Release an abandoned fork snapshot after the truncate lands, not before.
  Clearing the bootstrap column is a DB write a retry can redo; deleting the
  snapshot is not, and a delete that exhausted its retries left the fork with
  no workspace to boot from under a full transcript.
- Re-acquire the rewind lock when a renew call errors, not just when it
  reports the lease lost. Sitting the tick out let a short Redis outage expire
  the 2m key mid-reset and hand the session to a second rewind.
- Keep the history cursor when a post-rewind reload fails. Clearing it sent
  the next scroll-up back to the newest page, which the kept prefix already
  holds, instead of to older messages.

Efficiency:

- Add ListAssistantCheckpointsUpTo and use it to answer "which checkpoint does
  kept history still reach". Rewind was reading every message body and artifact
  row in the session to find one SHA.
- Take the send-side turn lease once per request. The handler already holds it
  before persisting; QARequest.TurnLeaseHeld lets the QA services skip their
  own, so IM/MCP still take one and HTTP send stops doubling every rewind check
  and lease round trip.
- Batch the suggestion delete behind DeleteByMessageIDs instead of one query
  per deleted message while the rewind lock is held.
- Resolve the rewind affordance for the whole transcript in one pass rather
  than two list scans per rendered row on every streamed token.

Cleanup:

- Drop the SANDBOX_REPLACED skip reason, which the service never produces (it
  fail-closes with a 409), along with its copy in all five locales.
- Drop GetMessagesBySession from rewindMessageStore; rewind never calls it.
- Drop the unused chat.rewind.confirmTitle copy from all five locales.
- Restore the chatHeader.toggleSandboxPanel wording this branch had changed
  unrelated to rewind.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-20 21:01:00 +08:00
Captain a5bdf82a02
fix(storage): let object storage transfers outlive 30 seconds (#3368)
* fix(storage): let object storage transfers outlive 30 seconds

The S3, COS, KS3, OBS and OSS file services build their SDK client from
DefaultSSRFSafeHTTPClientConfig, whose 30-second Timeout is Go's
whole-request timeout, body included. So every request had to finish
within 30 seconds whatever it carried: a 40 MB PDF needs about
11 Mbit/s of uplink to make it as one COS Object.Put (OSS uploads in
10 MB parts, so there the cap was per part), and the same file coming
back from a distant region was cut the same way, while the local backend
took any size (#3306). MinIO and TOS, which only hand the SDK a
transport, never had the cap.

Give the object storage clients a config without the whole-request
timeout, and bound connection setup on their transport instead (TLS
handshake, response headers, idle connections), so a peer that accepts
the connection and then goes silent still fails within about a minute;
the header wait starts after the request body is written, so it does
not cap an upload. A transfer is then bounded by its request context.
The bucket probe every constructor runs, which had relied on the client
timeout, gets a 30-second context of its own, and the OSS and KS3
connectivity checks get the 10-second deadline the other backends
already set. KS3 called its SDK without a context at all; it now passes
the caller's, so the same bounds apply there.

* fix(storage): bound object transfers and split bucket probes

Dropping the 30s client timeout left asynq/API transfers unbounded, and Head+Create shared one setup context so a slow miss aborted create. Add a 30-minute fallback deadline, give each probe its own timeout, and only create a KS3 bucket on a real 404.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-19 21:30:55 +08:00
lyingbug 76a19f14c4
feat(mcp): add built-in MCP server with per-workspace endpoints (#3371)
* feat(mcp): add built-in MCP server with per-workspace endpoints

WeKnora previously exposed itself to MCP clients only through the Python
package under mcp-server/, which binds one process to one API key and
therefore to one workspace, and mirrors the REST API tool-for-tool.

This adds a native MCP server surface to the Go backend:

- New mcp_endpoints table (PostgreSQL 000100, SQLite 000020). A workspace
  can publish any number of endpoints, each with its own bearer token
  (SHA-256 stored, rotatable, shown once), knowledge-base allowlist, tool
  allowlist, default agent and per-minute rate limit.
- Management API under /api/v1/mcp-endpoints (Viewer reads, Admin writes,
  API keys need manage_channels) plus a tool catalog endpoint.
- internal/mcpserver hosts one mcp-go Streamable HTTP server in stateless
  mode at /mcp/:endpoint_id. tools/list is filtered per endpoint and
  tools/call re-checks the allowlist, applies rate limiting and touches
  last_used_at. MCPEndpointAuth resolves the token and injects the
  workspace, an mcp_endpoint principal and an API-key style scope derived
  from the endpoint, so existing knowledge-base and capability checks
  apply unchanged.
- Redesigned tool set (12 tools in four groups): retrieval and reading
  reuse the agent's native tools; ask runs the default or a named agent
  synchronously and returns a cited answer with a session_id for
  follow-ups; wiki tools cover search, read and index; write tools are
  off by default. Knowledge bases accept ids or names.
- Frontend: new "MCP Server" integrations tab (tab id mcpserver, to
  avoid clashing with the existing MCP-client settings section) with
  endpoint cards, a create/edit drawer, a one-time token dialog and
  ready-to-paste snippets for Cursor/VS Code/Claude Desktop, Claude Code
  and mcp-remote. Locales updated in all five languages.
- Deployment capability integrations.mcpserver wired on both sides.
- Docs: website-docs MCP page now leads with the built-in server; the
  Python mcp-server README carries a deprecation notice.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): wire the MCP server into DI/router and tighten authorization

The previous commit lost eight staged files to the repo pre-commit hook,
which resolves ROOT to the main checkout and re-adds the main-checkout
copies of tracked files into the worktree index. This commit restores them
(container providers, RouterParams and route registration, context key,
principal, clone table entry, deployment capability key, SQLite parity
test) and addresses the review findings:

- Knowledge-base scope is authorized, not trusted: endpoint allowlists and
  every tool call resolve each knowledge base through access.ResolveKB for
  the calling principal (ownership or organization share). Foreign IDs are
  rejected on create/update and dropped at call time.
- Write tools mint explicit editor grants via access.KBAccess.WithGrant
  before calling the knowledge service, so add/update/delete go through the
  same RequireKBWrite path as the REST API. add_document also checks the
  grant itself.
- ask no longer accepts agent_id. It runs only the endpoint's configured
  agent; tenant-owned agents and user-facing builtins qualify, internal
  builtins (wiki fixer, skill installer) are refused both on save and at
  call time. Web search is forced off for MCP calls.
- Tool annotations match behaviour: ask is not read-only, add_document is
  a mutation.
- CORS allows MCP-Protocol-Version / Mcp-Session-Id / Last-Event-ID and
  exposes Mcp-Session-Id for browser-based MCP clients.
- Rate limiter runs its local cleanup goroutine; SQLite migration gains the
  deleted_at index for parity with PostgreSQL; non-ASCII endpoint names get
  unique mcpServers keys derived from the endpoint id.
- Swagger annotations on the management handler.
- Tests: KB isolation (service and server), write grants, endpoint-agent
  restriction, tool annotations, and a router test that mounts the public
  /mcp/:endpoint_id surface and checks the deployment capability.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(mcp): renumber migrations after rebase onto main

main took 000100 / 000020 for chunks_index_diet, so the mcp_endpoints
migrations move to 000101 (PostgreSQL) and 000021 (SQLite); the SQLite
parity test and docs follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): run document tools under the knowledge base owner tenant

Shared knowledge bases resolved through access.ResolveKB were only half
usable: documents of an organization-shared knowledge base are stored under
the owner tenant, so the document-level tools that queried by the calling
tenant found nothing, and add_document would have created rows under the
caller's tenant inside someone else's knowledge base.

- knowledgeInScope loads documents tenant-agnostically and then requires
  the document to belong to an allowed knowledge base and to that base's
  tenant.
- scopedKBContext replaces grantContext: it resolves the grant, switches
  the execution tenant to the knowledge base owner (KBAccess.Context) and
  swaps TenantInfo to the owner so storage/model settings resolve there.
  list_documents, read_document and every write tool run under it.
- Locale text and docs no longer claim clients can pass agent_id.
- Go SDK gains MCP endpoint methods (client/mcp_endpoint.go) with docs, and
  the Swagger artifacts are regenerated.
- Tests: shared knowledge base read/write under the owner tenant, spoofed
  document tenant rejected, viewer share cannot write.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* chore(mcp): renumber migrations after rebase onto main

main now carries 000101 / 000021 (knowledge_profiles), so the
mcp_endpoints migrations move to 000102 (PostgreSQL) and 000022 (SQLite);
the SQLite parity test expects version 22 and the docs follow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* fix(mcp): guard last_used touch without a repository and wrap SDK line

The transport tests build the server without a repository; the background
last_used_at write then dereferenced a nil interface and, on the slower CI
runner, took the test binary down. touchLastUsed now returns early when no
repository is wired, and the tests inject a recording repository and assert
the touch instead. Also wraps a 123-column SDK signature for the client
module's lll check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-17 22:40:05 +08:00
lyingbug 5574ebc011
feat(knowledge): derive knowledge base descriptions from document profiles (#3370)
* feat(knowledge): derive knowledge base descriptions from document profiles

Knowledge bases only had a hand-written description, which most users
leave empty, so agents bound to several knowledge bases had nothing to
route on and searched all of them. This adds a generated description
that is derived from per-document data instead of maintained by a model.

Document profile: the summary call now returns JSON with a 2-3 sentence
summary plus a gist, 3-5 topic keywords, a document type and one
typical question (knowledges.profile). Input is capped at 8k chars and
output at 1024 tokens; plain-text replies from custom templates still
work and simply carry no profile. The summary chunk embeds the gist and
short summary.

Knowledge base aggregate: a database-only fold over those rows (counts
per file type, tag, normalized topic and doc type; stratified samples of
titles and questions; a hash of the inputs). Deleting, moving or
re-parsing a document changes nothing but the rows, so the next rebuild
is exact without any model judgement.

Description text: one small model call over the aggregate yields a
gist, merged topics and typical questions (knowledge_bases.
generated_profile). The call is skipped when the aggregate hash is
unchanged; failures keep the previous text and record the error; an
empty knowledge base clears it. The manual description is never
touched, and the agent runtime context renders both so the model can
pass knowledge_base_ids instead of fanning out.

Refreshes are requested from summary task exit, post-process completion,
deletion, cross-KB moves and enabling the option, debounced per
knowledge base in 30 s windows via kb:profile on the summary queue.
POST /knowledge-bases/:id/profile/generate regenerates synchronously.
profile_config (enabled, model_id, custom_instructions) opts a document
knowledge base in; the settings dialog shows the generated card with
regenerate and adopt-as-description actions in five locales. Search
results now log hits per chunk type so the value of indexing summary
chunks can be measured.

Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the
SQLite schema test expects the new version and columns.

* fix(knowledge): keep generated descriptions out of duplicates and refresh clones

DuplicateKnowledgeBase copies the source row through JSON, which carried
generated_profile onto a knowledge base that has no documents yet, so
an empty duplicate advertised the source's topics. The duplicate now
starts without a generated description and keeps profile_config so it
derives its own once documents arrive.

CopyKnowledgeBase builds the target field by field and skipped
profile_config, and the clone task never asked for a description after
copying the documents, so a clone stayed undescribed until its next
upload. The target now carries a copy of profile_config and both clone
paths request a refresh on completion.

* fix(knowledge): refresh the loaded clone target and carry document profiles

ProcessKBClone requested the description refresh with the reservation
stub it builds before CopyKnowledgeBase runs. For the main create path
that stub has neither a type nor profile_config, so the refresh helper
rejected it and nothing was enqueued. The refresh now uses the loaded
target returned by CopyKnowledgeBase.

cloneKnowledge copied the summary text but not the structured profile,
so even a scheduled refresh would have aggregated titles without topics
until every clone was re-summarized. Cloned documents now carry an
independent copy of the profile.
2026-09-17 22:21:55 +08:00
nullptr-error 42e6163981
feat(session): fork conversations from a historical user message (#3304)
* feat(session): add fork schema and history queries

为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。

* feat(sandbox): add git, fork snapshots, and bootstrap hook

沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。

* feat(session): checkpoint workspace at the end of each turn

每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。

* feat(session): fork conversations and reap orphan snapshots

支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。

* feat(chat): add session fork entry, prefill, and badges

在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。

* style: add missing newline so session_fork_delete_test.go is gofmt-clean

* fix(session): prune later git history when rolling a fork workspace back

reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。

* style(session): clear golangci-lint findings on the session-fork diff

Pre-push lints the whole PR against main; wrap long lines, check c.Error,
and fix gofumpt/revive/staticcheck so the merge can be pushed.

* fix(chat): show the fork button on hover without shifting the thread

Overlay the action so revealing it no longer expands the user bubble and janks the messages below.

* fix(chat): drop the fork landing banners

Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread.

* fix(agent): dedupe artifacts by content hash, not size

Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only.

* feat(session): allow forking from an assistant answer

User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists.

* fix(session): resolve parent attachments from a forked chat

Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss.

* fix(agent): rewrite sandbox refs after a content-hash skip

Fork restores change mtime on unchanged files, so Collect returns empty
and named citations were left as sandbox: paths. Resolve those names
against session artifacts, notify only real uploads, and stamp restored
mtime only on the matching content hash.

* fix(session): drop a dead fork snapshot and retry checkpoint rewrite

Create from a missing template now abandons bootstrap so the next
Resolve uses a normal sandbox. A failed checkpoint rewrite no longer
marks the bootstrap consumed, which previously made nested forks
SANDBOX_REPLACED.

* fix(session): load forked parent attachments into the prompt

ResolveForPrompt used session-scoped lookup, so copied parent attachment
IDs passed Get but never reached the model. Use the same resolveInSession
path as preview.

* fix(chat): keep the user-message fork button visible

The control was hover-only, which hid it on touch. Always show it, and
drop the unused degrade tooltip.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-17 14:20:18 +08:00
serply e80f5dfbc7 feat(web-search): add Serply provider
Serply returns Google results through a keyed JSON API, which fills the
gap between Google Custom Search (needs an engine ID and a restricted
scope) and the scrapers. The adapter follows brave.go: hardcoded
endpoint, redirects refused so the key never leaves the official host,
4MB response cap, and per-call country and freshness filters. country
maps to Google's gl parameter and pd/pw/pm/py map to tbs qdr:d/w/m/y.
Serply has no date-range filter, so that form returns an error instead
of being silently dropped. num is clamped to 10 because the API serves
one results page per call.

Wiring matches the Bocha provider: type constant and metadata, registry
factory, service validation, frontend union, swagger enum entries (edited
by hand; a full swag regen drags in unrelated drift), and the web search
docs table.
2026-09-17 12:25:09 +08:00
jacentsao 01f0700b35
feat(datasource): add Confluence sync (#3287)
* feat(datasource): add Confluence sync

* fix(confluence): resume full sync and list Server pages without CQL

CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.

* fix(confluence): satisfy package lint for pre-push against main

* fix(confluence): parse Server page lists and keep version expand

The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.

* fix(confluence): classify errors and fail closed on empty listings

Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-16 14:38:16 +08:00
nullptr-error 6ccc5cedf1
feat(sandbox): add a graphical desktop to session sandboxes (#3245) 2026-09-14 20:37:37 +08:00
DEMO 5d3114fc9e
feat(datasource): add DingTalk Docs integration with safe sync and localization (#2313)
* feat(datasource): add DingTalk Docs integration

* fix(datasource): harden DingTalk sync and complete translations

* fix(datasource): close DingTalk review gaps in sync and rendering

Prefer millisecond node timestamps, keep full-sync deletion reconciliation, and stop dropping official block payloads so incremental updates and Markdown import stay complete.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-14 19:53:29 +08:00
lyingbug 7e53c31671
feat(browser): connect local Chrome with task controls and display modes (#3180)
* feat(browser): integrate BrowserSkill local task tabs and account pairing

* feat(browser): refine task routing, lifecycle, and display modes

* fix(browser): reclaim temporary task tabs and groups

* test(database): cover browser authorization migration schema
2026-09-11 17:59:27 +08:00
lyingbug 72581cb462 Revert "feat(sandbox): add safe live file management to session sandboxes (#3…"
This reverts commit 01205b3d56.
2026-09-11 17:42:08 +08:00
DEMO 01205b3d56
feat(sandbox): add safe live file management to session sandboxes (#3181)
* feat(sandbox): add safe live file management to session sandboxes

* fix(sandbox): keep live files from waking paused sandboxes

Serialize directory entries as "directory", refuse Files-tab Connect on paused runtimes, and cap list/delete trees so oversized workspaces fail closed.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-11 17:31:53 +08:00
nullptr-error 277d6ee851
feat(sandbox): add an interactive terminal and files tab beside chat (#3110)
* feat(sandbox): stream interactive PTYs from Cube and E2B

Give session sandboxes a provider-neutral terminal capability so a later
WebSocket layer can attach without knowing Cube vs E2B. Long-lived
streams use a timeout-free client; idle terminals keep the sandbox alive
by refreshing TTL, and a workspace can set how long to wait before
disconnect.

* feat(session): bridge browser terminals to the session sandbox

Mint a short-lived ticket over a normal authenticated POST so the browser
WebSocket never puts the access JWT in the query string. After upgrade,
recheck the bound token, membership, and session ownership so logout or
revocation tears the PTY down. Provisioning stays opt-in: a GET
handshake only creates a sandbox when the client sends provision=1.

* feat(chat): add a sandbox terminal panel beside the conversation

Open a side panel with an xterm attached to the session sandbox. Starting
the terminal is an explicit click so merely opening the panel does not
create or resume a microVM; reconnects stay lookup-only until the user
confirms create-and-start. Persist the remote shell PID in
sessionStorage so a refresh reattaches instead of spawning an orphan.

* feat(sandbox): let workspaces set terminal idle disconnect

Expose the idle-disconnect duration in the sandbox config editor so an
operator can close unused terminals and let the provider TTL pause the
sandbox, instead of leaving PTYs open for the built-in 15 minutes.

* feat(chat): show generated files in the sandbox side panel

Move the per-message artifact list into the first tab of the right-hand panel so generated files stay in view with the terminal. Keep the old drawer only for embedded chat.

* fix(sandbox-terminal): keep tickets out of logs and live PTYs across refresh

Gin was copying the handshake JWT into response_body logs, silent access-token
rotation dropped open terminals as AUTH_REVOKED, and first-use provisioning
missed shared agents. Local echo now skips password prompts.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-08 17:43:15 +08:00
lyingbug 647848f395
fix(agent): improve web search and persistent page reading (#3086)
* fix(agent): improve web search and persistent page reading

* fix(agent): make web fetch snapshots waitable and retryable

Keep in-flight page fetches independent of a caller's short deadline so
content=true and continuations cannot cancel a shared download or miss a
snapshot that is still loading. Mark snapshot_expired as retryable, fetch
only the first three search hits in parallel, and stop forcing Brave's
country to US.
2026-09-07 23:09:27 +08:00
Fei.Chen 15727591a7
feat(web-search): add Bocha AI Search provider (#3036)
* feat(web-search): add Bocha AI Search provider

Add a Bocha (bochaai.com) web search provider following the Metaso
adapter pattern: hardcoded official endpoint (SSRF-safe), Bearer auth,
4MB response cap, error-detail extraction, and RFC3339 date parsing
(datePublished preferred, dateLastCrawled fallback).

Tenant-facing config fields:
- freshness: noLimit/oneDay/oneWeek/oneMonth/oneYear (default noLimit)
- summary: request long text summaries, preferred as snippets (default on)

Wiring: registry factory in container, provider type enum + /types
metadata (dynamic form fields), frontend provider union type, and
swagger docs enum entries.

* fix(web-search): match Bocha's real wire format for codes and error messages

Live endpoint probing showed errors return {"message": .., "code": "401"}
(string code, message field) instead of the documented {msg, code:int}.
Parse the code tolerantly (string or number) and extract the detail from
message/msg so auth and rate-limit failures surface their reason.

* fix(web-search): allow Bocha in provider CRUD validation

In-service integration testing caught a fourth wiring point the adapter,
registry, /types metadata and frontend covered but CRUD did not: the
service-layer whitelist (isValidProviderType) and parameter validation
switches rejected provider type 'bocha', so providers could be tested
via POST /web-search-providers/test but never created or updated.

Add Bocha to both switches (delegating to ValidateBochaParameters like
Metaso/Zhipu) with service-level test coverage.

* fix(web-search): correct Bocha legacy timestamp timezone

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-07 20:06:26 +08:00
lyingbug d5cb842580
feat(agent): unify sandbox tools and drop host preloaded skills (#3033)
* feat(agent): unify sandbox tools and drop host preloaded skills

Read workspace and skill files through read_file, and run installed or staged skills through shell_exec(skill_name). Remove the unused skills/preloaded tree and keep generated-file previews stable across re-renders.

* fix(frontend): do not assume classList in markdown sanitize hooks

Node unit tests stub anchors without a DOM classList. Detect download cards from class/download attributes instead.
2026-09-05 16:45:59 +08:00
wizardchen cd3f8b4a9a feat(skills): store skills in a tenant catalog and install per sandbox
A skill is a workspace definition, not a sandbox setting. Keep one catalog
row and run a full image install for each backend, so agents only enable
skills that are already ready on the sandbox they use.
2026-08-28 15:17:37 +08:00
hailongzhao 188200e3b1 feat(sandbox): inject per-user and workspace skill env vars at execution
Skills declare the environment variables they read at run time, so a
credential no longer has to be an admin-set value shared by everyone
using the same skill image.

The installer agent writes a declaration the server validates against the
bundle. Admins fill workspace-wide values from the skill drawer, members
fill their own under Settings, and a successful shell_exec captures the
declared names a run already used for the current speaker. Resolution is
keyed by Principal so IM callers do not collapse onto one synthetic
account, and values are injected only into the process of that one run.

Values are AES-GCM encrypted at rest and never returned by any endpoint.
2026-08-27 14:32:00 +08:00
nullptr-error 18d22b014b
feat(sandbox): expose tenant skill management over HTTP and in the console (#2789)
* feat(sandbox): enhance skill management in sandbox configurations

Updated the skill management system to allow listing, installing, and removing skills within sandbox configurations. The ListSkills function now accepts a sandboxConfigID to filter skills based on the selected configuration. Introduced a new SandboxSkillsDrawer component for managing skills through the UI, along with necessary updates to the API and localization files to support these features.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix: keep skill maintenance sessions out of the console session list

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: store and expose the installer transcript locators

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: record the skill installer agent's conversation

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: stream and persist the installer transcript during a skill install

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: expose install transcript locators on the skill type

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: add the floating skill install transcript window

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: open the install transcript from the sandbox skills drawer

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): 重构技能安装体验,支持内联时间线、编辑第4步及保留镜像指针

- 替换右下角浮窗为内联安装时间线 (SkillInstallTimeline.vue)
  - 新增 GET /api/v1/sandbox-configs/:id/skills/:skillId/transcript 接口,支持回放与追尾
  - 移除 /sessions/continue-stream 依赖,避免 404 问题
  - 事件日志以 install_prompt 开头,不再将工具调用 preamble 拼入最终答案
  - 删除 SkillInstallTranscript.vue,技能条目下可展开/折叠时间线

- 编辑向导新增第 4 步「管理技能」
  - Cube/E2B 环境:连接 → 模板 → 运行配置 → 技能(第4步)
  - Docker/Local 环境:无第4步(无快照镜像)
  - 首次创建必须按 1→2→3 顺序,第3步保存成功后自动进入第4步
  - 已有配置支持步骤条数字跳转,可直接访问任意步骤
  - 卡片菜单「管理技能」打开同一抽屉并定位至第4步
  - SandboxSkillsDrawer.vue 重命名为 SandboxSkillsPanel.vue,去除独立抽屉壳

- 修复保存运行配置时冲掉 skill_image 的问题
  - 顶部展示信息(基础模板/快照ID/版本/构建时间)读取 config.skill_image
  - 编辑器保存 payload 不再包含 skill_image,合并时保留已有 SkillImage 和 VolumeMount
  - MergeSandboxConfigForUpdate 增加对 SkillImage 和 VolumeMount 的保护
  - 已冲掉的配置需重新安装技能才能恢复镜像指针

- 「查看安装过程」在安装开始时即显示
  - 技能列表中出现「正在安装」状态即展示按钮
  - 若记录尚未写入,展示等待提示并自动重试,避免误报「没有留下记录」

* 沙盒内技能目录从 UUID 改成 SKILL.md 的 name。

* fix(sandbox): heal stuck skill rows and block planted maintenance sessions

The reaper left a mid-upgrade install spinning forever when the old snapshot was still live, and restored removals to ready after the pointer had already moved. Client session create/update can no longer hide chats behind the skill_maintenance: prefix.

* style: gofumpt the skill reaper test stubs

* style: satisfy golangci on the tenant skill admin API

* feat(sandbox): rebuild skill images at turn boundaries, not mid-exec

Hold a session turn lease through AgentQA so an install cannot tear down
the live sandbox between tool calls; the next turn's first resolve picks
up the new image. Configs can instead keep existing sandboxes and boot
the new snapshot only for new sessions.

* fix(sandbox): judge stuck skill runs from the snapshot chain

InstalledSnapshotID only records the generation that installed a skill; later
installs of other skills advance the live pointer without rewriting older rows.
Comparing those IDs made the reaper fail healthy installs and delete live
skills whose removal had stalled.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-25 11:28:12 +08:00
hailongzhao 67cdf26216 feat(sandbox): add tenant skill install and remove service layer
Introduce the hidden skill-installer agent and TenantSkillService so a
sandbox config can install or remove skills into snapshot images under a
per-config lock, without yet exposing HTTP or UI entry points.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 17:51:13 +08:00
hailongzhao 00fe035668 feat(sandbox): add tenant skill snapshot infrastructure
Give Cube/E2B a shared snapshot API and persist skill image pointers so later session sandboxes can inherit installed skills from a config snapshot instead of the base template.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 16:00:23 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00
joeyczheng 664d193ca7 feat(datasource): add Tencent IMA datasource
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-08-12 14:56:25 +08:00
Syt3s 5eaabf6bbb
feat: add Exa WebSearch (#2617)
* feat: add Exa WebSearch

* Fix test for Exa config options validation

* feat: add exa connection test
2026-08-11 17:05:52 +08:00
wizardchen 738fe1ec53 feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断
沙箱的后端类型、凭据、模板、超时和私网访问策略此前分散在部署级
WEKNORA_SANDBOX_* 环境变量里,一个实例只能有一套,租户之间无法各用
自己的 E2B 账号或自建 Cube 集群。现在这些配置全部落到空间维度的具名
沙箱配置上:部署环境不再提供覆盖项,未选择配置的智能体直接不启用脚本
执行,而不是静默回落到某个部署默认值——后者会让「这个智能体到底跑在
哪」无从判断。

同时把连接检测从「能不能连上控制面」扩展成可解释的诊断:
- E2B 标准模板显式带上 default 标签构建。创建沙箱时裸模板名/ID 只解析
  default 标签,构建带了别的标签就永远拿不到,表现为一个与「模板不存在」
  无法区分的 404。
- 模板状态不再从历史构建或 spawn 次数反推就绪,只认当前构建;构建完成
  却没有 default 标签的模板单独报成 untagged,因为它在列表里和「仍在构建」
  长得一样,但等待永远不会有结果。
- 创建失败带上 HTTP 状态,404 走模板解释路径,给出具体到「重建模板」还是
  「Key 属于其他团队」的结论。
- 探针脚本改在进程工作目录下暂存。系统临时目录在 macOS 上位于
  /var/folders,Docker 虚拟机并不共享,挂载进容器后是空目录,真实技能执行
  (脚本在 skills/ 下)却不会踩到,检测因此报假失败。
- 检测失败不再只说「命令输出与预期不符」,带上退出码与 stderr 首行。

前端把沙箱设置页与配置抽屉重做:凭据按「是否已配置」展示而非回显掩码,
删除与停用改为就地确认,占用情况走右侧抽屉,列表卡片展示模板状态、凭据、
超时等可判断健康度的事实。
2026-08-11 15:42:02 +08:00
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
Binks Riverton 355d161d64 feat: add Metaso web search provider 2026-08-10 13:33:55 +08:00
lkx 9426ee1470 feat: 增加自动添加标签的功能 2026-08-07 15:01:01 +08:00
Greenplumwine 439af21f9b
feat(feishu-driver): 新增飞书云盘数据源 (#2466)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-05 19:39:49 +08:00
qilifan 3507156929
feat(storage): support AWS S3 NoAk default credentials (#2008)
* feat(storage): support AWS S3 default credentials

* fix(storage): allow clearing S3 credentials

* fix static checks and trim locale diff

---------

Co-authored-by: qilifan <4359902+qilifan@users.noreply.github.com>
2026-08-05 11:59:02 +08:00
wizardchen 350d9e4eca perf(container): batch legacy storage alias lookup at startup
The startup migration probed every workspace/provider pair with First(),
so each miss emitted a GORM "record not found" log line and a separate
round trip. Load all aliases once and index them in memory instead.
2026-08-04 22:48:54 +08:00
ochan.kwon c15c3e972f fix(retriever): rebuild a missing vector store engine on demand
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:

  - Startup skips a store whose engine fails to construct and never
    retries it, so that instance cannot serve the store again -- not
    even after a restart, if the backend is slow to come up.
  - A store registered on one instance is absent on every other until
    they restart, because registration is not broadcast.
  - Registration can fail on the instance handling the create, leaving
    the store missing even there.

Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.

Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.

Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.

Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.

Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.

For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.

A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.

Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.

Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
2026-07-29 12:25:18 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00