Commit Graph

139 Commits (da4ef68d9d5327dfd236299d882f092943a1a165)

Author SHA1 Message Date
lyingbug da4ef68d9d
feat(plugin): standalone plugin hosts with Redis discovery (P2 batch 4) (#3725)
* feat(plugin): standalone plugin hosts with Redis discovery
2026-09-26 14:46:26 +08:00
wizardchen 27b0c84bfd fix(docparser): make the MinerU parse timeouts configurable
Self-hosted MinerU capped every parse at a hardcoded 1000s, which fired
before WEKNORA_DOCREADER_CALL_TIMEOUT and WEKNORA_DOCUMENT_PROCESS_TIMEOUT
could matter, so large PDFs always failed at 16m40s (#3091). MinerU Cloud
had the same problem with a hardcoded 600s polling deadline.

- WEKNORA_MINERU_TIMEOUT (default 1000s) bounds a self-hosted parse: the
  whole V1 job, or the legacy /file_parse request.
- WEKNORA_MINERU_CLOUD_TIMEOUT (default 600s) bounds how long a MinerU
  Cloud batch is polled.
- Both share the env parsing added for WEKNORA_PADDLEOCR_VL_TIMEOUT in
  #3537, now in helpers.go: invalid or non-positive values log a warning
  and fall back to the default.
2026-09-25 18:38:25 +08:00
wizardchen 1fa915d712 chore: update BrowserSkill extension version to 0.3.1 and enhance localization support
- Updated the BROWSERSKILL_EXTENSION_PATH in .env.example and Dockerfile.app to reference the new extension version 0.3.1.
- Modified localization files for English, Japanese, Korean, Russian, and Chinese to reflect the support for both Chrome and Edge browsers.
- Added checks for extension version compatibility in the BrowserConnectionSettings component to improve user experience.

This commit ensures that the application is aligned with the latest BrowserSkill extension and enhances the localization for better user guidance.
2026-09-23 17:12:19 +08:00
Frank Zhu acd1a2a5a4
docs(ollama): document embedding model config and RAM guidance (#3531)
* docs(ollama): document embedding model config and RAM guidance (#3480)

Local embedding and generation share the Ollama process at OLLAMA_BASE_URL.
The embedding model name is the local Embedding model's name; EMBEDDING_MODEL_NAME
is read only when builtin_models.yaml references it. The published 8 GB
starting point does not include Ollama weights or optional Neo4j.

Fixes #3480

* docs(ollama): shrink README/docs scope per maintainer review (#3480)
2026-09-23 12:02:42 +08:00
Captain 171097f06e
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS

A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).

SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.

Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.

Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.

Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.

docreader enforces the same switch in its own validator, and compose
passes the variable to both services.

Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.

The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.

Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.

The switch defaults to off, so nothing changes for deployments that do
not set it.

* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs

docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.

The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 12:02:06 +08:00
Leon 402729341a
fix(docparser): make PaddleOCR-VL request timeout configurable (#3537)
* fix(docparser): make PaddleOCR-VL request timeout configurable

* style(docparser): wrap timeout configuration and test lines
2026-09-23 11:58:59 +08:00
lyingbug 63ef76dd80
docs: 迁移有效文档至 website-docs 并清理旧文档 (#3580)
* docs: consolidate maintained documentation in website-docs

* docs: remove migrated legacy documentation and update remaining references
2026-09-22 23:25:40 +08:00
wizardchen a62a51e6b0 docs(db): stop claiming MySQL is a supported DB_DRIVER
initDatabase() only switches on postgres and sqlite; any other DB_DRIVER
returns "unsupported database driver". The .env.example comment listed
mysql as a valid value, so anyone following it hit a startup failure.

migrations/mysql/00-init-db.sql is the other source of that impression --
it is kept roughly in sync whenever the base tables change, but nothing
references it. Add a README stating it is unwired and listing what real
MySQL support would still need.

Also correct the table count in the database-schema doc: that script now
covers 10 tables, not 7.
2026-09-20 13:29:03 +08:00
Frank_zhu 68e21b6e74
docs(env): warn against SEARXNG_PORT colliding with APP_PORT (#3385)
Document that binding SearXNG to the same port as the app can make
localhost hit SearXNG's HTML 404 on login. Docs-only; no runtime change.
2026-09-20 11:14:42 +08:00
lyingbug b82fd6c0f9
fix(security): close the low-severity shared-space findings (#3416)
* fix(security): close the low-severity shared-space findings

Verified and fixed from the shared-space permission review:

- Embed channels kept knowledge_ids, tag_ids, mentioned_items, skill_names
  and summary_model_id from anonymous visitors. Mentions are only scoped
  for shared agents, so a visitor could query any KB (and pick any model)
  of the channel workspace by ID. The payload now drops them; the widget
  never sends them.
- PUT /knowledgebase/:kb_id/wiki/issues/:issue_id/status updated any issue
  by ID. The update is now scoped to the path's KB (404 otherwise).
- A direct org invite let the inviter attach any user of the target
  workspace (or any user by ID) as its representative, whose username and
  email the roster then showed to all members. Direct adds attach no
  representative, and the roster returns emails only for the caller's own
  workspace.
- COS and KS3 dropped the bucket (and COS the region) segment when turning
  a path into an object key, while the tenant check scans every segment:
  cos://42/<region>/prefix/7/... passed as tenant 42 and read tenant 7's
  object from a shared bucket. Both must now match the service's config.
- Shared agent listings returned the full agent to receivers. Prompts and
  the creator's user ID are withheld; capabilities and scope stay visible.
- Temporary document uploads let the caller pick the parser engine (and
  its credentials) of a shared agent's workspace; only the agent's rules
  choose one there now.
- Suggested questions used request / message-snapshot knowledge_ids
  without a permission check. Each document is now resolved to its KB and
  checked like KB IDs, including a shared agent's scope.
- IM channels accepted any knowledge_base_id (and any agent ID on create).
  Both must belong to the channel's workspace, the KB also to a restricted
  API key's allow-list.
- KB-restricted API keys could list org-shared KBs outside their allow-list
  in the organization views; those listings are now filtered.
- With enable_rbac=false, RequireKBAccess let cross-workspace denials
  through. That switch is for roles inside a workspace; the guard now
  always denies another workspace's KB, since handlers behind it load by
  ID without a tenant filter.
- The org member limit was counted before and enforced apart from the
  insert; it is now checked in the insert's transaction with the
  organization row locked (SQLite serializes writers instead).
- The UpdateOrganization route comment and API doc said owner-only; the
  service and UI allow org admins. Comment and doc corrected.

Documented, not changed: issued /r/ and presigned links stay valid until
they expire after a share is revoked.

* fix(security): address review of the low-severity shared-space fixes

- Embed payloads also drop question_origin. It selects a KB like an
  @mention for agents that retrieve only when mentioned, and the widget
  never sends it; the selection stays inside the agent's scope, but an
  anonymous visitor should not pick among its KBs. suggestion_attribution is
  kept (the widget sends it and it is validated).
- The receiver view of a shared agent also withholds the follow-up
  suggestion instruction, a prompt like the others. Starter questions are
  shown to receivers anyway, and the sharer's name is displayed by design.
- Follow-up suggestion generation moves into the agent's workspace with
  types.WithExecutionTenant instead of a bare tenant rewrite, so the
  caller is pinned before the document permission check runs. The auth
  middleware and the shared run already pinned it; this removes the
  dependency on that.
- KS3 CopyFile copies only from the service's own bucket (as COS and S3
  do); another bucket's path is a cross-backend copy.

Tests cover the dropped question_origin, the follow-up instruction and the
/shared-agents listing, a follow-up context with the tenant switched, a
repeated wiki issue status update, and the KS3 copy source.
2026-09-18 22:58:46 +08:00
wizardchen 84c9871d16 fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
lyingbug d7ccd5ba51
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling

* fix: preserve trusted Doris redirects and prepare browser tests

* fix: align proxy address checks and integration test coverage
2026-09-16 16:20:57 +08:00
Jonesxq 350b2ab364
fix(milvus): support multilingual BM25 retrieval (#3235)
* fix(milvus): support multilingual BM25 retrieval

* fix(milvus): preserve source metric and tighten collection matching

Read the source embedding index metric during BM25 migration, match
collections by {prefix}_{dimension}, and route Japanese/Korean text to
the ICU fallback.

* fix(milvus): satisfy lint on multilingual BM25 changes

---------

Co-authored-by: Jonesxq <239089032+Jonesxq@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-14 19:18:24 +08:00
lyingbug 7e53c31671
feat(browser): connect local Chrome with task controls and display modes (#3180)
* feat(browser): integrate BrowserSkill local task tabs and account pairing

* feat(browser): refine task routing, lifecycle, and display modes

* fix(browser): reclaim temporary task tabs and groups

* test(database): cover browser authorization migration schema
2026-09-11 17:59:27 +08:00
HUANG Cheng 6a55860431
feat(ui): multi-stage Docker frontend build (no host npm) (#3074)
* feat(ui): multi-stage Docker build without host npm

Build frontend assets inside the image with node:24-bookworm, then
copy into the existing digest-pinned nginx runtime so docker compose
and CI no longer require a host-side dist step.

* fix(ui): copy local xlsx package and raise Node heap in Docker build

npm ci needs packages/xlsx before install, and vite OOM'd under the
default Node heap during the multi-stage image build.

* fix(ui): native multi-arch frontend Docker build and layer cache

Keep Vite on BUILDPLATFORM so CI does not qemu-emulate Node, pin the
builder image, and move the commit ARG after npm ci so dependency
layers stay cacheable.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-07 19:33:57 +08:00
GUO Zhaogong 6f0ed94a47
feat(i18n): add Japanese (ja-JP) locale support (#3080)
* feat(i18n): add Japanese (ja-JP) locale support

Add a complete Japanese UI locale for the admin frontend and the web
embed widget, and register it everywhere the supported-locale list is
enumerated.

- frontend/src/i18n/locales/ja-JP.ts: full bundle (5,600+ keys) aligned
  with en-US/zh-CN key structure; verified by the locale key audit
- frontend/src/i18n/embed.ts: ja-JP block for the embed widget, plus
  `ja*` normalisation in normalizeEmbedLocale
- Register ja-JP in i18n index, SUPPORTED_LOCALES, localeKeyAudit
  bundles/order, TDesign locale map (ja_JP), login language picker,
  general settings dropdown and embed default-locale options
- Add `language.jaJP` key to every locale so the settings dropdown can
  use $t like its siblings
- Accept ja-JP as an embed channel default_locale on the backend and as
  DEFAULT_LOCALE in the frontend docker entrypoint
- Extend locale-enumerating tests and update accepted-value lists in
  .env.example, public/config.js, widget script and website-docs

* feat(i18n): add ja-JP names and descriptions for prompt templates

Add a ja-JP block to the i18n map of all 32 prompt templates
(system_prompt, context_template, rewrite, fallback, intent_prompts,
agent_system_prompt) so the template picker shows Japanese labels when
the UI locale is ja-JP instead of falling back to the Chinese names.
Prompt content is unchanged.

* feat(i18n): add ja-JP strings for the CLI integration page

main gained the CLI setup page (#3081) with 18 new integrations.cli.* /
integrations.tabs.cli keys; add their Japanese translations so the
locale key audit stays green after rebasing onto main.

* style(i18n): unify ja-JP spacing and long-vowel conventions

- Remove spaces between Latin tokens and adjacent Japanese text
  (e.g. "MCP サービス" -> "MCPサービス"), matching the majority style
  already used in the bundle
- Drop the trailing long-vowel mark on common technical katakana terms
  (サーバ, ユーザ, フォルダ, プロバイダ, パラメータ, ヘッダ, クラスタ, ...)
  so the whole locale follows one convention

* feat(i18n): add ja-JP labels for agent type presets

config/agent_type_presets.yaml carried default / zh-CN / zh-TW labels
only, so the agent editor showed English preset names in the Japanese
UI. Add a ja-JP label and description to all five presets.
2026-09-07 19:20:41 +08:00
lyingbug d5cb842580
feat(agent): unify sandbox tools and drop host preloaded skills (#3033)
* feat(agent): unify sandbox tools and drop host preloaded skills

Read workspace and skill files through read_file, and run installed or staged skills through shell_exec(skill_name). Remove the unused skills/preloaded tree and keep generated-file previews stable across re-renders.

* fix(frontend): do not assume classList in markdown sanitize hooks

Node unit tests stub anchors without a DOM classList. Detect download cards from class/download attributes instead.
2026-09-05 16:45:59 +08:00
wizardchen bbe95a6c1b refactor(skills): enhance skill upload size management and error handling
- Updated .env.example to clarify skill zip upload limits and their implications for Nginx configuration.
- Refactored frontend logic to differentiate between skill zip uploads and other uploads, ensuring accurate error messages for oversized requests.
- Improved utility functions for managing upload limits and added tests to validate behavior for oversized JSON bodies.
- Enhanced error handling in various handlers to provide clearer feedback on upload size constraints, particularly for sandbox and catalog installations.
2026-09-02 15:08:05 +08:00
wizardchen 17324c46e4 refactor(skills): improve skill bundle size handling and error management
- Updated .env.example and docker-compose.yml to clarify skill bundle size limits and their implications for GitHub source downloads.
- Enhanced frontend logic to differentiate between skill bundle uploads and knowledge uploads, ensuring proper error messages for oversized requests.
- Introduced new utility functions for managing upload limits and added tests to validate behavior for oversized JSON bodies.
- Refactored error handling in various handlers to provide clearer feedback on upload size constraints.
2026-09-02 15:08:05 +08:00
wizardchen 3a06a3b9e9 refactor(skills): improve skill bundle size management and nginx configuration
- Updated .env.example to clarify the skill bundle size limits and their impact on Nginx configuration.
- Refactored frontend/docker-entrypoint.sh to separate max file size settings for skills and general uploads.
- Enhanced nginx.conf to apply specific body size limits for skill uploads without affecting other endpoints.
- Added tests to ensure proper handling of skill bundle file indexing and size constraints.
- Improved error handling in tenant_source.go and tenant_sandbox_config.go to manage skill file limits effectively.
2026-09-02 15:08:05 +08:00
wizardchen 56bf53a1fd fix(skills): raise skill zip cap and keep uninstall progress stable
GitHub skill packages such as ppt-master exceed the 50MB knowledge-upload limit. Give skills their own 256MB download/upload cap, keep uninstalling visible if the list lags, and distinguish zip-entry vs skill-file errors.
2026-09-02 15:08:05 +08:00
wizardchen 2783d9cab8 fix(skills): count GitHub skill subtrees and polish catalog install UI
Large GitHub skill folders were rejected because the installer counted the whole repo zip. Count only the SKILL.md subtree, raise the file cap, and hide uninstall while install is running.
2026-09-02 15:08:05 +08:00
lyingbug 02269bd8f1
feat: require opt-in for the Docker sandbox backend (#2936)
* feat: require opt-in for the Docker sandbox backend

Docker engine access is host-root equivalent, so keep the backend off
until System Settings or WEKNORA_SANDBOX_DOCKER_ENABLED says otherwise.
Also move system-settings help into the title icon and add i18n for
tenant.auto_accept_invitation.

* fix(i18n): drop tenant wording from Docker host-risk copy

The workspace terminology check rejects 租户 / tenant in user-facing strings.
2026-09-01 15:42:28 +08:00
cainbit a55f4b7834
feat: support complex password (#2929)
* feat: support complex password

* fix: honor runtime complex-password policy on all password writes

Keep the service layer as the last check, share one resolver with invite
registration and admin user create, and drop the leftover startup debug print.

* fix: satisfy pre-push lint on password-policy error paths

The hook diffs against origin/main, so wrap the original error strings
and check c.Error on the new validation branches.

---------

Co-authored-by: wuwc.ew <wuwc.ew@cebwm.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-01 11:50:10 +08:00
wizardchen 5a7683cf38 fix: close OIDC id_token verification gaps for explicit endpoints
Discovery was skipped when authorization/token URLs were already set, so
jwks_uri and issuer never populated and unsigned id_token claims could
still provision accounts. Always discover missing JWKS/issuer, ignore
unverified tokens, and require kid/exp/iss/sub checks.
2026-09-01 11:26:30 +08:00
Binks Riverton af496ea782 docs: clarify Docker image release tags 2026-08-24 17:01:32 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
wizardchen d6d177f3ab fix(i18n): harden DEFAULT_LOCALE runtime config after #2658
Whitelist locale tags in docker-entrypoint to prevent config.js injection,
extract resolveDefaultLocale with trim/validation, add unit tests, and sync
local dev config plus docs for the new deployment default language setting.
2026-08-12 10:53:23 +08:00
sunmh 23849a0c66 feat(i18n): 前端默认语言支持经 DEFAULT_LOCALE 环境变量运行时配置 2026-08-12 10:49:04 +08:00
wizardchen 738fe1ec53 feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断
沙箱的后端类型、凭据、模板、超时和私网访问策略此前分散在部署级
WEKNORA_SANDBOX_* 环境变量里,一个实例只能有一套,租户之间无法各用
自己的 E2B 账号或自建 Cube 集群。现在这些配置全部落到空间维度的具名
沙箱配置上:部署环境不再提供覆盖项,未选择配置的智能体直接不启用脚本
执行,而不是静默回落到某个部署默认值——后者会让「这个智能体到底跑在
哪」无从判断。

同时把连接检测从「能不能连上控制面」扩展成可解释的诊断:
- E2B 标准模板显式带上 default 标签构建。创建沙箱时裸模板名/ID 只解析
  default 标签,构建带了别的标签就永远拿不到,表现为一个与「模板不存在」
  无法区分的 404。
- 模板状态不再从历史构建或 spawn 次数反推就绪,只认当前构建;构建完成
  却没有 default 标签的模板单独报成 untagged,因为它在列表里和「仍在构建」
  长得一样,但等待永远不会有结果。
- 创建失败带上 HTTP 状态,404 走模板解释路径,给出具体到「重建模板」还是
  「Key 属于其他团队」的结论。
- 探针脚本改在进程工作目录下暂存。系统临时目录在 macOS 上位于
  /var/folders,Docker 虚拟机并不共享,挂载进容器后是空目录,真实技能执行
  (脚本在 skills/ 下)却不会踩到,检测因此报假失败。
- 检测失败不再只说「命令输出与预期不符」,带上退出码与 stderr 首行。

前端把沙箱设置页与配置抽屉重做:凭据按「是否已配置」展示而非回显掩码,
删除与停用改为就地确认,占用情况走右侧抽屉,列表卡片展示模板状态、凭据、
超时等可判断健康度的事实。
2026-08-11 15:42:02 +08:00
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
sunmh 47d95cd720 feat(invitation): add auto-accept switch for inviting registered users
Add a global tenant switch (tenant.auto_accept_invitation /
WEKNORA_TENANT_AUTO_ACCEPT_INVITATION, default false). When enabled,
inviting an already-registered user adds them to the tenant as an
active member immediately, instead of creating a pending invitation
the invitee must accept from their inbox.

Backend:
- CreateInvitation reads the switch and, when on, calls the new shared
  addMemberAndRespond helper (extracted from TenantMemberHandler.AddMember)
  to persist the member and write the HTTP response. Both call sites
  share it so the sentinel->status mapping can't drift apart.
- Register the setting in the system_setting registry (bool, default
  false).
- Expose the switch as capabilities.auto_accept_invitation on
  GET /auth/me, so the SPA can adapt the invite popup without calling
  the super-admin settings API.
- Cover enabled/disabled, already-a-member, unknown email, owner role
  rejection, and nil member service with unit tests.

Frontend:
- The invite popup adapts to the mode. With auto-accept, the form
  submits directly (button reads "Send", no "confirm invitation" step)
  and refreshes the member roster silently (no toast; the new row is
  the feedback). The classic two-step flow is unchanged when off.
- Fix capabilities hydration: refreshFromAuthMe (the only reader of
  /auth/me capabilities) was invoked solely from the invitation-token
  flow, so on page reload can_create_tenant and auto_accept_invitation
  stayed at their initial false. Trigger it from initFromStorage once a
  token is restored so deployment switches are reconciled with the live
  server.
2026-08-10 15:04:29 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
wizardchen 267c847fb8 docs: complete docker compose pull guidance across deployment docs
Follow up on #2567 by documenting the upgrade workflow, syncing JA/KO
READMEs and installation guide, and clarifying that WEKNORA_VERSION
changes require docker compose pull to avoid stale cached images.
2026-08-06 19:58:02 +08:00
lyingbug fefc2c1dc0 Revert "docs: complete docker compose pull guidance across deployment docs"
This reverts commit 485454a9ea.
2026-08-06 19:51:39 +08:00
wizardchen 485454a9ea docs: complete docker compose pull guidance across deployment docs
Follow up on #2567 by documenting the upgrade workflow, syncing JA/KO
READMEs and installation guide, and clarifying that WEKNORA_VERSION
changes require docker compose pull to avoid stale cached images.
2026-08-06 19:30:55 +08:00
chenjunhong 8fd679a443 docs: add docker compose pull to deployment steps to prevent stale images
Users downloading a release and running `docker compose up -d` may get
cached old images because Docker doesn't pull by default. This caused
confusion when the UI displayed an older version than the release.

- Add `docker compose pull` before `up -d` in both README and README_CN
- Enable WEKNORA_VERSION in .env.example so users can pin a release tag

Closes #2560
2026-08-06 19:24:33 +08:00
Greenplumwine 439af21f9b
feat(feishu-driver): 新增飞书云盘数据源 (#2466)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-05 19:39:49 +08:00
wizardchen e71c2dce2e fix(api): address resource_urls review follow-ups
Harden public URL mode: stop logging signed URLs at INFO, copy messages
before rewriting history responses, reuse winning grants on insert races,
sync the Go client resource_urls query param, and add knowledge-search
coverage.
2026-08-05 14:42:51 +08:00
wizardchen eb6ac0110c fix(api): close the resource_urls=public gaps found in review
Follow-up on resource_urls=public. Review turned up one authorization gap plus a
set of correctness problems in the streaming path.

Authorization:

  - Embed channels were documented as excluded, but nothing enforced it. The
    delegating handlers hand the visitor's own gin context to the shared
    handlers, so ?resource_urls=public — and a deployment-wide
    RESOURCE_URL_MODE=public — reached anonymous traffic and produced
    shareable, credential-free URLs that bypass the channel-scoped
    /embed/:channel_id/files proxy. ensureEmbedSession now pins the request
    context to handle mode, which covers every delegated endpoint including
    future ones. The downgrade is silent so a client that forwards the
    parameter keeps working.

  - A knowledge-base-restricted API key is refused with 403. Such a key is
    already denied /files because a raw storage path cannot be bound to its
    allow-list, so handing it anonymous file URLs would reopen that hole from
    the other side. The default handle mode stays available to it.

Streaming:

  - Rewriter.ref holds its lock across resolution. Resolver implementations keep
    an unsynchronised per-provider cache, so the previous unlocked window was a
    real data race for any caller sharing a StreamRewriter between goroutines —
    which its own doc comment invited. Serialising also collapses a concurrent
    duplicate into one signature.

  - Release the holdback buffer on every path that ends a stream while the
    client is still connected: a user-requested stop, an error event, and
    giving up on the event store. Only completion flushed before, so the tail
    generated just before a stop was silently dropped.

  - A released tail carries the metadata of the event it was cut from instead of
    a bare event_id.

  - The maxHeldBytes safety valve aligns to a UTF-8 boundary rather than cutting
    a character in half, and an unclosed "![…](" only holds back a
    whitespace-free destination under 2048 bytes, so prose that merely mentions
    Markdown image syntax no longer stalls the stream behind the buffer.

Also resolves the mode before running a knowledge search instead of after, so an
invalid value no longer costs a full retrieval, and warns once per distinct bad
RESOURCE_URL_MODE value instead of once per process.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
lyingbug ef1f0b36c4 docs(api): document resource_urls and RESOURCE_URL_MODE
Adds a cross-cutting "文件与图片引用" section to docs/api/README.md covering the
parameter, the deployment default, the endpoints it applies to, and the two
caveats that matter in practice: it needs APP_EXTERNAL_URL (or a publicly
reachable storage backend) to produce a link at all, and the links it produces
are time-limited but anonymously readable.

Also annotates the affected endpoints for Swagger and refreshes the generated
files for those parameters only, leaving unrelated drift in the committed
swagger output alone.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
qilifan 3507156929
feat(storage): support AWS S3 NoAk default credentials (#2008)
* feat(storage): support AWS S3 default credentials

* fix(storage): allow clearing S3 credentials

* fix static checks and trim locale diff

---------

Co-authored-by: qilifan <4359902+qilifan@users.noreply.github.com>
2026-08-05 11:59:02 +08:00
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
nullkey 4e3b512af6 fix(im): render resource:// images in IM channels (nginx /r/ + config guards)
IM channels (DingTalk/Feishu) showed broken images because resource://
image references resolve to WeKnora capability links that were neither
reachable nor renderable:

- nginx never proxied /r/, so <APP_EXTERNAL_URL>/r/<token> links fell into
  the SPA fallback and returned a blank page. Add a /r/ proxy block
  mirroring /files, using the ${APP_SCHEME}/${APP_HOST}/${APP_PORT} vars.
- rewriteStorageURLs treated any "dst != src" as success and emitted
  non-HTTP results (e.g. an internal storage:// path when APP_EXTERNAL_URL
  is unset) that IM clients cannot fetch. Only substitute http(s) URLs;
  otherwise leave the reference unchanged and log an actionable WARN.
- Warn once at startup when IM channels are active but APP_EXTERNAL_URL is
  unset — the config state that silently breaks resource:// images on the
  default MinIO/local deployment.
- Document that IM knowledge-base images need an IM-reachable http URL via
  one of two paths: a publicly reachable storage backend, or APP_EXTERNAL_URL
  routing resource:// through nginx /r/.
2026-07-28 21:25:12 +08:00
wizardchen d283fabc99 fix(doris): harden Stream Load HTTP with SSRF-safe client
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
2026-07-24 15:10:44 +08:00
Greenplumwine fa7ae075ee
feat(compose): align env vars with code + fix helm GraphRAG bug (#2164)
* feat(compose): align env vars in docker-compose and .env.example with code

docker-compose.yml / docker-compose.dev.yml
- app.environment: remove deprecated vars Go app no longer reads
  (CRYPTO_MASTER_KEY, CRYPTO_SALT, TENANT_AES_KEY, ENABLE_GRAPH_RAG);
  add missing vars the code actually consumes (OSS/S3/TOS/COS completion,
  OpenSearch, Tencent VectorDB, Milvus credentials, Redis TLS/namespace,
  OIDC, audit/invitation/quota, LLM tuning, doc timeouts, etc.)
- docreader.environment: complete PDF tuning knobs, SSRF/LOG_LEVEL/gRPC
  params, GRPC_MTLS_REQUIRE_CLIENT_CERT; fix DOCREADER_PDF_JPEG_QUALITY
  default 90 -> 85 to match code (config.py)
- mcp.environment: add MCP_SERVER_AUTH_TOKEN (required by HTTP transport),
  WEKNORA_CHAT_TIMEOUT, WEKNORA_VERIFY_SSL, MCP_ALLOWED_UPLOAD_DIRS
- dev compose docreader.environment kept symmetric with prod

.env.example
- reorganize into A-J sections (deploy / data / retrieval / models /
  parsing / auth / agent / integrations / observability / security)
- remove the four deprecated vars, document why
- add missing vars, fix NEO4J_URI to bolt:// (matches compose + single-node),
  fix DORIS_COMPAT_MODE typo, complete docreader PDF tuning defaults

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(helm): inject NEO4J_ENABLE and drop deprecated TENANT_AES_KEY

helm graph bug fix
- NEO4J_ENABLE is the sole graph switch the Go app reads (since v0.1.6,
  ENABLE_GRAPH_RAG is no longer read). helm never injected NEO4J_ENABLE,
  so GraphRAG was silently broken: setting neo4j.enabled=true gave the
  app NEO4J_URI/USERNAME/PASSWORD but not the switch, and the UI rejected
  Node Extractor config with "请正确配置环境变量NEO4J_ENABLE".
- app.yaml: inject NEO4J_ENABLE=true inside the neo4j.enabled block;
  remove the dead ENABLE_GRAPH_RAG injection
- NOTES.txt: update GraphRAG instructions (just enable neo4j, no need
  to set the deprecated ENABLE_GRAPH_RAG)
- values.yaml: drop ENABLE_GRAPH_RAG, fix neo4j section comment

TENANT_AES_KEY cleanup (deprecated since v0.4.0, encryption now uses
SYSTEM_AES_KEY exclusively; Go app no longer reads it)
- secrets.yaml: stop generating/looking-up/writing TENANT_AES_KEY,
  fix stale comment about tenants.api_key rotation
- app.yaml: remove TENANT_AES_KEY secretKeyRef
- values.yaml: drop tenantAesKey, fix existingSecret key list + comments
- cloud-image/firstboot.sh: stop generating/writing/printing TENANT_AES
- cloud-image/README.md: drop TENANT_AES_KEY from credential list

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-21 15:55:52 +08:00
wizardchen ceb4d1a310 feat(chat): add session-scoped temporary attachments with async parsing
Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
2026-07-16 17:41:27 +08:00
wizardchen 41975abcc5 refactor(terminology): rename user-facing tenant labels to workspace
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
2026-07-14 16:14:43 +08:00
wizardchen 87ba3fc2de refactor(auth): unify OIDC provisioning under shared default_tenant_mode policy
Drop the standalone OIDC_AUTH_DEFAULT_TENANT_MODE / oidc_auth.default_tenant_mode
knob so OIDC auto-provisioning and local password registration share the single
auth.default_tenant_mode policy (resolved by the callback handler). Repair
tenantless sessions by falling back to the first active membership on the login
happy path and stale-preference paths, allow tenantless PUT /auth/me/preferences,
clear the tenant snapshot on /auth/me responses without a tenant, and default the
frontend canCreateTenant capability to fail-closed until /auth/me confirms it.
2026-07-13 19:14:49 +08:00
wizardchen ecf2c1fb9b feat(auth): support tenantless provisioning and gated self-service workspace creation
Allow deployments to register users without auto-creating a personal tenant, expose creation policy via /auth/me, and guide tenantless accounts through a workspace onboarding flow.
2026-07-13 18:13:51 +08:00