* cubesandbox support
* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system
- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
- shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
- sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
- sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
- ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
- /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
- ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
- DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.
* feat(agent):add remote sandbox client abstract & client errors
* feat(sandbox): implement Redis binding store with distributed lifecycle lock;
* feat(sandbox): add remote session lifecycle with ownership context
Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.
* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient
Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.
* refactor(sandbox): decouple SessionBoundManager from Cube
Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.
* refactor(agent): enhance session capability handling in sandbox manager
- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.
* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding
- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0
- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1
- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat
- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
(gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
hardcoded NewMemorySessionSandboxBindingStore() in production path
- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
(Cube and E2B). Sessions survive brief idle periods; providers still
handle TTL-based reaping.
- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
per-provider apply*Defaults helpers
- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python
- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide
Total: 29 files, +1188 / -3721
* feat(agent): bind collected skill artifacts to owning message in resource registry
When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.
- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
maybePersist, and add best-effort bindArtifactResource (binding failures are
logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
and for graceful degradation when Bind fails.
Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sandbox): harden remote sandbox security and artifact collection
Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.
Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.
* fix(ci): gofmt sandbox files and resolve artifact drawer type error
Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.
* fix(sandbox): adapt to upstream main after rebase
Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.
Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.
* feat(sandbox): add sandbox config entity, cordon lease and session pin field
* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease
* feat(sandbox): stamp sandbox config ID onto sandbox metadata
* feat(sandbox): scope sandbox listing, release and reaping to a single config
* feat(sandbox): resolve managers per config and refuse cordoned or missing configs
* feat(sandbox): implement session pinning and config identity change detection
- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions
* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs
Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation
HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints
* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config
- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic
* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy
Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.
* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch
- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.
* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities
- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter
* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube
Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.
## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.
Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.
HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
- Control plane traffic reuses the same `SharedTransport` as E2B.
- Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.
* feat(sandbox): sandbox execution DefaultUser change from root to user
* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults
- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
buildSessionCreateRequest for both Cube and E2B session sandboxes
* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.
* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant
Three defects found while reviewing the remote sandbox work:
- The Cube connectivity probe and the deployment-default Cube/E2B managers
built clients without a guarded transport, so an admin-supplied api_url or
proxy_url could reach link-local addresses via DNS rebinding even though
save-time validation passed. All three now dial through the same outbound
guard the per-tenant resolver enforces.
- Attachment staging was gated on the process-wide manager, so a workspace
pointing an agent at a named Cube/E2B config while the deployment default
stayed docker/local would run its tools remotely against an empty
/workspace/input. The gate now resolves the manager the session actually
runs on.
- Session to sandbox bindings were keyed by the tenant in context, which a
shared agent deliberately swaps for the agent owner's so models, KBs and
named configs resolve in the sharing workspace. Session deletion only ever
knows the session's own tenant, so teardown looked up a key nobody wrote
and abandoned a paused MicroVM that keeps billing. The binding tenant is
now carried separately and survives CloneContext.
* chore(sandbox): renumber migrations above upstream and de-collide test stub
Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.
Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.
---------
Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(security): harden outbound URL SSRF protection
* test(security): add SSRF validation tests for outbound URL caching
Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.
* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go
Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
The heartbeat loop and the detached callback handlers close the client's
current connection, but their goroutines can outlive the connection that
spawned them: heartbeatCancel only signals, and callback handlers run on a
context deliberately detached from the connection. A late-firing goroutine
therefore closed whatever connection the reconnect loop had just opened,
leaving the client to drop messages until the next read timeout.
Pass the owning connection into both callers and close through closeConnIf,
which only acts while that connection is still active. This matches the
yunzhijia and qqbot drivers, which already scope their heartbeat loops to a
single connection.
Also fold Stop() into closeConn so it no longer holds c.mu across Close.
Stop() was the one caller where the delay actually spread: the IM service
tears channels down while holding its channel-map lock, so a TLS close_notify
write to a stalled peer stalled every other channel operation.
The receive loop's deferred cleanup now closes only through closeConnIf,
dropping a redundant second Close of the same connection, and connectAndRun
re-checks the closed flag after publishing the connection so a Stop() racing
the dial cannot leave a receive loop running until its read deadline expires.
The IM channels rewrite resource:// and provider:// references into loadable
HTTP URLs because IM clients cannot attach WeKnora credentials to an image
fetch. That logic was private to internal/im, so no other surface could reuse
it.
Move the reference pattern, HTTP-result guard, tenant-aware storage backend
resolver and streaming holdback helpers into internal/storageurl, and add a
Rewriter that memoises resolutions (a resource:// handle costs one access-grant
row per resolution). internal/im now delegates; behaviour is unchanged.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
IM channels (DingTalk/Feishu) showed broken images because resource://
image references resolve to WeKnora capability links that were neither
reachable nor renderable:
- nginx never proxied /r/, so <APP_EXTERNAL_URL>/r/<token> links fell into
the SPA fallback and returned a blank page. Add a /r/ proxy block
mirroring /files, using the ${APP_SCHEME}/${APP_HOST}/${APP_PORT} vars.
- rewriteStorageURLs treated any "dst != src" as success and emitted
non-HTTP results (e.g. an internal storage:// path when APP_EXTERNAL_URL
is unset) that IM clients cannot fetch. Only substitute http(s) URLs;
otherwise leave the reference unchanged and log an actionable WARN.
- Warn once at startup when IM channels are active but APP_EXTERNAL_URL is
unset — the config state that silently breaks resource:// images on the
default MinIO/local deployment.
- Document that IM knowledge-base images need an IM-reachable http URL via
one of two paths: a publicly reachable storage backend, or APP_EXTERNAL_URL
routing resource:// through nginx /r/.
- Follow a single download redirect while re-validating the target stays
within the allowed host suffix; never forward the bearer token across it.
The shared http client disables auto-redirects for SSRF safety, so a 302
from downloadfileOpen previously failed the download outright.
- Cap in-memory download size at 32 MiB via a limited read-closer that
errors instead of silently truncating, avoiding OOM on large responses.
- Remove dead frontend host-suffix validation branch (normalize always
fills the default) and drop the now-unused i18n keys
yunzhijiaAllowedHostSuffixRequired / yunzhijiaRequiredCredentials.
- Add tests for redirect follow/reject and the size limit.
SendReply now sets param.formatType=markdown by default since WeKnora
replies are authored in Markdown. Callers can override via
reply.Extra["yunzhijia_format_type"] (empty string disables the param).
Lark and Feishu are the same product deployed on two isolated clouds
(open.larksuite.com and open.feishu.cn). WeKnora touches both from two
places — the IM bot and the Wiki data-source connector — and this brings
Lark to parity across both. Neither API surface differs, so rather than
duplicate either implementation, each is parametrized by a Region and
registered twice.
IM (internal/im/feishu):
Every API call routes through a.api(), which prefixes the region's
host, and the WebSocket client passes larkws.WithDomain — the SDK
defaults to open.feishu.cn, so Lark long connections would otherwise
fail to authenticate.
Data source (internal/datasource/connector/feishu):
Lark was half-supported: the client honoured a base_url override and a
LarkBaseURL constant existed, but the connector registered only as
"feishu" and the UI never exposed base_url, so Lark wikis could not be
configured at all. Lark is now its own connector type. base_url is kept
as an explicit override so data sources that pointed a "feishu"
connector at open.larksuite.com keep working.
Bugs found and fixed along the way:
- Wiki resource links were hardcoded to https://feishu.cn/wiki/{token} in
four places, so a Lark wiki linked into the wrong cloud entirely.
- Bot identity is now platform-prefixed (feishu:cli_x vs lark:cli_x).
BotIdentity carries a unique index, so without this a Lark channel
would be rejected as a duplicate of an unrelated Feishu bot sharing an
app_id.
- The card image cache is now keyed by app_id. image_keys are issued per
app on one cloud; a Feishu key handed to a Lark app fails the card
update with code=200570.
- The datasource doc claimed Lark "adapts automatically" — it could not,
since the UI had no way to reach base_url.
Permissions are where the clouds genuinely diverge. Identical APIs do not
mean identical permission catalogues: aily:file:read, aily:file:write and
corehr:file:download exist on Feishu but not on Lark, so pasting Feishu's
scope JSON into Lark fails to import. That JSON is also over-broad (it
covers products WeKnora never calls) and simultaneously incomplete (it
omits the drive:*/docx:* scopes the connector needs). Lark therefore gets
its own list, derived from the 17 endpoints the code actually calls and
checked identifier-by-identifier against Lark's own API reference, plus a
full Feishu→Lark mapping table marking each scope verified, unverified,
or absent. Both features use tenant_access_token only — no user_access_
token exists anywhere in the repo — so the user dimension stays empty.
Known limitation: Region.ThinkingText makes Lark's initial card
placeholder English, but once streaming starts the think-block headers
come from the shared MarkdownThinkStyle and are Chinese on every platform
(Lark, Slack, Telegram, Mattermost alike). Fixing that means threading a
locale through FormatIMDisplayContent, which has no platform context — a
cross-platform change, not a Lark one. Documented as such.
The channel-creation 400 message is now derived from validIMPlatforms so
the list cannot drift as platforms are added.
Tests: region invariants and per-region URL construction on both sides,
the Feishu/Lark bot-identity split, platform-to-channel mapping, base_url
override precedence, an end-to-end SendReply against a fake Open Platform
(token fetch, reply, thread-unsupported fallback, hard-error surfacing),
and a ListResources drive asserting wiki links never leak the other
cloud's host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce resource://<handle> as the stable application identity for stored
objects, decoupling API responses, persisted rich text, and LLM prompts from
physical storage paths. Adds a resource catalog service (register/resolve/bind,
revocable expiring access grants), a request-local alias registry that compacts
resource references while in the LLM context and restores them (including across
streaming chunks), plus router, IM, chat-pipeline, and multimodal integration.
Includes schema migrations (resources, resource_bindings, resource_access_grants)
for sqlite, and hardens URL/output security handling.
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
Previously the Feishu adapter sent replies with the "send message" API
(POST /im/v1/messages) without a root_id. In topic-enabled groups this
spawned a new topic for every reply instead of attaching to the thread
the user @-mentioned the bot from, fragmenting the conversation.
Switch SendReply (text) and sendCardByCardID (streaming card) to the
"reply message" API (POST /im/v1/messages/:message_id/reply), using
incoming.MessageID as the path parameter. Feishu automatically replies
in thread form when the replied-to message is itself a thread message,
so no reply_in_thread flag or session-mode plumbing is needed.
Add sendWithFallback to retry via the plain send-message API on
fallback-eligible error codes (230019 topic gone, 230054 unsupported
message type, 230071 group does not support reply-in-thread), and to
skip straight to the fallback when incoming.MessageID is empty. This
keeps replies deliverable even when the reply API cannot be used.
Also add a debug log in convertEvent (longconn.go) that prints every
raw received event (msg_type/chat_id/msg_id/root_id/parent_id/thread_id/
content) to aid diagnosing intermittent event-delivery issues.
No interface signature changes; other platform adapters untouched.
Co-Authored-By: Claude <noreply@anthropic.com>
Validate temporary download URLs against allowed DingTalk CDN hosts before fetching,
and accept pictureDownloadCode when downloadCode is absent so image messages ingest reliably.
The DingTalk adapter only ever classified incoming messages as text and
did not implement FileDownloader, so files sent to the bot were never
saved to the channel's configured knowledge base — unlike WeCom, which
worked. Users saw the file received and read, but never stored.
Parse file/picture msgtypes in both the webhook and stream paths (set
MessageTypeFile/Image, FileName, FileKey=downloadCode, robot_code), and
implement DownloadFile: exchange the downloadCode for a temporary URL via
robot/messageFiles/download, then fetch it. robotCode comes from the
callback (webhook) or the app client ID (stream). Pictures get a
MsgID-derived name, mirroring the WeCom adapter.
Emit search_source metadata from RAG progress events so quick-answer steps,
summaries, and IM labels no longer mislabel web-only results as knowledge base search.
Deleting or disabling a Feishu channel left its WebSocket long connection
alive: the Lark SDK's Start blocks on a bare select{} and never observes
ctx, and v3.6.1 exposed no public Close(), so the wsCancel() cleanup was a
no-op. The socket stayed open, the SDK kept auto-reconnecting, and Feishu
kept pushing messages — causing cross-environment message duplication when
the same bot was bound elsewhere.
Fix in three layers:
1. Connection teardown
- Upgrade oapi-sdk-go v3.6.1 -> v3.9.7 (adds Client.Close()).
- Add LongConnClient.Close() wrapping wsClient.Close(); factory's cleanup
now calls Close() first, then wsCancel() as a belt-and-braces fallback.
2. Multi-instance coordination
- wsLeaderRetryLoop: after acquiring leadership, re-check the DB before
starting; abort if the channel was deleted/disabled while waiting.
- wsLeaderRenewLoop: on each renewal, re-check the DB; step down if the
channel was deleted/disabled on another instance (HTTP request may have
been served elsewhere), so the leader doesn't keep the connection open
until process restart.
3. Robustness
- DB checks distinguish gorm.ErrRecordNotFound (channel truly gone ->
stop/abort) from transient DB errors (skip this round, retry next tick)
so a DB hiccup can't cause a spurious stop or permanent leader-abort.
- startChannelInternal now stops any existing channelState for the same
ID before overwriting, making it idempotent against concurrent start
paths and preventing adapter/long-connection leaks.
No interface changes; webhook and long-poll modes unaffected.
Co-Authored-By: Claude <noreply@anthropic.com>
IM channels cannot complete in-conversation OAuth prompts, so when an
MCP service needs authorization the reply now includes clickable
authorization URLs (when APP_EXTERNAL_URL and OAuth manager are
available) instead of only directing users to the admin console.
Restore logger config via t.Cleanup after t.Setenv so Yuque token tests
do not leak global state, add unit tests for MCP OAuth non-interactive
context helpers and IM auth notice formatting, and fix EmbedChatCore
indentation.
* feat: support multi-tag for document knowledge
- Add knowledge_tag_relations join table for many-to-many tag-knowledge mapping
- Migrate existing single tag_id data to the new table
- Update Knowledge struct, filter, payload types (TagID -> Tags/TagIDs)
- Rewrite repository layer: SetKnowledgeTags, GetKnowledgeTags, ListIDsByTagIDs
- Update tag count references to query knowledge_tag_relations
- Update service/handler layers for multi-tag create/list/batch-update
- Frontend: multi-select tag sidebar, multi-tag chip display, tag toggle in popup
- FAQ tags remain single-tag model (unchanged)
* feat(knowledge): tag edit dialog with dynamic overflow and inline creation
- Add TagEditDialog component with search, multi-select, and confirm
- Replace inline t-popup tag selector with dialog-based editing
- Tags overflow with +N badge, visible count adapts to container width via ResizeObserver
- Date/card layout: fixed date width, tags flex to fill remaining space
- Add persistent new-tag input bar in dialog (Enter to create, auto-select)
- Tag creation emits callback to refresh parent tag list immediately
- Apply same overflow + dialog trigger to both card view and list view
- Add i18n keys: tagEditDialogTitle, tagEditSearch, tagSelectedCount, tagOverflowTip, tagNewPlaceholder
* fix: exclude soft-deleted knowledge from tag reference counts
CountReferences, BatchCountReferences, and DeleteUnusedTags now
JOIN with knowledges table to filter out soft-deleted records
when counting knowledge_tag_relations. Previously they queried
knowledge_tag_relations directly without the deleted_at check,
causing stale tag counts after knowledge deletion.
* fix: refresh tag counts after async knowledge deletion completes
Add a second delayed tag list refresh (2.5s) after tag deletion to capture
updated reference counts for remaining tags. The original immediate refresh
handles removing the deleted tag from the list, while this delayed refresh
ensures other tags' counts are correct after the async deletion finishes.
---------
Co-authored-by: lyingbug <xiayula2014@gmail.com>
Add compact agent filter beside channel list titles with selected name
display, fix embed localStorage agentId read on rebind, add IM agent
rebind tests, and deep-link agent editor counts to the matching tab.
Add agent complete wait timeout and answer fallbacks for cancel paths,
restore Telegram Markdown on finalize with plain-text retry, and show
answer-only final IM frames instead of the collapsed think summary.
Wait for agent complete without a short timeout, unify Telegram stream
parse mode, remove obsolete FinalAnswer handling, isolate per-section
newline tracking, add lifecycle tests, and update IM streaming docs.
Mirror Web AgentStreamDisplay/RagPipelineProgress in IM text streams:
retract live answers into 思考过程, show Web-aligned tool titles with
query/summary from structured data, and collapse thinking on finalize
instead of dropping it entirely.
- Add IM service dependency to CustomAgentHandler for managing agent-related IM channels.
- Implement DeleteChannelsByAgent method to stop and soft-delete IM channels associated with a custom agent upon its removal.
- Update ListChannelsByTenant to exclude soft-deleted custom agents, ensuring consistent overview lists after agent removal.
DingTalk websocket (Stream) channels stopped delivering @-mentions after
a day or two and only recovered when the IM channel was toggled off/on.
The factory fired the SDK's Start once and relied entirely on the SDK's
internal auto-reconnect, with no liveness supervision on WeKnora's side.
On long-running connections the SDK can enter a zombie state where the
connection object stays alive but no longer receives messages, leaving no
way to recover automatically. The cancel func was also ineffective because
the SDK ignores the context after the initial dial, so stopping a channel
leaked the old connection.
Add a reusable RunSupervised helper that keeps a long connection alive and
proactively recycles it every 6h on top of the SDK's own reconnect, which
bounds the worst-case outage and clears stuck connections. Wire DingTalk to
it, add LongConnClient.Close (disables AutoReconnect before closing so the
SDK does not resurrect the connection), and make the cancel func actually
tear the connection down. Drop the now-unused stream client reference from
the adapter, which only sends replies via sessionWebhook or OpenAPI.
Introduce embed_channels storage, publish-token and session-token auth,
origin allowlist enforcement, shared Redis rate limiter, public embed API
routes, webhooks with SSRF validation, and session listing by embed source.
Three related improvements to the IM-session experience:
- Auto-generated titles: a new IM session starts untitled and gets a
content-based title from its first message via the same GenerateTitleAsync
flow web sessions use, instead of an identity title like "user · dm". Falls
back to the identity title when the first message has no text. All platforms.
- In-flight reply recovery (web chat): opening a session whose reply is still
being generated on an IM channel made the web client attach via
continue-stream and fail (the IM answer never streams through this server),
flashing a "stream failed" toast and forcing a manual refresh. Treat a failed
attach to an IM reply (channel === 'im') as recoverable — poll until it
completes, then reload the thread; on timeout surface the standard error.
Web/api replies keep the original behaviour, untouched.
- Source-filter classification: the sidebar filter treats "web" as sessions
with no IM mapping, but the list join excluded soft-deleted mappings, so
/clear'd IM sessions leaked into the user's own web chats. Include soft-deleted
mappings so a session ever bound to an IM channel stays under its platform.
Adds a repository regression test.
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
IM callback routes bypass the auth middleware because IM platforms verify
their own signatures, so the downstream QA pipeline never received a
UserID/TenantRole. The Organization-shared knowledge base resolution gates
on a non-empty UserID, so IM-channel questions silently skipped shared KBs
while the Web path (which sets UserID via JWT auth) worked.
Inject a synthetic "system-<tenantID>" identity (mirroring the API-key path,
recognised by types.IsSyntheticUserID) plus a Viewer role when building the
IM session context. Viewer is the least privilege sufficient to retrieve
shared KBs.
Closes#1600
- Eliminated all instances of the final_answer tool from the agent's configuration and logic, transitioning to a model where the agent concludes responses with plain text instead.
- Updated event handling and response analysis to reflect this change, ensuring that the agent's output is now directly delivered as text without relying on a dedicated final answer tool.
- Adjusted localization strings across multiple languages to remove references to final_answer, enhancing clarity and consistency in user-facing messages.
- Improved the overall structure of the agent's response handling to streamline the process and reduce potential confusion regarding tool usage.
refactor(agent): streamline answer handling and preamble logic
- Updated the agent's response handling to treat any text streamed before a tool call as a preamble, ensuring it is not included in the final answer.
- Removed the superseded marker logic from the event emission process, simplifying the flow of answer content and enhancing clarity in the UI.
- Adjusted the event handling to ensure that preamble segments are correctly retracted and relocated into the steps tree, improving the overall user experience.
- Enhanced the agent's internal logic to better manage answer segments and their states during streaming, leading to a more coherent response structure.
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.
Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.
Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.
Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
This commit introduces a new test suite for the IM file service, including a stub implementation for testing purposes. It adds tests for resolving file services based on storage providers and ensures proper fallback mechanisms for MinIO URLs. Additionally, the `rewriteStorageURLs` and `cleanIMContent` functions have been refactored to utilize a resolver for improved caching and efficiency. These changes enhance the robustness of file service handling and improve test coverage for various storage scenarios.