Reject out-of-range and duplicate response indexes before mapping them to batch inputs. Return an error when a requested input has no response item instead of silently passing an incomplete batch downstream.
Add focused coverage for malformed responses and valid out-of-order results.
Signed-off-by: PP1 <74917296+pengpengyi92@users.noreply.github.com>
New endpoint GET /api/v1/auth/oidc/start — unlike /auth/oidc/url which
returns JSON and requires frontend JS to navigate, this endpoint directly
returns a 302 redirect to the OIDC Provider's authorization page. This
enables external platforms (enterprise portals, Nexus, etc.) to give a
single clickable link that triggers the full OIDC authorization code
flow and, when the IdP has an active SSO session, achieves passwordless
login without any frontend involvement.
- backend derives the callback URL from the request origin automatically
(no caller-supplied redirect_uri needed)
- OIDC nonce cookie is bound identically to /auth/oidc/url (CSRF/replay
protection)
- registered in the auth middleware public-path whitelist (otherwise
the global auth middleware returns 401 before routing reaches the
handler)
- refactored shared nonce-cookie helper out of GetOIDCAuthorizationURL
for reuse by OIDCStart
* cubesandbox support
* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system
- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
- shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
- sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
- sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
- ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
- /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
- ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
- DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.
* feat(agent):add remote sandbox client abstract & client errors
* feat(sandbox): implement Redis binding store with distributed lifecycle lock;
* feat(sandbox): add remote session lifecycle with ownership context
Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.
* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient
Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.
* refactor(sandbox): decouple SessionBoundManager from Cube
Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.
* refactor(agent): enhance session capability handling in sandbox manager
- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.
* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding
- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0
- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1
- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat
- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
(gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
hardcoded NewMemorySessionSandboxBindingStore() in production path
- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
(Cube and E2B). Sessions survive brief idle periods; providers still
handle TTL-based reaping.
- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
per-provider apply*Defaults helpers
- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python
- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide
Total: 29 files, +1188 / -3721
* feat(agent): bind collected skill artifacts to owning message in resource registry
When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.
- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
maybePersist, and add best-effort bindArtifactResource (binding failures are
logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
and for graceful degradation when Bind fails.
Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(sandbox): harden remote sandbox security and artifact collection
Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.
Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.
* fix(ci): gofmt sandbox files and resolve artifact drawer type error
Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.
* fix(sandbox): adapt to upstream main after rebase
Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.
Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.
* feat(sandbox): add sandbox config entity, cordon lease and session pin field
* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease
* feat(sandbox): stamp sandbox config ID onto sandbox metadata
* feat(sandbox): scope sandbox listing, release and reaping to a single config
* feat(sandbox): resolve managers per config and refuse cordoned or missing configs
* feat(sandbox): implement session pinning and config identity change detection
- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions
* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs
Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation
HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints
* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config
- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic
* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy
Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.
* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch
- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.
* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities
- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter
* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube
Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.
## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.
Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.
HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
- Control plane traffic reuses the same `SharedTransport` as E2B.
- Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.
* feat(sandbox): sandbox execution DefaultUser change from root to user
* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults
- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
buildSessionCreateRequest for both Cube and E2B session sandboxes
* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.
* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant
Three defects found while reviewing the remote sandbox work:
- The Cube connectivity probe and the deployment-default Cube/E2B managers
built clients without a guarded transport, so an admin-supplied api_url or
proxy_url could reach link-local addresses via DNS rebinding even though
save-time validation passed. All three now dial through the same outbound
guard the per-tenant resolver enforces.
- Attachment staging was gated on the process-wide manager, so a workspace
pointing an agent at a named Cube/E2B config while the deployment default
stayed docker/local would run its tools remotely against an empty
/workspace/input. The gate now resolves the manager the session actually
runs on.
- Session to sandbox bindings were keyed by the tenant in context, which a
shared agent deliberately swaps for the agent owner's so models, KBs and
named configs resolve in the sharing workspace. Session deletion only ever
knows the session's own tenant, so teardown looked up a key nobody wrote
and abandoned a paused MicroVM that keeps billing. The binding tenant is
now carried separately and survives CloneContext.
* chore(sandbox): renumber migrations above upstream and de-collide test stub
Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.
Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.
---------
Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Deduplicate the two inline SearchResult rebuilds (SSE references and
agent-stream references) into one searchResultFromMap helper.
Add a JSON round-trip test pinning all fields.
- Reconcile stale pending invites after auto-accept AddMember
- Adopt invited workspace as home tenant when invitee is tenantless
- Await refreshFromAuthMe on SPA boot; sync auto_accept in hydrateSessionFromToken
- Document capabilities.auto_accept_invitation; extend client SDK and OpenAPI
Add a global tenant switch (tenant.auto_accept_invitation /
WEKNORA_TENANT_AUTO_ACCEPT_INVITATION, default false). When enabled,
inviting an already-registered user adds them to the tenant as an
active member immediately, instead of creating a pending invitation
the invitee must accept from their inbox.
Backend:
- CreateInvitation reads the switch and, when on, calls the new shared
addMemberAndRespond helper (extracted from TenantMemberHandler.AddMember)
to persist the member and write the HTTP response. Both call sites
share it so the sentinel->status mapping can't drift apart.
- Register the setting in the system_setting registry (bool, default
false).
- Expose the switch as capabilities.auto_accept_invitation on
GET /auth/me, so the SPA can adapt the invite popup without calling
the super-admin settings API.
- Cover enabled/disabled, already-a-member, unknown email, owner role
rejection, and nil member service with unit tests.
Frontend:
- The invite popup adapts to the mode. With auto-accept, the form
submits directly (button reads "Send", no "confirm invitation" step)
and refreshes the member roster silently (no toast; the new row is
the feedback). The classic two-step flow is unchanged when off.
- Fix capabilities hydration: refreshFromAuthMe (the only reader of
/auth/me capabilities) was invoked solely from the invitation-token
flow, so on page reload can_create_tenant and auto_accept_invitation
stayed at their initial false. Trigger it from initFromStorage once a
token is restored so deployment switches are reconciled with the live
server.
* fix(security): harden outbound URL SSRF protection
* test(security): add SSRF validation tests for outbound URL caching
Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.
* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go
Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
The VLM path sent max_tokens and temperature, both rejected by reasoning
and GPT-5 models. go-openai's reasoning validator failed the request
client-side, so every OCR and caption call errored with "this model is
not supported MaxTokens, please use MaxCompletionTokens" and no image
chunk was ever created.
The chat path already solved this in shapeOpenAIReasoning (issue #1283);
the VLM path was never wired to it. Reuse the shared
provider.IsOpenAIReasoningOrGPT5Model predicate so both paths agree on
which models are affected. Migrating max_tokens alone is not enough --
the VLM default temperature of 0.1 is independently rejected.
Also report an empty completion with finish_reason=length as an error.
max_completion_tokens covers reasoning tokens, so an exhausted budget
returns empty content rather than an API error, which was previously
recorded as no_extracted_content and looked identical to an image with
no text.
Fixes#2537
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Shared-agent replies can reference resources in the source workspace. Route
those images through a session/message file proxy that derives authorization
from the persisted message and active share, and align the chat renderer to
use the assistant message ID during streaming hydration.
Address post-merge review: verify old password before policy checks,
reject same-password rotations, mark OIDC-provisioned users in preferences
and hide the self-service form with guidance, map API error details to
i18n, sync docs/api, Swagger, and the Go client SDK.
Expose the existing change-password API in the User Profile settings
page with current / new / confirm fields, shared password policy
validation, and i18n. Align the service layer with registration policy
(8-32 chars, letter + number) and map invalid-old / weak-password
errors cleanly. After success, revoke sessions and return the user to
login.
Fixes#2267
Automatic tagging shipped with a few paths that quietly disabled it and a
prompt that spent most of its budget on data the model never needed.
Robustness:
- Treat an omitted confidence as a deliberate selection instead of zero.
Models frequently drop the field, and the 0.75 floor then filtered out
every match, silently turning the feature off. Percentage-scale values
are rescaled rather than accepted as out-of-range certainty.
- Classify the first 500 tags when a knowledge base has more, logging a
warning, instead of dropping the task entirely. Large knowledge bases
previously got no auto-tagging at all with no user-visible signal.
- Reuse common.ParseLLMJsonResponse for the model reply, and give that
helper a balanced-scan fallback. The local extraction cut the payload at
the last "}" or "]", so a trailing citation such as "see [1]" produced
invalid JSON and burned the task's retries.
Prompt:
- Address candidates by ordinal rather than UUID. Tag names are short while
a UUID is 36 characters, so this cuts candidate tokens roughly tenfold,
keeps the model from having to reproduce (and corrupt) an id, and is what
makes the raised 500 ceiling affordable. Ordinals are range-checked and
mapped back to ids server-side.
- Drop the rationale field. Nothing read it, and at max_tags=10 it competed
with the 1024-token output budget and risked truncating the JSON.
- Fence document text in <document> so file content cannot be read as
instructions.
Behaviour:
- Add auto_tag_config.skip_if_tagged, defaulting to true, so documents the
user already tagged at upload time keep their classification. Existing
tags are now loaded before the model call, so a skipped document costs no
LLM round-trip. Configs stored without the field adopt the default.
Also promote AddKnowledgeTagRelations to KnowledgeRepository instead of
reaching for it through a type assertion that failed at runtime, and route
the max_tags bounds through AutoTagConfig.Normalize so the prompt, the match
filter and the persisted config cannot drift apart.
Adds handler-level coverage for scope, cancellation, idempotency, ordinal
validation and the new toggle, plus SDK, Swagger and API doc updates.
RemoveMember left users.tenant_id and LastActiveTenantID pointing at the
removed workspace, so login synthesised it back into the space switcher
while RBAC returned 403 on every request.
After a successful soft-delete, clear dangling home/preference pointers
and revoke outstanding sessions. On the login path, validate that home
still has an active membership and stop synthesising memberships from
users.tenant_id when the membership table is authoritative and empty.
Fixes#2586
SanitizeForLog is for log output and rewrites tabs/newlines/control
characters. Applying it to req.Password during public registration
changed the credential that was hashed, so login with the original
password failed. Match the invite registration path and pass the
password through unchanged.
Fixes#2521
* chore(types): add internal ContentRewritten to types.SearchResult
* test(chunks): pin rewritten results out of the position path
Pipeline-replaced content (parent or neighbor expansion) must not trust
stale coordinates even when the replacement satisfies the length
invariant, and must not extend the merged group's range.
* fix(chunks): keep pipeline-rewritten results off the position path
Parent and neighbor expansion replace Content after retrieval. Without a
marker their stale coordinates could be incorrectly trusted.
Ensure ContentRewritten is false in chunkTrusted.
Position-aware merging still routed trimming through AppendWithOverlap,
which searches for the longest suffix match inside a window. On periodic
text (table rows, log lines) that search mistakes the repetition for the
overlap and drops real content, and with a zero overlap there is nothing
to search for in the first place.
When coordinates are trusted the overlap is already known, so verify it
character for character and cut exactly, falling back to the text matcher
only when the bodies disagree inside the overlap.
- Overlaps wider than the 400-rune text window must be trimmed once.
- Adjacent chunks built from one repeating row must keep every row.
- A length-preserving text mismatch must fall back and keep both bodies.
The merge pipeline treats revision 0 as "never edited", so the field fails
open to the position path wherever it is not carried over. Make the column
explicit for direct row scans and document the remaining JSON gap.
The heartbeat loop and the detached callback handlers close the client's
current connection, but their goroutines can outlive the connection that
spawned them: heartbeatCancel only signals, and callback handlers run on a
context deliberately detached from the connection. A late-firing goroutine
therefore closed whatever connection the reconnect loop had just opened,
leaving the client to drop messages until the next read timeout.
Pass the owning connection into both callers and close through closeConnIf,
which only acts while that connection is still active. This matches the
yunzhijia and qqbot drivers, which already scope their heartbeat loops to a
single connection.
Also fold Stop() into closeConn so it no longer holds c.mu across Close.
Stop() was the one caller where the delay actually spread: the IM service
tears channels down while holding its channel-map lock, so a TLS close_notify
write to a stalled peer stalled every other channel operation.
The receive loop's deferred cleanup now closes only through closeConnIf,
dropping a redundant second Close of the same connection, and connectAndRun
re-checks the closed flag after publishing the connection so a Stop() racing
the dial cannot leave a receive loop running until its read deadline expires.
Chunk editing made parser coordinates unreliable, so all merges were
downgraded to pure text matching, including unedited documents.
Restore position-aware merging for pairs whose coordinates are still
trustworthy, and keep the text fallback for edited or stale content so
it is never dropped on coordinates.
- Feature contracts: trusted pairs trim overlaps, join adjacent chunks
seamlessly, and stay separate across position gaps.
- Safety contracts: never drop current content on coordinates alone.
* feat(datasource): add Feishu/Lark Drive (云盘) connector
Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.
Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
/ feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
(FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
the wiki connector's three cursor semantics (resume fast-path,
failure-does-not-advance-cursor, toSyncCursor snapshot), plus
ListResources / ResolveResourceAncestors (top-down shared traversal,
best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata
Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".
Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
Drive root folder_token input + load button before the lazy-load tree,
edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
input group
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): refine Feishu Drive connector UI, channel and root folder
A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:
Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
"飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.
Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
switch); the tree area shows a "load a folder first" placeholder until
the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
actionable i18n hints.
Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
to resolve the root folder's human-readable name; the list API only
returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
suffix) so it matches the saved resource_id form and the direct
children's ParentID, which previously broke selection matching on edit.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): sync a single selected Drive file without folder error
When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".
The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.
A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks extraction design spec
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks implementation plan
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): sync feishu drive docx via blocks API with export fallback
- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
sub-items), falling back to the export API on blocks errors or empty
renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): label feishu drive knowledge as 飞书云盘 in list
The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): polish feishu drive resource-step UI in datasource editor
- Replace the two stacked t-alert banners under the folder_token input:
shareHint becomes an always-visible form-desc line, rootNotSupportedHint
moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
status + tips on the input (replacing the global MessagePlugin) when
empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
fixed and the resource region (placeholder / loading / empty / tree)
fills the remaining drawer height, with the tree scrolling internally.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu drive datasource integration guide
User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine
Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.
Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.
Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): split feishu connector into core/wiki/drive packages
Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.
container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).
Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).
Co-Authored-By: Claude <noreply@anthropic.com>
* test(datasource): migrate feishu tests to core/wiki/drive packages
Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.
All tests pass: core 4s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test
Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).
Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.
All tests pass: core 3.5s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): repair feishu cleanup fallout
Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.
All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(frontend): feishu drive data source UI and i18n
Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: add trellis journal merge=union rule
Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): rename feishu/drive drive_connector.go to connector.go
Aligns naming with feishu/wiki subpackage (connector.go).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): allow data:image base64 URIs in sanitized markdown
DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path
The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export
Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.
- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
they don't break under the new default
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout
- 飞书云盘接入说明: add section on docx parse mode (export default vs
blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
add pros/cons table, fix source-file table to core/wiki/drive layout
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: 修复合并main后丢失新增的国际化问题
* fix(i18n): 修复误删的国际化内容
* docs: 删除误提交的文件
---------
Co-authored-by: Claude <noreply@anthropic.com>
A caller that resends CreateFAQEntry before the first call answers could
create several rows for the same standard question. Indexing runs inline and
the embedding call retries with exponential backoff, so one create can take
far longer than an impatient caller waits; meanwhile the duplicate-question
check only looked at ChunkStatusIndexed, so a sibling create still mid-flight
was invisible and every retry passed the check.
- Serialize concurrent creates of the same question behind a Redis SetNX
guard, keyed by tenant + knowledge base + question hash, with an in-process
fallback when Redis is unavailable so an outage cannot block FAQ authoring.
- Widen the duplicate check to every non-deleted status. This also fixes
ChunkStatusDefault entries, which are listed to users but never blocked
duplicates.
- Bound the indexing step of an interactive create so a degraded embedding
service cannot stretch the request out; bulk and background indexing keep
the full retry budget.
- Report a failed rollback instead of discarding the error, since a leftover
stored chunk is hidden from the list yet now matches the duplicate check.
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
Harden public URL mode: stop logging signed URLs at INFO, copy messages
before rewriting history responses, reuse winning grants on insert races,
sync the Go client resource_urls query param, and add knowledge-search
coverage.
Follow-up on resource_urls=public. Review turned up one authorization gap plus a
set of correctness problems in the streaming path.
Authorization:
- Embed channels were documented as excluded, but nothing enforced it. The
delegating handlers hand the visitor's own gin context to the shared
handlers, so ?resource_urls=public — and a deployment-wide
RESOURCE_URL_MODE=public — reached anonymous traffic and produced
shareable, credential-free URLs that bypass the channel-scoped
/embed/:channel_id/files proxy. ensureEmbedSession now pins the request
context to handle mode, which covers every delegated endpoint including
future ones. The downgrade is silent so a client that forwards the
parameter keeps working.
- A knowledge-base-restricted API key is refused with 403. Such a key is
already denied /files because a raw storage path cannot be bound to its
allow-list, so handing it anonymous file URLs would reopen that hole from
the other side. The default handle mode stays available to it.
Streaming:
- Rewriter.ref holds its lock across resolution. Resolver implementations keep
an unsynchronised per-provider cache, so the previous unlocked window was a
real data race for any caller sharing a StreamRewriter between goroutines —
which its own doc comment invited. Serialising also collapses a concurrent
duplicate into one signature.
- Release the holdback buffer on every path that ends a stream while the
client is still connected: a user-requested stop, an error event, and
giving up on the event store. Only completion flushed before, so the tail
generated just before a stop was silently dropped.
- A released tail carries the metadata of the event it was cut from instead of
a bare event_id.
- The maxHeldBytes safety valve aligns to a UTF-8 boundary rather than cutting
a character in half, and an unclosed ", so an existing grant cannot be reused by
looking it up — the plaintext token is unrecoverable. Derive the token instead:
HMAC-SHA256(SYSTEM_AES_KEY, resource id + time window). The same resource in the
same window derives the same token, so the row can be found by its hash and
reused; only the first request in a window inserts. The window is half the TTL,
so a reused grant always has at least TTL/2 of life left.
This keeps the storage properties intact: the table still holds only hashes and
the plaintext cannot be reconstructed from a dump without SYSTEM_AES_KEY, while
authorization stays entirely on the row, so expiry and revocation still decide
whether a token resolves. Without SYSTEM_AES_KEY, grants stay random and
per-request as before. A stable URL per window also lets client and CDN caches
hit.
Stop deleting revoked grants during cleanup. A revoked row has to survive until
its expiry, otherwise a token derived for the same resource and window would
re-create it and revive the access that was just revoked. grant.revoked_at has
no writer yet, so this changes nothing today, but the derived token makes the
tombstone load-bearing.
Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>