Commit Graph

236 Commits (06eeb95b4e050ede712ef16fb6053e4fa5fcfbde)

Author SHA1 Message Date
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
shiquan_zhan b39f00633c fix: replace all exact wiki text matches 2026-08-05 19:30:40 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wizardchen b581094c7b fix(web_fetch): address review follow-ups for structured fetch results
Restore HTTP-only 15s pipeline fetching, reject SPA shell pages when
browser rendering fails, surface per-URL failure fields in the UI, and
keep structured model output when all URLs fail.
2026-07-28 22:17:31 +08:00
langcaiye a3da5cb57a fix(agent): preserve web evidence when page fetches fail 2026-07-28 22:17:31 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
wizardchen 80a5003cc9 feat(wiki): add page revision history with diff, revert, and manual editing
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
2026-07-27 15:03:24 +08:00
wizardchen 4a5fc4487b fix(agent): keep batched wiki_read_page results within output budget
Batch wiki reads used to lose middle pages when the registry applied
head/tail truncation to the combined output. wiki_read_page now budgets
per page, caps neighbour link summaries, and names omitted slugs explicitly.
Also align registry truncation with rune-based limits and raise the default
tool output ceiling to 24k characters.
2026-07-27 13:22:16 +08:00
koishi 9e66389126 Add files via upload 2026-07-27 11:39:06 +08:00
wizardchen 718b7347d0 fix(agent): harden search-target scope and model-handle policies
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
2026-07-27 11:05:43 +08:00
wizardchen 5e413c6478 refactor(agent): introduce modelcontext registry and fix handle-codec bugs
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.

Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
  UnresolvedToolHandles)
2026-07-27 11:05:43 +08:00
wizardchen 84f65a809f fix(agent): ensure wiki summary slugs are preserved during citation compaction
This update addresses the issue where wiki summary-page slugs (summary/<knowledgeID>) were being mangled during the citation compaction process, leading to dead links. The changes include:

- Reordering the message encoding process to ensure resource references are aliased before source references, preventing the UUID from being replaced in the summary slug.
- Implementing a new function to selectively replace document IDs while preserving the integrity of summary slugs.
- Adding unit tests to verify that summary slugs remain intact and are not altered during compaction.

This fix enhances the reliability of cross-links in the wiki system and prevents potential dead links from being generated.
2026-07-24 15:57:08 +08:00
wizardchen 5d0326f987 fix(wiki): harden slug integrity across ingest and agent write paths
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:

- Ingest dedup now groups each new item with only its own trigram
  candidates and validates merges against that per-item scope, rejecting
  cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
  translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
  system-owned summary pages, and auto-repairs dead [[slug]] links
  (rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
  storage handles so they round-trip safely through chat.

Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
2026-07-24 15:57:08 +08:00
wizardchen 6fb858105a feat(llm): add prompt-cache observability and wiki cache-friendly prompts
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
2026-07-23 19:17:30 +08:00
wizardchen 8ae82573ab fix(mcp): coordinate OAuth token refresh and authorization polling
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
2026-07-23 15:35:24 +08:00
wizardchen 8ca9c117bb feat(knowledge): enhance FAQ entry filtering and tagging
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.

This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
2026-07-23 10:50:32 +08:00
wizardchen eedd7d4641 feat(agent): enhance suggested questions API with tag scopes
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
2026-07-17 10:19:43 +08:00
wizardchen b862c2e45d feat(agent): add per-agent citation output toggle
Move citation protocol injection to the system layer so custom prompts no
longer own citation syntax. Agents can disable inline citations while source
aliases still encode and decode tool arguments correctly.
2026-07-17 10:19:43 +08:00
wizardchen 014164ae44 feat(llm): introduce request-local source reference aliases
Replace raw chunk/document/KB IDs in LLM context with short cN/dN/bN/wN
aliases, decode tool arguments on the way in, and expand compact <ref/>
citations to the public <kb>/<web> format before user-facing output.
2026-07-17 10:19:43 +08:00
wizardchen ceb4d1a310 feat(chat): add session-scoped temporary attachments with async parsing
Let users upload documents in chat without ingesting them into a knowledge base: files are parsed in the background, referenced by ID on send, and expire after a configurable TTL.
2026-07-16 17:41:27 +08:00
wizardchen fbc9db85d1 fix(chat): enforce retrieved image output in answers
When retrieved context or tool results contain Markdown images, dynamically
append output requirements to system prompts and final-answer instructions.
Also normalize fullwidth image parentheses in the frontend renderer so
localized model output still renders correctly.
2026-07-15 18:10:31 +08:00
wizardchen 951f95903b fix(chat): use Markdown for LLM image context and normalize legacy XML
Chat and agent tool context now emits answer-ready Markdown images instead
of internal <image> XML so copied content renders safely. The frontend also
normalizes legacy XML blocks in persisted conversations during streaming.
2026-07-15 18:10:31 +08:00
wizardchen 0f4500b98f fix(tests): update resource alias syntax from 'res:' to 'res://' in tests and registry
Refactor tests and registry to use the new 'res://' syntax for resource aliases, ensuring consistency across the codebase. This change addresses alias handling in various test cases, including stream processing and registry encoding, to prevent data loss and improve resource reference management.
2026-07-15 17:02:43 +08:00
wizardchen 38c594aec3 feat(chat): implement orphan alias detection and logging in chat completion and streaming
Add functionality to detect unresolvable resource aliases emitted by the model during chat completion and streaming. Introduce logging for these orphan aliases to aid in debugging and improve the handling of resource references. Update the stream processing to ensure held aliases are flushed upon cancellation, preventing data loss. Include tests to verify the correct reporting of orphan aliases.
2026-07-15 17:02:43 +08:00
wizardchen 6d14cc3536 feat(resource): add stable resource registry with LLM-context alias compaction
Introduce resource://<handle> as the stable application identity for stored
objects, decoupling API responses, persisted rich text, and LLM prompts from
physical storage paths. Adds a resource catalog service (register/resolve/bind,
revocable expiring access grants), a request-local alias registry that compacts
resource references while in the LLM context and restores them (including across
streaming chunks), plus router, IM, chat-pipeline, and multimodal integration.

Includes schema migrations (resources, resource_bindings, resource_access_grants)
for sqlite, and hardens URL/output security handling.
2026-07-15 17:02:43 +08:00
wizardchen f8b3e5f148 feat(storage): add multi-instance storage backends per workspace
Introduce tenant-scoped storage backend registry so workspaces can register
multiple storage instances and bind each knowledge base to a specific backend,
with migration from legacy tenant-level storage config and new settings UI.
2026-07-15 14:54:39 +08:00
wizardchen 41975abcc5 refactor(terminology): rename user-facing tenant labels to workspace
Align product copy, API docs, Swagger descriptions, and i18n strings
with workspace terminology while keeping internal tenant identifiers
and headers unchanged for API compatibility.
2026-07-14 16:14:43 +08:00
wizardchen eacc278b9a fix(wiki): strip internal chunk aliases from page content
Wiki ingest prompts previously encouraged inline [cNNN] citations that
leak internal chunk metadata into reader-facing Markdown. Strip legacy
aliases on read/write paths and update the modify prompt to keep prose
clean while source links remain in ChunkRefs.
2026-07-13 15:59:09 +08:00
wizardchen c1cddb4d4d Add instruction fields to knowledge base configuration 2026-07-13 12:16:19 +08:00
wizardchen 6ac1703198 Simplify KB duplicate API and improve naming/docs coverage.
Drop client-supplied target_id, generate IDs server-side, localize duplicate
names from request language, document the endpoint, and expand handler/client tests.
2026-07-08 20:55:54 +08:00
wizardchen 58b79876ae Add knowledge base duplicate flow 2026-07-08 20:55:54 +08:00
wizardchen 3209ed77fd feat(quick-qa): add references drawer and polish retrieval source UI
Add a side panel for web and knowledge-base citations with citation-badge
highlighting, refine collapsed RAG pipeline summary layout, and emit
knowledge_base_id from agent knowledge_search results.
2026-07-08 00:26:34 +08:00
cao_j b16cb0bb56 fix(agent): treat end_turn as natural stop 2026-07-06 23:16:18 +08:00
wizardchen 8d40959f88 fix lint and test failures 2026-07-01 17:31:42 +08:00
wizardchen 4c962a9403 feat(agent): add configurable MCP OAuth wait timeout per agent
Allow agents to set how long in-conversation OAuth prompts wait before
auto-skipping, with UI in the agent editor and bounded gate behavior.
2026-07-01 14:37:37 +08:00
wizardchen cb99736d72 feat: scope API MCP OAuth by principal 2026-06-30 20:01:05 +08:00
wizardchen 505195865a fix(mcp): trigger OAuth prompt during tool registration and dedupe helpers
OAuth authorization was only handled in MCPTool.Execute, so unauthenticated
services failed silently during RegisterMCPTools before any tool reached the
agent. Wire EventBus/session metadata into registration and reuse the same
getOrCreateMCPClientWithOAuthRetry path used at execution time.

Extract MCP OAuth session/wait/connect logic into mcp_oauth.go and remove the
duplicate waitForOAuthAndRetry + RegisterMCPContext layering in mcp_tool.go.
2026-06-29 23:22:28 +08:00
wizardchen 9fd7a7f522 feat(mcp): prompt and resume OAuth authorization mid-conversation
When an agent invokes an OAuth-enabled MCP service that the current user has
not authorized yet, the tool call now pauses and surfaces an "Authorize" card
in the chat instead of failing with an opaque error. After the user completes
the per-user OAuth popup, the backend verifies a token exists and resumes the
paused tool call automatically.

Backend:
- Add mcp_oauth_required / mcp_oauth_resolved events and response types.
- Add OAuthPendingRequest + Gate.RequestOAuthAndWait, reusing the existing
  approval waiter, Resolve, Redis cross-instance fan-out and timeout.
- MCPTool.Execute pauses on an authorization-required error for OAuth services
  and retries once (with a fresh ctx and recycled connection) after the user
  authorizes.
- Forward the new events to the SSE stream.
- Add ResolveMCPOAuth handler + route; it verifies authorization before
  unblocking so a premature resolve cannot resume into another failure.

Frontend:
- Add McpOAuthCard with an Authorize button reusing the popup + status-poll
  flow, then resolve the paused run.
- Handle the new stream events (dedup multiple pending cards per service).
- Add resolveMCPOAuth API client and i18n keys (zh/en/ko/ru).
2026-06-29 23:22:28 +08:00
wizardchen c29761331c fix(agent): enforce tag scopes across knowledge tools 2026-06-29 19:17:42 +08:00
wizardchen 17ffd902ac fix(agent): avoid expanding tag scopes for grep chunks 2026-06-29 19:17:42 +08:00
wizardchen d3071fbe03 test(agent): add unit tests for buildMustUseBlock functionality
- Introduced tests for buildMustUseBlock to validate multi-word service prefix handling and name sanitization.
- Ensured that service names are correctly sanitized to prevent newline injection in must_use blocks.
- Verified that the generated block includes the correct tool name prefixes and does not contain unwanted characters.
2026-06-29 18:09:44 +08:00
wizardchen d24e830c85 refactor(agent): fix grep scope loss and consolidate per-turn @mention scoping
- Fix grep_chunks dropping the full-KB scope when a turn mixes @KB with
  @tag/@file: extract scopeClause and OR knowledge-ID and (kb,tenant)
  predicates together, matching knowledge_search's fan-out.
- Consolidate the three scattered per-request @Skill/@MCP scope blocks in
  buildAgentConfig into applyPerRequestSkillScope / applyPerRequestMCPScope,
  each narrowing the whitelist and computing the must_use pin in one place.
- Dedupe the two identical selection-mode normalizers in Input-field.vue.
- Fix knowledge_create_test passing "" for the tagIDs []string parameter,
  which made the whole service test package fail to compile.
- Add unit tests for scopeClause and the per-request scope helpers.
2026-06-29 18:09:44 +08:00
wizardchen b86efe3f92 fix(agent): harden @mention scope and simplify must_use hints
Tighten per-turn @MCP/@Skill/@tag handling after review: enforce agent
MCP/Skill whitelist, ignore mentions when selection mode is none, tag-scope
grep_chunks, and simplified English must_use lines without tool dumps.
Skip persisting scope envelopes to history rendered_content.
2026-06-29 18:09:44 +08:00
wizardchen 049f85139d feat(chat): add @Skill/@MCP mentions with scoped agent runtime
Wire per-turn @mentions for skills, MCP services, tags, and files through
QA handlers into agent config, runtime_context/must_use user blocks, and
tool registration. Fix tag-only scopes dropping KB tools, refresh suggested
questions on mention changes, and strengthen MCP must_use so @selected
services are invoked before local KB search.
2026-06-29 18:09:44 +08:00
wizardchen 7f3f219f58 feat(mcp): add OAuth2 authorization for MCP services
Support connecting to MCP servers that require OAuth2, with a Cursor-like
zero-config flow: the user provides only a URL and the system performs
metadata discovery, dynamic client registration (DCR), and the
authorization code flow with PKCE on first connect.

- Persist per-user access/refresh tokens and per-service DCR client
  credentials, encrypted at rest (AES-256-GCM)
- Store the PKCE verifier/state during the flow via Redis with an
  in-memory fallback for Lite mode
- Cache MCP clients per (service, user) so tokens stay user-isolated
- Add endpoints for authorize-url, callback, status, and token revoke;
  the callback is allowlisted for unauthenticated access
- Surface a structured "authorization required" hint when a tool call
  hits an unauthorized MCP service
- Frontend: add an auth-type selector with OAuth config, inline
  authorization status/actions, and render the connection test result
  inline in the editor drawer instead of a stacked centered dialog
2026-06-22 20:19:31 +08:00
wizardchen 83eeec0ddf feat(i18n): add optional embedding model descriptions for multiple locales
Enhance the internationalization support by adding optional descriptions for the embedding model in English, Korean, Russian, and Chinese. Update the UI to reflect these changes, ensuring users are informed about the optional nature of the embedding model when configuring knowledge base settings.
2026-06-16 23:51:29 +08:00
wizardchen 588f00b18a refactor(wiki): enhance taxonomy planning and streamline category handling
Revamp the wiki taxonomy planning process to assign coherent directory paths to entity/concept pages in a single pass, ensuring consistent reuse of existing folders. This change eliminates the need for per-page category assignments during the reduce phase, preserving user-curated categories.

Consolidate category path parsing and deduplication into a shared utility, improving consistency across the service and repository layers. Update related functions and tests to reflect these changes, ensuring efficient handling of category paths in the wiki ingestion process.
2026-06-16 20:26:06 +08:00
wizardchen fe33db9493 feat(wiki): add page hierarchy/category metadata and retrieval APIs
Persist category and hierarchy metadata on wiki pages (new migration), expose
category-path retrieval and related repository/service/handler/router APIs, and
update wiki ingestion (batch, cite, taxonomy) plus agent wiki prompts to build
and consume the taxonomy. Regenerate swagger docs for the new endpoints.
2026-06-16 20:26:06 +08:00