Commit Graph

30 Commits (22cbcdfa5789b33dbb8d47532c21b24da2fdf5aa)

Author SHA1 Message Date
lyingbug c40a9dd194
refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470)
把集中式的 OpenAI 方言客户端重建为三层:协议层 internal/models/api/<protocol>(OpenAI Chat Completions / OpenAI Responses / Anthropic Messages / Google GenAI),目录层 internal/models/catalog,厂商层 internal/models/vendors/<id>(27 家,go:embed)。27 家厂商逐条对照官方文档核验,厂商事实以数据声明,协议层不出现厂商名。

既有行的运行时行为会变的四处:
1. extra_config.api 变为保留的协议选择器,非法取值被拒绝;
2. Azure 未填 api_version 的行改走 /openai/v1 GA 数据面;
3. api.openai.com 的一方流量改走 Responses 协议(中转/网关不受影响);
4. 7 家厂商的输出上限字段按文档纠正。

不需要数据库迁移。详见 PR 描述与 website-docs/03-features/06-models.md。
2026-09-20 21:28:23 +08:00
wizardchen ad5d6a8fc2 fix(agent): make read_document queries keyword-based and bound its pages
read_document's in-document query was matched as one literal string, so a
keyword query such as "sunflower f(n,k)" found nothing in a 708-chunk
document even when both words were there. The zero-match result gave the
model no next step, and the UI rendered it as "0 / 708 chunks loaded",
which reads like a failed load.

Paging mode also had no size bound: the registry truncates
ToolResult.Output, but the model sees the result rendered from Data, so a
limit=100 page reached ~68k characters in the model context.

- A plain query is split on whitespace; a chunk matches when it contains
  every word, in any order. regex=true still matches one expression.
- A zero-match result carries a <hint> suggesting fewer/different words
  or reading a chunk from search_knowledge results.
- A page stops adding chunks at 80% of the output budget (counting
  per-chunk rendering overhead); next_offset resumes at the first chunk
  not returned, and an oversized first chunk is still returned.
- The UI summary shows in-document search results ("no matches" /
  "N matches") and, for offset paging, the chunk range instead of a page
  number that no longer holds.
2026-09-18 15:09:25 +08:00
lyingbug 92e7c0ab37
refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381)
* refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents

The agent's retrieval surface had grown to five overlapping tools whose
behaviour was scattered across long descriptions, interleaved "tips" in
tool output (most of which modelcontext discarded before the model saw
them), and a grep_chunks implementation that ran an unindexed regex scan
over the chunks table ordered by created_at, so large knowledge bases lost
recall and paid a full-table scan on every call.

Tool surface
- search_knowledge replaces knowledge_search and grep_chunks: one query,
  mode = hybrid | semantic | keyword, optional knowledge_base_ids, limit.
  Keyword mode runs through the retrieval engine's keyword index instead of
  the DB regex scan. Modes unavailable for the scope are refused with a
  reason. Rerank, MMR and de-duplication are kept; already_seen markers,
  retrieval_statistics and the zero-result "CRITICAL" block are gone.
- read_document replaces list_knowledge_chunks, get_document_info and
  wiki_read_source_doc: id accepts a dN or cN handle, always returns the
  document header, pages by exact offset (fixing the old floor-to-page
  bug), finds passages with query (literal by default, regex opt-in) and
  reads neighbours around a chunk with context.
- list_documents pages through one knowledge base, honouring @file/@tag
  scope.
- wiki_search takes a single query plus regex, knowledge_base_ids, limit;
  literal queries are escaped so the model never reasons about JSON
  escaping. Legacy queries / knowledge_base_id are still accepted.
- query_knowledge_graph is registered only when a graph-enabled KB is in
  scope and its description/tips were trimmed.

Compatibility
- Retired names in stored allowlists, presets and API calls are mapped to
  their successors by NormalizeAllowedTools at registration time; no data
  migration. Stored histories with old names still decode, render and
  redact. Calls to retired names return a replacement hint.
- modelcontext gains an "id" key space that decodes against both the chunk
  and document handle tables (decode-only), plus rendering for the
  document header, chunk roles, in-document matches and next_offset.

Prompts and surfaces
- The RAG, wiki researcher and hybrid templates now name the tools and the
  search -> read -> cite order; tool descriptions are short and contain no
  workflow instructions.
- Engine display names, history redaction, grounding prompt, IM rendering,
  the built-in MCP server (search_knowledge gains mode/limit, grep_chunks
  becomes keyword search, read_document gains query, wiki_search gains
  regex), frontend editor/renderers/i18n, presets, website docs and the
  changelog are updated accordingly.

Tests cover name mapping, mode validation and refusal, output shape,
read_document paging/context/query/scope, list_documents, wiki_search
escaping and legacy arguments, modelcontext decoding and rendering, and
registration-time aliasing and graph gating.

* fix(agent): address review findings on read_document, list_documents and model rendering

- read_document: fetch the second page whenever the first page did not
  reach the end of the document, so an unaligned window near the end no
  longer drops the trailing chunks (also fixes context windows around a
  chunk near the end); a query passed together with a cN handle now
  searches the owning document instead of being ignored; query mode stops
  collecting once the output budget is nearly spent and reports truncated;
  the document row is authorized in place instead of being fetched twice;
  the context read takes the total from the window fetch instead of a
  separate count query.
- list_documents: resolve the page's document scope in one batched tag
  lookup instead of one call per row.
- modelcontext: render read_document output once and read next_offset
  through the existing intValue helper.
- search_knowledge: point title/file-name lookups at list_documents.
- Tests for the boundary window, chunk context near the end, cN + query,
  and budget-bound query results.

* fix(agent): close compatibility gaps in the consolidated retrieval tools

- Wiki-only scopes: read_document and list_documents are registered
  whenever a wiki KB is in scope, since every KB stores chunks. Their
  capability requirement includes wiki but is marked auxiliary, so they do
  not widen a RAG agent's derived "all knowledge bases" filter to wiki-only
  bases (Go and frontend derivations both honour it).
- search_knowledge resolves the retrieval mode per knowledge base instead
  of refusing the call: FAQ and vector-only bases answer keyword requests
  semantically, keyword-only bases answer semantic requests by keyword.
  Results carry requested_mode and mode_fallbacks, the empty-result text
  names the fallback, and the call is refused only when no base in scope
  has a chunk index.
- The model rendering uses the mode the search actually ran with (it was
  always labelled semantic) and emits <mode_fallback> per degraded base.
- list_documents applies the @file/@tag scope before paging: tags are
  pushed into the database filter, pinned documents are loaded by ID, and
  total/next_page count only readable documents.
- read_document resolves chunk context by chunk_index through a new
  ChunkRepository.ListChunkNeighbors, so parent, summary and image chunks
  sharing the index sequence no longer shift the neighbourhood; offset is
  documented as a reading-order position, not a chunk index.
- wiki_search (agent and MCP) keeps its regular-expression default and
  only matches literally when the text is not a valid expression;
  regex=false forces a literal match and regex=true rejects invalid input.
- MCP grep_chunks keeps grep semantics: literal terms from the pattern
  fetch candidates from the keyword index (semantic for bases without
  one), and every returned chunk is verified against the pattern.

* docs(changelog): align the consolidation entry with regex wiki_search and grep_chunks semantics

* docs(mcp): note that grep_chunks recall is bounded by index candidates
2026-09-18 14:12:22 +08:00
lyingbug 182f67bd86
refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372)
* refactor(wiki): consolidate source_refs parsing and resolution

source_refs entries ("uuid" or "uuid|title") were parsed by ten
independent implementations that disagreed on whitespace handling, on
whether a model-supplied title could be trusted, and on how a leading
pipe was treated. The scope-enforced and legacy agent resolvers also
duplicated the lookup and the routing step looked every document up a
second time.

- types.ParseWikiSourceRef / FormatWikiSourceRef / WikiSourceRefMatchesKnowledge
  are now the only readers and writer of the format.
- resolveWikiSourceDocuments replaces resolveAuthorizedSourceRefs,
  resolveSourceRefs and wikiKnowledgeBasesForSourceRefs: one lookup per
  document, titles always rebuilt from server data, dedupe by canonical ID,
  unknown documents fail closed in every mode.
- removeSourceRef now matches padded or titled refs of a deleted document.

* refactor(data_analysis): expose the selected document as a fixed table name

The tool required the document ID inside the SQL text and blindly
string-replaced it with the physical DuckDB table name, which forced the
model-context codec to rewrite handles inside quoted SQL and let the
UUID-derived k_<uuid> table name reach the model through stored schema
summaries.

The document is now selected only by knowledge_id. Every query runs on a
dedicated connection where "dataset" is a temporary view over the
physical table (DuckDB temporary objects are connection-local, so
concurrent sessions stay isolated). The physical name remains accepted
for internal callers; any other table fails validation with an error
naming the table the model should use. Schema summaries, the schema
tool, the analyst prompt and the tool schema all name the same table, so
data_analysis.sql no longer carries source handles and its text codec
policy is removed.

* feat(modelcontext): report raw identifiers that survive encoding

Text-level compaction exists because raw UUIDs reach model context through
producers the registry never sees: error strings, slugs, results of
unknown shape. Retiring it requires knowing where those producers are.

Registry.LeakedIdentifiers scans an encoded message list for UUID-shaped
values in every application-composed field (user messages are skipped),
classifies each as registered (codec missed a rewrite) or unregistered
(producer emitted a raw ID), and SummarizeLeaks renders one line per
field. The agent engine and the three RAG pipeline call sites log the
report as a warning after EncodeMessages.

* fix(chat_pipeline): align RAG data analysis with the fixed table contract

The RAG data-analysis plugin still asked the planning model to fill in
knowledge_id and to write SQL from a prompt that exposed the raw
knowledge UUID, then passed the model's JSON straight to Execute. Under
the new data_analysis contract a query that names anything but the
dataset table is rejected, and the plugin logged that rejection at error
level and moved on, so the analysis vanished without any signal. A
"no analysis needed" plan took the same error path.

The pipeline now owns document selection: the model returns only
needs_analysis and sql, the prompt names the dataset table and never
shows the knowledge ID or physical table, and the plugin builds the tool
input itself. A rejected query is fed back to the model once for repair;
a query that still fails is reported through pipelineWarn with the SQL
and the tool error, and "not needed" is logged as such instead of as a
failure.

Covered by unit tests with a scripted model and executor: not-needed,
pipeline-owned knowledge_id, one-shot repair, persistent failure
surfaced, unparseable plan rejected.
2026-09-17 22:08:16 +08:00
lyingbug 5780cb0a71
fix(sandbox): separate file access from artifact publishing (#3356)
* fix(sandbox): separate file access from artifact publishing

* test(sandbox): align service coverage and pre-push checks with CI
2026-09-17 16:18:01 +08:00
lyingbug 7a98a8e3de
feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193)
* feat(browser): enhance BrowserIcon component and update localization strings

- Updated BrowserIcon component to accept dynamic width and height props.
- Revised localization strings in English, Japanese, Korean, Russian, and Chinese for improved clarity and user guidance regarding browser connection and usage instructions.
- Enhanced BrowserConnectionSettings view with better structure and user feedback elements.

* Default BrowserSkill tasks to a separate window

* refactor(prompts): streamline agent system prompts and context templates

- Updated agent system prompts to enhance clarity and specificity in user interactions.
- Revised context templates to better separate metadata and user requests, improving overall structure.
- Adjusted fallback and intent prompts to ensure consistent language handling and response requirements.
- Introduced new localization strings for browser search instructions across multiple languages.
- Added tests for agent prompt serialization and hydration to ensure template integrity and functionality.

* chore: wrap prompt strings to satisfy golangci-lint

Keep the settings, separate-window, and prompt commits pushable together
without rewriting their history.

* fix(frontend): match agent editor save test to serialized payload

The prompt-inheritance save path still closes after update; the test
should assert that payload, not the unsanitized form object.

* feat(browser): implement browser connection management and related tests

- Introduced a new `browserConnection` store to manage the state of browser connections, including online status and device information.
- Updated the `Input-field` component to handle browser connection states, enabling/disabling features based on the connection status.
- Added tests for the `browserConnection` store to ensure correct behavior under various scenarios.
- Enhanced the `Input-field` and `BrowserConnectionSettings` components to provide user feedback based on the browser connection state.
- Adjusted related components and views to integrate the new browser connection logic, improving overall user experience.

* fix(frontend): update model-save assertion for serialized agent payload

The agent editor still persists the selected model ID via a spread of
formData; the source test should match that payload, not the raw form.
2026-09-11 21:28:58 +08:00
lyingbug b60351f861
fix(agent): prevent MCP answers from citing unrelated KB sources (#3144)
* fix(agent): keep MCP citations separate from historical KB sources

* style(modelcontext): wrap citation prompts and regression tests

* test(service): align fallback citations with current evidence policy
2026-09-10 00:31:43 +08:00
wizardchen 4b04cd83ee fix(mcp): normalize JSON-encoded argument envelopes 2026-09-09 16:11:52 +08:00
wizardchen 73462de222 feat(mcp)!: streamline setup and use short routing handles
BREAKING CHANGE: Updates that explicitly include usage_instructions must supply nonblank text of at most 16000 characters. Omitting the field preserves existing instructions; legacy descriptions remain readable.
2026-09-09 16:11:52 +08:00
wizardchen 694f0413dd refactor(agent): discover and invoke MCP tools on demand 2026-09-08 12:01:36 +08:00
lyingbug 647848f395
fix(agent): improve web search and persistent page reading (#3086)
* fix(agent): improve web search and persistent page reading

* fix(agent): make web fetch snapshots waitable and retryable

Keep in-flight page fetches independent of a caller's short deadline so
content=true and continuations cannot cancel a shared download or miss a
snapshot that is still loading. Mark snapshot_expired as retryable, fetch
only the first three search hits in parallel, and stop forcing Brave's
country to US.
2026-09-07 23:09:27 +08:00
lyingbug d5cb842580
feat(agent): unify sandbox tools and drop host preloaded skills (#3033)
* feat(agent): unify sandbox tools and drop host preloaded skills

Read workspace and skill files through read_file, and run installed or staged skills through shell_exec(skill_name). Remove the unused skills/preloaded tree and keep generated-file previews stable across re-renders.

* fix(frontend): do not assume classList in markdown sanitize hooks

Node unit tests stub anchors without a DOM classList. Detect download cards from class/download attributes instead.
2026-09-05 16:45:59 +08:00
wizardchen dd647fefad refactor(skills): introduce write_skill_file and edit_skill_file tools for skill directory management
- Added write_skill_file and edit_skill_file tools to facilitate writing and editing files directly within the skill directory during installation.
- Updated the agent service to register these tools exclusively for the built-in skill installer, ensuring they operate within the correct scope.
- Enhanced error handling and validation to prevent unauthorized file access outside the designated skill directory.
- Improved documentation and prompts to clarify usage and limitations of the new tools, emphasizing the importance of proving import resolutions by executing them.
- Added tests to validate the functionality and security of the new file management tools.
2026-09-02 15:08:05 +08:00
wizardchen 21244d451a feat(agent): write workspace scripts and run them with installed skills
Generated PPT/report scripts overflowed the shell_exec command cap, and
session agents kept mutating the frozen skill tree or failing SkillHub
installs on display titles. Add write/edit sandbox tools, attach the
skill venv to /workspace scripts, and accept slug-based install names.
2026-08-29 22:45:18 +08:00
wizardchen f79052fe67 fix(agent): keep skill-script stdout on failure for the model and UI
Failed execute_skill_script results were reduced to "exited with code 1" plus a retry hint, so the chat card hid the real stderr/stdout and the next model round could not correct arguments like a missing chart axis.
2026-08-28 15:05:42 +08:00
wizardchen 10cc694fed feat(memory): introduce long-term memory feature with i18n updates
- Added a new configuration option for agents to enable or disable long-term memory.
- Implemented UI elements in the Agent Editor for managing long-term memory settings.
- Updated internationalization files for English, Korean, Russian, and Chinese to include new labels and descriptions related to long-term memory.
- Enhanced backend logic to support memory retrieval and management based on user and agent preferences.
- Introduced a new tool for searching long-term memory, ensuring it respects the agent's memory settings.
- Added tests to verify the functionality of the new memory features and their integration with existing systems.
2026-08-13 18:00:02 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
wizardchen b581094c7b fix(web_fetch): address review follow-ups for structured fetch results
Restore HTTP-only 15s pipeline fetching, reject SPA shell pages when
browser rendering fails, surface per-URL failure fields in the UI, and
keep structured model output when all URLs fail.
2026-07-28 22:17:31 +08:00
langcaiye 75a2eabe20 fix(agent): restore safe rendered web fetch fallback 2026-07-28 22:17:31 +08:00
langcaiye a3da5cb57a fix(agent): preserve web evidence when page fetches fail 2026-07-28 22:17:31 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen 718b7347d0 fix(agent): harden search-target scope and model-handle policies
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
2026-07-27 11:05:43 +08:00
wizardchen 6663268504 refactor(modelcontext): unify handle terminology and document the merged package
Internal-only rename of the remaining alias-era names (ChunkAlias ->
ChunkHandle, aliasForRealValue -> handleForDurable, realForAlias ->
durableForHandle, OrphanAliases -> OrphanHandles, shortSourceAliasRE ->
shortSourceHandleRE, locals and comments). Exported API names are
unchanged. The README now describes the single-package layout, the
handleTable/streamHold primitives and the sourceKeySpaces policy table.
2026-07-27 11:05:43 +08:00
wizardchen 69b792398e refactor(modelcontext): consolidate stream decoding on one suffix-hold primitive
streamHold captures the shared shape of streaming-safe handle decoding
(withhold a trailing byte run that could still become a handle, decode
what is released, decide the fate of a held tail at close). The resource
decoder, HandleStreamDecoder and the orphan-resource filter become
instances whose hold/flush functions are the original logic moved
verbatim; the citation stream expander stays a dedicated tag parser.

The duplicate res://\d+ regexes merge into resourceHandleShapeRE, all
stream types now live in stream.go, and orphan_stream.go is removed.
2026-07-27 11:05:43 +08:00
wizardchen 213ca0b42e refactor(modelcontext): rebuild every handle space on one generic table
handleTable[M] is the single bidirectional primitive behind all model
handles: source chunks/documents/KBs/web pages, res:// resource handles,
and the exported HandleTable (iN issue handles, ingest ref-N/c000
handles). Metadata lives under the same lock as the mapping, web
references dedup on the canonical URL while decoding to the raw URL, and
the append-only counter preserves historical numbering exactly.

Behavior is unchanged and proven by the moved test suites; the per-space
text-rewrite strategies (plain substring for resources, word-boundary
regex for HandleTable, shape-regex lookup for sources) deliberately stay
in their wrappers because each is load-bearing wire behavior. The unused
ChunkReference.Alias field is dropped — handles are derived from the
table, never stored in metadata.
2026-07-27 11:05:43 +08:00
wizardchen 286eadac0a refactor(modelcontext): unify source-key gating behind one dispatch table
sourceKeySpaces becomes the single table of ID-bearing keys: it drives
handle registration (registerSourceIDByKey) and the alias-decode gate
(walkJSON / unresolved-handle audit), replacing the separate decodableKeys
list that had to be kept in sync by hand. The per-tool sourceIDKeys policy
remains the outer gate; a new test asserts every policy key has a
registration space.

The generic nil-policy wrappers (sourceRegistry.DecodeToolCalls,
EncodeMessages) are removed; the two tests that exercised anonymous-tool
decoding now go through the public Registry with real tool names.
2026-07-27 11:05:43 +08:00
wizardchen 3b9de5de8e refactor(modelcontext): fold llmreference into modelcontext as source codec
Mechanical move: internal/llmreference becomes in-package code.
registry.go splits into sources.go (handle registration, tool-argument
codec, replay encoding) and citations.go (protocol prompts, <ref/> ->
<kb/>/<web/> expansion, stream expander); model_output.go moves verbatim.

Renames only: Registry -> sourceRegistry, StreamExpander ->
citationStreamExpander, ProtocolPrompt -> sourceProtocolPrompt,
ToolArgumentPolicy -> toolArgumentPolicy. ChunkReference becomes a native
modelcontext type (was an alias). Unreferenced DocumentAlias,
KnowledgeBaseAlias and ExpandResponse are dropped. Tests move intact.
2026-07-27 11:05:43 +08:00
wizardchen c01b00313b refactor(modelcontext): fold llmresource into modelcontext as resource codec
Mechanical move: internal/llmresource/registry.go becomes
internal/modelcontext/resources.go with Registry renamed to
resourceRegistry and StreamDecoder to resourceStreamDecoder. No logic
changes; the unused DecodeResponse method is dropped. Tests move intact.
2026-07-27 11:05:43 +08:00
wizardchen 5e413c6478 refactor(agent): introduce modelcontext registry and fix handle-codec bugs
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.

Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
  UnresolvedToolHandles)
2026-07-27 11:05:43 +08:00