Commit Graph

79 Commits (06eeb95b4e050ede712ef16fb6053e4fa5fcfbde)

Author SHA1 Message Date
nullptr-error 5df788dc77
feat(sandbox) (#2301)
* cubesandbox support

* feat(agent): add sandbox shell execution, file management tools, and Artifacts collection/download system

- Add three sandbox agent tools: shell_exec / sandbox_ls / sandbox_read
  - shell_exec: safely execute shell commands in session sandbox with timeout & output truncation
  - sandbox_ls: enumerate files under sandbox artifact output directory with recursive listing & pagination
  - sandbox_read: read sandbox file content with offset-based chunked reading
- Add Artifacts system:
  - ArtifactCollector: real-time collection of sandbox tool output files, attached to messages
  - /api/artifacts/{session_id}/{message_id}/{filename}: download endpoint for frontend
  - ChatArtifactsDrawer.vue: frontend artifacts sidebar UI
  - DB migration 000066_message_artifacts: add artifacts column to messages table
- Attachment sandbox staging: upload user attachments to session sandbox on QA request for skill tool access
- Update Agent Prompts: add shell/sandbox tool usage constraints and safety rules
- Extend Agent Observe: include agent steps in Langfuse observability tracing
- Refactor Sandbox SessionManager: support session-scoped file listing, reading, and status queries
- Frontend: render artifact events in AgentStreamDisplay, i18n updates for 4 locales
- Add comprehensive tests: shell_exec_test, sandbox_read_test, artifact_collector_test, etc.

* feat(agent):add remote sandbox client abstract  & client errors

* feat(sandbox): implement Redis binding store with distributed lifecycle lock;

* feat(sandbox): add remote session lifecycle with ownership context

Implement remoteSessionLifecycle to coordinate persistent remote sandboxes
across multi-instance deployments. Supports Resolve (lazy provision,
recovery, race-safe create) and Destroy with compare-delete semantics.
Add ownership context isolation to decouple remote resource cleanup from
request cancellation.

* feat(sandbox): add CubeRemoteClient adapter for RemoteSandboxClient

Implement CubeRemoteClient as a provider-neutral RemoteSandboxClient
implementation. Error normalization, timeout mapping, and metadata
support are fully integrated. SDK details remain private to the
adapter.

* refactor(sandbox): decouple SessionBoundManager from Cube

Remove internal/sandbox/cube.go (CubeSandbox, sync.Map reaper). Add
RemoteSandbox as provider-neutral Sandbox interface implementation.
Rewrite SessionBoundManager to depend on RemoteSandboxClient +
SessionSandboxBindingStore + SessionExistenceChecker, with all
lifecycle ops delegated to remoteSessionLifecycle (Redis + distributed
lock). Application-layer interfaces migrated to RemoteDirEntry/StatEntry
with time.Time ModTime. Cleanup delegated to provider idle-timeout.

* refactor(agent): enhance session capability handling in sandbox manager

- Introduced sessionFileStore and sessionShellExecutor functions to centralize capability checks for session-scoped file and shell execution.
- Updated agent service to register tools based on the presence of session filesystem capabilities, improving compatibility with various sandbox backends.
- Refactored tests to validate behavior when session capabilities are absent, ensuring robust handling of different sandbox types.
- Removed Cube-specific checks, promoting a more provider-neutral approach to session management.

* refactor(sandbox): add E2B backend, modernize Cube adapter, and enable multi-instance Redis binding

- Add e2b_remote_client.go (~850 lines) — full RemoteSandboxClient implementation
  covering lifecycle, execution, and filesystem ops (ListDir/Stat/MakeDir/Remove)
- Wire E2B into NewManagerFromType and container/bootstrap path (buildE2BManager)
- Dependency: matiasinsaurralde/go-e2b v0.1.0

- Cut ~1500 lines of legacy SDK wrapper (cube_client.go, cube_envd.go,
  cube_defaults.go) and replace with direct *cubesandbox.Client usage
- Parse CubeProxy URL directly in adapter (parseProxyURL), removing the
  old hand-rolled envd transport layer
- Pass lifecycle extra fields (onTimeout, autoResume) via SDK CreateOptions.Extra
- Target Cube deployment v0.5.1

- Add RemoteNetworkPolicy to RemoteCreateRequest with AllowInternetAccess,
  AllowPublicTraffic, AllowOut/DenyOut — shared by Cube and E2B adapters
- Both adapters default to permissive (internet+public on) for backwards compat

- Add selectSessionBindingStore() — Redis-backed or in-memory, controlled by
  WEKNORA_SANDBOX_REDIS_NAMESPACE / WEKNORA_REDIS_NAMESPACE
- Add repositorySessionExistenceChecker — checks session existence via DB
  (gorm.ErrRecordNotFound → not exists, transient errors → preserve binding)
- newSandboxManager() now accepts redisClient + sessionRepo; deprecated
  hardcoded NewMemorySessionSandboxBindingStore() in production path

- buildSessionCreateRequest default timeout action: Kill → Pause+AutoResume
  (Cube and E2B). Sessions survive brief idle periods; providers still
  handle TTL-based reaping.

- buildSessionCreateRequest() and effectiveHTTPTimeout() parameterized
  by RemoteProvider (was hardcoded to Cube Config fields)
- NewSessionBoundManager validates provider and delegates defaults to
  per-provider apply*Defaults helpers

- Legacy SDK wrapper: cube_client.go, cube_envd.go, cube_defaults.go,
  cube_envd_test.go, cube_net_diag_test.go, cube_test.go (~2500 lines)
- Unused helpers: DefaultManager.validateExecution, IsRemoteTerminal,
  IsRemoteUnsupported, IsRemoteTransient, ShouldPreserveRemoteBinding
- Dead Config fields: CubeEnvdPort, CubeIdleTTL, AllowedCmds
- Dead ExecuteResult method: GetOutput
- Unused test: session_artifacts_integration_test.go
- Preloaded skill: sandbox-test-python

- Extract shared httpErrorKind() from per-adapter normalize functions
- Add SupportsFilesystemEnumeration capability flag — gates SessionFileStore
  advertisement (E2B returns true post-reconnect, Cube always true)
- Fix stale comments: Cube-only → remote backends, remove "tomorrow"/"future"
- Bump session sandbox TTL defaults: 604800s → 300s (.env.example)
- Update docs/agent-skills.md: full multi-instance deployment guide

Total: 29 files, +1188 / -3721

* feat(agent): bind collected skill artifacts to owning message in resource registry

When ArtifactCollector persists a skill-generated file it now registers a
resource:// binding (owner_type=message, owner_id=assistantMessageID,
relation=artifact) in addition to the messages.artifacts JSONB record. This
mirrors how knowledge uploads and chat attachments bind their resources, so
artifacts can be enumerated and garbage-collected through the resource
registry instead of only via the message JSONB column.

- ArtifactCollector: inject ResourceCatalog, add messageID to Collect/
  maybePersist, and add best-effort bindArtifactResource (binding failures are
  logged, never drop the artifact; only resource:// refs are bound).
- agent_stream_handler: pass assistantMessageID into Collect.
- Renumber message_artifacts migration 000066 -> 000077 to resolve the
  version collision with existing migrations (upstream now uses 000075/000076).
- Tests: update collector signatures and add coverage for the binding path
  and for graceful degradation when Bind fails.

Note: binding applies to newly generated artifacts only; existing rows are
not backfilled.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(sandbox): harden remote sandbox security and artifact collection

Disable Cube local fallback so skill scripts never run on the host when
the remote provider is unavailable. Require Redis for cube/e2b binding
unless WEKNORA_SANDBOX_ALLOW_MEMORY_BINDING=true for single-node dev.

Collect artifacts with WithoutCancel so user stop does not drop files,
reject shell_exec work_dir outside /workspace, and refuse session-scoped
Execute when a local fallback would be engaged.

* fix(ci): gofmt sandbox files and resolve artifact drawer type error

Apply gofmt to PR-touched Go files and narrow ChatArtifactsDrawer
props in AgentStreamDisplay so session-id is always a string.

* fix(sandbox): adapt to upstream main after rebase

Renumber the message_artifacts migration 000077 -> 000079: upstream now
uses 000077_remove_wiki_log and 000078_chunk_editing_and_custom_metadata.

Declare empty model-handle policies for shell_exec / list_sandbox_files /
read_sandbox_file. Upstream added a coverage test requiring every
UI-exposed tool to declare one; sandbox output is arbitrary skill-authored
content, so it gets the same treatment as execute_skill_script.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat(sandbox): replace single tenant sandbox config column with a configs table to support multi-sandbox config per tenant.

* feat(sandbox): add sandbox config entity, cordon lease and session pin field

* feat(sandbox): add sandbox config repository with tenant scoping and cordon lease

* feat(sandbox): stamp sandbox config ID onto sandbox metadata

* feat(sandbox): scope sandbox listing, release and reaping to a single config

* feat(sandbox): resolve managers per config and refuse cordoned or missing configs

* feat(sandbox): implement session pinning and config identity change detection

- Add SessionSandboxPinner with read/pin/clear to bind session to config
- Refactor four entry points (execute/attachment/artifact/destroy) to use pinned config
- Add config_identity.go to detect identity changes by effective values (APIURL/APIKey/Domain/ProxyURL)
- Add SandboxIdentityChanged with key-masked comparison to prevent false 409 conflicts
- Fix artifact collection for default backend and destroy for legacy empty-pin sessions

* feat(sandbox): add multi-tenant sandbox config management and HTTP APIs

Service layer:
- Add sandbox config business service supporting CRUD and inventory listing
- For non-identity changes (e.g., template, TTL), save directly without locking or remote interactions
- For identity changes (e.g., API key, endpoint), reject save if there are still live sandboxes under this config on the remote side,
  and return structured inventory details (sandbox count, affected sessions, related agents) for frontend prompting
- Before deleting a config, check remote live sandboxes; if occupied, reject and prompt user to end sessions or create a new config
- Identity comparison based on effective values to avoid normal saves being misidentified as credential rotation

HTTP interfaces and routing:
- Add REST endpoints for sandbox config (List/Create/Get/Update/Delete/Inventory)
- Mask sensitive credentials (API key, etc.) in all read responses
- Return 409 with structured inventory details on rejection scenarios
- sandbox-check probing supports specifying an existing config_id directly
- Remove old handler file, consolidate validation logic into service layer
- DI wiring and route registration for all six endpoints

* feat(sandbox): add sandbox_config_id field to allow agent referencing named sandbox config

- Add SandboxConfigID field to CustomAgentConfig with omitempty; empty value falls back to global defaults
- Validate config existence on create/update; return 400 if not found
- Inject sandboxConfigService into CustomAgentHandler
- Add unit tests covering field serialization and validation logic

* feat(sandbox, agent): Migrate sandbox config to multi-config CRUD, support named backend selection and workspace scripts policy

Migrate frontend API from legacy KV single-config to /sandbox-configs REST interface, supplement type definitions, detection methods and policy fields; refactor sandbox settings page to multi-config card grid, manage configs with cube/e2b tabs, add editor drawer for create/edit/delete, connection testing and occupancy hints, and add workspace scripts policy toggle; add sandbox config dropdown in Agent editor skill section, integrate config caching, and support selecting named config or falling back to deployment default.

* feat(sandbox): Unify sandbox disable semantics and add tenant-level policy master switch

- Backend: Unify interception in resolveTenantSandboxForConfig with priority: global disable > tenant disable > default fallback > named config; add deploymentSandboxDisabled() using env var to avoid mis‑impacting named E2B; update WorkspaceSandboxPolicy comment as master switch for entire workspace; add unit tests for various scenarios.
- Frontend: Sync i18n copy: button text changed to "Disable/Enable sandbox execution"; confirmation dialog clarifies impact scope; success/failure messages aligned.
- Semantic layering: WEKNORA_SANDBOX_MODE=disabled disables globally; tenant‑level "Disable sandbox execution" only disables for the current workspace.

* feat(sandbox): Add SSRF protection, Agent binding to named configs, and reference query capabilities

- Add outbound URL security guard to validate Cube/E2B API/Proxy addresses against SSRF (including cloud metadata), support allowing private endpoints via env var, link-local always blocked
- Add named backend type validation, only cube/e2b can be stored as workspace named config
- Add sandbox config ID field to Agent at runtime, resolve sandbox for attachments and execution based on Agent's selected named config instead of always global default
- Add query to list Agents referencing a sandbox config ID, useful for deletion/changes to show dependents
- Adapt artifact collector and other callers to fill missing resolve signature parameter

* fix(sandbox): fix session teardown and config-update, add HTTP pool sharing for Cube

Session teardown & deletion ordering
- Move sandbox destroy (and pin `Clear`) before soft‑delete operations (`DeleteSession`/`BatchDeleteSessions`/`DeleteAllSessions`) to prevent remote sandbox leaks and stale pins.
- Pass deployment‑wide `SessionBoundManager` as fallback into `resolveTenantSandboxForConfig` for sentinel pins (`"-"`) to support deployment‑default sessions.

## Config update edge cases
- When identity fields change but old credentials cannot build a client or list config sandboxes, log a warning and proceed with the update instead of returning `sandbox_inventory_unverifiable`; live sandboxes still return 409.
- UI: move `sandboxes_still_live` / `unverifiable` alerts to the top of the sandbox config drawer and auto‑scroll on save refusal and connectivity check results.

Artifact collection for Cube skills
- Pre‑create `/workspace/output` via envd with correct ownership/permissions for `DefaultSandboxExecUser` to ensure Cube skill artifacts are collected properly by `ArtifactCollector`.

HTTP pool sharing for named Cube configs
- Named sandbox configs rebuilt a client on every `Resolve`, causing fresh TCP connections to CubeAPI and the envd proxy on each request.
- Add `CubeTransportPool` and `cubeSplitTransport`:
  - Control plane traffic reuses the same `SharedTransport` as E2B.
  - Data plane traffic uses a guarded transport cached per proxy host:port, preserving the sandbox `Host` header.
- Wire the pool through the tenant resolver `buildClient`; keep SDK defaults for the deployment‑default manager and connectivity probes.

* feat(sandbox): sandbox execution DefaultUser change from root to user

* fix(multi-sandbox-config-per-tenant): align deployment config with tenant resolver and centralize defaults

- Merge form overrides onto DeploymentConfig() via WEKNORA_SANDBOX_* env vars
  instead of DefaultConfig() hardcoded values, allowing empty fields to inherit
  from .env rather than Go constants
- Centralize deployment baseline parsing in sandbox/defaults.go with
  DeploymentConfig, CubeConfigFromEnv, E2BConfigFromEnv
- Delegate container startup to shared helpers to avoid drift
- Propagate tenant env_vars through ResolveEffectiveConfig into
  buildSessionCreateRequest for both Cube and E2B session sandboxes

* fix(multi-sandbox-config-per-tenant): fix some bugs frontend.

* fix(sandbox): guard check-path dials, fix staging gate and shared-agent binding tenant

Three defects found while reviewing the remote sandbox work:

- The Cube connectivity probe and the deployment-default Cube/E2B managers
  built clients without a guarded transport, so an admin-supplied api_url or
  proxy_url could reach link-local addresses via DNS rebinding even though
  save-time validation passed. All three now dial through the same outbound
  guard the per-tenant resolver enforces.

- Attachment staging was gated on the process-wide manager, so a workspace
  pointing an agent at a named Cube/E2B config while the deployment default
  stayed docker/local would run its tools remotely against an empty
  /workspace/input. The gate now resolves the manager the session actually
  runs on.

- Session to sandbox bindings were keyed by the tenant in context, which a
  shared agent deliberately swaps for the agent owner's so models, KBs and
  named configs resolve in the sharing workspace. Session deletion only ever
  knows the session's own tenant, so teardown looked up a key nobody wrote
  and abandoned a paused MicroVM that keeps billing. The binding tenant is
  now carried separately and survives CloneContext.

* chore(sandbox): renumber migrations above upstream and de-collide test stub

Upstream main took 000079 (knowledge_folder_path) and 000080
(knowledge_base_auto_tag_config) while this branch was open, so the three
sandbox migrations move to 000081-000083. golang-migrate requires unique,
ordered versions, and the RAISE NOTICE labels move with the file names.

Main also added a handler test declaring stubSessionService in the same
package; the artifact download test's configurable variant is renamed rather
than merged, since the two stubs answer GetSession differently.

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-10 21:28:20 +08:00
KKtwo 3011f132e9 feat: support Volcengine rerank 2026-07-20 19:58:48 +08:00
shenk-b 5ee96019e0 feat(tracing): migrate langfuse client to OTLP/OTel wire protocol
Migrate WeKnora's Langfuse tracing from the legacy /api/public/ingestion
batch API to the OpenTelemetry Go SDK emitting OTLP/HTTP to
/api/public/otel/v1/traces — the Langfuse v3+ standard ingestion protocol.

Why: Langfuse v3 deprecated the legacy ingestion API in favor of OTLP, and
OTel-only backends (e.g. the LiteFuse Doris fork) dropped it entirely, so the
legacy client was incompatible. Migrating to the OTel SDK also adds W3C
traceparent cross-service propagation: an upstream caller's trace id is
inherited so its run and the WeKnora call land under one trace in the UI.

What:
- Manager owns an OTel TracerProvider backed by an otlptracehttp exporter
  (Basic auth + the x-langfuse-ingestion-version gate header). Spans are
  buffered by BatchSpanProcessor and exported complete on End, so a long
  trace no longer fragments across flush batches.
- *Trace/*Span/*Generation wrap OTel spans; the public API (StartTrace/
  Span/Generation, handles, InjectTracing, asynq/gin middleware, Config,
  TokenUsage) is unchanged, so callers (agent/chat/handler/scheduler/asynqdl)
  need no edits.
- W3C traceparent: GinMiddleware extracts traceparent from incoming requests;
  AsynqMiddleware + InjectTracing propagate it across the asynq payload.
  types.TracingContext gains a LangfuseTraceparent field.
- logger.CloneContext preserves the active OTel span across context rebuilds
  so child spans (agent engine, summary generation) stitch to the HTTP trace
  instead of orphaning into separate traces.
- StartSpan/StartGeneration re-establish the parent from the *Trace handle
  when the ctx lost the active OTel span, covering background/derived contexts.

Compatibility: works with Langfuse v3+ (cloud / self-hosted / the bundled
docker-compose profile) and LiteFuse. Drops Langfuse v2 (legacy-only) support.
Uses the OTel Go SDK already (indirectly) in go.mod, promoting
otlptracehttp to direct; no third-party langfuse-go library.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 16:30:45 +08:00
wizardchen ac8c5c5540 chore(deps): resolve open Dependabot security alerts across Go, npm, and pip.
Bump vulnerable runtime and transitive dependencies in docreader, frontend, and the Go module graph to address current GitHub security advisories.
2026-07-11 12:15:39 +08:00
wizardchen 63198c5d9f feat(models): cap background LLM concurrency per model
Background ingestion/enrichment stages (summary, question, graph,
multimodal) all target the same chat model, and asynq queue weights are
scheduling priority rather than throttling. A large upload could therefore
burst up to the entire worker pool of concurrent chat calls against a
single provider, unbounded across replicas, with no model-side rate limit.

Govern concurrency at the chat-client layer (the one place that sees every
task type), keyed by model ID:

- internal/models/limiter: Redis ZSET + Lua distributed semaphore with a
  lease heartbeat, expired-lease reclaim for crashed holders, and fail-open
  on any backend error or context cancellation.
- chat: concurrencyChat decorator (outermost in NewChat) acquires a
  per-model slot only for background tasks; interactive chat is never gated.
- router/task: asynq middleware tags every worker execution as background.
- container: wire the Redis limiter with WEKNORA_MODEL_MAX_CONCURRENCY
  (model.max_concurrency), default 8; enabled only when Redis is present.

Tested with miniredis: concurrency cap, expired-lease reclaim, per-key
isolation, and all fail-open paths (nil client, backend down, cancelled ctx).
2026-07-09 20:40:51 +08:00
养乐多 baada32cb6 fix(im/feishu): actually close WebSocket long connection when channel is stopped
Deleting or disabling a Feishu channel left its WebSocket long connection
alive: the Lark SDK's Start blocks on a bare select{} and never observes
ctx, and v3.6.1 exposed no public Close(), so the wsCancel() cleanup was a
no-op. The socket stayed open, the SDK kept auto-reconnecting, and Feishu
kept pushing messages — causing cross-environment message duplication when
the same bot was bound elsewhere.

Fix in three layers:

1. Connection teardown
   - Upgrade oapi-sdk-go v3.6.1 -> v3.9.7 (adds Client.Close()).
   - Add LongConnClient.Close() wrapping wsClient.Close(); factory's cleanup
     now calls Close() first, then wsCancel() as a belt-and-braces fallback.

2. Multi-instance coordination
   - wsLeaderRetryLoop: after acquiring leadership, re-check the DB before
     starting; abort if the channel was deleted/disabled while waiting.
   - wsLeaderRenewLoop: on each renewal, re-check the DB; step down if the
     channel was deleted/disabled on another instance (HTTP request may have
     been served elsewhere), so the leader doesn't keep the connection open
     until process restart.

3. Robustness
   - DB checks distinguish gorm.ErrRecordNotFound (channel truly gone ->
     stop/abort) from transient DB errors (skip this round, retry next tick)
     so a DB hiccup can't cause a spurious stop or permanent leader-abort.
   - startChannelInternal now stops any existing channelState for the same
     ID before overwriting, making it idempotent against concurrent start
     paths and preventing adapter/long-connection leaks.

No interface changes; webhook and long-poll modes unaffected.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-06 23:10:45 +08:00
wizardchen e40ad802d7 feat(datasource): add RSS feed connector with full-text ingestion
Enable private RSS feeds as a knowledge base data source with custom
headers, article body fetching, and Markdown output for incremental sync.
2026-06-23 17:30:38 +08:00
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00
wizardchen dda16436d0 feat(lkeap): integrate Tencent Cloud LKEAP Rerank support
- Added support for Tencent Cloud LKEAP Rerank, including new configuration options for SecretId and SecretKey.
- Updated frontend components to handle LKEAP-specific fields and enhance user experience with appropriate labels and placeholders.
- Enhanced backend logic to manage LKEAP credentials and integrate with the new rerank API.
- Improved localization for LKEAP-related terms across multiple languages, ensuring clarity and consistency in user-facing messages.
- Introduced tests for LKEAP Reranker functionality to ensure reliability and correctness in handling requests.
2026-06-04 17:22:18 +08:00
wizardchen 7d1fe6f11b chore(deps): add opensearch-go v4.6.0 to go.mod and update go.sum
- Added the opensearch-go v4.6.0 dependency to the go.mod file.
- Removed the indirect requirement for opensearch-go from go.sum as it is now a direct dependency.
2026-05-26 21:13:56 +08:00
ochan.kwon 11c3236e52 feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3)
First half of the gated OpenSearch k-NN driver introduced in PR 1
(#1445) by way of #1440. PR 2a ships a hollow, interface-compliant
shell of the `internal/application/repository/retriever/opensearch/`
package — every behavioural method (Save / BatchSave / DeleteBy* /
Retrieve, plus the previously-stubbed CopyIndices / BatchUpdate* /
EstimateStorageSize / swapToVersion) returns `ErrFeatureNotEnabled`
or a conservative sentinel value. PR 2b lands the real read/write
implementations in dedicated files (`query.go` + `retrieve.go` +
`crud.go`) and replaces the stubs accordingly.

Strict feature-gate (unchanged from PR 1): no entry is added to
validEngineTypes / GetVectorStoreTypes / retrieverEngineMapping /
BuildEnvVectorStores / container env path / engine factory switch,
so the driver remains unreachable. Attempting to register an
`engine_type=opensearch` VectorStore continues to fail with the
existing "not a valid engine type" error.

What lands in PR 2a
-------------------

Driver skeleton (6 production files + 2 test files, ~1170 + ~1115 LoC):

- `repository.go` — Repository struct + NewRepository constructor
  that validates cluster reachability + OS version (2.4+ / 3.x;
  primary tested 3.3.2) + k-NN plugin presence on every cluster
  node. sync.Once-guarded ensureReady(ctx, dim) for lazy per-
  dimension index creation, with transient errors not cached so a
  momentary cluster blip does not permanently poison a dim.
  sanitizeIndexName enforces a strict OS-compatible name spec.
  probeVersion uses robust strings.Split/Atoi parsing for
  pre-release suffixes and missing-patch versions. EngineType
  returns the PR 1 constant; Support returns [keywords, vector].
- `transport.go` — newOpenSearchClient ships TLS posture
  (MinVersion TLS 1.2, opt-in InsecureSkipVerify, forward-secrecy-
  only cipher list) and transport tuning for the driver. Caller
  exists only in PR 3 (container.go + engine_factory.go); PR 2a
  remains gated dead code.
- `mapping.go` — buildIndexMapping(cfg, dim) produces the full
  knn_vector + HNSW + content-analyzer mapping with every *_id
  field as an explicit keyword and source_type as integer.
  buildKeywordsMapping ships the dim-less keyword-only index
  mapping used by the no-embedding save path. createIndexAndAlias
  creates <alias>_v1 and aliases <alias> to it, with best-effort
  orphan cleanup and mapping-drift detection.
- `config.go` — internalCfg (value type) applying OpenSearch
  defaults (hnsw_m=16, ef_construction=100, ef_search=100,
  shards=4, replicas=1, engine=lucene).
- `errors.go` — nine sentinels (ErrIndexNotFound,
  ErrDimensionMismatch, ErrAuth, ErrTransport,
  ErrVersionUnsupported, ErrConfigInvalid, ErrFeatureNotEnabled,
  ErrBatchTooLarge, ErrCircuitBreaker). Repository never imports
  apperrors; PR 3's engine factory wraps these to typed AppError
  2200/2201.
- `stubs.go` — every behavioural method returns
  ErrFeatureNotEnabled. EstimateStorageSize returns a conservative
  HNSW lower-bound estimate (not 0) so the Phase 2 KB-delete guard
  fails-closed for non-empty KBs.

Tests (~1115 LoC, 50 cases):

- `repository_test.go` — interface satisfaction, sentinel mapping,
  sanitizeIndexName positive/negative matrix, semver parsing
  (pre-release / missing-patch), buildIndexMapping JSON shape pin
  (Lucene + Faiss + Keywords), probeVersion matrix (OS 1.x / 2.2 /
  2.5 / 2.11 / 3.x / 3.0.0-rc1 / ES rejection), probeKNNPlugin
  multi-node coverage, ensureReady concurrency + per-dim isolation
  + transient retry, NewRepository storeID validation, all 11
  stubs (CopyIndices, BatchUpdate*, EstimateStorageSize,
  SwapToVersion + Save / BatchSave / Retrieve / DeleteBy*),
  wrapTransport sentinel mapping + leak guard, isNotFound /
  isAlreadyExistsError, drainAndClose / limitedDecode helpers.
- `transport_test.go` — TLS defaults / opt-in InsecureSkipVerify /
  TLS 1.2 pinning / cipher list / transport tuning.

Single dependency addition: github.com/opensearch-project/
opensearch-go/v4 v4.6.0 in go.mod/go.sum.

SDK quirks discovered (opensearch-go v4.6.0)
--------------------------------------------

PR 2a includes the workarounds for two of three SDK limitations
that landed during full implementation (the third, Refresh:*bool,
only affects the delete path that ships in PR 2b):

- AliasExists method passes dataPointer=nil to its internal do(),
  which means non-2xx responses come back as a plain
  *errors.errorString ("status: 404 Not Found") rather than as
  *opensearch.StructError. aliasExists therefore inspects
  resp.StatusCode directly (resp is returned even when err is
  non-nil) and only falls back to wrapTransport for the "no
  response at all" case.
- sync.OnceReset is not in the standard library; the keyword-only
  index uses a mutex + ready/err flag pattern so transient failures
  can be retried by the next caller. The per-dimension path uses
  the `once map[int]*sync.Once` delete-and-recreate trick.

Test fixes folded in
--------------------

While doing a full `go test ./...` against PR 1-merged main, two
deterministic regressions surfaced that block a clean run-everything
signal. Both are unrelated to the driver and are folded into PR 2a
so the PR's own CI run is green:

(1) Follow-up to #1445 — fanout test missed the new normalizer policy
    (internal/application/service/knowledgebase_search_fanout_test.go,
    +46 / -6). #1445 changed EngineAwareNormalizer for ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite /
    Qdrant / TencentVectorDB / Doris from (score+1)/2 to clamp01
    passthrough (those engines surface non-negative cosine to the
    normalizer per Lucene script_score non-negative invariant for
    ES, k-NN plugin SpaceType.COSINESIMIL.scoreTranslation for
    OpenSearch, engine-internal or IR-normalized conversions for
    the rest). Milvus is now the only engine that still surfaces
    raw signed cosine in [-1, 1].

    TestRetrieveFromStores_MixedEngine_Normalizes still asserted
    the old cosine-shift behaviour for ES (raw -0.4 → expected 0.3)
    which under passthrough now becomes clamp01(-0.4) = 0. The
    normalizer's own _test.go was updated at #1445 time, but this
    fan-out integration test was not.

    Fix: rewrite the godoc to spell out the two engine groups;
    restate sub-case 2 as ES passthrough on a production-possible
    mid-range cosine (0.3 → 0.3, PG out-ranks ES); add sub-case 3
    pinning the cosine-shift branch via Milvus -0.4 → 0.3.

(2) Pre-existing — SSRF whitelist singleton race surfaced by this run
    (internal/utils/security.go + internal/utils/security_test.go +
    internal/infrastructure/web_search/searxng_test.go,
    +33 / -9). loadSSRFWhitelist in internal/utils/security.go is
    cached via sync.Once on first call. The internal reset helper
    resetSSRFWhitelistForTest was unexported, so tests in other
    packages could not reset and saw whatever whitelist was cached
    by the first sync.Once.Do() in the same test binary. In
    internal/infrastructure/web_search/, TestValidateProxyURL runs
    before TestValidateSearxngBaseURL alphabetically and exercises
    ValidateURLForSSRF with no SSRF_WHITELIST set, caching an empty
    whitelist; the later setenv in searxng_test then has no effect
    and 127.0.0.1 is rejected with "hostname 127.0.0.1 is restricted".
    Pre-existing on main; surfaced now because this PR was the
    first to do a full `go test ./...` run on top of #1445.

    Fix: capitalize the helper to ResetSSRFWhitelistForTest (the
    ForTest suffix is the test-only contract); update in-package
    callers; in web_search/searxng_test.go import internal/utils
    and call ResetSSRFWhitelistForTest around the env mutation in
    both TestValidateSearxngBaseURL and TestSearxngProvider_Search.
    No production code path changes.

Roadmap
-------

- PR 2b (next, depends on this PR) — read/write implementations:
  query.go + retrieve.go + crud.go land their real bodies; stubs
  for Save / BatchSave / DeleteBy* / Retrieve in stubs.go are
  removed; corresponding CRUD/retrieve/filter test cases (~430
  LoC) join repository_test.go.
- PR 3 — activation switch + async paths (CopyIndices,
  BatchUpdate*, large-batch async deletes) + i18n + docker-compose
  dev profile. After PR 3 merges, the OpenSearch driver becomes
  reachable via either `engine_type=opensearch` VectorStore or
  `RETRIEVE_DRIVER=opensearch` env.

Backward compatibility
----------------------

- New package — additive only. No existing file modified except
  go.mod / go.sum, the two test files in (1)/(2), and the
  test-only export rename in utils/security.go.
- Driver is unreachable: no registry path activates it.
- No SQL migration.
- The PR 1 normalizer case for OpenSearch remains unreachable
  here (no driver instance produces a result yet).

Test plan
---------

- [x] go build ./... clean
- [x] go vet ./... clean
- [x] go test -race -count=1 ./internal/application/repository/retriever/opensearch/... passes
- [x] grep -r "case types.OpenSearchRetrieverEngineType" internal/
      shows only PR 1's normalizer case + this driver's EngineType()
      and tests — no activation path.
- [x] grep -r "case \"opensearch\"" internal/ shows no hits.
2026-05-26 20:54:58 +08:00
wizardchen 5fcdc30914 chore: update Go version to 1.26.0 in go.mod 2026-05-20 16:55:18 +08:00
knight 513e589494 feat: add vLLM server URL configuration for MinerU
Add support for configuring vLLM server URL when using vlm-http-client or hybrid-http-client backend in MinerU.

Changes:
- internal/types/tenant.go: Add MinerUVLMServerURL field to ParserEngineConfig
- internal/infrastructure/docparser/mineru_converter.go: Pass server_url to MinerU API when backend is vlm-http-client or hybrid-http-client
- frontend/src/api/system/index.ts: Add TypeScript type definition
- frontend/src/views/settings/ParserEngineSettings.vue: Add vLLM server URL input field
- frontend/src/i18n/locales/*.ts: Add translations (zh-CN, en-US, ko-KR)
2026-05-20 16:33:12 +08:00
dependabot[bot] 6b72ea9254 chore(deps): bump the server-deps group with 25 updates
Bumps the server-deps group with 25 updates:

| Package | From | To |
| --- | --- | --- |
| [github.com/JohannesKaufmann/html-to-markdown/v2](https://github.com/JohannesKaufmann/html-to-markdown) | `2.5.0` | `2.5.1` |
| [github.com/PuerkitoBio/goquery](https://github.com/PuerkitoBio/goquery) | `1.10.3` | `1.12.0` |
| [github.com/elastic/go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch) | `8.18.0` | `8.19.6` |
| [github.com/gin-contrib/cors](https://github.com/gin-contrib/cors) | `1.7.5` | `1.7.7` |
| [github.com/gin-gonic/gin](https://github.com/gin-gonic/gin) | `1.11.0` | `1.12.0` |
| [github.com/golang-jwt/jwt/v5](https://github.com/golang-jwt/jwt) | `5.3.0` | `5.3.1` |
| [github.com/hibiken/asynq](https://github.com/hibiken/asynq) | `0.25.1` | `0.26.0` |
| [github.com/larksuite/oapi-sdk-go/v3](https://github.com/larksuite/oapi-sdk-go) | `3.5.3` | `3.6.1` |
| [github.com/mark3labs/mcp-go](https://github.com/mark3labs/mcp-go) | `0.43.0` | `0.52.0` |
| [github.com/panjf2000/ants/v2](https://github.com/panjf2000/ants) | `2.11.3` | `2.12.0` |
| [github.com/pganalyze/pg_query_go/v6](https://github.com/pganalyze/pg_query_go) | `6.1.0` | `6.2.2` |
| [github.com/qdrant/go-client](https://github.com/qdrant/go-client) | `1.16.1` | `1.18.1` |
| [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) | `9.14.0` | `9.14.1` |
| [github.com/slack-go/slack](https://github.com/slack-go/slack) | `0.18.0-rc2` | `0.23.1` |
| [github.com/spf13/viper](https://github.com/spf13/viper) | `1.20.1` | `1.21.0` |
| [github.com/tencentyun/cos-go-sdk-v5](https://github.com/tencentyun/cos-go-sdk-v5) | `0.7.65` | `0.7.73` |
| [github.com/weaviate/weaviate](https://github.com/weaviate/weaviate) | `1.37.2` | `1.37.3` |
| [github.com/weaviate/weaviate-go-client/v5](https://github.com/weaviate/weaviate-go-client) | `5.5.0` | `5.7.3` |
| [github.com/yanyiwu/gojieba](https://github.com/yanyiwu/gojieba) | `1.4.5` | `1.4.7` |
| [go.opentelemetry.io/otel/exporters/stdout/stdouttrace](https://github.com/open-telemetry/opentelemetry-go) | `1.35.0` | `1.43.0` |
| [go.uber.org/dig](https://github.com/uber-go/dig) | `1.18.1` | `1.19.0` |
| [golang.org/x/mod](https://github.com/golang/mod) | `0.35.0` | `0.36.0` |
| [golang.org/x/net](https://github.com/golang/net) | `0.53.0` | `0.54.0` |
| [golang.org/x/time](https://github.com/golang/time) | `0.14.0` | `0.15.0` |
| [google.golang.org/api](https://github.com/googleapis/google-api-go-client) | `0.265.0` | `0.278.0` |


Updates `github.com/JohannesKaufmann/html-to-markdown/v2` from 2.5.0 to 2.5.1
- [Release notes](https://github.com/JohannesKaufmann/html-to-markdown/releases)
- [Commits](https://github.com/JohannesKaufmann/html-to-markdown/compare/v2.5.0...v2.5.1)

Updates `github.com/PuerkitoBio/goquery` from 1.10.3 to 1.12.0
- [Release notes](https://github.com/PuerkitoBio/goquery/releases)
- [Commits](https://github.com/PuerkitoBio/goquery/compare/v1.10.3...v1.12.0)

Updates `github.com/elastic/go-elasticsearch/v8` from 8.18.0 to 8.19.6
- [Release notes](https://github.com/elastic/go-elasticsearch/releases)
- [Changelog](https://github.com/elastic/go-elasticsearch/blob/v8.19.6/CHANGELOG.md)
- [Commits](https://github.com/elastic/go-elasticsearch/compare/v8.18.0...v8.19.6)

Updates `github.com/gin-contrib/cors` from 1.7.5 to 1.7.7
- [Release notes](https://github.com/gin-contrib/cors/releases)
- [Commits](https://github.com/gin-contrib/cors/compare/v1.7.5...v1.7.7)

Updates `github.com/gin-gonic/gin` from 1.11.0 to 1.12.0
- [Release notes](https://github.com/gin-gonic/gin/releases)
- [Changelog](https://github.com/gin-gonic/gin/blob/master/CHANGELOG.md)
- [Commits](https://github.com/gin-gonic/gin/compare/v1.11.0...v1.12.0)

Updates `github.com/golang-jwt/jwt/v5` from 5.3.0 to 5.3.1
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Commits](https://github.com/golang-jwt/jwt/compare/v5.3.0...v5.3.1)

Updates `github.com/hibiken/asynq` from 0.25.1 to 0.26.0
- [Release notes](https://github.com/hibiken/asynq/releases)
- [Changelog](https://github.com/hibiken/asynq/blob/master/CHANGELOG.md)
- [Commits](https://github.com/hibiken/asynq/compare/v0.25.1...v0.26.0)

Updates `github.com/larksuite/oapi-sdk-go/v3` from 3.5.3 to 3.6.1
- [Release notes](https://github.com/larksuite/oapi-sdk-go/releases)
- [Changelog](https://github.com/larksuite/oapi-sdk-go/blob/v3_main/changelog.md)
- [Commits](https://github.com/larksuite/oapi-sdk-go/compare/v3.5.3...v3.6.1)

Updates `github.com/mark3labs/mcp-go` from 0.43.0 to 0.52.0
- [Release notes](https://github.com/mark3labs/mcp-go/releases)
- [Commits](https://github.com/mark3labs/mcp-go/compare/v0.43.0...v0.52.0)

Updates `github.com/panjf2000/ants/v2` from 2.11.3 to 2.12.0
- [Release notes](https://github.com/panjf2000/ants/releases)
- [Commits](https://github.com/panjf2000/ants/compare/v2.11.3...v2.12.0)

Updates `github.com/pganalyze/pg_query_go/v6` from 6.1.0 to 6.2.2
- [Changelog](https://github.com/pganalyze/pg_query_go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pganalyze/pg_query_go/compare/v6.1.0...v6.2.2)

Updates `github.com/qdrant/go-client` from 1.16.1 to 1.18.1
- [Release notes](https://github.com/qdrant/go-client/releases)
- [Commits](https://github.com/qdrant/go-client/compare/v1.16.1...v1.18.1)

Updates `github.com/redis/go-redis/v9` from 9.14.0 to 9.14.1
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/v9.14.1/RELEASE-NOTES.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.14.0...v9.14.1)

Updates `github.com/slack-go/slack` from 0.18.0-rc2 to 0.23.1
- [Release notes](https://github.com/slack-go/slack/releases)
- [Changelog](https://github.com/slack-go/slack/blob/master/CHANGELOG.md)
- [Commits](https://github.com/slack-go/slack/compare/v0.18.0-rc2...v0.23.1)

Updates `github.com/spf13/viper` from 1.20.1 to 1.21.0
- [Release notes](https://github.com/spf13/viper/releases)
- [Commits](https://github.com/spf13/viper/compare/v1.20.1...v1.21.0)

Updates `github.com/tencentyun/cos-go-sdk-v5` from 0.7.65 to 0.7.73
- [Release notes](https://github.com/tencentyun/cos-go-sdk-v5/releases)
- [Changelog](https://github.com/tencentyun/cos-go-sdk-v5/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tencentyun/cos-go-sdk-v5/compare/v0.7.65...v0.7.73)

Updates `github.com/weaviate/weaviate` from 1.37.2 to 1.37.3
- [Release notes](https://github.com/weaviate/weaviate/releases)
- [Commits](https://github.com/weaviate/weaviate/compare/v1.37.2...v1.37.3)

Updates `github.com/weaviate/weaviate-go-client/v5` from 5.5.0 to 5.7.3
- [Release notes](https://github.com/weaviate/weaviate-go-client/releases)
- [Commits](https://github.com/weaviate/weaviate-go-client/compare/v5.5.0...v5.7.3)

Updates `github.com/yanyiwu/gojieba` from 1.4.5 to 1.4.7
- [Release notes](https://github.com/yanyiwu/gojieba/releases)
- [Changelog](https://github.com/yanyiwu/gojieba/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yanyiwu/gojieba/compare/v1.4.5...v1.4.7)

Updates `go.opentelemetry.io/otel/exporters/stdout/stdouttrace` from 1.35.0 to 1.43.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.35.0...v1.43.0)

Updates `go.uber.org/dig` from 1.18.1 to 1.19.0
- [Release notes](https://github.com/uber-go/dig/releases)
- [Changelog](https://github.com/uber-go/dig/blob/master/CHANGELOG.md)
- [Commits](https://github.com/uber-go/dig/compare/v1.18.1...v1.19.0)

Updates `golang.org/x/mod` from 0.35.0 to 0.36.0
- [Commits](https://github.com/golang/mod/compare/v0.35.0...v0.36.0)

Updates `golang.org/x/net` from 0.53.0 to 0.54.0
- [Commits](https://github.com/golang/net/compare/v0.53.0...v0.54.0)

Updates `golang.org/x/time` from 0.14.0 to 0.15.0
- [Commits](https://github.com/golang/time/compare/v0.14.0...v0.15.0)

Updates `google.golang.org/api` from 0.265.0 to 0.278.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.265.0...v0.278.0)

---
updated-dependencies:
- dependency-name: github.com/JohannesKaufmann/html-to-markdown/v2
  dependency-version: 2.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/PuerkitoBio/goquery
  dependency-version: 1.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/elastic/go-elasticsearch/v8
  dependency-version: 8.19.6
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/gin-contrib/cors
  dependency-version: 1.7.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/gin-gonic/gin
  dependency-version: 1.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/golang-jwt/jwt/v5
  dependency-version: 5.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/hibiken/asynq
  dependency-version: 0.26.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/larksuite/oapi-sdk-go/v3
  dependency-version: 3.6.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/mark3labs/mcp-go
  dependency-version: 0.52.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/panjf2000/ants/v2
  dependency-version: 2.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/pganalyze/pg_query_go/v6
  dependency-version: 6.2.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/qdrant/go-client
  dependency-version: 1.18.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/redis/go-redis/v9
  dependency-version: 9.14.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/slack-go/slack
  dependency-version: 0.23.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/spf13/viper
  dependency-version: 1.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/tencentyun/cos-go-sdk-v5
  dependency-version: 0.7.73
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/weaviate/weaviate
  dependency-version: 1.37.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: github.com/weaviate/weaviate-go-client/v5
  dependency-version: 5.7.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: github.com/yanyiwu/gojieba
  dependency-version: 1.4.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: server-deps
- dependency-name: go.opentelemetry.io/otel/exporters/stdout/stdouttrace
  dependency-version: 1.43.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: go.uber.org/dig
  dependency-version: 1.19.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: golang.org/x/mod
  dependency-version: 0.36.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: golang.org/x/net
  dependency-version: 0.54.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: golang.org/x/time
  dependency-version: 0.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
- dependency-name: google.golang.org/api
  dependency-version: 0.278.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: server-deps
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-12 13:30:44 +08:00
dependabot[bot] 91cf237dd9 chore(deps): bump golang.org/x/crypto from 0.50.0 to 0.51.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.50.0 to 0.51.0.
- [Commits](https://github.com/golang/crypto/compare/v0.50.0...v0.51.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.51.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 20:07:12 +08:00
wizardchen f016eff79a chore(deps): update github.com/go-openapi/swag to v0.26.0 and remove unused dependencies
Updated the go.mod and go.sum files to bump the version of github.com/go-openapi/swag and its related packages to v0.26.0. Removed the indirect dependency on github.com/josharian/intern as it is no longer needed.
2026-05-09 19:52:21 +08:00
dependabot[bot] 2fd2a139e0 chore(deps): bump github.com/jackc/pgx/v5 from 5.7.2 to 5.9.2
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.7.2 to 5.9.2.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v5.7.2...v5.9.2)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-version: 5.9.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 19:42:26 +08:00
dependabot[bot] 4bac6e30d2 chore(deps): bump github.com/milvus-io/milvus/client/v2
Bumps [github.com/milvus-io/milvus/client/v2](https://github.com/milvus-io/milvus) from 2.6.2 to 2.6.4.
- [Release notes](https://github.com/milvus-io/milvus/releases)
- [Commits](https://github.com/milvus-io/milvus/compare/v2.6.2...v2.6.4)

---
updated-dependencies:
- dependency-name: github.com/milvus-io/milvus/client/v2
  dependency-version: 2.6.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:36:17 +08:00
dependabot[bot] 46abae96f4 chore(deps): bump github.com/ollama/ollama from 0.11.4 to 0.23.2
Bumps [github.com/ollama/ollama](https://github.com/ollama/ollama) from 0.11.4 to 0.23.2.
- [Release notes](https://github.com/ollama/ollama/releases)
- [Commits](https://github.com/ollama/ollama/compare/v0.11.4...v0.23.2)

---
updated-dependencies:
- dependency-name: github.com/ollama/ollama
  dependency-version: 0.23.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:57 +08:00
dependabot[bot] bcd1700240 chore(deps): bump gorm.io/driver/postgres from 1.5.11 to 1.6.0
Bumps [gorm.io/driver/postgres](https://github.com/go-gorm/postgres) from 1.5.11 to 1.6.0.
- [Commits](https://github.com/go-gorm/postgres/compare/v1.5.11...v1.6.0)

---
updated-dependencies:
- dependency-name: gorm.io/driver/postgres
  dependency-version: 1.6.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:39 +08:00
dependabot[bot] 80731a2726 chore(deps): bump google.golang.org/grpc from 1.80.0 to 1.81.0
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.80.0 to 1.81.0.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.80.0...v1.81.0)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.81.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:33:28 +08:00
langcaiye 74b1342440 feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
dependabot[bot] 3293d5464b chore(deps): bump github.com/chromedp/chromedp from 0.14.2 to 0.15.1
Bumps [github.com/chromedp/chromedp](https://github.com/chromedp/chromedp) from 0.14.2 to 0.15.1.
- [Release notes](https://github.com/chromedp/chromedp/releases)
- [Commits](https://github.com/chromedp/chromedp/compare/v0.14.2...v0.15.1)

---
updated-dependencies:
- dependency-name: github.com/chromedp/chromedp
  dependency-version: 0.15.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:03:48 +08:00
dependabot[bot] ea4a2a39f2 chore(deps): bump github.com/aws/aws-sdk-go-v2/config
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.14 to 1.32.17.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.32.14...config/v1.32.17)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-version: 1.32.17
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:03:25 +08:00
dependabot[bot] d2c741aeff chore(deps): bump github.com/weaviate/weaviate
Bumps [github.com/weaviate/weaviate](https://github.com/weaviate/weaviate) from 1.33.0-rc.1 to 1.37.2.
- [Release notes](https://github.com/weaviate/weaviate/releases)
- [Commits](https://github.com/weaviate/weaviate/compare/v1.33.0-rc.1...v1.37.2)

---
updated-dependencies:
- dependency-name: github.com/weaviate/weaviate
  dependency-version: 1.37.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 13:00:27 +08:00
dependabot[bot] 7fe708557b chore(deps): bump gorm.io/gorm from 1.30.0 to 1.31.1
Bumps [gorm.io/gorm](https://github.com/go-gorm/gorm) from 1.30.0 to 1.31.1.
- [Release notes](https://github.com/go-gorm/gorm/releases)
- [Commits](https://github.com/go-gorm/gorm/compare/v1.30.0...v1.31.1)

---
updated-dependencies:
- dependency-name: gorm.io/gorm
  dependency-version: 1.31.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:56:13 +08:00
dependabot[bot] bb63d7ed59 chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials
Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.19.11 to 1.19.16.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.19.11...credentials/v1.19.16)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-version: 1.19.16
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:56:02 +08:00
dependabot[bot] 83db5782bc chore(deps): bump github.com/go-openapi/strfmt from 0.25.0 to 0.26.2
Bumps [github.com/go-openapi/strfmt](https://github.com/go-openapi/strfmt) from 0.25.0 to 0.26.2.
- [Release notes](https://github.com/go-openapi/strfmt/releases)
- [Commits](https://github.com/go-openapi/strfmt/compare/v0.25.0...v0.26.2)

---
updated-dependencies:
- dependency-name: github.com/go-openapi/strfmt
  dependency-version: 0.26.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:55:44 +08:00
dependabot[bot] 5c8978b72b chore(deps): bump github.com/duckdb/duckdb-go/v2 from 2.5.4 to 2.10502.0
Bumps [github.com/duckdb/duckdb-go/v2](https://github.com/duckdb/duckdb-go) from 2.5.4 to 2.10502.0.
- [Release notes](https://github.com/duckdb/duckdb-go/releases)
- [Commits](https://github.com/duckdb/duckdb-go/compare/v2.5.4...v2.10502.0)

---
updated-dependencies:
- dependency-name: github.com/duckdb/duckdb-go/v2
  dependency-version: 2.10502.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:54:09 +08:00
dependabot[bot] f3a6a6e6bd chore(deps): bump github.com/aliyun/alibabacloud-oss-go-sdk-v2
Bumps [github.com/aliyun/alibabacloud-oss-go-sdk-v2](https://github.com/aliyun/alibabacloud-oss-go-sdk-v2) from 1.4.1 to 1.5.1.
- [Release notes](https://github.com/aliyun/alibabacloud-oss-go-sdk-v2/releases)
- [Changelog](https://github.com/aliyun/alibabacloud-oss-go-sdk-v2/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aliyun/alibabacloud-oss-go-sdk-v2/compare/v1.4.1...v1.5.1)

---
updated-dependencies:
- dependency-name: github.com/aliyun/alibabacloud-oss-go-sdk-v2
  dependency-version: 1.5.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:51:49 +08:00
dependabot[bot] 0354d50492 chore(deps): bump github.com/sashabaranov/go-openai
Bumps [github.com/sashabaranov/go-openai](https://github.com/sashabaranov/go-openai) from 1.40.5 to 1.41.2.
- [Release notes](https://github.com/sashabaranov/go-openai/releases)
- [Commits](https://github.com/sashabaranov/go-openai/compare/v1.40.5...v1.41.2)

---
updated-dependencies:
- dependency-name: github.com/sashabaranov/go-openai
  dependency-version: 1.41.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:51:33 +08:00
dependabot[bot] 458dcb4f7a chore(deps): bump github.com/minio/minio-go/v7 from 7.0.91 to 7.1.0
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.91 to 7.1.0.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.91...v7.1.0)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-version: 7.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:51:05 +08:00
dependabot[bot] 01b6381f82 chore(deps): bump github.com/aws/aws-sdk-go-v2/service/s3
Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.83.0 to 1.101.0.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.83.0...service/s3/v1.101.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.101.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:49:43 +08:00
dependabot[bot] d2a1d844b8 chore(deps): bump github.com/volcengine/ve-tos-golang-sdk/v2
Bumps [github.com/volcengine/ve-tos-golang-sdk/v2](https://github.com/volcengine/ve-tos-golang-sdk) from 2.7.23 to 2.9.4.
- [Release notes](https://github.com/volcengine/ve-tos-golang-sdk/releases)
- [Changelog](https://github.com/volcengine/ve-tos-golang-sdk/blob/main/CHANGELOG.md)
- [Commits](https://github.com/volcengine/ve-tos-golang-sdk/compare/v2.7.23...v2.9.4)

---
updated-dependencies:
- dependency-name: github.com/volcengine/ve-tos-golang-sdk/v2
  dependency-version: 2.9.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:49:19 +08:00
dependabot[bot] 484d08da36 chore(deps): bump github.com/google/jsonschema-go from 0.4.2 to 0.4.3
Bumps [github.com/google/jsonschema-go](https://github.com/google/jsonschema-go) from 0.4.2 to 0.4.3.
- [Release notes](https://github.com/google/jsonschema-go/releases)
- [Commits](https://github.com/google/jsonschema-go/compare/v0.4.2...0.4.3)

---
updated-dependencies:
- dependency-name: github.com/google/jsonschema-go
  dependency-version: 0.4.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-09 12:48:53 +08:00
issunion 4cce6f2e99 feat(retriever): 接入 Apache Doris 4.1 作为向量数据库
为 RetrieveEngine 体系新增 Doris 后端,与现有 Qdrant/Milvus/Weaviate
等保持完整能力对齐:向量检索、关键词检索、健康检查、环境变量与多实例
DB 配置、前端类型注册、单元测试、Docker Compose 模板。

实现要点:
- 协议分工:主链路用 MySQL 协议(database/sql + go-sql-driver/mysql)
  做 DDL / 查询 / 删除;批量更新走 Stream Load HTTP API,并启用
  partial_update=true、merge_type=APPEND,自动按 1MiB 切分批次并处理
  307 重定向。
- 表结构:UNIQUE KEY(id) + enable_unique_key_merge_on_write=true 以
  支持 upsert/部分列更新;按维度分表(<base>_<dim>),每张表上建
  HNSW ANN 索引(metric_type=cosine_distance)和 INVERTED 索引
  (parser=chinese)。
- 分数语义:使用 cosine_distance_approximate,再以 1 - dist 转换为
  "越大越相似",与现有 KVHybridRetrieveEngine 约定一致。
- 异步索引:ANN 索引为后台构建,ensureTable 通过轮询 SHOW INDEX 等
  待索引就绪后再放行写入,避免首次检索召回为空。
- ARRAY<FLOAT> 序列化:go-sql-driver/mysql 不支持数组占位符,
  embeddingLiteral 将 []float32 转成 SQL 字面量字符串再拼接。

新增文件:
- internal/application/repository/retriever/doris/{structs,schema,
  query,repository,streamload,repository_test}.go
- scripts/e2e-doris.sh:E2E 验证清单
- docs/wiki/集成扩展/Doris改动与上游同步.md:fork-and-rebase 工作流
  与改动清单

修改文件(接线 + 文档):
- internal/types/{retriever,tenant,vectorstore}.go:新增
  DorisRetrieverEngineType、env 解析、表单 schema 与索引参数校验
- internal/container/{container,engine_factory}.go:环境变量驱动
  与 VectorStore 配置驱动两条路径都支持 Doris
- internal/application/service/vectorstore{,_healthcheck}.go:连接
  校验 + Ping/Version 健康检查
- docker-compose.yml:新增 doris-fe / doris-be 服务(profile=doris)
- .env.example:DORIS_* 环境变量与示例
- docs/{使用其他向量数据库,wiki/集成扩展/集成向量数据库}.md:
  使用说明与索引/分数行为说明

依赖:go.mod/go.sum 新增 github.com/go-sql-driver/mysql(运行时)和
github.com/DATA-DOG/go-sqlmock(测试)。

测试:repository 层 SQL 形状、Stream Load HTTP 行为、whereBuilder
逻辑、embeddingLiteral 往返、健康检查错误路径均有单测覆盖。
2026-05-08 21:59:35 +08:00
ChenRussell 42348eb40d feat(ks3): add KS3 object storage support
Add Kingsoft Cloud KS3 as a supported storage provider, including
  backend service implementation, storage config UI, and provider
  scheme recognition throughout the codebase.
2026-05-05 00:22:27 +08:00
wizardchen 002d98a640 test(agent): add end-to-end DuckDB excel tests and fix sheet enumeration (#1007)
Catching up on a real-world check: when a multi-sheet .xlsx fixture is
fed through LoadFromExcel against an in-memory DuckDB with the spatial
and excel extensions, the first iteration's sheet enumeration quietly
fell back because st_read_meta doesn't expose a scalar 'layer_name'
column — it returns a LIST<STRUCT> column called 'layers'. We now
UNNEST(layers).name so enumeration actually works, and assert the full
data path with three DuckDB-backed tests:

  - multi-sheet workbook yields the sum of all rows and per-sheet
    breakdown via __sheet_name (including schema drift: columns only
    present in one sheet are NULL for the other)
  - single-sheet workbook still works and tags rows with its sheet name
  - sheet name containing a single quote (Q1'24) survives the SQL
    literal round-trip

excelize/v2 is added as a test-time helper to build the fixtures
deterministically. If DuckDB's spatial/excel extensions can't be
installed (offline CI), the tests skip rather than fail.

Refs: https://github.com/Tencent/WeKnora/issues/1007
2026-04-24 19:57:56 +08:00
bingxiang.cheng a2900d5540 chore(deps): 更新并新增go.mod依赖项
- 新增aliyun alibabacloud-oss-go-sdk-v2依赖
- 删除部分冗余依赖,调整依赖列表结构
- 添加jackc/pgx/v5作为直接依赖
- 统一和规范间接依赖声明顺序
- 移除go.opentelemetry.io/contrib的版本替换声明
2026-04-14 17:33:12 +08:00
wizardchen a655a25cf8 fix: revert to neo4j v6 and workaround Go 1.24 Windows compiler bug with -p=1
Go 1.24 has an internal compiler race condition on Windows that causes
'package without types' errors during parallel compilation. Setting
GOFLAGS="-p=1" serializes the build and avoids triggering the bug.
2026-04-11 22:08:02 +08:00
wizardchen 9525f08038 fix: downgrade neo4j driver from v6 to v5 to fix Go 1.24 Windows internal compiler bug 2026-04-11 21:59:03 +08:00
wizardchen 02fd3aad49 fix: upgrade neo4j driver to v6.0.0 to fix Go 1.24 Windows compiler error 2026-04-11 19:32:21 +08:00
wizardchen 4d244f51d2 feat: integrate lumberjack for log file management, enhancing logging capabilities with rotation and compression 2026-04-11 18:51:16 +08:00
wizardchen 9874fcee7c feat: implement automatic update checking and downloading functionality in the desktop app, enhancing user experience with seamless updates and corresponding UI settings 2026-04-11 18:51:16 +08:00
wizardchen 5c4229745d fix: restore indirect dependency for golang.org/x/sys and refine drag selection logic in desktop app 2026-04-11 18:51:16 +08:00
wizardchen 1ca784be97 chore: add godotenv dependency for environment variable management 2026-04-11 18:51:16 +08:00
wizardchen 96fb36ebb3 feat: introduce macOS desktop app using Wails wrapper 2026-04-11 18:51:16 +08:00
wizardchen 6f48e356b9 feat: update readme & docs 2026-03-30 21:38:45 +08:00
wizardchen 41fcdf1a83 chore(go.mod): update dependencies to include tiktoken-go and golang.org/x/sys
- Added github.com/tiktoken-go/tokenizer v0.7.0 to the required dependencies for improved tokenization support.
- Included golang.org/x/sys v0.40.0 to enhance system call functionalities.
- Removed indirect references to these packages to streamline the dependency management.
2026-03-25 22:12:50 +08:00
wizardchen 0bb1a581de feat(agent): enhance token management and context handling in AgentEngine
- Added token usage tracking to the AgentEngine, allowing for better estimation of current context token counts.
- Implemented a new method to estimate current tokens based on previous usage and newly appended messages, improving context management.
- Updated context window management to utilize estimated token counts for more efficient message consolidation.
- Enhanced streaming methods to include token usage information, providing better insights into LLM interactions.
- Refactored the token estimator to support BPE tokenization, ensuring accurate token counts for messages.
- Added unit tests to validate new token estimation and context management functionalities.
2026-03-25 22:08:29 +08:00