Commit Graph

41 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
lyingbug e179f44445
refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589)
* refactor(models): separate providers, catalog and runtime configuration

* refactor(models): isolate provider definitions and runtime catalogs

* docs: exclude refactoring analysis notes from the pull request
2026-09-23 14:20:08 +08:00
lyingbug 9030f556f2
refactor(embedding): move the wire formats into protocol packages, and correct the vendor facts (#3500) 2026-09-21 11:55:38 +08:00
lyingbug c40a9dd194
refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470)
把集中式的 OpenAI 方言客户端重建为三层:协议层 internal/models/api/<protocol>(OpenAI Chat Completions / OpenAI Responses / Anthropic Messages / Google GenAI),目录层 internal/models/catalog,厂商层 internal/models/vendors/<id>(27 家,go:embed)。27 家厂商逐条对照官方文档核验,厂商事实以数据声明,协议层不出现厂商名。

既有行的运行时行为会变的四处:
1. extra_config.api 变为保留的协议选择器,非法取值被拒绝;
2. Azure 未填 api_version 的行改走 /openai/v1 GA 数据面;
3. api.openai.com 的一方流量改走 Responses 协议(中转/网关不受影响);
4. 7 家厂商的输出上限字段按文档纠正。

不需要数据库迁移。详见 PR 描述与 website-docs/03-features/06-models.md。
2026-09-20 21:28:23 +08:00
lyingbug 50bbf9ad5f
fix(embedding): two wire-level bugs in the remote embedders (#3484)
* fix(embedding): stop four embedders from returning (nil, nil) on a dead upstream

doRequestWithRetry keeps resp and err in the enclosing scope and returns
(nil, err) once the retries run out. Writing `req, err := ...` inside the
loop introduces a loop-local err, so `resp, err = client.Do(req)` updates
that copy and the enclosing err stays nil. Every retry failing therefore
returned (nil, nil): BatchEmbed's `if err != nil` did not fire and the next
line dereferenced resp.Body, a nil-pointer panic that took the process down
instead of surfacing "connection refused". Stopping the embedding upstream
and asking any RAG question reproduced it.

The bug was found, documented and fixed in openai.go, then re-fixed in
zhipu.go and again, differently, in azure_openai.go; jina.go, nvidia.go,
aliyun.go and volcengine.go kept the original spelling because the retry
loop exists in eight near-identical copies and a fix to one reaches none of
the others.

TestEmbeddersReportUnreachableUpstream now drives all eight against a
refused port with the retry budget zeroed, so the next copy of this loop is
covered the moment it appears.

* fix(embedding): read the documented index field of DashScope multimodal vectors

The multimodal embedding response carries each vector's position in `index`:

  {"output": {"embeddings": [{"index": 0, "embedding": [...], "type": "text"},
                             {"index": 1, "embedding": [...], "type": "image"}]}}

The decoder read `text_index`, which is the field name of DashScope's *text*
embedding API one page over. A missing field decodes as 0, so every vector in
a batch was written to slot 0 and the rest of the slice stayed empty: with the
default BATCH_EMBED_SIZE of 5, four of every five chunks were indexed with an
empty vector. Only models routed to this client are affected — an Aliyun
embedding model whose name contains "vision" or "multimodal"; the text-only
models go through the OpenAI-compatible path.

An index outside the batch, or a response that covers fewer inputs than were
sent, is now an error. Handing back an empty vector stores it and poisons
retrieval without anything failing.

The regression test replays the response example from the reference page, so
the decoder is pinned against the vendor's own document rather than against
our structs.

https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference
2026-09-20 21:12:33 +08:00
wizardchen ca88fe74ae fix(embedding): preserve batching and concurrency limits for WeKnoraCloud 2026-09-19 20:26:46 +08:00
PP1 06eeb95b4e fix(embedding): validate WeKnoraCloud response indexes
Reject out-of-range and duplicate response indexes before mapping them to batch inputs. Return an error when a requested input has no response item instead of silently passing an incomplete batch downstream.

Add focused coverage for malformed responses and valid out-of-order results.

Signed-off-by: PP1 <74917296+pengpengyi92@users.noreply.github.com>
2026-08-11 16:59:48 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
ljl 2b31d3681b fix(embedding): prevent ants worker panic and mutex deadlock on empty/mismatched results
If the embedding provider returns an empty slice of vectors (e.g. length 0)
due to api mismatch or empty input, BatchEmbed will return length 0.
Accessing embedding[i] directly in BatchEmbedWithPool results in an out of bounds panic.
Because the panic occurs inside a mu.Lock() block, the mutex is never unlocked,
leading to a permanent deadlock of the document processing queue.

This fix checks the returned embeddings slice length and returns an error
instead of panicking, releasing the mutex properly.
2026-08-04 16:46:57 +08:00
wizardchen 146e341748 refactor(http): share SSRF-safe transport across model clients
Extract NewSSRFSafeTransport and NewSSRFSafeHTTPClientWithTransport so
embedding and rerank can reuse one connection pool without allocating a
throwaway transport per client. SSRF dial and redirect policies are
unchanged; each client still keeps its own timeout.
2026-07-23 12:00:05 +08:00
zhangzherui ccf02e1a2a fix(embedding): reuse SSRF-safe HTTP transport 2026-07-23 11:51:54 +08:00
wizardchen b9cccc7bee feat(system): expose model concurrency stats on runtime queues dashboard
Surface per-model active/waiting/limit from the background task limiter so admins can distinguish task scheduling from model-service throttling, with full i18n coverage.
2026-07-11 07:17:28 +08:00
wizardchen 20256287ec feat(models): add max concurrency setting for background tasks
Introduce a new `max_concurrency` parameter to the model configuration, allowing control over the maximum number of concurrent background tasks (ingestion/enrichment) per model. This feature is applicable to chat, embedding, and VLM models, with a value of 0 or undefined falling back to the global default. Update UI components and localization files to support this new setting, ensuring a seamless user experience.
2026-07-09 20:40:51 +08:00
wizardchen 200c344065 feat(models): cap background embedding concurrency per model
Extend the per-model background concurrency governor to embedding, the
highest-volume background model call (document ingestion vectorises every
chunk, so a batch upload can burst the whole worker pool against one provider).

The wrapper sits innermost (below debug/langfuse) because BatchEmbedWithPool
fans a batch out into per-sub-batch BatchEmbed calls via the pooler, and the
pooler invokes BatchEmbed on whichever Embedder is threaded down as `model`;
sitting innermost routes those per-sub-batch provider round-trips through Gate
so the semaphore bounds real concurrent calls rather than one coarse per-doc
unit. Only background (asynq) calls are throttled; interactive query embedding
is never gated.
2026-07-09 20:40:51 +08:00
wizardchen 6b60d292f8 fix(embedding): add SSRF validation and safe HTTP client for embedders
Align embedding outbound requests with chat module defenses by validating
base URLs at construction time and using SSRFSafeDialContext with redirect checks.
2026-06-26 22:41:04 +08:00
wizardchen 53e0333a53 Add embedding dimension override support 2026-06-15 19:18:19 +08:00
MaoMeng cf0a64b88e
fix:dimensions not sent in embedding API requests (#1654)
fix:dimensions not sent in embedding API requests
2026-06-15 17:32:23 +08:00
Yeongpil Yoon 8f8a276120 fix(embedding): support native Gemini embeddings 2026-06-02 21:37:58 +08:00
wizardchen 5141a1230c fix(embedding): repair broken comment in Zhipu embedder
A stray `/` line split what should have been `//`, which prevented `go build` from succeeding in CI.
2026-05-17 21:31:54 +08:00
Wellbek 4fef6826cf feat(embedding): add Zhipu AI embedder implementation 2026-05-17 20:14:51 +08:00
toy0116 8955bf1aa8 fix(embedding): OpenAIEmbedder.doRequestWithRetry shadows err · returns (nil, nil) on connection failure · caller nil-derefs and SIGSEGVs
Reproduce:

  1. Run WeKnora pointing OPENAI_BASE_URL at any OpenAI-compatible
     embedding endpoint.
  2. Stop that endpoint.
  3. Issue any RAG-mode chat / knowledge_search query — the pipeline
     reaches chunk_search_parallel → knowledgeBaseService.GetQueryEmbedding
     → OpenAIEmbedder.Embed → OpenAIEmbedder.BatchEmbed.
  4. Observe the entire WeKnora process SIGSEGV instead of returning a
     graceful 5xx.

Stack (real, observed in production):

  panic: runtime error: invalid memory address or nil pointer dereference
  [signal SIGSEGV: segmentation violation code=0x2 addr=0x40]

  internal/models/embedding/openai.go:195 +0x458   BatchEmbed
  internal/models/embedding/openai.go:93  +0xb0    Embed
  internal/application/service/knowledgebase_search.go:35
                                                   GetQueryEmbedding
  internal/application/service/chat_pipeline/search.go:371
                                                   searchByTargets.func1

Root cause — variable shadowing in doRequestWithRetry:

  func (e *OpenAIEmbedder) doRequestWithRetry(...) (*http.Response, error) {
      var resp *http.Response
      var err error                                        // outer err
      ...
      for i := 0; i <= e.maxRetries; i++ {
          ...
          req, err := http.NewRequestWithContext(...)      // ← `:=` shadows
                                                           //   outer `err`
                                                           //   because `req`
                                                           //   is new in
                                                           //   the loop scope
          if err != nil { ... continue }
          ...
          resp, err = e.httpClient.Do(req)                 // writes to the
                                                           // shadowed
                                                           // loop-local err
          if err == nil { return resp, nil }
          logger.Errorf("attempt %d failed: %v", i+1, err) // logs the inner err
      }
      return nil, err                                      // returns OUTER
                                                           // err — never set
  }

When httpClient.Do returns connection errors on every attempt, the loop
logs them faithfully but the outer err stays nil. The function returns
(nil, nil). Caller in BatchEmbed reads:

  resp, err := e.doRequestWithRetry(ctx, jsonData)
  if err != nil { return nil, ... }       // skipped (err is nil)
  if resp.Body != nil {                   // ← line 195 · resp itself is nil
      defer resp.Body.Close()             //   nil pointer dereference
  }                                       //   → SIGSEGV → process dies

Fix — declare `req` separately so the shared `err` variable is not shadowed:

  var req *http.Request
  req, err = http.NewRequestWithContext(...)               // outer err

After the fix, when all retries fail, doRequestWithRetry returns the real
error (e.g. `dial tcp 127.0.0.1:11434: connect: connection refused`),
the caller propagates it as a regular error, the chat pipeline returns a
5xx to the client, and the process keeps running.

Verified by stopping the embedding upstream while WeKnora was serving
queries · process stayed up · returned graceful errors · resumed normal
operation when the upstream came back. Without the patch the same
sequence killed the process within one query.
2026-05-04 23:18:42 +08:00
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
wizardchen fd3e2992b1 feat(models): support custom HTTP headers across all remote model calls
Introduce per-model `custom_headers` config (similar to OpenAI Python SDK's
`extra_headers`) so users can inject gateway auth tokens, trace IDs, etc.
into every outbound API request for chat / embedding / rerank / VLLM / ASR
models. Reserved headers like Authorization / Content-Type are always
preserved to avoid breaking auth or signing flows.

Along the way, unify production and "test connection" paths onto a single
`ConfigFromModel(*types.Model, appID, appSecret)` constructor in each
model package. Both `service.modelService.GetXxxModel` and the four
`handler.initialization.Check*Model` / `TestEmbeddingModel` endpoints now
go through the same field mapping, so new parameters only need to be
added in one place.

Backend:
- Add `CustomHeaders map[string]string` to `types.ModelParameters` with
  JSON/YAML `omitempty` tags for forward compatibility.
- New `internal/utils/extraheaders.go` providing `ApplyCustomHeaders`
  (for hand-rolled HTTP paths) and `WrapHTTPClientWithHeaders` /
  `CustomHeadersRoundTripper` (for SDK paths like go-openai). Reserved
  headers (Authorization, api-key, Content-Type, Accept, Host,
  Content-Length, User-Agent) are filtered out.
- Inject headers in every remote model path: chat (SDK + raw HTTP),
  embedding (openai/aliyun/jina/volcengine/nvidia/azure_openai), rerank
  (remote_api/aliyun/jina/nvidia/zhipu — via shared `customHeaderSetter`
  interface), VLM and ASR (via http.Client wrapping).
- Add `ConfigFromModel` to each of chat/embedding/rerank/vlm/asr,
  consolidating field mapping (ExtraConfig, CustomHeaders, InterfaceType
  defaulting, WeKnoraCloud credentials, etc.) and cover it with
  dedicated unit tests per package.
- Refactor `service/model.go`: replace ~100 lines of hand-written Config
  literals with one-line `ConfigFromModel` calls; drop the now-unused
  `stringMapToAnyMap` helper.
- Refactor `handler/initialization.go` test-connection endpoints: merge
  four ad-hoc request structs into one `ModelTestRequest`, extract
  `buildTestModel` and `resolveTenantWeKnoraCloudCreds` helpers, and
  route all four endpoints through `ConfigFromModel` + `NewXxx` so the
  test path is now behaviorally identical to production.

Frontend:
- Add `custom_headers` to the `ModelConfig` API type.
- `ModelEditorDialog.vue`: new Key-Value header editor (add/remove rows),
  auto-converts map <-> array when loading/saving, and passes the
  current header set to the `Test Connection` button so the preview
  reflects exactly what production will send.
- `ModelSettings.vue`: serialize the header array back into a map for
  the backend (drops empty rows).
- i18n: add custom-header labels / descriptions / placeholders to
  zh-CN, en-US, ru-RU, ko-KR.

Tests:
- `internal/utils/extraheaders_test.go` covers reserved-header filtering
  and round-tripper wrapping (including nil-client fallback).
- New `config_from_model_test.go` in chat / embedding / rerank / vlm /
  asr verifies all fields (CustomHeaders, ExtraConfig, InterfaceType
  defaulting, AppID/AppSecret) are propagated end-to-end.
2026-04-23 23:36:40 +08:00
wizardchen 53e8e17df4 feat(logger): implement LLM debug logging functionality
- Added a new logger for LLM calls, enabling detailed logging of model interactions, including request and response data.
- Introduced configuration options for enabling and specifying the log directory for LLM debug logs.
- Implemented cleanup for old log files to manage disk space effectively.
- Wrapped existing chat implementations to log calls when debug logging is enabled, enhancing traceability for model interactions.
2026-04-16 23:15:02 +08:00
begoniezhao 5082460cc0 fix(embedding): remove hardcoded TruncatePromptTokens in BatchEmbed
Remove the hardcoded TruncatePromptTokens: 512 limit from WeKnoraCloud
BatchEmbed to allow the API to use its default truncation behavior,
avoiding unexpected truncation of longer input texts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-14 16:29:49 +08:00
begoniezhao d4edd374df refactor: replace CryptoService with utils AES helpers and simplify encryption
- Remove infrastructure/crypto package (CryptoService, config, PBKDF2-based encryption)
- Replace with lightweight utils.GetAESKey/EncryptAESGCM/DecryptAESGCM helpers
- Remove cryptoSvc dependency from modelService, tenantService, weKnoraCloudService
- Remove crypto state persistence logic from container initialization
- Add TruncatePromptTokens field to WeKnoraCloud embed request
- Update frontend hint text for WeKnoraCloud credential errors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-13 20:55:39 +08:00
begoniezhao 2c0c1b8bab feat: integrate ChatBot provider and docreader 2026-04-13 17:47:24 +08:00
liweixin 2b10cc54de test: remove unnecessary OpenAI dimensions regression test 2026-04-09 19:19:25 +08:00
liweixin 5233394d01 fix: gate Azure OpenAI dimensions support 2026-04-09 19:19:25 +08:00
liweixin c663182a00 feat: add support for dimensions parameter in Azure OpenAI and OpenAI embedding requests 2026-04-09 19:19:25 +08:00
liweixin 2eb2dea634 feat: add Azure OpenAI support for embedding models 2026-04-09 19:19:25 +08:00
ChenRussell 0fe5d67b6a fix: enforce 7500 char limit in chunker to prevent embedding API errors
Prevents 400 Bad Request errors when uploading documents with large
tables or code blocks by adding hard limits in buildUnitsWithProtection
and mergeUnits functions.
2026-03-15 23:59:34 +08:00
Manx98 5afc81d00b feat: support NVIDIA model API 2026-03-13 18:14:08 +08:00
begoniezhao 4610a4d1ee feat: Use num_ctx instead of truncate for embedding truncation 2026-02-04 14:24:01 +08:00
joeyczheng b1633c836a fix: concurrent map writes due to dig.Container is not thread safe
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-01-15 20:07:11 +08:00
nullkey 7947305ff0 fix: 增加魔搭社区嵌入模型所需的EncodingFormat参数 2026-01-09 11:08:57 +08:00
nullkey e9a2a7b3e5 feat: 新增对多家平台的模型适配 2025-12-29 12:18:42 +08:00
wizardchen eb8bfddfd1 refactor: Update function signatures across multiple files to improve readability by adding context parameters and enhancing code structure 2025-11-28 16:17:52 +08:00
wizardchen 64c44c9e27 refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
wizardchen 057cfd9eb0 feat: Enhance FAQ import functionality and UI improvements
- Added support for tracking the status of FAQ import tasks, including progress indicators for ongoing imports.
- Introduced new localization strings for processing tasks and batch operations in multiple languages.
- Updated the FAQ entry manager to display import progress and improved user feedback during the import process.
- Enhanced the KnowledgeBase component to reflect the processing state of knowledge items, improving user experience during document uploads.
- Implemented batch deletion functionality for chunks, streamlining the management of knowledge base content.
2025-11-23 17:32:50 +08:00
wizardchen 3bf2a14499 feat: Enhance agent configuration and localization for web search prompts
- Updated agent configuration to support separate system prompts for web search enabled and disabled states.
- Removed deprecated agent configuration parameters to streamline settings management.
- Enhanced UI components in AgentSettings.vue to allow configuration of custom prompts based on web search status.
- Improved localization in English, Russian, and Chinese for new prompt settings and UI elements.
- Refactored related API and service logic to accommodate changes in agent configuration structure.
2025-11-19 19:27:25 +08:00
wizardchen 56eb2bce33 init commit 2025-08-05 15:08:07 +08:00