WeKnora/internal/models/api
lyingbug c3f077ed18
fix(models): align configuration and multimodal embedding workflows (#4080)
* fix(models): align configuration defaults and multimodal workflows

* feat(image): enhance image retrieval diagnostics and filtering

- Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results.
- Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores.
- Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording.
- Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling.

* feat(embeddings): enhance image handling with new formats and message support

- Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats.
- Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows.
- Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements.
- Added tests to validate the new image message handling and ensure correct request formatting for different scenarios.
- Updated documentation to reflect changes in image input handling and configuration options.

* feat(similarity): add embedding similarity mode for enhanced document scoring

- Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query.
- Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type.
- Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity.
- Added tests to validate the functionality of the new similarity mode in embedding models.
- Updated localization files to include new strings related to similarity mode in multiple languages.

* feat(embeddings): add Qwen3-VL-Embedding support for multimodal input

- Introduced Qwen3-VL-Embedding model with support for both text and image inputs.
- Updated model configuration files to include compatibility settings for the new embedding type.
- Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs.
- Updated documentation to reflect the new model's capabilities and usage instructions.

* test(models): wrap embedding assertions to satisfy lint

* test(opensearch): isolate HTTP transports between parallel tests
2026-10-10 11:06:00 +08:00
..
anthropicmessages fix(models): align configuration and multimodal embedding workflows (#4080) 2026-10-10 11:06:00 +08:00
arkembeddings feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
cohererank feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
dashscopeembeddings feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
dashscoperank refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
googleembeddings feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
googlegenai fix(llm): treat cut-off streams as broken and tighten LLM retries 2026-09-25 18:22:14 +08:00
nimrerank feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
openaichataudio refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
openaicompletions fix(models): emit explicit empty content for assistant tool-call turns (#3605) 2026-10-08 17:43:05 +08:00
openaiembeddings fix(models): align configuration and multimodal embedding workflows (#4080) 2026-10-10 11:06:00 +08:00
openairesponses fix(llm): treat cut-off streams as broken and tighten LLM retries 2026-09-25 18:22:14 +08:00
openaitranscriptions refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
pineconerank feat(rerank): support Pinecone Inference protocol (#3957) 2026-10-08 17:54:15 +08:00
teirank feat(rerank): support Hugging Face TEI protocol (#3956) 2026-10-08 14:02:49 +08:00
tencentlkeap refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
volcengineknowledge refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
batch.go refactor(rerank): move the wire formats into protocol packages, and correct the vendor facts (#3488) 2026-09-21 10:36:26 +08:00
batch_test.go refactor(rerank): move the wire formats into protocol packages, and correct the vendor facts (#3488) 2026-09-21 10:36:26 +08:00
compat_settings.go refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
credentials.go refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
embeddings.go feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
embeddings_settings.go fix(models): align configuration and multimodal embedding workflows (#4080) 2026-10-10 11:06:00 +08:00
embeddings_test.go refactor(embedding): move the wire formats into protocol packages, and correct the vendor facts (#3500) 2026-09-21 11:55:38 +08:00
endpoint.go fix(models): keep Retry-After on HTTPError and honour it in agent LLM retries 2026-09-29 01:22:15 +08:00
endpoint_test.go fix(frontend): 修复复制模型时模型名称被改写的问题 (#3830) 2026-09-29 19:32:08 +08:00
image_input.go feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
images.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
images_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
json_field_extractor.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
json_field_extractor_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
multipart.go refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
options.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
options_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
prompt_cache.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
prompt_cache_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
raw_dump.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
rerank.go feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
rerank_settings.go feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
retry.go refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
retry_test.go refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
sandbox_file_progress.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
sandbox_file_progress_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
special_tokens.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
special_tokens_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
sse.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
sse_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
stream_assembler.go fix(llm): treat cut-off streams as broken and tighten LLM retries 2026-09-25 18:22:14 +08:00
stream_assembler_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
thinking_emitter.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
transcriptions.go refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
transcriptions_settings.go refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
transport.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
transport_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
types.go refactor(models): separate provider definitions, model catalogs, and runtime configuration (#3589) 2026-09-23 14:20:08 +08:00
usage.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
usage_log.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
usage_log_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00