fix(models): align configuration and multimodal embedding workflows (#4080)

* fix(models): align configuration defaults and multimodal workflows

* feat(image): enhance image retrieval diagnostics and filtering

- Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results.
- Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores.
- Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording.
- Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling.

* feat(embeddings): enhance image handling with new formats and message support

- Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats.
- Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows.
- Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements.
- Added tests to validate the new image message handling and ensure correct request formatting for different scenarios.
- Updated documentation to reflect changes in image input handling and configuration options.

* feat(similarity): add embedding similarity mode for enhanced document scoring

- Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query.
- Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type.
- Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity.
- Added tests to validate the functionality of the new similarity mode in embedding models.
- Updated localization files to include new strings related to similarity mode in multiple languages.

* feat(embeddings): add Qwen3-VL-Embedding support for multimodal input

- Introduced Qwen3-VL-Embedding model with support for both text and image inputs.
- Updated model configuration files to include compatibility settings for the new embedding type.
- Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs.
- Updated documentation to reflect the new model's capabilities and usage instructions.

* test(models): wrap embedding assertions to satisfy lint

* test(opensearch): isolate HTTP transports between parallel tests
main
lyingbug 2026-10-10 11:06:00 +08:00 committed by GitHub
parent 40eb34bc84
commit c3f077ed18
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
54 changed files with 2631 additions and 191 deletions

View File

@ -235,6 +235,8 @@ export interface ModelDebugOptions {
thinking?: boolean
// Graded thinking level; takes precedence over the boolean when set.
reasoning_effort?: ReasoningEffortLevel | string
// Embedding only: score `documents` (and the file) against `input` as a query.
similarity?: boolean
}
export interface ModelDebugResult {

View File

@ -76,33 +76,40 @@
<template v-if="selectedModel">
<section class="setting-drawer__section">
<h4 class="setting-drawer__section-title">{{ $t('modelSettings.debug.groupInput') }}</h4>
<div v-if="selectedModel.type !== 'ASR'" class="form-item">
<div v-if="hasImageMode || isEmbedding" class="form-item">
<t-radio-group v-model="inputMode" @change="clearFile">
<t-radio-button value="text">{{ isEmbedding ? $t('modelSettings.debug.embeddingInput') : $t('modelSettings.debug.documents') }}</t-radio-button>
<t-radio-button v-if="hasImageMode" value="image">{{ $t('modelSettings.debug.imageFile') }}</t-radio-button>
<t-radio-button v-if="isEmbedding" value="similarity">{{ $t('modelSettings.debug.similarityMode') }}</t-radio-button>
</t-radio-group>
</div>
<div v-if="selectedModel.type !== 'ASR' && !(isEmbedding && useImageMode)" class="form-item">
<label class="form-label">{{ inputLabel }}</label>
<t-textarea
v-model="input"
:placeholder="inputPlaceholder"
:autosize="{ minRows: 4, maxRows: 8 }"
:autosize="{ minRows: useSimilarityMode ? 2 : 4, maxRows: 8 }"
/>
</div>
<div v-if="selectedModel.type === 'Rerank'" class="form-item">
<label class="form-label">{{ $t('modelSettings.debug.documents') }}</label>
<div v-if="(selectedModel.type === 'Rerank' && !useImageMode) || useSimilarityMode" class="form-item">
<label class="form-label">{{ useSimilarityMode ? $t('modelSettings.debug.similarityCandidates') : $t('modelSettings.debug.documents') }}</label>
<t-textarea
v-model="documentsText"
:placeholder="$t('modelSettings.debug.documentsPlaceholder')"
:placeholder="useSimilarityMode ? $t('modelSettings.debug.similarityCandidatesPlaceholder') : $t('modelSettings.debug.documentsPlaceholder')"
:autosize="{ minRows: 4, maxRows: 8 }"
/>
<p class="form-desc">{{ $t('modelSettings.debug.documentsHint') }}</p>
<p class="form-desc">{{ useSimilarityMode ? $t('modelSettings.debug.similarityHint') : $t('modelSettings.debug.documentsHint') }}</p>
</div>
<div v-if="needsFile" class="form-item">
<label class="form-label">{{ fileLabel }}</label>
<div v-if="showFilePicker" class="form-item">
<label class="form-label">{{ useSimilarityMode ? $t('modelSettings.debug.similarityImage') : fileLabel }}</label>
<div class="file-picker">
<input
ref="fileInputRef"
class="file-picker__input"
type="file"
:accept="selectedModel.type === 'VLLM' ? 'image/*' : 'audio/*'"
:accept="selectedModel.type === 'ASR' ? 'audio/*' : 'image/*'"
@change="onNativeFileChange"
>
<t-button variant="outline" size="small" @click="fileInputRef?.click()">
@ -110,7 +117,11 @@
{{ $t('modelSettings.debug.chooseFile') }}
</t-button>
</div>
<p v-if="file" class="form-desc">{{ file.name }} · {{ formatBytes(file.size) }}</p>
<div v-if="file" class="selected-file">
<img v-if="imagePreview" :src="imagePreview" :alt="file.name" class="image-preview">
<p class="form-desc">{{ file.name }} · {{ formatBytes(file.size) }}</p>
<t-button variant="text" size="small" @click="clearFile">{{ $t('common.clear') }}</t-button>
</div>
</div>
</section>
@ -196,6 +207,23 @@
<p v-if="result.error" class="result-error">{{ result.error }}</p>
<div v-if="similarityResults.length > 0" class="similarity-list">
<div v-for="(item, rank) in similarityResults" :key="item.index" class="similarity-row">
<span class="similarity-row__rank">{{ rank + 1 }}</span>
<span class="similarity-row__content">
<template v-if="item.kind === 'image'">
<img v-if="imagePreview" :src="imagePreview" :alt="file?.name" class="similarity-row__image">
{{ $t('modelSettings.debug.similarityImageCandidate') }}
</template>
<template v-else>{{ item.content }}</template>
</span>
<span class="similarity-row__bar">
<span :style="{ width: `${Math.max(0, item.similarity) * 100}%` }" />
</span>
<span class="similarity-row__score">{{ item.similarity.toFixed(4) }}</span>
</div>
</div>
<t-tabs v-model="resultTab" class="result-tabs">
<t-tab-panel value="response" :label="$t('modelSettings.debug.rawResponse')" />
<t-tab-panel value="request" :label="$t('modelSettings.debug.requestPreview')" />
@ -254,6 +282,8 @@ const selectedModelId = ref('')
const input = ref('')
const documentsText = ref('')
const file = ref<File | null>(null)
const inputMode = ref<'text' | 'image' | 'similarity'>('text')
const imagePreview = ref('')
const fileInputRef = ref<HTMLInputElement | null>(null)
const reasoningEffort = ref<ReasoningLevel>('off')
const temperature = ref(0.7)
@ -277,14 +307,29 @@ const isChat = computed(() => selectedModel.value?.type === 'KnowledgeQA')
// (remote chat models carry `capabilities` computed by the backend).
const supportsThinking = computed(() => modelCanThink(selectedModel.value?.capabilities))
const reasoningOptions = computed(() => supportedLevels(selectedModel.value?.capabilities))
const needsFile = computed(() => ['VLLM', 'ASR'].includes(selectedModel.value?.type || ''))
const acceptsImages = computed(() => {
const model = selectedModel.value
if (!model || model.type === 'ASR') return false
return model.type === 'VLLM'
|| model.capabilities?.input?.includes('image') === true
|| (model.type === 'KnowledgeQA' && model.parameters.supports_vision === true)
})
const hasImageMode = computed(() => acceptsImages.value && ['Embedding', 'Rerank'].includes(selectedModel.value?.type || ''))
const useImageMode = computed(() => hasImageMode.value && inputMode.value === 'image')
const isEmbedding = computed(() => selectedModel.value?.type === 'Embedding')
const useSimilarityMode = computed(() => isEmbedding.value && inputMode.value === 'similarity')
// In similarity mode the image is an optional extra candidate.
const showFilePicker = computed(() => selectedModel.value?.type === 'ASR'
|| (acceptsImages.value && (!hasImageMode.value || useImageMode.value || useSimilarityMode.value)))
const needsFile = computed(() => useImageMode.value || ['VLLM', 'ASR'].includes(selectedModel.value?.type || ''))
const documents = computed(() => documentsText.value.split('\n').map(item => item.trim()).filter(Boolean))
const canRun = computed(() => {
if (!selectedModel.value) return false
if (useSimilarityMode.value) return !!input.value.trim() && (documents.value.length > 0 || !!file.value)
if (needsFile.value && !file.value) return false
if (selectedModel.value.type === 'ASR') return true
if (selectedModel.value.type === 'Rerank') return !!input.value.trim() && documents.value.length > 0
return !!input.value.trim()
if (['ASR', 'VLLM'].includes(selectedModel.value.type) || (selectedModel.value.type === 'Embedding' && useImageMode.value)) return true
if (selectedModel.value.type === 'Rerank') return !!input.value.trim() && (useImageMode.value ? !!file.value : documents.value.length > 0)
return !!input.value.trim() || (isChat.value && acceptsImages.value && !!file.value)
})
const allModelTypeOptions = computed(() => {
@ -319,6 +364,7 @@ const vendorLabel = (model: ModelConfig) => {
}
const inputLabel = computed(() => {
if (useSimilarityMode.value) return t('modelSettings.debug.similarityQuery')
if (selectedModel.value?.type === 'Embedding') return t('modelSettings.debug.embeddingInput')
if (selectedModel.value?.type === 'VLLM') return t('modelSettings.debug.vlmPrompt')
if (selectedModel.value?.type === 'Rerank') return t('modelSettings.debug.query')
@ -326,15 +372,16 @@ const inputLabel = computed(() => {
})
const inputPlaceholder = computed(() => {
if (useSimilarityMode.value) return t('modelSettings.debug.similarityQueryPlaceholder')
if (selectedModel.value?.type === 'Embedding') return t('modelSettings.debug.embeddingPlaceholder')
if (selectedModel.value?.type === 'VLLM') return t('modelSettings.debug.vlmPromptPlaceholder')
return t('modelSettings.debug.queryPlaceholder')
})
const fileLabel = computed(() =>
selectedModel.value?.type === 'VLLM'
? t('modelSettings.debug.imageFile')
: t('modelSettings.debug.audioFile'),
selectedModel.value?.type === 'ASR'
? t('modelSettings.debug.audioFile')
: t('modelSettings.debug.imageFile'),
)
const formattedResult = computed(() => {
@ -345,6 +392,19 @@ const formattedResult = computed(() => {
return JSON.stringify(value, null, 2)
})
interface SimilarityResult {
index: number
kind: 'text' | 'image'
content?: string
similarity: number
}
const similarityResults = computed<SimilarityResult[]>(() => {
const response = result.value?.raw_response as { results?: unknown } | null | undefined
if (!isEmbedding.value || !result.value?.ok || !response || !Array.isArray(response.results)) return []
return response.results as SimilarityResult[]
})
const OBSERVATION_LABELS: Record<string, string> = {
api: 'modelSettings.debug.metrics.api',
thinking_format: 'modelSettings.debug.metrics.thinkingFormat',
@ -418,11 +478,9 @@ watch(() => selectedModel.value?.id, () => {
reasoningEffort.value = clampLevel(reasoningEffort.value, reasoningOptions.value)
}, { immediate: true })
watch(() => selectedModel.value?.type, () => {
file.value = null
result.value = null
history.value = []
resultTab.value = 'response'
watch(() => selectedModel.value?.id, () => {
inputMode.value = 'text'
clearFile()
})
const resetResult = () => {
@ -441,6 +499,17 @@ const selectModelType = (type: DebugModelType) => {
resetResult()
}
const clearFile = () => {
file.value = null
if (fileInputRef.value) fileInputRef.value.value = ''
resetResult()
}
watch(file, value => {
if (imagePreview.value) URL.revokeObjectURL(imagePreview.value)
imagePreview.value = value && selectedModel.value?.type !== 'ASR' ? URL.createObjectURL(value) : ''
})
const onNativeFileChange = (event: Event) => {
const target = event.target as HTMLInputElement
const selectedFile = target.files?.[0] || null
@ -473,9 +542,9 @@ const runDebug = async () => {
try {
const level: ReasoningLevel | null = supportsThinking.value ? reasoningEffort.value : null
const nextResult = await debugModel(selectedModel.value.id, {
input: input.value.trim(),
documents: documents.value,
file: file.value,
input: isEmbedding.value && useImageMode.value ? '' : input.value.trim(),
documents: (selectedModel.value.type === 'Rerank' && !useImageMode.value) || useSimilarityMode.value ? documents.value : [],
file: showFilePicker.value ? file.value : null,
options: isChat.value ? {
system_prompt: systemPrompt.value.trim() || undefined,
temperature: temperature.value,
@ -483,7 +552,7 @@ const runDebug = async () => {
max_tokens: maxTokens.value,
// reasoning_effort is authoritative; the boolean keeps older backends working.
...(level ? { reasoning_effort: level, thinking: levelEnablesThinking(level) } : { thinking: false }),
} : {},
} : useSimilarityMode.value ? { similarity: true } : {},
})
result.value = nextResult
history.value.unshift({
@ -506,6 +575,7 @@ const copyResult = async () => {
}
onBeforeUnmount(() => {
if (imagePreview.value) URL.revokeObjectURL(imagePreview.value)
if (document.activeElement instanceof HTMLElement) {
document.activeElement.blur()
}
@ -632,6 +702,20 @@ onBeforeUnmount(() => {
}
}
.selected-file {
display: flex;
align-items: center;
gap: 8px;
margin-top: 8px;
}
.image-preview {
width: 80px;
height: 80px;
object-fit: contain;
border-radius: var(--app-radius-sm);
}
.history-list {
display: flex;
flex-wrap: wrap;
@ -720,6 +804,65 @@ onBeforeUnmount(() => {
font-size: var(--app-text-sm);
}
.similarity-list {
display: flex;
flex-direction: column;
gap: 4px;
margin-top: 10px;
}
.similarity-row {
display: grid;
grid-template-columns: 20px minmax(0, 1fr) 80px 56px;
align-items: center;
gap: 8px;
padding: 6px 8px;
border-radius: var(--app-radius-sm);
background: var(--td-bg-color-secondarycontainer);
font-size: var(--app-text-sm);
&__rank {
color: var(--td-text-color-placeholder);
text-align: right;
}
&__content {
display: flex;
align-items: center;
gap: 6px;
overflow: hidden;
color: var(--td-text-color-primary);
text-overflow: ellipsis;
white-space: nowrap;
}
&__image {
width: 24px;
height: 24px;
object-fit: cover;
border-radius: var(--app-radius-xs);
}
&__bar {
height: 6px;
overflow: hidden;
border-radius: 3px;
background: var(--td-component-stroke);
span {
display: block;
height: 100%;
background: var(--td-brand-color);
}
}
&__score {
color: var(--td-text-color-secondary);
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
text-align: right;
}
}
.result-error {
margin: 10px 0 0;
color: var(--td-error-color);

View File

@ -87,10 +87,11 @@
<span class="source-option__label">{{ $t('model.editor.sourceRemote') }}</span>
</button>
<button
v-if="activeModelType !== 'rerank'"
type="button"
class="source-option"
:class="{ 'is-active': formData.source === 'local', 'is-disabled': ollamaServiceStatus === false || activeModelType === 'rerank' }"
:disabled="ollamaServiceStatus === false || activeModelType === 'rerank'"
:class="{ 'is-active': formData.source === 'local', 'is-disabled': ollamaServiceStatus === false }"
:disabled="ollamaServiceStatus === false"
role="radio"
:aria-checked="formData.source === 'local'"
@click="formData.source = 'local'"
@ -100,14 +101,8 @@
</button>
</div>
<!-- ReRank模型不支持Ollama的提示信息 -->
<div v-if="activeModelType === 'rerank'" class="ollama-unavailable-tip rerank-tip">
<t-icon name="info-circle-filled" class="tip-icon info" />
<span class="tip-text">{{ $t('model.editor.ollamaNotSupportRerank') }}</span>
</div>
<!-- Ollama不可用时的提示信息 -->
<div v-else-if="shouldShowOllamaUnavailableTip(formData.source, activeModelType, ollamaServiceStatus)"
<div v-if="shouldShowOllamaUnavailableTip(formData.source, activeModelType, ollamaServiceStatus)"
class="ollama-unavailable-tip">
<t-icon name="error-circle-filled" class="tip-icon" />
<span class="tip-text">{{ $t('model.editor.ollamaUnavailable') }}</span>
@ -1394,6 +1389,8 @@ const getModalDescription = () => {
// 获取模型名称占位符
const getModelNamePlaceholder = () => {
const catalogModel = catalogEntries.value[0]
if (formData.value.source === 'remote' && catalogModel) return catalogModel.id
if (activeModelType.value === 'vllm') {
return formData.value.source === 'local'
? t('model.editor.modelNamePlaceholder.localVllm')
@ -1461,52 +1458,29 @@ const goToOllamaSettings = async () => {
const lastOpenedModelId = ref<string | null>(null)
const selectModelType = async (type: EditorModelType) => {
if (isEdit.value || draftModelType.value === type) return
if (saving.value || isEdit.value || draftModelType.value === type) return
draftModelType.value = type
if (type === 'rerank') {
formData.value.source = 'remote'
}
if (type !== 'embedding') {
formData.value.dimension = undefined
formData.value.supportsDimensionOverride = false
dimensionChecked.value = false
dimensionSuccess.value = false
dimensionMessage.value = ''
}
if (type !== 'chat') {
formData.value.supportsVision = false
}
remoteChecked.value = false
remoteAvailable.value = false
remoteMessage.value = ''
// A different model type starts a new connection, including credentials
// and advanced overrides. Clear the old draft before waiting for its catalog.
resetForm()
const draft = formData.value
await loadProviders()
const supported = providerOptions.value.some(p => p.value === formData.value.provider)
if (!supported) {
formData.value.provider = 'generic'
formData.value.baseUrl = ''
// 只丢厂商相关的 extra_config;协议 / 远端模型名是用户对这次接入的选择,
// 与厂商无关,和 handleProviderChange 保持同一套规则。
formData.value.extraConfig = keepVendorNeutralExtraConfig()
formData.value.appSecret = ''
} else {
handleProviderChange(formData.value.provider || 'generic')
if (props.visible && !isEdit.value && formData.value === draft && activeModelType.value === type) {
applyExtraFieldDefaults()
}
}
// 监听 visible 变化,初始化表单
watch(() => props.visible, (val) => {
if (val) {
const typeChanged = draftModelType.value !== props.modelType
// Sync the entry type before loading its catalog; the previous draft may
// belong to another settings tab (including the initial chat default).
draftModelType.value = props.modelType
// 检查Ollama服务状态
checkOllamaServiceStatus()
// 从 API 加载 Model Provider 列表(编辑已有行时顺便补齐额外字段默认值)。
// Catalogs can be published by another administrator while this page is
// open, so refresh this type without dropping other types' cached lists.
loadProviders(true).then(() => {
if (props.visible && !isEdit.value) applyExtraFieldDefaults()
})
advancedOpen.value = false
// 每次打开都清理上一次遗留的校验/检测结果,避免编辑别的模型时
@ -1521,7 +1495,6 @@ watch(() => props.visible, (val) => {
dimensionMessage.value = ''
const currentId = props.modelData?.id ?? null
draftModelType.value = props.modelType
hydratingForm.value = true
try {
@ -1549,11 +1522,11 @@ watch(() => props.visible, (val) => {
: [],
}
legacyThinkingControlLoaded.value = !!props.modelData.thinkingControl
} else if (lastOpenedModelId.value !== null || !formData.value.id) {
// 上次是编辑某个模型,或第一次新增 → 重置成空白
} else if (typeChanged || lastOpenedModelId.value !== null || !formData.value.id) {
// 切换类型、上次是编辑某个模型,或第一次新增 → 重置成空白
resetForm()
}
// 否则:连续两次"新增"打开(中间是点遮罩/ESC 关闭的)→ 保留上次填写
// 同一类型连续两次"新增"打开(中间是点遮罩/ESC 关闭的)→ 保留上次填写
lastOpenedModelId.value = currentId
@ -1571,6 +1544,16 @@ watch(() => props.visible, (val) => {
hydratingForm.value = false
})
}
// Catalogs can be published while this page is open. Refresh this type,
// but do not apply defaults to a different draft if the request finishes late.
const draft = formData.value
const type = activeModelType.value
loadProviders(true).then(() => {
if (props.visible && !isEdit.value && formData.value === draft && activeModelType.value === type) {
applyExtraFieldDefaults()
}
})
}
})
@ -1621,6 +1604,17 @@ const resetForm = () => {
customHeaders: [],
appSecret: '',
}
catalogFilled.value = {}
advancedOpen.value = false
saveError.value = ''
searchKeyword.value = ''
invalidateConnectionTest(false)
resolveRevision++
if (resolveTimer) {
clearTimeout(resolveTimer)
resolveTimer = null
}
resolving.value = false
resolved.value = null
resolveFailed.value = false
resolveError.value = ''
@ -1632,6 +1626,7 @@ const resetForm = () => {
dimensionChecked.value = false
dimensionSuccess.value = false
dimensionMessage.value = ''
nextTick(() => formRef.value?.clearValidate?.())
}
// 处理厂商选择变化 (自动填充默认 URL)
@ -1676,13 +1671,11 @@ const resetModelSelectionForVendor = () => {
const handleProviderChange = (value: string) => {
const provider = providerOptions.value.find(opt => opt.value === value)
if (provider?.defaultUrls) {
// 根据当前模型类型获取对应的默认 URL
const defaultUrl = provider.defaultUrls[activeModelType.value]
if (defaultUrl) {
formData.value.baseUrl = defaultUrl
}
}
// Older catalogs omit inherited type URLs. Mirror GetDefaultURL's chat
// fallback, preserving an explicitly empty URL and never the previous vendor's.
formData.value.baseUrl = provider?.defaultUrls?.[activeModelType.value]
?? provider?.defaultUrls?.chat
?? ''
// 重置校验状态:它描述的是上一家厂商的连通性,跟新厂商无关。放在 defaultUrls
// 判断之外,否则切到没有默认地址的厂商时会留着一条"连接正常"的旧结论。
remoteChecked.value = false
@ -2930,17 +2923,6 @@ const handleCancel = () => {
line-height: 1.5;
}
// ReRank提示使用主题绿色风格,与主页面保持一致
&.rerank-tip {
background: var(--td-success-color-light);
border: 1px solid var(--td-success-color-focus);
border-left: 3px solid var(--td-brand-color);
.tip-text {
color: var(--td-success-color);
}
}
:deep(.tip-link) {
color: var(--td-brand-color);
font-size: var(--app-text-md);

View File

@ -0,0 +1,177 @@
import assert from 'node:assert/strict'
import { readFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { fileURLToPath } from 'node:url'
import { runInNewContext } from 'node:vm'
import test from 'node:test'
import { compileScript, parse } from '@vue/compiler-sfc'
import ts from 'typescript'
import { createRenderer, h, nextTick, reactive } from 'vue'
const require = createRequire(import.meta.url)
const filename = fileURLToPath(new URL('./ModelDebugDrawer.vue', import.meta.url))
const { descriptor } = parse(readFileSync(filename, 'utf8'), { filename })
const script = compileScript(descriptor, { id: 'model-debug-images' }).content
.replace('__expose();', '')
.replace('return __returned__', '__expose(__returned__); return __returned__')
const compiled = ts.transpileModule(script, {
compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022 },
}).outputText
async function fixture(type: string, images = true, legacyVision = false) {
const requests: any[] = []
const revoked: string[] = []
const props = reactive({ visible: false, models: [
{ id: 'image-model', name: 'model', type, source: 'remote', parameters: { supports_vision: legacyVision }, capabilities: { input: images ? ['text', 'image'] : ['text'], thinking_levels: [] } },
{ id: 'text-model', name: 'text', type, source: 'remote', parameters: {}, capabilities: { input: ['text'], thinking_levels: [] } },
] })
const exports: any = {}
runInNewContext(compiled, {
exports, console, document: { activeElement: null }, HTMLElement: class {},
URL: { createObjectURL: () => 'blob:preview', revokeObjectURL: (url: string) => revoked.push(url) },
require(name: string) {
if (name === 'vue') return require('vue')
if (name === 'vue-i18n') return { useI18n: () => ({ t: (key: string) => key, locale: { value: 'en-US' } }) }
if (name === '@/api/model') return { debugModel: async (id: string, data: any) => { requests.push({ id, ...data }); return { ok: true, elapsed_ms: 1, raw_response: [], observations: {} } } }
if (name === '@/stores/modelProviders') return { useModelProvidersStore: () => ({ ensureLoaded: async () => {}, labelFor: () => '' }) }
if (name === '@/utils') return { fileSizeVerification: () => false }
if (name === '@/utils/reasoningEffort') return require('../utils/reasoningEffort.ts')
if (name === '@/utils/contextWindow') return require('../utils/contextWindow.ts')
if (name === 'tdesign-vue-next') return { MessagePlugin: { error() {} } }
return { default: {} }
},
})
exports.default.render = () => null
const renderer = createRenderer<any, any>({
createElement: () => ({}), createText: () => ({}), createComment: () => ({}),
insert() {}, remove() {}, setElementText() {}, setText() {}, patchProp() {},
parentNode: () => null, nextSibling: () => null,
})
let vm: any
const app = renderer.createApp({ setup: () => () => h(exports.default, { ...props, ref: (value: any) => { vm = value } }) })
app.mount({})
props.visible = true
await nextTick()
return { vm, requests, revoked, dispose: () => app.unmount() }
}
const file = { name: 'cat.png', size: 100, type: 'image/png' }
for (const type of ['Embedding', 'Rerank']) {
test(`${type} image mode sends the upload without stale text documents`, async () => {
const f = await fixture(type)
try {
assert.equal(f.vm.hasImageMode, true)
assert.equal(f.vm.showFilePicker, false)
f.vm.input = 'a cat'
f.vm.documentsText = 'old candidate'
f.vm.inputMode = 'image'
assert.equal(f.vm.canRun, false)
f.vm.onNativeFileChange({ target: { files: [file] } })
await nextTick()
assert.equal(f.vm.imagePreview, 'blob:preview')
assert.equal(f.vm.canRun, true)
await f.vm.runDebug()
assert.equal(f.requests[0].file.name, file.name)
assert.equal(f.requests[0].input, type === 'Embedding' ? '' : 'a cat')
assert.equal(f.requests[0].documents.length, 0)
f.vm.clearFile()
await nextTick()
assert.equal(f.vm.canRun, false)
assert.deepEqual(f.revoked, ['blob:preview'])
} finally { f.dispose() }
})
}
test('embedding similarity sends the query, candidates and an optional image', async () => {
const f = await fixture('Embedding')
try {
f.vm.inputMode = 'similarity'
await nextTick()
assert.equal(f.vm.useSimilarityMode, true)
assert.equal(f.vm.showFilePicker, true)
assert.equal(f.vm.canRun, false)
f.vm.input = 'a dog'
assert.equal(f.vm.canRun, false, 'a query alone has nothing to compare against')
f.vm.documentsText = 'a puppy\n\na car'
assert.equal(f.vm.canRun, true)
await f.vm.runDebug()
assert.equal(f.requests[0].input, 'a dog')
assert.deepEqual([...f.requests[0].documents], ['a puppy', 'a car'])
assert.equal(f.requests[0].options.similarity, true)
assert.equal(f.requests[0].file, null)
f.vm.documentsText = ''
f.vm.onNativeFileChange({ target: { files: [file] } })
assert.equal(f.vm.canRun, true, 'an image is a candidate on its own')
await f.vm.runDebug()
assert.equal(f.requests[1].file.name, file.name)
} finally { f.dispose() }
})
test('embedding similarity is offered on text-only models without an image picker', async () => {
const f = await fixture('Embedding', false)
try {
assert.equal(f.vm.hasImageMode, false)
f.vm.inputMode = 'similarity'
await nextTick()
assert.equal(f.vm.showFilePicker, false)
f.vm.input = 'a dog'
f.vm.documentsText = 'a puppy'
await f.vm.runDebug()
assert.equal(f.requests[0].file, null)
assert.equal(f.requests[0].options.similarity, true)
} finally { f.dispose() }
})
test('image-capable chat sends text and an optional image, including image-only input', async () => {
const f = await fixture('KnowledgeQA')
try {
assert.equal(f.vm.showFilePicker, true)
assert.equal(f.vm.hasImageMode, false)
f.vm.onNativeFileChange({ target: { files: [file] } })
assert.equal(f.vm.canRun, true)
await f.vm.runDebug()
assert.equal(f.requests[0].input, '')
assert.equal(f.requests[0].file.name, file.name)
f.vm.input = 'describe'
await f.vm.runDebug()
assert.equal(f.requests[1].input, 'describe')
assert.equal(f.requests[1].file.name, file.name)
} finally { f.dispose() }
})
for (const type of ['KnowledgeQA', 'Embedding', 'Rerank']) {
test(`${type} clears the file when switching to a text-only model of the same type`, async () => {
const f = await fixture(type)
try {
f.vm.inputMode = 'image'
f.vm.onNativeFileChange({ target: { files: [file] } })
await nextTick()
f.vm.selectedModelId = 'text-model'
await nextTick()
assert.equal(f.vm.file, null)
assert.equal(f.vm.inputMode, 'text')
assert.equal(f.vm.showFilePicker, false)
assert.equal(f.vm.imagePreview, '')
} finally { f.dispose() }
})
}
test('legacy vision flag enables chat image uploads', async () => {
const f = await fixture('KnowledgeQA', false, true)
try { assert.equal(f.vm.showFilePicker, true) } finally { f.dispose() }
})
for (const type of ['VLLM', 'ASR']) {
test(`${type} still requires its file`, async () => {
const f = await fixture(type, false)
try {
assert.equal(f.vm.showFilePicker, true)
assert.equal(f.vm.canRun, false)
f.vm.onNativeFileChange({ target: { files: [file] } })
assert.equal(f.vm.canRun, true)
await f.vm.runDebug()
assert.equal(f.requests[0].file.name, file.name)
} finally { f.dispose() }
})
}

View File

@ -28,10 +28,13 @@ function deferred<T>() {
type Result = { available: boolean; message?: string; dimension?: number }
async function fixture(options: {
type?: string
initialType?: string
edit?: boolean
check?: (payload: any) => Promise<Result>
save?: (payload: any) => Promise<void>
providers?: any[]
providersByType?: Record<string, any[]>
loadProviders?: (type?: string) => Promise<void>
resolve?: (params: any) => Promise<any>
} = {}) {
const requests: any[] = []
@ -39,9 +42,18 @@ async function fixture(options: {
const resolves: any[] = []
const providers: any[] = options.providers || []
const catalogEvents: string[] = []
const catalogRequests: Array<{ type?: string; force?: boolean }> = []
const loadedProviders = reactive<Record<string, any[]>>({})
const providersStore = {
ensureLoaded: async (_type?: string, force?: boolean) => { catalogEvents.push(force ? 'refresh' : 'load'); return providers },
providersFor: () => providers,
ensureLoaded: async (type?: string, force?: boolean) => {
catalogEvents.push(force ? 'refresh' : 'load')
catalogRequests.push({ type, force })
const entries = options.providersByType?.[type || ''] ?? providers
await options.loadProviders?.(type)
loadedProviders[type || ''] = entries
return entries
},
providersFor: (type?: string) => options.providersByType ? loadedProviders[type || ''] || [] : providers,
isLoading: () => false,
providerById: (id: string) => providers.find((p) => p.value === id),
iconFor: () => '',
@ -52,7 +64,7 @@ async function fixture(options: {
const visibility: boolean[] = []
const props = reactive<{ visible: boolean; modelType: string; modelData: any }>({
visible: false,
modelType: options.type || 'chat',
modelType: options.initialType || options.type || 'chat',
modelData: options.edit ? {
id: 'saved-model', modelName: 'old-model', name: '', source: 'remote',
baseUrl: 'https://example.com/v1', provider: 'generic', isDefault: false,
@ -109,15 +121,178 @@ async function fixture(options: {
}),
})
app.mount({})
props.modelType = options.type || 'chat'
props.visible = true
await nextTick()
await nextTick()
Object.assign(vm.formData, { modelName: 'draft-model', baseUrl: 'https://example.com/v1', apiKey: options.edit ? '' : 'draft-key' })
vm.formRef = { validate: async () => true }
await nextTick()
return { vm, props, requests, saves, toasts, visibility, resolves, catalogEvents, close: () => app.unmount() }
return { vm, props, requests, saves, toasts, visibility, resolves, catalogEvents, catalogRequests, close: () => app.unmount() }
}
for (const type of ['embedding', 'rerank', 'vllm', 'asr']) {
test(`${type}: opening from its settings tab loads the matching providers and models`, async () => {
const provider = {
value: 'vendor', label: 'Vendor', modelTypes: [type],
models: [{ id: `${type}-model`, type: type === 'vllm' ? 'chat' : type }],
}
const f = await fixture({ initialType: 'chat', type, providersByType: { [type]: [provider] } })
try {
assert.deepEqual(f.catalogRequests, [{ type, force: true }])
assert.equal(f.vm.activeModelType, type)
assert.deepEqual(Array.from(f.vm.providerOptions, (p: any) => p.value), ['vendor'])
f.vm.formData.provider = 'vendor'
f.vm.formData.modelName = ''
assert.deepEqual(Array.from(f.vm.catalogModelOptions, (m: any) => m.value), [`${type}-model`])
} finally { f.close() }
})
}
test('reopening after changing the draft type refreshes the newly selected settings tab', async () => {
const f = await fixture()
try {
await f.vm.selectModelType('rerank')
f.props.visible = false
await nextTick()
f.catalogRequests.length = 0
f.props.modelType = 'embedding'
f.props.visible = true
await nextTick()
assert.deepEqual(f.catalogRequests, [{ type: 'embedding', force: true }])
assert.equal(f.vm.activeModelType, 'embedding')
} finally { f.close() }
})
for (const type of ['chat', 'embedding', 'rerank', 'vllm', 'asr']) {
test(`${type}: changing type resets the whole connection form and its results`, async () => {
const f = await fixture({ type: type === 'chat' ? 'rerank' : 'chat', providers: [{
value: 'vendor', modelTypes: ['chat', 'embedding', 'rerank', 'vllm', 'asr'], models: [],
}] })
try {
Object.assign(f.vm.formData, {
source: 'local', provider: 'vendor', modelName: 'old-model', displayName: 'Old display',
baseUrl: 'https://example.com/rerank', apiKey: 'old-key', appSecret: 'old-secret',
customHeaders: [{ key: 'X-Old', value: 'old' }], extraConfig: { api: 'old-api', remote_model_name: 'old-name' },
dimension: 1024, supportsDimensionOverride: true, supportsVision: true, interfaceType: 'ollama',
contextWindow: 64000, maxOutputTokens: 4096, maxConcurrency: 8, isDefault: true,
thinkingControl: 'old-thinking', spec: { context_window: 64000 }, specCompat: '{"old":true}',
})
await nextTick()
f.vm.remoteChecked = true
f.vm.remoteAvailable = true
f.vm.remoteMessage = 'old result'
f.vm.dimensionChecked = true
f.vm.dimensionSuccess = true
f.vm.dimensionMessage = 'old dimension'
f.vm.modelChecked = true
f.vm.modelAvailable = true
f.vm.catalogFilled = { contextWindow: 64000 }
f.vm.advancedOpen = true
f.vm.saveError = 'old error'
let cleared = 0
f.vm.formRef = { clearValidate: () => { cleared++ } }
await f.vm.selectModelType(type)
await nextTick()
assert.equal(f.vm.activeModelType, type)
assert.equal(f.vm.formData.source, 'remote')
assert.equal(f.vm.formData.provider, 'generic')
for (const field of ['modelName', 'displayName', 'baseUrl', 'apiKey', 'appSecret', 'thinkingControl', 'specCompat']) {
assert.equal(f.vm.formData[field], '', field)
}
for (const field of ['dimension', 'interfaceType', 'contextWindow', 'maxOutputTokens', 'maxConcurrency']) {
assert.equal(f.vm.formData[field], undefined, field)
}
for (const field of ['supportsDimensionOverride', 'supportsVision', 'isDefault']) {
assert.equal(f.vm.formData[field], false, field)
}
assert.equal(f.vm.formData.spec, null)
assert.equal(f.vm.formData.customHeaders.length, 0)
assert.equal(Object.keys(f.vm.formData.extraConfig).length, 0)
assert.equal(Object.keys(f.vm.catalogFilled).length, 0)
for (const field of ['remoteChecked', 'remoteAvailable', 'remoteStale', 'dimensionChecked', 'dimensionSuccess', 'modelChecked', 'modelAvailable', 'advancedOpen']) {
assert.equal(f.vm[field], false, field)
}
for (const field of ['remoteMessage', 'dimensionMessage', 'saveError']) assert.equal(f.vm[field], '', field)
assert.ok(cleared > 0)
} finally { f.close() }
})
}
test('a delayed type catalog cannot rewrite a newer connection draft', async () => {
const pending = deferred<void>()
const f = await fixture({
loadProviders: type => type === 'embedding' ? pending.promise : Promise.resolve(),
providersByType: { asr: [{ value: 'vendor', defaultUrls: { asr: 'https://default.example.com' }, models: [] }] },
})
try {
const earlier = f.vm.selectModelType('embedding')
assert.equal(f.vm.formData.apiKey, '', 'reset must not wait for the catalog')
await f.vm.selectModelType('asr')
f.vm.formData.provider = 'vendor'
f.vm.formData.baseUrl = 'https://custom.example.com'
f.vm.formData.extraConfig = { region: 'custom' }
pending.resolve()
await earlier
assert.equal(f.vm.activeModelType, 'asr')
assert.equal(f.vm.formData.baseUrl, 'https://custom.example.com')
assert.equal(f.vm.formData.extraConfig.region, 'custom')
} finally { pending.resolve(); f.close() }
})
test('same-type reopening keeps the draft, but another settings tab starts a fresh form', async () => {
const f = await fixture()
try {
f.props.visible = false
await nextTick()
f.props.visible = true
await nextTick()
assert.equal(f.vm.formData.modelName, 'draft-model')
f.props.visible = false
await nextTick()
f.props.modelType = 'embedding'
f.props.visible = true
await nextTick()
assert.equal(f.vm.formData.modelName, '')
assert.equal(f.vm.formData.apiKey, '')
assert.equal(f.vm.formData.baseUrl, '')
} finally { f.close() }
})
test('type reset discards pending connection and catalog diagnosis results', async () => {
const connection = deferred<Result>()
const diagnosis = deferred<any>()
const f = await fixture({ check: () => connection.promise, resolve: () => diagnosis.promise })
try {
const checking = f.vm.checkRemoteAPI()
const resolving = f.vm.runResolve()
await f.vm.selectModelType('asr')
connection.resolve({ available: true })
diagnosis.resolve({ cataloged: true, model: { id: 'old-model' } })
await Promise.all([checking, resolving])
assert.equal(f.vm.remoteChecked, false)
assert.equal(f.vm.remoteStale, false)
assert.equal(f.vm.checking, false)
assert.equal(f.vm.resolved, null)
assert.equal(f.vm.resolving, false)
} finally { f.close() }
})
test('selecting the current type, editing, or saving does not reset the draft', async () => {
for (const state of ['same-type', 'editing', 'saving']) {
const f = await fixture({ edit: state === 'editing' })
try {
const original = JSON.stringify(f.vm.formData)
const requests = f.catalogRequests.length
if (state === 'saving') f.vm.saving = true
await f.vm.selectModelType(state === 'same-type' ? 'chat' : 'asr')
assert.equal(JSON.stringify(f.vm.formData), original, state)
assert.equal(f.catalogRequests.length, requests, state)
} finally { f.close() }
}
})
for (const type of ['chat', 'embedding', 'rerank', 'vllm', 'asr']) {
test(`${type}: connection tests use unsaved form values without saving`, async () => {
const f = await fixture({ type })
@ -791,6 +966,54 @@ test('switching vendor clears a connection result that described the old one', a
} finally { f.close() }
})
test('ASR vendor changes replace the previous URL and show the selected catalog model as the hint', async () => {
const f = await fixture({ type: 'asr', providers: [
{ value: 'openai', defaultUrls: { asr: 'https://api.openai.com/v1' }, models: [{ id: 'whisper-1', type: 'asr' }] },
{ value: 'zhipu', defaultUrls: { asr: 'https://open.bigmodel.cn/api/paas/v4' }, models: [{ id: 'glm-asr-2512', type: 'asr' }] },
{ value: 'generic', models: [] },
] })
try {
for (const [vendor, url, hint] of [
['openai', 'https://api.openai.com/v1', 'whisper-1'],
['zhipu', 'https://open.bigmodel.cn/api/paas/v4', 'glm-asr-2512'],
['generic', '', 'model.editor.modelNamePlaceholder.remoteAsr'],
]) {
f.vm.formData.provider = vendor
f.vm.handleProviderChange(vendor)
assert.equal(f.vm.formData.baseUrl, url, vendor)
assert.equal(f.vm.getModelNamePlaceholder(), hint, vendor)
}
} finally { f.close() }
})
test('ASR inherits the vendor chat base URL from older catalog responses', async () => {
const f = await fixture({ type: 'asr', providers: [
{ value: 'mimo', defaultUrls: { chat: 'https://api.xiaomimimo.com/v1' }, models: [{ id: 'mimo-v2.5-asr', type: 'asr' }] },
{ value: 'zhipu', defaultUrls: { chat: 'https://open.bigmodel.cn/api/paas/v4' }, models: [] },
{ value: 'custom', defaultUrls: { chat: 'https://chat.example.com', asr: 'https://speech.example.com' }, models: [] },
] })
try {
for (const [vendor, url] of [
['mimo', 'https://api.xiaomimimo.com/v1'],
['zhipu', 'https://open.bigmodel.cn/api/paas/v4'],
['custom', 'https://speech.example.com'],
]) {
f.vm.formData.provider = vendor
f.vm.handleProviderChange(vendor)
assert.equal(f.vm.formData.baseUrl, url, vendor)
}
} finally { f.close() }
})
test('an explicitly empty type URL clears the old endpoint without inheriting the chat URL', async () => {
const f = await fixture({ type: 'asr', providers: [{ value: 'custom', defaultUrls: { chat: 'https://chat.example.com', asr: '' } }] })
try {
f.vm.formData.provider = 'custom'
f.vm.handleProviderChange('custom')
assert.equal(f.vm.formData.baseUrl, '')
} finally { f.close() }
})
// A VLM entry is a chat model that accepts images, so the backend — which
// already scopes the provider list to the requested model type — returns it
// typed "chat". Re-filtering on that type in the editor emptied the picker
@ -844,7 +1067,7 @@ for (const [type, modelName] of [['chat', 'chat'], ['embedding', 'embedding'], [
})
}
test('switching WeKnora Cloud model types updates the managed choices', async () => {
test('reselecting WeKnora Cloud after a type reset offers the new managed choices', async () => {
const f = await fixture({ providers: [cloudProvider] })
try {
f.vm.formData.provider = 'weknoracloud'
@ -853,6 +1076,9 @@ test('switching WeKnora Cloud model types updates the managed choices', async ()
f.vm.formData.modelName = 'chat'
await f.vm.selectModelType('vllm')
assert.equal(f.vm.formData.modelName, '')
assert.equal(f.vm.formData.provider, 'generic')
f.vm.formData.provider = 'weknoracloud'
f.vm.handleProviderChange('weknoracloud')
assert.deepEqual(Array.from(f.vm.catalogModelOptions, (o: any) => o.value), ['vlm'])
} finally { f.close() }
})

View File

@ -1009,6 +1009,11 @@ export default {
chunkLoadFailed: 'Failed to load chunks'
},
uploadConfirm: {
imageEmbeddingModel: "Image indexing embedding model",
imageIndexInherited: "Image indexing uses the embedding model assigned to this knowledge base. Change it in knowledge base settings.",
imageVectorIndex: "Image vector indexing for this import",
imageVectorIndexHint: "Generate image vectors for this upload or reparse without changing knowledge base defaults. Retrieval still follows the knowledge base image vector search setting.",
imageVectorRequiresIndex: "Enable vector indexing in knowledge base settings first.",
documentSummary: "Document summary",
documentSummaryDescription: "Choose whether to automatically summarize documents in this import.",
generateSummary: "Generate document summaries",
@ -5508,6 +5513,14 @@ export default {
documentsPlaceholder: 'Enter one candidate document per line',
documentsHint: 'Each non-empty line is sent as a separate document to the ReRank model',
imageFile: 'Image file',
similarityMode: 'Similarity',
similarityQuery: 'Query',
similarityQueryPlaceholder: 'Enter a query, e.g. a dog running on grass',
similarityCandidates: 'Candidates',
similarityCandidatesPlaceholder: 'One candidate text per line',
similarityHint: 'The query is embedded as a search query and the candidates as documents, as in retrieval; results are sorted by cosine similarity, highest first',
similarityImage: 'Candidate image (optional)',
similarityImageCandidate: 'Image',
audioFile: 'Audio file',
chooseFile: 'Choose file',
parameters: 'Request parameters',

View File

@ -1009,6 +1009,11 @@ export default {
chunkLoadFailed: 'チャンクの読み込みに失敗しました'
},
uploadConfirm: {
imageEmbeddingModel: "画像インデックス用 Embedding モデル",
imageIndexInherited: "画像インデックスにはナレッジベースの Embedding モデルを使用します。変更はナレッジベース設定で行ってください。",
imageVectorIndex: "今回の画像ベクトル索引",
imageVectorIndexHint: "今回のアップロードまたは再解析で画像ベクトルを生成します。ナレッジベースの既定設定は変更しません。検索への利用は画像ベクトル検索設定に従います。",
imageVectorRequiresIndex: "先にナレッジベース設定でベクトル索引を有効にしてください。",
documentSummary: "ドキュメントの要約",
documentSummaryDescription: "今回インポートするドキュメントの要約を自動生成するか選択します。",
generateSummary: "ドキュメントの要約を生成",
@ -5508,6 +5513,14 @@ export default {
documentsPlaceholder: '候補ドキュメントを1行に1件ずつ入力してください',
documentsHint: '空でない各行が個別のドキュメントとしてリランクモデルに送信されます',
imageFile: '画像ファイル',
similarityMode: '類似度',
similarityQuery: 'クエリ',
similarityQueryPlaceholder: 'クエリを入力(例:芝生を走る犬)',
similarityCandidates: '候補テキスト',
similarityCandidatesPlaceholder: '候補テキストを1行に1件ずつ入力してください',
similarityHint: 'クエリは検索クエリとして、候補はドキュメントとしてベクトル化されます(実際の検索と同じ)。結果はコサイン類似度の高い順に並びます',
similarityImage: '候補画像(任意)',
similarityImageCandidate: '画像',
audioFile: '音声ファイル',
chooseFile: 'ファイルを選択',
parameters: 'リクエストパラメータ',

View File

@ -2592,6 +2592,14 @@ export default {
documentsPlaceholder: '후보 문서를 한 줄에 하나씩 입력하세요',
documentsHint: '비어 있지 않은 각 줄은 ReRank 모델에 별도 문서로 전송됩니다',
imageFile: '이미지 파일',
similarityMode: '유사도',
similarityQuery: '쿼리',
similarityQueryPlaceholder: '쿼리를 입력하세요. 예: 잔디 위를 달리는 개',
similarityCandidates: '후보 텍스트',
similarityCandidatesPlaceholder: '한 줄에 하나의 후보 텍스트를 입력하세요',
similarityHint: '쿼리는 검색 쿼리로, 후보는 문서로 벡터화됩니다(실제 검색과 동일). 결과는 코사인 유사도가 높은 순으로 정렬됩니다',
similarityImage: '후보 이미지(선택)',
similarityImageCandidate: '이미지',
audioFile: '오디오 파일',
chooseFile: '파일 선택',
parameters: '요청 매개변수',
@ -7009,6 +7017,11 @@ export default {
}
},
uploadConfirm: {
imageEmbeddingModel: "이미지 인덱싱 Embedding 모델",
imageIndexInherited: "이미지 인덱싱에는 지식 베이스에 연결된 Embedding 모델을 사용합니다. 변경하려면 지식 베이스 설정으로 이동하세요.",
imageVectorIndex: "이번 가져오기의 이미지 벡터 인덱싱",
imageVectorIndexHint: "이번 업로드 또는 재분석에서 이미지 벡터를 생성하며 지식 베이스 기본 설정은 변경하지 않습니다. 검색 사용 여부는 지식 베이스의 이미지 벡터 검색 설정을 따릅니다.",
imageVectorRequiresIndex: "먼저 지식 베이스 설정에서 벡터 인덱싱을 활성화하세요.",
documentSummary: "문서 요약",
documentSummaryDescription: "이번에 가져오는 문서의 요약을 자동으로 생성할지 선택합니다.",
generateSummary: "문서 요약 생성",

View File

@ -2592,6 +2592,14 @@ export default {
documentsPlaceholder: 'Введите один документ на строку',
documentsHint: 'Каждая непустая строка отправляется в модель ReRank как отдельный документ',
imageFile: 'Файл изображения',
similarityMode: 'Сходство',
similarityQuery: 'Запрос',
similarityQueryPlaceholder: 'Введите запрос, например: собака бежит по траве',
similarityCandidates: 'Кандидаты',
similarityCandidatesPlaceholder: 'По одному тексту-кандидату в строке',
similarityHint: 'Запрос векторизуется как поисковый запрос, кандидаты — как документы, как при реальном поиске; результаты отсортированы по косинусному сходству по убыванию',
similarityImage: 'Изображение-кандидат (необязательно)',
similarityImageCandidate: 'Изображение',
audioFile: 'Аудиофайл',
chooseFile: 'Выбрать файл',
parameters: 'Параметры запроса',
@ -7009,6 +7017,11 @@ export default {
}
},
uploadConfirm: {
imageEmbeddingModel: "Модель эмбеддингов для индексации изображений",
imageIndexInherited: "Индексация изображений использует модель эмбеддингов базы знаний. Изменить её можно в настройках базы знаний.",
imageVectorIndex: "Индексация изображений для этого импорта",
imageVectorIndexHint: "Создавать векторы изображений при этой загрузке или повторном разборе без изменения настроек базы знаний. Поиск по ним по-прежнему зависит от настройки поиска изображений базы знаний.",
imageVectorRequiresIndex: "Сначала включите векторную индексацию в настройках базы знаний.",
documentSummary: "Сводка документа",
documentSummaryDescription: "Выберите, создавать ли сводки документов при этом импорте.",
generateSummary: "Создавать сводки документов",

View File

@ -2594,6 +2594,14 @@ export default {
documentsPlaceholder: '每行输入一个候选文档',
documentsHint: '每个非空行会作为一个独立文档发送给 ReRank 模型',
imageFile: '图片文件',
similarityMode: '相似度',
similarityQuery: '查询文本',
similarityQueryPlaceholder: '输入一条查询,例如:草地上奔跑的狗',
similarityCandidates: '候选文本',
similarityCandidatesPlaceholder: '每行输入一个候选文本',
similarityHint: '查询按检索查询方式向量化,候选按文档方式向量化,与线上检索一致;结果按余弦相似度从高到低排列',
similarityImage: '候选图片(可选)',
similarityImageCandidate: '图片',
audioFile: '音频文件',
chooseFile: '选择文件',
parameters: '请求参数',
@ -7011,6 +7019,11 @@ export default {
}
},
uploadConfirm: {
imageEmbeddingModel: "图像索引 Embedding 模型",
imageIndexInherited: "图像索引使用知识库绑定的 Embedding 模型;如需更换,请前往知识库设置。",
imageVectorIndex: "本次图片向量索引",
imageVectorIndexHint: "为本次上传或重新解析的图片生成向量,不改变知识库默认配置。是否参与检索仍由知识库的“图片向量检索”开关决定。",
imageVectorRequiresIndex: "请先在知识库设置中开启向量索引。",
documentSummary: "文档摘要",
documentSummaryDescription: "为本次导入的文档选择是否自动生成摘要。",
generateSummary: "生成文档摘要",

View File

@ -2594,6 +2594,14 @@ export default {
documentsPlaceholder: '每行輸入一個候選文件',
documentsHint: '每個非空行會作為一個獨立文件傳送給 ReRank 模型',
imageFile: '圖片檔案',
similarityMode: '相似度',
similarityQuery: '查詢文字',
similarityQueryPlaceholder: '輸入一條查詢,例如:草地上奔跑的狗',
similarityCandidates: '候選文字',
similarityCandidatesPlaceholder: '每行輸入一個候選文字',
similarityHint: '查詢按檢索查詢方式向量化,候選按文件方式向量化,與線上檢索一致;結果按餘弦相似度由高到低排列',
similarityImage: '候選圖片(可選)',
similarityImageCandidate: '圖片',
audioFile: '音訊檔案',
chooseFile: '選擇檔案',
parameters: '請求參數',
@ -7011,6 +7019,11 @@ export default {
}
},
uploadConfirm: {
imageEmbeddingModel: "圖像索引 Embedding 模型",
imageIndexInherited: "圖像索引使用知識庫綁定的 Embedding 模型;如需更換,請前往知識庫設定。",
imageVectorIndex: "本次圖片向量索引",
imageVectorIndexHint: "為本次上傳或重新解析的圖片產生向量,不改變知識庫預設設定。是否參與檢索仍由知識庫的「圖片向量檢索」開關決定。",
imageVectorRequiresIndex: "請先在知識庫設定中開啟向量索引。",
documentSummary: "文件摘要",
documentSummaryDescription: "為本次匯入的文件選擇是否自動生成摘要。",
generateSummary: "生成文件摘要",

View File

@ -74,6 +74,8 @@ export interface KnowledgeProcessOverrides {
// Per-upload override of the knowledge base's attribute-observed image
// pipeline switch.
image_attrs_enabled?: boolean
// Controls indexing for this document, without changing KB retrieval.
image_vector_enabled?: boolean
// Per-task attribute -> work table; merged per action key on top of the
// knowledge base's table.
image_actions?: ImageActionsConfig

View File

@ -421,6 +421,35 @@
/>
</div>
</div>
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
<div class="setting-info">
<label>{{ t('uploadConfirm.imageEmbeddingModel') }}</label>
<p class="desc">{{ t('uploadConfirm.imageIndexInherited') }}</p>
<p v-if="uiState.imageVectorEnabled && !embeddingTakesImages" class="warn">
{{ t('knowledgeEditor.advanced.multimodal.imageVectorModelUnsupported') }}
</p>
</div>
<div class="setting-control">
<ModelSelector
model-type="Embedding"
:selected-model-id="kbInfo?.embedding_model_id || ''"
:all-models="allModels"
disabled
:placeholder="t('uploadConfirm.notSet')"
/>
</div>
</div>
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
<div class="setting-info">
<label>{{ t('uploadConfirm.imageVectorIndex') }}</label>
<p class="desc">{{ t('uploadConfirm.imageVectorIndexHint') }}</p>
<p v-if="!kbInfo?.indexing_strategy?.vector_enabled" class="warn">{{ t('uploadConfirm.imageVectorRequiresIndex') }}</p>
</div>
<div class="setting-control">
<t-switch v-model="uiState.imageVectorEnabled" :disabled="!kbInfo?.indexing_strategy?.vector_enabled" size="medium" />
</div>
</div>
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
<div class="setting-info">
<label>{{ t('knowledgeEditor.advanced.multimodal.imageAttrsLabel') }}</label>
@ -705,6 +734,7 @@ interface UploadUIState {
chunkingConfig: ChunkingUIConfig
multimodalConfig: { enabled: boolean; vllmModelId: string; descriptionLanguage?: string; customInstructions?: string }
imageAttrsEnabled: boolean
imageVectorEnabled: boolean
imageActions: ImageActionsConfig
asrConfig: { enabled: boolean; modelId: string; language: string }
questionGenerationConfig: { enabled: boolean; questionCount: number; customInstructions?: string }
@ -775,6 +805,11 @@ const editorResources = useEditorResourcesStore()
const uiStore = useUIStore()
const allModels = ref<any[]>([])
const embeddingTakesImages = computed(() => {
const model = allModels.value.find(m => m.id === props.kbInfo?.embedding_model_id)
// A shared model may be absent from the model list; the backend resolves it.
return !model || !!model.capabilities?.input?.includes('image')
})
const localFiles = ref<File[]>([])
const localUrls = ref<string[]>([])
const availableTags = ref<Array<{ id: string; name: string }>>([])
@ -1197,6 +1232,7 @@ function createDefaultUIState(): UploadUIState {
},
multimodalConfig: { enabled: false, vllmModelId: '', descriptionLanguage: '', customInstructions: '' },
imageAttrsEnabled: false,
imageVectorEnabled: false,
imageActions: mergeImageActions(),
asrConfig: { enabled: false, modelId: '', language: '' },
questionGenerationConfig: { enabled: true, questionCount: 3, customInstructions: '' },
@ -1242,6 +1278,7 @@ function initFromKbInfo(kb: any) {
},
// 默认跟随知识库的图片属性观察设置;用户可对本次任务单独覆盖。
imageAttrsEnabled: !!kb.image_processing_config?.image_attrs_enabled,
imageVectorEnabled: !!kb.image_processing_config?.image_vector_enabled,
imageActions: mergeImageActions(kb.image_processing_config?.image_actions),
asrConfig: {
enabled: !!kb.asr_config?.enabled,
@ -1292,6 +1329,7 @@ function buildProcessOverrides(): KnowledgeProcessOverrides {
},
enable_multimodel: state.multimodalConfig.enabled,
image_attrs_enabled: state.imageAttrsEnabled,
image_vector_enabled: state.imageVectorEnabled,
image_actions: {
ocr: {
// The KB's own conditions (mergeImageActions keeps a custom list), so
@ -1357,6 +1395,7 @@ function applyOverridesToState(o?: KnowledgeProcessOverrides | null) {
if (o.parser_engine_rules) s.chunkingConfig.parserEngineRules = o.parser_engine_rules
if (o.enable_multimodel != null) s.multimodalConfig.enabled = o.enable_multimodel
if (o.image_attrs_enabled != null) s.imageAttrsEnabled = o.image_attrs_enabled
if (o.image_vector_enabled != null) s.imageVectorEnabled = o.image_vector_enabled
if (o.image_actions) s.imageActions = mergeImageActions(o.image_actions)
if (o.vlm_config) {
if (o.vlm_config.enabled != null) s.multimodalConfig.enabled = o.vlm_config.enabled

View File

@ -0,0 +1,90 @@
import assert from 'node:assert/strict'
import { readFileSync } from 'node:fs'
import { createRequire } from 'node:module'
import { fileURLToPath } from 'node:url'
import { runInNewContext } from 'node:vm'
import test from 'node:test'
import { compileScript, parse } from '@vue/compiler-sfc'
import ts from 'typescript'
import { createRenderer, h } from 'vue'
const require = createRequire(import.meta.url)
const filename = fileURLToPath(new URL('./UploadConfirmDialog.vue', import.meta.url))
const { descriptor } = parse(readFileSync(filename, 'utf8'), { filename })
const script = compileScript(descriptor, { id: 'upload-image-vector-test' }).content
.replace('__expose();', '')
.replace('return __returned__', '__expose(__returned__); return __returned__')
const compiled = ts.transpileModule(script, {
compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022 },
}).outputText
function fixture(kbInfo: any) {
const exports: any = {}
runInNewContext(compiled, {
exports, console,
require(name: string) {
if (name === 'vue') return require('vue')
if (name === 'vue-i18n') return { useI18n: () => ({ t: (key: string) => key, te: () => false }) }
if (name === '@/stores/chatResources') return { useChatResourcesStore: () => ({}) }
if (name === '@/stores/editorResources') return { useEditorResourcesStore: () => ({}) }
if (name === '@/stores/ui') return { useUIStore: () => ({}) }
if (name === '@/api/knowledge-base') return { mergeImageActions: () => ({ ocr: { on: [], on_unobserved: true } }) }
return { default: {} }
},
})
exports.default.render = () => null
const renderer = createRenderer<any, any>({
createElement: () => ({}), createText: () => ({}), createComment: () => ({}),
insert() {}, remove() {}, setElementText() {}, setText() {}, patchProp() {},
parentNode: () => null, nextSibling: () => null,
})
let vm: any
const app = renderer.createApp({ setup: () => () => h(exports.default, { visible: false, kbInfo, ref: (value: any) => { vm = value } }) })
app.mount({})
vm.initFromKbInfo(kbInfo)
return { vm, dispose: () => app.unmount() }
}
for (const enabled of [false, true]) {
test(`upload inherits ${enabled} and submits the explicit opposite without changing the KB`, () => {
const kb = { embedding_model_id: 'shared-embedding', image_processing_config: { image_vector_enabled: enabled } }
const f = fixture(kb)
try {
assert.equal(f.vm.uiState.imageVectorEnabled, enabled)
f.vm.uiState.imageVectorEnabled = !enabled
const payload = JSON.parse(JSON.stringify(f.vm.buildProcessOverrides()))
assert.equal(payload.image_vector_enabled, !enabled)
assert.equal(kb.image_processing_config.image_vector_enabled, enabled)
assert.equal('embedding_model_id' in payload, false)
} finally { f.dispose() }
})
test(`reparse restores an explicit ${enabled} over the KB default`, () => {
const f = fixture({ image_processing_config: { image_vector_enabled: !enabled } })
try {
f.vm.applyOverridesToState({ image_vector_enabled: enabled })
assert.equal(f.vm.uiState.imageVectorEnabled, enabled)
assert.equal(f.vm.buildProcessOverrides().image_vector_enabled, enabled)
} finally { f.dispose() }
})
}
test('old documents inherit the KB value and old KBs default off', () => {
const f = fixture({ image_processing_config: { image_vector_enabled: true } })
try {
f.vm.applyOverridesToState({ summary_enabled: false })
assert.equal(f.vm.uiState.imageVectorEnabled, true)
f.vm.initFromKbInfo({})
assert.equal(f.vm.uiState.imageVectorEnabled, false)
} finally { f.dispose() }
})
test('image capability warning uses the KB embedding model and tolerates shared models absent from the list', () => {
const f = fixture({ embedding_model_id: 'embedding' })
try {
assert.equal(f.vm.embeddingTakesImages, true)
f.vm.allModels = [{ id: 'embedding', capabilities: { input: ['text'] } }]
assert.equal(f.vm.embeddingTakesImages, false)
f.vm.allModels = [{ id: 'embedding', capabilities: { input: ['text', 'image'] } }]
assert.equal(f.vm.embeddingTakesImages, true)
} finally { f.dispose() }
})

View File

@ -9,6 +9,7 @@ import (
"io"
"net/http"
"net/http/httptest"
"net/http/httptrace"
"strings"
"sync"
"sync/atomic"
@ -31,9 +32,14 @@ import (
func newTestClient(t *testing.T, url string) *osapi.Client {
t.Helper()
// httptest.Server.Close touches http.DefaultTransport. Give each client
// its own pool so parallel server cleanup cannot interrupt its requests.
transport := http.DefaultTransport.(*http.Transport).Clone()
t.Cleanup(transport.CloseIdleConnections)
c, err := osapi.NewClient(osapi.Config{
Client: opensearch.Config{
Addresses: []string{url},
Transport: transport,
},
})
if err != nil {
@ -42,6 +48,32 @@ func newTestClient(t *testing.T, url string) *osapi.Client {
return c
}
// Closing any httptest server also closes idle connections on the global
// transport. A sibling test must not be able to disrupt this client's pool.
func TestNewTestClient_IsolatesTransportFromOtherServers(t *testing.T) {
t.Parallel()
ts := httptest.NewServer(versionHandler("opensearch", "3.3.2"))
defer ts.Close()
client := newTestClient(t, ts.URL)
if err := probeVersion(context.Background(), client); err != nil {
t.Fatal(err)
}
sibling := httptest.NewServer(http.NotFoundHandler())
sibling.Close()
var reused bool
ctx := httptrace.WithClientTrace(context.Background(), &httptrace.ClientTrace{
GotConn: func(info httptrace.GotConnInfo) { reused = info.Reused },
})
if err := probeVersion(ctx, client); err != nil {
t.Fatal(err)
}
if !reused {
t.Fatal("closing another test server discarded this client's idle connection")
}
}
// newTestRepo builds a Repository directly (bypassing NewRepository's probe
// calls) for unit tests that exercise post-construction behavior. The
// caller supplies the http handler that simulates the cluster.

View File

@ -691,8 +691,8 @@ func (s *ImageMultimodalService) indexChunks(
// indexImageVector embeds the image itself with a multimodal embedding model
// and stores the vector under an image_vector chunk, so a query can find the
// image by what it shows even where the caption left that out. It returns a
// short status for the trace, "" when the knowledge base has not opted in
// (ImageProcessingConfig.ImageVectorEnabled) or the model takes no images.
// short status for the trace, "" when image indexing is disabled for this
// document (falling back to the KB setting) or the model takes no images.
//
// The caption and OCR chunks are already indexed, so a failure here only
// loses the extra recall and never fails the task.
@ -700,9 +700,13 @@ func (s *ImageMultimodalService) indexImageVector(
ctx context.Context, kb *types.KnowledgeBase, payload types.ImageMultimodalPayload, img []byte,
chunks []*types.Chunk, model embedding.Embedder, engine *retriever.CompositeRetrieveEngine,
) string {
// Off unless the knowledge base asked for it: a model that takes images
// is often chosen for text alone, and this costs a call per image.
if !kb.IsImageVectorEnabled() {
// The task's explicit choice wins over the KB default, including false.
// A vector index is still required; old tasks inherit the KB setting.
enabled := kb.IsImageVectorEnabled()
if payload.ImageVectorEnabled != nil {
enabled = kb.IsVectorEnabled() && *payload.ImageVectorEnabled
}
if !enabled {
return ""
}
imageModel, ok := embedding.AsImageEmbedder(model)

View File

@ -3,6 +3,7 @@ package service
import (
"bytes"
"context"
"encoding/json"
"image"
"image/png"
"testing"
@ -239,3 +240,65 @@ func TestIndexImageVectorNeedsTheKnowledgeBaseToOptIn(t *testing.T) {
})
}
}
// Follow the saved per-document choice through metadata, config resolution,
// queue serialization and the worker, without changing KB retrieval defaults.
func TestImageVectorUploadOverrideReachesWorker(t *testing.T) {
for _, tc := range []struct {
name string
kbEnabled, vectorEnabled bool
override *bool
want bool
}{
{"enable one upload", false, true, processConfigBoolPtr(true), true},
{"disable one upload", true, true, processConfigBoolPtr(false), false},
{"inherit enabled", true, true, nil, true},
{"inherit disabled", false, true, nil, false},
{"no vector index", true, false, processConfigBoolPtr(true), false},
} {
t.Run(tc.name, func(t *testing.T) {
knowledge, kb, images, chunks := slotReleaseFixture(1)
knowledge.ID = "image-vector-" + tc.name
useLiteCounter(t, knowledge.ID)
kb.IndexingStrategy.VectorEnabled = tc.vectorEnabled
kb.ImageProcessingConfig.ImageVectorEnabled = tc.kbEnabled
require.NoError(t, knowledge.SetProcessOverrides(&types.KnowledgeProcessOverrides{
ImageVectorEnabled: tc.override,
}))
overrides, err := knowledge.ProcessOverrides()
require.NoError(t, err)
eff := ResolveProcessConfig(kb, overrides)
wantSetting := tc.kbEnabled
if tc.override != nil {
wantSetting = *tc.override
}
require.Equal(t, wantSetting, eff.ImageVectorEnabled)
enqueuer := &metadataUpdateTaskEnqueuer{}
enqueueService := &knowledgeService{task: enqueuer}
require.NoError(t, enqueueService.enqueueImageMultimodalTasks(
context.Background(), knowledge, kb, images, chunks, nil,
))
require.Len(t, enqueuer.tasks, 1)
var payload types.ImageMultimodalPayload
require.NoError(t, json.Unmarshal(enqueuer.tasks[0].Payload(), &payload))
require.NotNil(t, payload.ImageVectorEnabled, "explicit false must survive JSON serialization")
require.Equal(t, wantSetting, *payload.ImageVectorEnabled)
// The queued choice survives a later change to the KB default.
kb.ImageProcessingConfig.ImageVectorEnabled = !tc.kbEnabled
svc, repo, recorder, engine := imageVectorFixture(t)
model := &imageModel{dims: 3}
status := svc.indexImageVector(
context.Background(), kb, payload, testPNG(t), multimodalChunks(), model, engine,
)
if tc.want {
require.Equal(t, "indexed", status)
require.Len(t, model.images, 1)
require.Len(t, recorder.rows, 1)
} else {
require.Empty(t, status)
require.Empty(t, model.images)
require.Empty(t, repo.chunks)
}
})
}
}

View File

@ -4443,20 +4443,21 @@ func (s *knowledgeService) enqueueImageMultimodalTasks(
}
payload := types.ImageMultimodalPayload{
TenantID: knowledge.TenantID,
KnowledgeID: knowledge.ID,
KnowledgeBaseID: kb.ID,
ChunkID: chunkID,
ImageURL: img.ServingURL,
EnableOCR: true,
EnableCaption: true,
ImageAttrsEnabled: eff.ImageAttrsEnabled,
ImageActions: eff.ImageActions,
Language: lang,
ImageSourceType: metadata["image_source_type"],
Attempt: attempt,
ImageIndex: idx,
SourceLocators: img.SourceLocators,
TenantID: knowledge.TenantID,
KnowledgeID: knowledge.ID,
KnowledgeBaseID: kb.ID,
ChunkID: chunkID,
ImageURL: img.ServingURL,
EnableOCR: true,
EnableCaption: true,
ImageAttrsEnabled: eff.ImageAttrsEnabled,
ImageVectorEnabled: &eff.ImageVectorEnabled,
ImageActions: eff.ImageActions,
Language: lang,
ImageSourceType: metadata["image_source_type"],
Attempt: attempt,
ImageIndex: idx,
SourceLocators: img.SourceLocators,
}
langfuse.InjectTracing(ctx, &payload)

View File

@ -49,6 +49,7 @@ func ResolveProcessConfig(kb *types.KnowledgeBase, overrides *types.KnowledgePro
GraphEnabled: kb.IsGraphEnabled(),
ExtractConfig: derefExtractConfig(kb.ExtractConfig),
ImageAttrsEnabled: imageCfg.ImageAttrsEnabled,
ImageVectorEnabled: imageCfg.ImageVectorEnabled,
ImageActions: types.ResolveImageActions(imageCfg.ImageActions),
}
if overrides == nil {
@ -93,6 +94,9 @@ func ResolveProcessConfig(kb *types.KnowledgeBase, overrides *types.KnowledgePro
if overrides.ImageAttrsEnabled != nil {
eff.ImageAttrsEnabled = *overrides.ImageAttrsEnabled
}
if overrides.ImageVectorEnabled != nil {
eff.ImageVectorEnabled = *overrides.ImageVectorEnabled
}
if overrides.ImageActions != nil {
base := eff.ImageActions
// Same rule as types.MergeImageActions: the OCR clause is a unit keyed

View File

@ -135,6 +135,8 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
return nil, err
}
recordImageRecallDecision(ctx, "candidate_cut", nil, summarizeImageCandidateCut(chunks, params.MatchCount))
// Truncate to the primary-match cap. MatchCount is guaranteed positive by
// the normalization at the top of this function; the slice bound below
// depends on that.

View File

@ -141,12 +141,19 @@ func (s *knowledgeBaseService) retrieveFromStores(
// (refillPastDroppedImages).
func retrieveGroup(ctx context.Context, g *storeGroup) ([]*types.RetrieveResult, error) {
params := paramsWithTopK(g)
res, err := g.Engine.Retrieve(ctx, params)
res, err := retrieveObservedImagePool(ctx, g, params, "initial")
res, err = retainPartialResults(ctx, g, res, err)
if err != nil {
return nil, err
}
refillPastDroppedImages(ctx, g, params, res)
for _, rr := range res {
if rr != nil {
out := summarizeImagePool(rr, g)
out["retriever"], out["kb_ids"] = rr.RetrieverType, g.KBIDs
recordImageRecallDecision(ctx, "filter", nil, out)
}
}
filterImageHits(res, g)
return res, nil
}

View File

@ -15,10 +15,9 @@ import (
// passage saying the same thing (the modality gap). So they get their own
// threshold, and vector retrieval widens its pool to leave them room.
const (
// imageVectorThreshold is the most an image hit is asked to score. It is
// empirical, a ceiling below the text defaults (0.15–0.2); a lower text
// threshold, or none, still wins.
imageVectorThreshold = 0.1
// imageVectorThreshold caps the image recall threshold at an empirical
// value; a lower text threshold, or none, still wins.
imageVectorThreshold = 0.2
// imageRecallWidening is how much larger the vector pool is when image
// hits compete in it: one half again, so text keeps close to the pool
// it had before images arrived.
@ -135,27 +134,38 @@ func filterImageHits(results []*types.RetrieveResult, g *storeGroup) {
}
func keepHit(hit *types.IndexWithScore, retriever types.RetrieverType, g *storeGroup) bool {
if hit == nil {
return false
}
if retriever == types.VectorRetrieverType {
// Recorded here, before normalization and fusion rewrite Score.
if hit != nil && retriever == types.VectorRetrieverType {
// Preserve the raw score before normalization and fusion.
hit.VectorScore = hit.Score
}
return imageHitDropReason(hit, retriever, g) == ""
}
// imageHitDropReason is shared by filtering and diagnostics so the trace uses
// the actual admission rules. It does not mutate scores or candidate order.
func imageHitDropReason(hit *types.IndexWithScore, retriever types.RetrieverType, g *storeGroup) string {
if hit == nil {
return "nil_hit"
}
image := hit.SourceType == types.ImageSourceType
switch retriever {
case types.KeywordsRetrieverType:
return !image
if image {
return "duplicate_image_keyword"
}
case types.VectorRetrieverType:
if image {
return !staleImage(hit, g) && hit.Score >= imageThreshold(g.VectorThreshold)
if staleImage(hit, g) {
return "image_recall_disabled"
}
if !(hit.Score >= imageThreshold(g.VectorThreshold)) {
return "below_image_threshold"
}
} else if g.imageRecall() && !(hit.Score >= g.VectorThreshold) {
return "below_text_threshold"
}
if !g.imageRecall() {
return true
}
return hit.Score >= g.VectorThreshold
}
return true
return ""
}
// staleImage reports whether a vector hit is an image row of a KB that does
@ -259,7 +269,7 @@ func refillPool(ctx context.Context, g *storeGroup, retriever types.RetrieverTyp
// refillSearch runs one refill search and reports whether its answer is
// complete; refillPool keeps the pool it has otherwise.
func refillSearch(ctx context.Context, g *storeGroup, p types.RetrieveParams) (*types.RetrieveResult, bool) {
more, err := g.Engine.Retrieve(ctx, []types.RetrieveParams{p})
more, err := retrieveObservedImagePool(ctx, g, []types.RetrieveParams{p}, "refill")
if err == nil && len(more) == 1 && more[0] != nil && more[0].Error != nil {
err = more[0].Error
}

View File

@ -0,0 +1,140 @@
package service
import (
"context"
"fmt"
"slices"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
)
const imageRecallSampleLimit = 10
// Every engine call (including refills) is observed before application filters
// mutate its rows. This adds no database/model requests. Missing rows cannot be
// attributed to the threshold vs the engine TopK from returned rows alone.
func retrieveObservedImagePool(ctx context.Context, g *storeGroup, params []types.RetrieveParams, phase string) (
[]*types.RetrieveResult, error,
) {
res, err := g.Engine.Retrieve(ctx, params)
for _, rr := range res {
if rr == nil {
continue
}
out := summarizeImagePool(rr, g)
requests := make([]map[string]interface{}, 0, len(params))
for _, p := range params {
if p.RetrieverType == rr.RetrieverType {
requests = append(requests, map[string]interface{}{
"top_k": p.TopK, "threshold": p.Threshold, "knowledge_type": p.KnowledgeType,
})
}
}
imageKBs := make([]string, 0, len(g.ImageKBIDs))
for id := range g.ImageKBIDs {
imageKBs = append(imageKBs, id)
}
slices.Sort(imageKBs)
if len(requests) == 1 {
out["pool_limit_reached"] = len(rr.Results) >= requests[0]["top_k"].(int)
}
out["phase"], out["requests"] = phase, requests
out["engine"], out["retriever"] = rr.RetrieverEngineType, rr.RetrieverType
out["kb_ids"], out["image_kb_ids"] = g.KBIDs, imageKBs
out["result_complete"] = err == nil && rr.Error == nil
out["unreturned_images"] = "unknown: engine threshold, top_k, or no eligible indexed images"
recordImageRecallDecision(ctx, "engine_pool", nil, out)
}
return res, err
}
func summarizeImagePool(rr *types.RetrieveResult, g *storeGroup) map[string]interface{} {
reasons := map[string]int{}
images := make([]map[string]interface{}, 0)
imageCount, keptImages := 0, 0
var minScore, maxScore float64
hasScore := false
for i, hit := range rr.Results {
reason := imageHitDropReason(hit, rr.RetrieverType, g)
if reason == "" {
reason = "kept"
}
reasons[reason]++
if hit == nil {
continue
}
if !hasScore || hit.Score < minScore {
minScore = hit.Score
}
if !hasScore || hit.Score > maxScore {
maxScore = hit.Score
}
hasScore = true
if hit.SourceType != types.ImageSourceType {
continue
}
imageCount++
if reason == "kept" {
keptImages++
}
if len(images) < imageRecallSampleLimit {
images = append(images, map[string]interface{}{
"pool_rank": i + 1, "chunk_id": hit.ChunkID, "knowledge_id": hit.KnowledgeID,
"raw_score": fmt.Sprintf("%.4f", hit.Score), "filter_decision": reason,
})
}
}
out := map[string]interface{}{
"returned_count": len(rr.Results), "returned_image_count": imageCount,
"image_count_passing_filter": keptImages, "filter_decision_counts": reasons,
"has_returned_scores": hasScore,
"returned_min_score": fmt.Sprint(minScore), "returned_max_score": fmt.Sprint(maxScore),
"image_recall_enabled": g.imageRecall(),
"image_samples": images, "image_samples_truncated": imageCount - len(images),
}
if g.imageRecall() {
out["text_threshold"] = g.VectorThreshold
out["image_threshold"] = imageThreshold(g.VectorThreshold)
}
return out
}
// The fusion score cut is separate from the raw vector threshold. Retain image
// samples even when none appears in the generic top-hit preview.
func summarizeImageCandidateCut(hits []*types.IndexWithScore, limit int) map[string]interface{} {
images := make([]map[string]interface{}, 0)
before, after := 0, 0
for i, hit := range hits {
if hit == nil || hit.SourceType != types.ImageSourceType {
continue
}
before++
decision := "match_count_cut"
if i < limit {
after++
decision = "kept"
}
if len(images) < imageRecallSampleLimit {
images = append(images, map[string]interface{}{
"rank": i + 1, "chunk_id": hit.ChunkID,
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
"fusion_score": fmt.Sprintf("%.4f", hit.Score), "decision": decision,
})
}
}
return map[string]interface{}{
"input_count": len(hits), "match_count": limit, "output_count": min(len(hits), limit),
"input_image_count": before, "output_image_count": after, "dropped_image_count": before - after,
"image_samples": images, "image_samples_truncated": before - len(images),
}
}
func recordImageRecallDecision(ctx context.Context, stage string, input, output map[string]interface{}) {
logger.GetLogger(ctx).WithFields(logger.Fields(output)).Info("image recall " + stage)
_, span := langfuse.GetManager().StartSpan(ctx, langfuse.SpanOptions{
Name: "retrieval.image_" + stage, Input: input,
})
span.Finish(output, nil, nil)
}

View File

@ -0,0 +1,65 @@
package service
import (
"encoding/json"
"fmt"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestImagePoolDiagnosticsUseActualFilterRulesWithoutMutatingHits(t *testing.T) {
g := &storeGroup{ImageKBIDs: recalls("kb"), VectorThreshold: 0.5}
rr := &types.RetrieveResult{RetrieverType: types.VectorRetrieverType, Results: []*types.IndexWithScore{
hit("text-low", 0.4, types.ChunkSourceType),
hit("image-low", 0.199, types.ImageSourceType),
hit("image-boundary", 0.2, types.ImageSourceType),
kbHit("disabled", "image-disabled", 0.9, types.ImageSourceType),
}}
out := summarizeImagePool(rr, g)
require.Equal(t, 3, out["returned_image_count"])
require.Equal(t, 1, out["image_count_passing_filter"])
require.Equal(t, map[string]int{
"below_text_threshold": 1, "below_image_threshold": 1, "kept": 1, "image_recall_disabled": 1,
}, out["filter_decision_counts"])
require.Len(t, rr.Results, 4)
require.Zero(t, rr.Results[2].VectorScore, "observability must not alter raw scores")
filterImageHits([]*types.RetrieveResult{rr}, g)
require.Equal(t, []string{"image-boundary"}, ids(rr.Results))
require.Equal(t, 0.2, rr.Results[0].VectorScore)
}
func TestImagePoolDiagnosticsCoverKeywordExclusionAndBoundSamples(t *testing.T) {
g := &storeGroup{ImageKBIDs: recalls("kb")}
rr := &types.RetrieveResult{RetrieverType: types.KeywordsRetrieverType}
for i := 0; i < 13; i++ {
h := hit(fmt.Sprint(i), 10, types.ImageSourceType)
h.Content = "private image description"
rr.Results = append(rr.Results, h)
}
out := summarizeImagePool(rr, g)
require.Equal(t, 13, out["returned_image_count"])
require.Equal(t, 0, out["image_count_passing_filter"])
require.Equal(t, 3, out["image_samples_truncated"])
require.Equal(t, map[string]int{"duplicate_image_keyword": 13}, out["filter_decision_counts"])
b, err := json.Marshal(out)
require.NoError(t, err)
require.NotContains(t, string(b), "private image description")
}
func TestImageCandidateCutDiagnosticsDistinguishFusionCutFromVectorThreshold(t *testing.T) {
image := hit("image", 0.3, types.ImageSourceType)
image.VectorScore = 0.2688
hits := []*types.IndexWithScore{hit("text", 0.9, types.ChunkSourceType), image}
out := summarizeImageCandidateCut(hits, 1)
require.Equal(t, 1, out["input_image_count"])
require.Equal(t, 0, out["output_image_count"])
rows := out["image_samples"].([]map[string]interface{})
require.Equal(t, "match_count_cut", rows[0]["decision"])
require.Equal(t, "0.2688", rows[0]["vector_score"])
require.Equal(t, "0.3000", rows[0]["fusion_score"])
require.Equal(t, 2, rows[0]["rank"])
require.Len(t, hits, 2)
require.Equal(t, 1, summarizeImageCandidateCut(hits, 2)["output_image_count"])
}

View File

@ -81,6 +81,8 @@ func TestFilterImageHitsHoldsEachKindToItsOwnThreshold(t *testing.T) {
vector := &types.RetrieveResult{RetrieverType: types.VectorRetrieverType, Results: []*types.IndexWithScore{
hit("text-strong", 0.6, types.ChunkSourceType),
hit("image-strong", 0.25, types.ImageSourceType),
hit("image-boundary", 0.2, types.ImageSourceType),
hit("image-below-boundary", 0.199, types.ImageSourceType),
hit("text-weak", 0.2, types.ChunkSourceType), // only reached because the query threshold was lowered
hit("image-weak", 0.05, types.ImageSourceType),
}}
@ -89,7 +91,7 @@ func TestFilterImageHitsHoldsEachKindToItsOwnThreshold(t *testing.T) {
hit("image-kw", 0.9, types.ImageSourceType),
}}
filterImageHits([]*types.RetrieveResult{vector, keyword}, g)
assert.Equal(t, []string{"text-strong", "image-strong"}, ids(vector.Results))
assert.Equal(t, []string{"text-strong", "image-strong", "image-boundary"}, ids(vector.Results))
assert.Equal(t, []string{"text-kw"}, ids(keyword.Results),
"an image row's Content is the caption, already indexed as text")
}

View File

@ -194,9 +194,8 @@ type ModelResponse struct {
// caller is a system administrator. See MCPServiceResponse.Credentials.
Credentials map[string]CredentialFieldMetadata `json:"credentials,omitempty"`
// Capabilities is the catalog view of the model. Chat models get the
// protocol, thinking levels and context window; embedding models get the
// input modalities, which decide whether images can be embedded in the
// same space as text.
// protocol, thinking levels and context window; embedding and rerank
// models get input modalities, including support for image input.
Capabilities *modelruntime.Capabilities `json:"capabilities,omitempty"`
}
@ -311,7 +310,7 @@ func NewModelResponse(ctx context.Context, m *types.Model) *ModelResponse {
}
var caps *modelruntime.Capabilities
switch m.Type {
case types.ModelTypeKnowledgeQA, types.ModelTypeVLLM, types.ModelTypeEmbedding:
case types.ModelTypeKnowledgeQA, types.ModelTypeVLLM, types.ModelTypeEmbedding, types.ModelTypeRerank:
if resolved, err := modelruntime.Resolve(modelruntime.Ref{
Provider: m.Parameters.Provider, Model: m.Name, BaseURL: m.Parameters.BaseURL,
ModelType: m.Type, Extra: m.Parameters.ExtraConfig, Override: m.Parameters.Spec,

View File

@ -16,6 +16,9 @@ import (
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/api"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/models/embedding"
"github.com/Tencent/WeKnora/internal/models/imageprep"
"github.com/Tencent/WeKnora/internal/models/rerank"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
secutils "github.com/Tencent/WeKnora/internal/utils"
@ -293,6 +296,9 @@ type ModelDebugOptions struct {
Thinking *bool `json:"thinking,omitempty"`
// ReasoningEffort is the graded level; when set it overrides Thinking.
ReasoningEffort string `json:"reasoning_effort,omitempty"`
// Similarity makes an embedding test score the documents (and an
// optional image) against the input as a search query.
Similarity bool `json:"similarity,omitempty"`
}
func parseModelDebugOptions(raw string) (ModelDebugOptions, error) {
@ -433,6 +439,18 @@ func consumeModelDebugChatStream(stream <-chan types.StreamResponse) (*modelDebu
return result, nil
}
// Model metadata and the legacy vision switch both allow image chat tests.
func modelDebugChatAcceptsImages(model *types.Model) bool {
if model.Parameters.SupportsVision {
return true
}
if model.Source == types.ModelSourceLocal {
return false
}
resolved, err := chat.Resolve(chat.ConfigFromModel(model, "", ""))
return err == nil && resolved.Spec.AcceptsImages()
}
// DebugModel executes a saved model through the same service constructors used
// by production calls and returns the complete normalized response. Credentials
// stay server-side; the request preview contains only non-secret fields.
@ -516,13 +534,26 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
fileSize = int64(len(fileBytes))
}
var image api.EmbedImage
if fileName != "" && model.Type != types.ModelTypeASR {
image, err = imageprep.Prepare(fileBytes, imageprep.Limits{})
if err != nil {
_ = c.Error(errors.NewBadRequestError(err.Error()))
return
}
}
requestPreview := modelDebugRequestPreview(model, input, documents, opts, fileName, fileSize)
observations := gin.H{}
switch model.Type {
case types.ModelTypeKnowledgeQA:
if strings.TrimSpace(input) == "" {
c.Error(errors.NewBadRequestError("query cannot be empty"))
if len(fileBytes) > 0 && !modelDebugChatAcceptsImages(model) {
_ = c.Error(errors.NewBadRequestError("chat model does not accept images"))
return
}
if strings.TrimSpace(input) == "" && len(fileBytes) == 0 {
_ = c.Error(errors.NewBadRequestError("query cannot be empty"))
return
}
instance, callErr := h.service.GetChatModel(ctx, id)
@ -534,7 +565,17 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
if strings.TrimSpace(opts.SystemPrompt) != "" {
messages = append(messages, chat.Message{Role: "system", Content: opts.SystemPrompt})
}
messages = append(messages, chat.Message{Role: "user", Content: input})
message := chat.Message{Role: "user", Content: input}
if len(fileBytes) > 0 {
message.Content = ""
if strings.TrimSpace(input) != "" {
message.MultiContent = append(message.MultiContent, chat.MessageContentPart{Type: "text", Text: input})
}
message.MultiContent = append(message.MultiContent, chat.MessageContentPart{
Type: "image_url", ImageURL: &chat.ImageURL{URL: image.DataURI()},
})
}
messages = append(messages, message)
chatOpts := &chat.ChatOptions{}
if opts.Temperature != nil {
chatOpts.Temperature = *opts.Temperature
@ -575,8 +616,12 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
}
writeModelDebugResult(c, started, requestPreview, resp, callErr, observations)
case types.ModelTypeEmbedding:
if strings.TrimSpace(input) == "" {
c.Error(errors.NewBadRequestError("input cannot be empty"))
if opts.Similarity {
h.debugEmbeddingSimilarity(c, id, input, documents, fileBytes, started, requestPreview, observations)
return
}
if strings.TrimSpace(input) == "" && len(fileBytes) == 0 {
_ = c.Error(errors.NewBadRequestError("input cannot be empty"))
return
}
instance, callErr := h.service.GetEmbeddingModel(ctx, id)
@ -584,12 +629,35 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
writeModelDebugResult(c, started, requestPreview, nil, callErr, observations)
return
}
if len(fileBytes) > 0 {
if strings.TrimSpace(input) != "" {
_ = c.Error(errors.NewBadRequestError("test either text or an image for embedding"))
return
}
imageModel, ok := embedding.AsImageEmbedder(instance)
if !ok {
_ = c.Error(errors.NewBadRequestError(embedding.ErrImagesUnsupported.Error()))
return
}
prepared, prepErr := imageprep.Prepare(fileBytes, imageModel.ImageLimits())
if prepErr != nil {
_ = c.Error(errors.NewBadRequestError(prepErr.Error()))
return
}
vectors, callErr := imageModel.BatchEmbedImages(ctx, []embedding.Image{prepared})
observations["result_count"] = len(vectors)
if len(vectors) > 0 {
observations["dimension"] = len(vectors[0])
}
writeModelDebugResult(c, started, requestPreview, vectors, callErr, observations)
return
}
vector, callErr := instance.Embed(ctx, input)
observations["dimension"] = len(vector)
writeModelDebugResult(c, started, requestPreview, vector, callErr, observations)
case types.ModelTypeRerank:
if strings.TrimSpace(input) == "" || len(documents) == 0 {
c.Error(errors.NewBadRequestError("query and documents cannot be empty"))
if strings.TrimSpace(input) == "" || (len(documents) == 0 && len(fileBytes) == 0) {
_ = c.Error(errors.NewBadRequestError("query and documents cannot be empty"))
return
}
instance, callErr := h.service.GetRerankModel(ctx, id)
@ -597,6 +665,26 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
writeModelDebugResult(c, started, requestPreview, nil, callErr, observations)
return
}
if len(fileBytes) > 0 {
if len(documents) > 0 {
_ = c.Error(errors.NewBadRequestError("test either text documents or an image for reranking"))
return
}
imageModel, ok := rerank.AsImageReranker(instance)
if !ok {
_ = c.Error(errors.NewBadRequestError(rerank.ErrImagesUnsupported.Error()))
return
}
prepared, prepErr := imageprep.Prepare(fileBytes, imageModel.ImageLimits())
if prepErr != nil {
_ = c.Error(errors.NewBadRequestError(prepErr.Error()))
return
}
results, callErr := imageModel.RerankImages(ctx, input, []rerank.Image{prepared})
observations["result_count"] = len(results)
writeModelDebugResult(c, started, requestPreview, results, callErr, observations)
return
}
results, callErr := instance.Rerank(ctx, input, documents)
observations["result_count"] = len(results)
writeModelDebugResult(c, started, requestPreview, results, callErr, observations)

View File

@ -104,6 +104,9 @@ func providerDTO(v *modelruntime.Provider, modelType types.ModelType, includeMod
modelTypes := make([]string, 0, len(v.ModelTypes))
for _, mt := range v.ModelTypes {
modelTypes = append(modelTypes, modelTypeToFrontend(mt))
// Use the same fallback as runtime resolution. Several ASR vendors
// share their chat base URL without declaring a separate ASR entry.
defaultURLs[modelTypeToFrontend(mt)] = v.GetDefaultURL(mt)
}
dto := ModelProviderDTO{
Value: v.ID,

View File

@ -8,10 +8,74 @@ import (
"strings"
"testing"
"github.com/Tencent/WeKnora/internal/models/api"
"github.com/Tencent/WeKnora/internal/models/providers"
modelruntime "github.com/Tencent/WeKnora/internal/models/runtime"
"github.com/Tencent/WeKnora/internal/types"
"github.com/gin-gonic/gin"
)
func TestAliyunMultimodalEmbeddingsAreSelectable(t *testing.T) {
vendor, ok := modelruntime.Get(providers.AliyunID)
if !ok {
t.Fatal("Aliyun provider is not registered")
}
entries := providerDTO(vendor, types.ModelTypeEmbedding, true).Models
for id, dimension := range map[string]int{
"qwen3-vl-embedding": 2560,
"qwen2.5-vl-embedding": 1024,
"tongyi-embedding-vision-flash": 768,
"multimodal-embedding-v1": 1024,
} {
t.Run(id, func(t *testing.T) {
var found bool
for _, entry := range entries {
if entry.ID == id {
found = true
if entry.Dimension != dimension || entry.Type != "embedding" {
t.Errorf("unexpected picker metadata: %+v", entry)
}
}
}
if !found {
t.Fatalf("%s is missing from the model picker", id)
}
resolved, err := modelruntime.Resolve(modelruntime.Ref{
Provider: providers.AliyunID, Model: id, ModelType: types.ModelTypeEmbedding,
})
if err != nil {
t.Fatal(err)
}
if resolved.EmbeddingAPI != api.EmbeddingDashScope {
t.Errorf("model must keep the native DashScope protocol, got %s", resolved.EmbeddingAPI)
}
})
}
}
func TestProviderDTOIncludesEffectiveDefaultURLs(t *testing.T) {
for _, vendor := range modelruntime.List() {
t.Run(vendor.ID, func(t *testing.T) {
got := providerDTO(vendor, types.ModelTypeASR, true)
for _, kind := range vendor.ModelTypes {
want := vendor.GetDefaultURL(kind)
if got.DefaultURLs[modelTypeToFrontend(kind)] != want {
t.Errorf("%s default URL = %q, want runtime default %q",
kind, got.DefaultURLs[modelTypeToFrontend(kind)], want)
}
}
if vendor.ID == providers.ZhipuID {
if got.DefaultURLs["asr"] != providers.ZhipuBaseURL {
t.Errorf("Zhipu ASR must prefill its own endpoint, got %q", got.DefaultURLs["asr"])
}
if len(got.Models) != 1 || got.Models[0].ID != "glm-asr-2512" {
t.Errorf("unexpected Zhipu ASR models: %+v", got.Models)
}
}
})
}
}
// listProvidersAs drives the real handler with one tenant role.
func listProvidersAs(t *testing.T, role types.TenantRole) []map[string]any {
t.Helper()

View File

@ -0,0 +1,382 @@
package handler
import (
"bytes"
"context"
"encoding/json"
"image"
"image/png"
"mime/multipart"
"net/http"
"net/http/httptest"
"testing"
"github.com/Tencent/WeKnora/internal/handler/dto"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/models/embedding"
"github.com/Tencent/WeKnora/internal/models/imageprep"
"github.com/Tencent/WeKnora/internal/models/rerank"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/gin-gonic/gin"
"github.com/stretchr/testify/require"
)
type debugImageService struct {
interfaces.ModelService
row *types.Model
embed *debugImageEmbedder
similarity *similarityEmbedder
rank *debugImageReranker
chat *debugImageChat
}
func (s *debugImageService) GetModelByID(context.Context, string) (*types.Model, error) {
return s.row, nil
}
func (s *debugImageService) GetEmbeddingModel(context.Context, string) (embedding.Embedder, error) {
if s.similarity != nil {
return s.similarity, nil
}
return s.embed, nil
}
func (s *debugImageService) GetRerankModel(context.Context, string) (rerank.Reranker, error) {
return s.rank, nil
}
func (s *debugImageService) GetChatModel(context.Context, string) (chat.Chat, error) {
return s.chat, nil
}
type debugImageEmbedder struct {
embedding.Embedder
accepts bool
images []embedding.Image
text string
}
func (e *debugImageEmbedder) AcceptsImages() bool { return e.accepts }
func (e *debugImageEmbedder) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
func (e *debugImageEmbedder) BatchEmbedImages(_ context.Context, images []embedding.Image) ([][]float32, error) {
e.images = images
return [][]float32{{1, 2, 3}}, nil
}
func (e *debugImageEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
e.text = text
return []float32{1, 2, 3}, nil
}
type debugImageReranker struct {
rerank.Reranker
accepts bool
images []rerank.Image
query string
}
func (r *debugImageReranker) AcceptsImages() bool { return r.accepts }
func (r *debugImageReranker) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
func (r *debugImageReranker) RerankImages(
_ context.Context, query string, images []rerank.Image,
) ([]rerank.RankResult, error) {
r.query = query
r.images = images
return []rerank.RankResult{{Index: 0, RelevanceScore: 0.9}}, nil
}
type debugChatBase interface{ chat.Chat }
type debugImageChat struct {
debugChatBase
messages []chat.Message
}
func (m *debugImageChat) ChatStream(
_ context.Context, messages []chat.Message, _ *chat.ChatOptions,
) (<-chan types.StreamResponse, error) {
m.messages = messages
ch := make(chan types.StreamResponse, 1)
ch <- types.StreamResponse{ResponseType: types.ResponseTypeAnswer, Content: "an image"}
close(ch)
return ch, nil
}
func runImageDebug(
t *testing.T, svc *debugImageService, input string, data []byte, documents string,
) (map[string]any, string) {
t.Helper()
var body bytes.Buffer
w := multipart.NewWriter(&body)
require.NoError(t, w.WriteField("input", input))
require.NoError(t, w.WriteField("documents", documents))
if data != nil {
f, err := w.CreateFormFile("file", "test.png")
require.NoError(t, err)
_, err = f.Write(data)
require.NoError(t, err)
}
require.NoError(t, w.Close())
req := httptest.NewRequest(http.MethodPost, "/models/test/debug", &body)
req.Header.Set("Content-Type", w.FormDataContentType())
recorder := httptest.NewRecorder()
c, _ := gin.CreateTestContext(recorder)
c.Request = req
c.Params = gin.Params{{Key: "id", Value: "test"}}
(&ModelHandler{service: svc}).DebugModel(c)
if len(c.Errors) > 0 {
return nil, c.Errors.Last().Error()
}
require.Equal(t, http.StatusOK, recorder.Code)
var result struct {
Data map[string]any `json:"data"`
}
require.NoError(t, json.Unmarshal(recorder.Body.Bytes(), &result))
return result.Data, ""
}
func debugPNG(t *testing.T) []byte {
t.Helper()
var b bytes.Buffer
require.NoError(t, png.Encode(&b, image.NewRGBA(image.Rect(0, 0, 2, 2))))
return b.Bytes()
}
func TestModelDebugImageEmbedding(t *testing.T) {
data := debugPNG(t)
for _, tc := range []struct {
name, input string
file []byte
accepts bool
wantErr string
}{
{"image only", "", data, true, ""},
{"text only", "hello", nil, true, ""},
{"unsupported image", "", data, false, "does not accept images"},
{"invalid upload", "", []byte("not an image"), true, "not an image"},
{"no input", "", nil, true, "input cannot be empty"},
{"mixed input", "hello", data, true, "either text or an image"},
} {
t.Run(tc.name, func(t *testing.T) {
embed := &debugImageEmbedder{accepts: tc.accepts}
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeEmbedding}, embed: embed}
result, errText := runImageDebug(t, svc, tc.input, tc.file, "")
if tc.wantErr != "" {
require.Contains(t, errText, tc.wantErr)
require.Empty(t, embed.images)
require.Empty(t, embed.text)
return
}
require.Empty(t, errText)
require.Equal(t, true, result["ok"])
require.Equal(t, float64(3), result["observations"].(map[string]any)["dimension"])
if tc.file != nil {
require.Len(t, embed.images, 1)
require.Equal(t, data, embed.images[0].Data)
require.Equal(t, "image/png", embed.images[0].MIMEType)
} else {
require.Equal(t, "hello", embed.text)
}
})
}
}
// similarityEmbedder places each text on a fixed axis so the expected cosine
// is known, and records which side every call was embedded as.
type similarityEmbedder struct {
embedding.Embedder
accepts bool
imageWidth int
queries []string
documents []string
images int
}
var similarityAxes = map[string][]float32{
"dog": {1, 0, 0}, "puppy": {0.9, 0.1, 0}, "car": {0, 1, 0}, "image": {0.6, 0.8, 0},
}
func (e *similarityEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
if types.IsEmbedQuery(ctx) {
e.queries = append(e.queries, text)
}
return similarityAxes[text], nil
}
func (e *similarityEmbedder) BatchEmbed(ctx context.Context, texts []string) ([][]float32, error) {
if !types.IsEmbedQuery(ctx) {
e.documents = append(e.documents, texts...)
}
out := make([][]float32, len(texts))
for i, text := range texts {
out[i] = similarityAxes[text]
}
return out, nil
}
func (e *similarityEmbedder) AcceptsImages() bool { return e.accepts }
func (e *similarityEmbedder) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
func (e *similarityEmbedder) BatchEmbedImages(context.Context, []embedding.Image) ([][]float32, error) {
e.images++
if e.imageWidth > 0 {
return [][]float32{make([]float32, e.imageWidth)}, nil
}
return [][]float32{similarityAxes["image"]}, nil
}
func runSimilarityDebug(
t *testing.T, embed *similarityEmbedder, query, documents string, file []byte,
) (map[string]any, string) {
t.Helper()
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeEmbedding}}
svc.similarity = embed
var body bytes.Buffer
w := multipart.NewWriter(&body)
require.NoError(t, w.WriteField("input", query))
require.NoError(t, w.WriteField("documents", documents))
require.NoError(t, w.WriteField("options", `{"similarity":true}`))
if file != nil {
f, err := w.CreateFormFile("file", "test.png")
require.NoError(t, err)
_, err = f.Write(file)
require.NoError(t, err)
}
require.NoError(t, w.Close())
req := httptest.NewRequest(http.MethodPost, "/models/test/debug", &body)
req.Header.Set("Content-Type", w.FormDataContentType())
recorder := httptest.NewRecorder()
c, _ := gin.CreateTestContext(recorder)
c.Request = req
c.Params = gin.Params{{Key: "id", Value: "test"}}
(&ModelHandler{service: svc}).DebugModel(c)
if len(c.Errors) > 0 {
return nil, c.Errors.Last().Error()
}
var result struct {
Data map[string]any `json:"data"`
}
require.NoError(t, json.Unmarshal(recorder.Body.Bytes(), &result))
return result.Data, ""
}
func TestModelDebugEmbeddingSimilarity(t *testing.T) {
embed := &similarityEmbedder{accepts: true}
result, errText := runSimilarityDebug(t, embed, "dog", `["car","puppy"]`, debugPNG(t))
require.Empty(t, errText)
require.Equal(t, true, result["ok"], result["error"])
require.Equal(t, []string{"dog"}, embed.queries, "the query is embedded as a search query")
require.Equal(t, []string{"car", "puppy"}, embed.documents, "candidates are embedded as content")
require.Equal(t, 1, embed.images)
results := result["raw_response"].(map[string]any)["results"].([]any)
require.Len(t, results, 3)
var order []string
for _, r := range results {
item := r.(map[string]any)
content, _ := item["content"].(string)
order = append(order, item["kind"].(string)+":"+content)
}
require.Equal(t, []string{"text:puppy", "image:", "text:car"}, order, "sorted by cosine, highest first")
require.InDelta(t, 0.6, results[1].(map[string]any)["similarity"], 1e-6)
require.Equal(t, float64(2), results[1].(map[string]any)["index"], "the image follows the documents")
require.Equal(t, float64(3), result["observations"].(map[string]any)["result_count"])
}
func TestModelDebugEmbeddingSimilarityRefusals(t *testing.T) {
for _, tc := range []struct {
name, query, documents string
file bool
embed *similarityEmbedder
wantErr string
}{
{"no query", "", `["car"]`, false, &similarityEmbedder{}, "at least one candidate"},
{"no candidate", "dog", "", false, &similarityEmbedder{}, "at least one candidate"},
{"image on a text model", "dog", "", true, &similarityEmbedder{}, "does not accept images"},
} {
t.Run(tc.name, func(t *testing.T) {
var file []byte
if tc.file {
file = debugPNG(t)
}
_, errText := runSimilarityDebug(t, tc.embed, tc.query, tc.documents, file)
require.Contains(t, errText, tc.wantErr)
require.Empty(t, tc.embed.queries, "nothing is sent for a refused test")
})
}
// An image vector of another width is a failed call, not a cosine of 0.
result, errText := runSimilarityDebug(t, &similarityEmbedder{accepts: true, imageWidth: 5}, "dog", "", debugPNG(t))
require.Empty(t, errText)
require.Equal(t, false, result["ok"])
require.Contains(t, result["error"], "5 dimensions, the query 3")
}
func TestModelDebugImageChat(t *testing.T) {
for _, tc := range []struct {
name, input string
vision bool
wantErr bool
}{
{"gpt-4o", "describe", false, false},
{"gpt-4o", "", false, false},
{"unknown-model", "describe", true, false},
{"unknown-model", "describe", false, true},
} {
t.Run(tc.name+tc.input, func(t *testing.T) {
model := &debugImageChat{}
svc := &debugImageService{
row: &types.Model{
Type: types.ModelTypeKnowledgeQA, Source: types.ModelSourceRemote, Name: tc.name,
Parameters: types.ModelParameters{Provider: "openai", SupportsVision: tc.vision},
},
chat: model,
}
result, errText := runImageDebug(t, svc, tc.input, debugPNG(t), "")
if tc.wantErr {
require.Contains(t, errText, "does not accept images")
require.Empty(t, model.messages)
return
}
require.Empty(t, errText)
require.Equal(t, true, result["ok"])
require.Len(t, model.messages, 1)
parts := model.messages[0].MultiContent
require.True(t, len(parts) > 0)
require.Contains(t, parts[len(parts)-1].ImageURL.URL, "data:image/png;base64,")
if tc.input != "" {
require.Equal(t, tc.input, parts[0].Text)
}
})
}
}
func TestModelDebugImageRerank(t *testing.T) {
for _, tc := range []struct {
accepts bool
documents, wantErr string
}{{true, "", ""}, {false, "", "does not accept images"}, {true, `["text"]`, "either text documents or an image"}} {
rank := &debugImageReranker{accepts: tc.accepts}
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeRerank}, rank: rank}
result, errText := runImageDebug(t, svc, "a cat", debugPNG(t), tc.documents)
if tc.wantErr != "" {
require.Contains(t, errText, tc.wantErr)
require.Empty(t, rank.images)
continue
}
require.Empty(t, errText)
require.Equal(t, true, result["ok"])
require.Len(t, rank.images, 1)
require.Equal(t, "a cat", rank.query)
}
}
func TestRerankResponseIncludesImageCapabilities(t *testing.T) {
row := &types.Model{
Type: types.ModelTypeRerank, Source: types.ModelSourceRemote,
Name: "nvidia/llama-nemotron-rerank-vl-1b-v2", Parameters: types.ModelParameters{Provider: "nvidia"},
}
response := dto.NewModelResponse(context.Background(), row)
require.NotNil(t, response.Capabilities)
require.Contains(t, response.Capabilities.Input, "image")
}

View File

@ -0,0 +1,110 @@
package handler
import (
"fmt"
"sort"
"strings"
"time"
"github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/models/embedding"
"github.com/Tencent/WeKnora/internal/models/imageprep"
"github.com/Tencent/WeKnora/internal/types"
"github.com/gin-gonic/gin"
)
// embeddingSimilarity is one scored candidate of an embedding similarity test.
type embeddingSimilarity struct {
// Index is the candidate's position in the request: the documents in
// order, then the image.
Index int `json:"index"`
Kind string `json:"kind"`
Content string `json:"content,omitempty"`
Similarity float64 `json:"similarity"`
}
// debugEmbeddingSimilarity embeds the input as a search query and every
// candidate as indexed content, the two sides retrieval uses, and returns
// the candidates by cosine similarity.
func (h *ModelHandler) debugEmbeddingSimilarity(
c *gin.Context, id, query string, documents []string, fileBytes []byte,
started time.Time, requestPreview, observations gin.H,
) {
ctx := c.Request.Context()
if strings.TrimSpace(query) == "" || (len(documents) == 0 && len(fileBytes) == 0) {
_ = c.Error(errors.NewBadRequestError("query and at least one candidate are required"))
return
}
instance, err := h.service.GetEmbeddingModel(ctx, id)
if err != nil {
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
return
}
var (
imageModel embedding.ImageEmbedder
prepared embedding.Image
)
if len(fileBytes) > 0 {
var ok bool
if imageModel, ok = embedding.AsImageEmbedder(instance); !ok {
_ = c.Error(errors.NewBadRequestError(embedding.ErrImagesUnsupported.Error()))
return
}
if prepared, err = imageprep.Prepare(fileBytes, imageModel.ImageLimits()); err != nil {
_ = c.Error(errors.NewBadRequestError(err.Error()))
return
}
}
queryVector, err := instance.Embed(types.WithEmbedQuery(ctx), query)
if err != nil {
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
return
}
observations["dimension"] = len(queryVector)
results := make([]embeddingSimilarity, 0, len(documents)+1)
score := func(index int, kind, content string, vector []float32) error {
// A width that differs from the query's means the two sides are not
// in one space; a cosine of 0 would hide that.
if len(vector) != len(queryVector) {
return fmt.Errorf("%s candidate %d has %d dimensions, the query %d",
kind, index, len(vector), len(queryVector))
}
results = append(results, embeddingSimilarity{
Index: index, Kind: kind, Content: content,
Similarity: types.CosineSimilarity(queryVector, vector),
})
return nil
}
if len(documents) > 0 {
vectors, err := instance.BatchEmbed(ctx, documents)
if err == nil && len(vectors) != len(documents) {
err = fmt.Errorf("%d vectors for %d documents", len(vectors), len(documents))
}
for i := 0; err == nil && i < len(documents); i++ {
err = score(i, "text", documents[i], vectors[i])
}
if err != nil {
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
return
}
}
if imageModel != nil {
vectors, err := imageModel.BatchEmbedImages(ctx, []embedding.Image{prepared})
if err == nil && len(vectors) != 1 {
err = fmt.Errorf("%d vectors for 1 image", len(vectors))
}
if err == nil {
err = score(len(documents), "image", "", vectors[0])
}
if err != nil {
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
return
}
}
sort.SliceStable(results, func(i, j int) bool { return results[i].Similarity > results[j].Similarity })
observations["result_count"] = len(results)
writeModelDebugResult(c, started, requestPreview, gin.H{"results": results}, nil, observations)
}

View File

@ -132,6 +132,7 @@ func TestURLResolution(t *testing.T) {
"https://api.anthropic.com": "https://api.anthropic.com/v1/messages",
"https://api.anthropic.com/v1": "https://api.anthropic.com/v1/messages",
"https://api.minimaxi.com/anthropic": "https://api.minimaxi.com/anthropic/v1/messages",
"https://api.minimax.cn/anthropic": "https://api.minimax.cn/anthropic/v1/messages",
"https://open.bigmodel.cn/api/anthropic": "https://open.bigmodel.cn/api/anthropic/v1/messages",
"https://proxy.example.com/v1/messages": "https://proxy.example.com/v1/messages",
}

View File

@ -19,13 +19,36 @@ type EmbeddingsCompat struct {
MaxInputChars *int `json:"max_input_chars,omitempty"`
// AcceptsTruncatePromptTokens marks a vLLM-class runtime, the only kind
// that implements the `truncate_prompt_tokens` extension.
AcceptsTruncatePromptTokens *bool `json:"accepts_truncate_prompt_tokens,omitempty"`
RequestTimeout *int `json:"request_timeout_seconds,omitempty"`
ExtraBody map[string]any `json:"extra_body,omitempty"`
ImageField *string `json:"image_field,omitempty"`
MaxImageBatchSize *int `json:"max_image_batch_size,omitempty"`
MaxImageBytes *int `json:"max_image_bytes,omitempty"`
ImageMIMETypes []string `json:"image_mime_types,omitempty"`
AcceptsTruncatePromptTokens *bool `json:"accepts_truncate_prompt_tokens,omitempty"`
RequestTimeout *int `json:"request_timeout_seconds,omitempty"`
ExtraBody map[string]any `json:"extra_body,omitempty"`
ImageField *string `json:"image_field,omitempty"`
MaxImageBatchSize *int `json:"max_image_batch_size,omitempty"`
MaxImageBytes *int `json:"max_image_bytes,omitempty"`
ImageMIMETypes []string `json:"image_mime_types,omitempty"`
ImageFormat *EmbeddingImageFormat `json:"image_format,omitempty"`
ImagePrompt *string `json:"image_prompt,omitempty"`
TextAsMessages *bool `json:"text_as_messages,omitempty"`
}
// EmbeddingImageFormat is how an image travels on the OpenAI embedding
// shape, which has no standard image input of its own.
type EmbeddingImageFormat string
const (
// EmbeddingImageObject puts {ImageField: data URI} in `input`, as Jina
// and SGLang document.
EmbeddingImageObject EmbeddingImageFormat = "object"
// EmbeddingImageMessages sends a chat conversation in `messages` instead
// of `input`, which is how vLLM serves multimodal embedding models
// (https://docs.vllm.ai/en/latest/models/pooling_models.html). One
// conversation yields one vector, so every image is its own request.
EmbeddingImageMessages EmbeddingImageFormat = "messages"
)
// Known reports whether f is a format this build can send; empty is known.
func (f EmbeddingImageFormat) Known() bool {
return f == "" || f == EmbeddingImageObject || f == EmbeddingImageMessages
}
// EmbeddingsSettings is the resolved (fully defaulted) form.
@ -76,6 +99,30 @@ type EmbeddingsSettings struct {
// has no standard image input, so an endpoint that names none takes no
// images), plus the documented per-request and per-image limits.
ImageInput
// ImageFormat chooses between ImageField's object in `input` and a chat
// conversation in `messages`. Empty means object when ImageField is set.
ImageFormat EmbeddingImageFormat `json:",omitempty"`
// ImagePrompt is the text sent beside each image in the messages format;
// some chat templates (VLM2Vec's) expect an instruction there. Empty
// sends the image alone.
ImagePrompt string `json:",omitempty"`
// TextAsMessages sends texts in the messages format too, one per
// request. A model whose chat template frames its inputs only places a
// text and an image in the same space when both pass through it; a
// plain `input` string skips the template.
TextAsMessages bool `json:",omitempty"`
}
// OpenAIImageFormat is the effective image format on the OpenAI shape, empty
// when the endpoint takes no images there.
func (s EmbeddingsSettings) OpenAIImageFormat() EmbeddingImageFormat {
switch {
case s.ImageFormat == EmbeddingImageMessages:
return EmbeddingImageMessages
case s.ImageField != "":
return EmbeddingImageObject
}
return ""
}
// BatchLimits renders the documented ceilings for SplitBatches.

View File

@ -10,7 +10,9 @@
// - `encoding_format` is absent from Zhipu's reference;
// - the parameter that separates a search query from an indexed document is
// `input_type` on NIM, `task` on Jina and `taskType` on Gemini, and most
// vendors have none.
// vendors have none;
// - images have no standard place: Jina and SGLang take an object in
// `input`, vLLM takes a chat conversation in `messages`.
//
// This package contains no vendor names.
package openaiembeddings
@ -75,7 +77,7 @@ type response struct {
// BuildRequestBody is the golden-test entry point: it returns the exact JSON
// object that would be sent.
func (c *Client) BuildRequestBody(texts []string, kind api.EmbedInputType) map[string]any {
return c.body(texts, kind)
return c.body("input", texts, kind)
}
// BuildImageRequestBody is the golden-test entry point for images. The shape
@ -87,14 +89,35 @@ func (c *Client) BuildImageRequestBody(images []api.EmbedImage, kind api.EmbedIn
for _, img := range images {
input = append(input, map[string]any{c.cfg.Settings.ImageField: img.DataURI()})
}
return c.body(input, kind)
return c.body("input", input, kind)
}
func (c *Client) body(input any, kind api.EmbedInputType) map[string]any {
// BuildImageMessageBody is the golden-test entry point for the messages
// format: one user turn carrying the image, then the configured prompt, as
// in vLLM's multimodal embedding example.
func (c *Client) BuildImageMessageBody(img api.EmbedImage, kind api.EmbedInputType) map[string]any {
content := []any{map[string]any{"type": "image_url", "image_url": map[string]any{"url": img.DataURI()}}}
if prompt := c.cfg.Settings.ImagePrompt; prompt != "" {
content = append(content, map[string]any{"type": "text", "text": prompt})
}
return c.messageBody(content, kind)
}
// BuildTextMessageBody is the golden-test entry point for a text sent in the
// messages format.
func (c *Client) BuildTextMessageBody(text string, kind api.EmbedInputType) map[string]any {
return c.messageBody([]any{map[string]any{"type": "text", "text": text}}, kind)
}
func (c *Client) messageBody(content []any, kind api.EmbedInputType) map[string]any {
return c.body("messages", []any{map[string]any{"role": "user", "content": content}}, kind)
}
func (c *Client) body(key string, input any, kind api.EmbedInputType) map[string]any {
s := c.cfg.Settings
body := map[string]any{
"model": c.cfg.Endpoint.Model,
"input": input,
key: input,
}
if s.SendEncodingFormat {
body["encoding_format"] = "float"
@ -127,22 +150,46 @@ func (c *Client) body(input any, kind api.EmbedInputType) map[string]any {
func (c *Client) Embed(
ctx context.Context, texts []string, kind api.EmbedInputType,
) ([][]float32, error) {
if c.cfg.Settings.TextAsMessages {
return c.postEach(ctx, len(texts), func(i int) map[string]any {
return c.BuildTextMessageBody(texts[i], kind)
})
}
return c.post(ctx, c.BuildRequestBody(texts, kind), len(texts))
}
// AcceptsImages reports whether the vendor names an image field. Without
// one a plain OpenAI-compatible server would read the objects as malformed
// text input.
func (c *Client) AcceptsImages() bool { return c.cfg.Settings.ImageField != "" }
// AcceptsImages reports whether the vendor declares an image format. Without
// one a plain OpenAI-compatible server would read image objects as
// malformed text input.
func (c *Client) AcceptsImages() bool { return c.cfg.Settings.OpenAIImageFormat() != "" }
// EmbedImages vectorizes one batch of images, in the order it was given.
func (c *Client) EmbedImages(
ctx context.Context, images []api.EmbedImage, kind api.EmbedInputType,
) ([][]float32, error) {
if !c.AcceptsImages() {
return nil, fmt.Errorf("this embedding endpoint declares no image input")
switch c.cfg.Settings.OpenAIImageFormat() {
case api.EmbeddingImageObject:
return c.post(ctx, c.BuildImageRequestBody(images, kind), len(images))
case api.EmbeddingImageMessages:
return c.postEach(ctx, len(images), func(i int) map[string]any {
return c.BuildImageMessageBody(images[i], kind)
})
}
return c.post(ctx, c.BuildImageRequestBody(images, kind), len(images))
return nil, fmt.Errorf("this embedding endpoint declares no image input")
}
// postEach sends one request per input, for the messages format where a
// conversation yields exactly one vector.
func (c *Client) postEach(ctx context.Context, n int, body func(int) map[string]any) ([][]float32, error) {
out := make([][]float32, n)
for i := range n {
vectors, err := c.post(ctx, body(i), 1)
if err != nil {
return nil, err
}
out[i] = vectors[0]
}
return out, nil
}
func (c *Client) post(ctx context.Context, body map[string]any, want int) ([][]float32, error) {

View File

@ -3,8 +3,10 @@ package openaiembeddings
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/Tencent/WeKnora/internal/models/api"
@ -122,6 +124,92 @@ func TestTruncatePromptTokensReachesOnlyVLLMClassRuntimes(t *testing.T) {
assert.Equal(t, 512, vllm.BuildRequestBody([]string{"a"}, api.EmbedDocument)["truncate_prompt_tokens"])
}
// vLLM's multimodal embedding example sends one user turn in `messages`:
// the image part, then the instruction its chat template expects. The rest
// of the body is the same as for `input`.
func TestVLLMImageMessageMatchesItsExample(t *testing.T) {
c := newClient("http://vllm.internal/v1", api.EmbeddingsSettings{
SendEncodingFormat: true, AcceptsTruncatePromptTokens: true, TruncatePromptTokens: 512,
ImageFormat: api.EmbeddingImageMessages, ImagePrompt: "Represent the given image.",
}, 0)
require.True(t, c.AcceptsImages())
body := c.BuildImageMessageBody(api.EmbedImage{Data: []byte{0x89}, MIMEType: "image/png"}, api.EmbedDocument)
assert.Equal(t, map[string]any{
"model": "m", "encoding_format": "float", "truncate_prompt_tokens": 512,
"messages": []any{map[string]any{"role": "user", "content": []any{
map[string]any{"type": "image_url", "image_url": map[string]any{"url": "data:image/png;base64,iQ=="}},
map[string]any{"type": "text", "text": "Represent the given image."},
}}},
}, body)
bare := newClient("http://vllm.internal/v1", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0)
content := bare.BuildImageMessageBody(
api.EmbedImage{Data: []byte{0x89}, MIMEType: "image/png"}, api.EmbedDocument,
)["messages"].([]any)[0].(map[string]any)["content"]
assert.Len(t, content, 1, "no prompt configured, so the image goes alone")
}
// The object format needs a field name; without one there is no image input.
func TestImageFormatDecidesWhetherImagesAreAccepted(t *testing.T) {
assert.False(t, newClient("u", api.EmbeddingsSettings{}, 0).AcceptsImages())
assert.False(t, newClient("u", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageObject}, 0).AcceptsImages())
assert.True(t, newClient("u", api.EmbeddingsSettings{
ImageInput: api.ImageInput{ImageField: "image"},
}, 0).AcceptsImages())
assert.True(t, newClient("u", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0).AcceptsImages())
}
// A conversation yields one vector, so a batch is one request per item, and
// texts follow images into `messages` only when the row asks.
func TestMessagesFormatSendsOneRequestPerItem(t *testing.T) {
t.Setenv("SSRF_WHITELIST", "127.0.0.1")
var bodies []map[string]any
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var body map[string]any
_ = json.NewDecoder(r.Body).Decode(&body)
bodies = append(bodies, body)
if input, ok := body["input"].([]any); ok {
parts := make([]string, len(input))
for i := range input {
parts[i] = fmt.Sprintf(`{"embedding":[0],"index":%d}`, i)
}
_, _ = fmt.Fprintf(w, `{"data":[%s]}`, strings.Join(parts, ","))
return
}
_, _ = fmt.Fprintf(w, `{"data":[{"embedding":[%d],"index":0}]}`, len(bodies))
}))
defer server.Close()
c := newClient(server.URL+"/v1", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0)
got, err := c.EmbedImages(context.Background(), []api.EmbedImage{
{Data: []byte{1}, MIMEType: "image/png"}, {Data: []byte{2}, MIMEType: "image/png"},
}, api.EmbedDocument)
require.NoError(t, err)
assert.Equal(t, [][]float32{{1}, {2}}, got)
require.Len(t, bodies, 2)
assert.Contains(t, bodies[0], "messages")
assert.NotContains(t, bodies[0], "input")
bodies = nil
_, err = c.Embed(context.Background(), []string{"a", "b"}, api.EmbedQuery)
require.NoError(t, err)
require.Len(t, bodies, 1, "texts stay in one `input` request unless the row opts in")
assert.Equal(t, []any{"a", "b"}, bodies[0]["input"])
bodies = nil
c = newClient(server.URL+"/v1", api.EmbeddingsSettings{
ImageFormat: api.EmbeddingImageMessages, TextAsMessages: true,
}, 0)
got, err = c.Embed(context.Background(), []string{"a", "b"}, api.EmbedQuery)
require.NoError(t, err)
assert.Equal(t, [][]float32{{1}, {2}}, got)
require.Len(t, bodies, 2)
assert.Equal(t, []any{map[string]any{"role": "user", "content": []any{
map[string]any{"type": "text", "text": "a"},
}}}, bodies[0]["messages"])
}
func TestDecodesArkTextResponse(t *testing.T) {
server, path, body := serve(t, arkTextResponse)
defer server.Close()

View File

@ -816,6 +816,24 @@
"max_image_bytes": 3000000
}
},
{
"id": "tongyi-embedding-vision-flash",
"name": "tongyi-embedding-vision-flash",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 768,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "",
"max_image_batch_size": 8,
"max_image_bytes": 3000000
}
},
{
"name": "tongyi-embedding-vision-flash (fixed 768 dimensions)",
"match": "tongyi-embedding-vision-flash",
@ -851,6 +869,25 @@
"max_image_bytes": 3000000
}
},
{
"id": "qwen3-vl-embedding",
"name": "qwen3-vl-embedding",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 2560,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "dimension",
"max_batch_size": 20,
"max_image_batch_size": 10,
"max_image_bytes": 10000000
}
},
{
"name": "qwen3-vl-embedding (independent vectors by default)",
"match": "qwen3-vl-embedding*",
@ -869,6 +906,25 @@
"max_image_bytes": 10000000
}
},
{
"id": "qwen2.5-vl-embedding",
"name": "qwen2.5-vl-embedding",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 1024,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "dimension",
"max_batch_size": 1,
"max_image_batch_size": 1,
"max_image_bytes": 5000000
}
},
{
"name": "qwen2.5-vl-embedding (fused vectors only)",
"match": "qwen2.5-vl-embedding*",
@ -887,6 +943,25 @@
"max_image_bytes": 5000000
}
},
{
"id": "multimodal-embedding-v1",
"name": "multimodal-embedding-v1",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 1024,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "",
"max_batch_size": 20,
"max_image_batch_size": 1,
"max_image_bytes": 3000000
}
},
{
"name": "multimodal-embedding-v1",
"match": "multimodal-embedding*",
@ -1979,6 +2054,22 @@
"xhigh": "high",
"max": "high"
}
},
{
"name": "Qwen3-VL-Embedding (self-hosted)",
"match": "*qwen3-vl-embedding*",
"type": "Embedding",
"input": [
"text",
"image"
],
"source": "https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B",
"compat": {
"text_as_messages": true,
"extra_body": {
"add_special_tokens": true
}
}
}
],
"gpustack": [],

View File

@ -362,6 +362,16 @@
"max_image_bytes": 3000000
}
},
{
"id": "tongyi-embedding-vision-flash",
"type": "Embedding",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "",
"max_image_batch_size": 8,
"max_image_bytes": 3000000
}
},
{
"match": "tongyi-embedding-vision-flash",
"type": "Embedding",
@ -383,6 +393,17 @@
"max_image_bytes": 3000000
}
},
{
"id": "qwen3-vl-embedding",
"type": "Embedding",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "dimension",
"max_batch_size": 20,
"max_image_batch_size": 10,
"max_image_bytes": 10000000
}
},
{
"match": "qwen3-vl-embedding*",
"type": "Embedding",
@ -394,6 +415,17 @@
"max_image_bytes": 10000000
}
},
{
"id": "qwen2.5-vl-embedding",
"type": "Embedding",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "dimension",
"max_batch_size": 1,
"max_image_batch_size": 1,
"max_image_bytes": 5000000
}
},
{
"match": "qwen2.5-vl-embedding*",
"type": "Embedding",
@ -405,6 +437,17 @@
"max_image_bytes": 5000000
}
},
{
"id": "multimodal-embedding-v1",
"type": "Embedding",
"compat": {
"api": "dashscope-embeddings",
"dimensions_field": "",
"max_batch_size": 20,
"max_image_batch_size": 1,
"max_image_bytes": 3000000
}
},
{
"match": "multimodal-embedding*",
"type": "Embedding",
@ -927,6 +970,16 @@
"xhigh": "high",
"max": "high"
}
},
{
"match": "*qwen3-vl-embedding*",
"type": "Embedding",
"compat": {
"text_as_messages": true,
"extra_body": {
"add_special_tokens": true
}
}
}
],
"hunyuan": [

View File

@ -592,6 +592,18 @@
"dimension": 1152,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"id": "tongyi-embedding-vision-flash",
"name": "tongyi-embedding-vision-flash",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 768,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"name": "tongyi-embedding-vision-flash (fixed 768 dimensions)",
"match": "tongyi-embedding-vision-flash",
@ -614,6 +626,18 @@
],
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"id": "qwen3-vl-embedding",
"name": "qwen3-vl-embedding",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 2560,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"name": "qwen3-vl-embedding (independent vectors by default)",
"match": "qwen3-vl-embedding*",
@ -625,6 +649,18 @@
],
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"id": "qwen2.5-vl-embedding",
"name": "qwen2.5-vl-embedding",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 1024,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"name": "qwen2.5-vl-embedding (fused vectors only)",
"match": "qwen2.5-vl-embedding*",
@ -636,6 +672,18 @@
],
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"id": "multimodal-embedding-v1",
"name": "multimodal-embedding-v1",
"type": "Embedding",
"input": [
"text",
"image",
"video"
],
"dimension": 1024,
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
},
{
"name": "multimodal-embedding-v1",
"match": "multimodal-embedding*",
@ -1342,6 +1390,16 @@
"match": "o4*",
"reasoning": true,
"source": "https://platform.openai.com/docs/guides/reasoning"
},
{
"name": "Qwen3-VL-Embedding (self-hosted)",
"match": "*qwen3-vl-embedding*",
"type": "Embedding",
"input": [
"text",
"image"
],
"source": "https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B"
}
],
"gpustack": [],

View File

@ -105,11 +105,26 @@ func TestImageEmbeddingWireFormatPerVendor(t *testing.T) {
"input": []any{map[string]any{"image": uri}},
},
},
{
name: "a generic (vLLM) row sends each image as a messages conversation", provider: "generic",
model: "TIGER-Lab/VLM2Vec-Full", base: "/v1", images: 2, wantRequests: 2,
wantPath: "/v1/embeddings",
wantBody: map[string]any{
"model": "TIGER-Lab/VLM2Vec-Full", "encoding_format": "float", "truncate_prompt_tokens": float64(511),
"messages": []any{map[string]any{"role": "user", "content": []any{
map[string]any{"type": "image_url", "image_url": map[string]any{"url": uri}},
}}},
},
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
up := newUpstream(t)
ie, ok := AsImageEmbedder(imageEmbedder(t, up, tc.provider, tc.model, tc.base, nil))
var spec *types.ModelSpecOverride
if tc.provider == "generic" {
spec = &types.ModelSpecOverride{Input: []string{"text", "image"}}
}
ie, ok := AsImageEmbedder(imageEmbedder(t, up, tc.provider, tc.model, tc.base, spec))
require.True(t, ok, "the catalog lists this model as taking images")
images := pngs(tc.images)
@ -137,9 +152,15 @@ func TestImageEmbeddingNeedsBothTheModelAndTheEndpoint(t *testing.T) {
}{
{"text model on a vendor with an image field", "jina", "jina-embeddings-v3", nil, false},
{"text model on the OpenAI shape", "openai", "text-embedding-3-small", nil, false},
// A generic OpenAI-compatible server has no image input to send to,
// whatever the row declares.
{"declared, but the endpoint names no image field", "generic", "my-clip", declare, false},
{"generic endpoint, image input not declared", "generic", "my-clip", nil, false},
{"declared on a generic (vLLM) endpoint", "generic", "my-clip", declare, true},
{
"declared, but the row switched to an object format without a field", "generic", "my-clip",
&types.ModelSpecOverride{
Input: []string{"text", "image"}, Compat: map[string]any{"image_format": "object"},
},
false,
},
{"declared on a vendor that names one", "jina", "my-jina-omni", declare, true},
{
"the row narrows a multimodal catalog entry", "volcengine", "doubao-embedding-vision-251215",
@ -163,6 +184,37 @@ func TestImageEmbeddingNeedsBothTheModelAndTheEndpoint(t *testing.T) {
}
}
// Qwen3-VL-Embedding on vLLM matches its model card only when text and
// image both pass the chat template and the prompt ends in <|endoftext|>,
// the token its last-token pooling reads. A plain `input` string skips the
// template and lands in a different space from the images.
func TestSelfHostedQwen3VLEmbeddingSendsTextAndImagesThroughTheTemplate(t *testing.T) {
up := newUpstream(t)
e := imageEmbedder(t, up, "generic", "Qwen/Qwen3-VL-Embedding-8B", "/v1", nil)
ie, ok := AsImageEmbedder(e)
require.True(t, ok, "the catalog declares image input without a row override")
got, err := e.BatchEmbed(types.WithEmbedQuery(context.Background()), []string{"ab", "c"})
require.NoError(t, err)
assert.Equal(t, [][]float32{{2}, {1}}, got)
_, err = ie.BatchEmbedImages(context.Background(), pngs(1))
require.NoError(t, err)
message := func(part map[string]any) map[string]any {
return map[string]any{
"model": "Qwen/Qwen3-VL-Embedding-8B", "encoding_format": "float",
"truncate_prompt_tokens": float64(511), "add_special_tokens": true,
"messages": []any{map[string]any{"role": "user", "content": []any{part}}},
}
}
require.Len(t, up.requests, 3, "the messages format carries one input per request")
assert.Equal(t, message(map[string]any{"type": "text", "text": "ab"}), up.requests[0].body)
assert.Equal(t, message(map[string]any{"type": "text", "text": "c"}), up.requests[1].body)
assert.Equal(t, message(map[string]any{
"type": "image_url", "image_url": map[string]any{"url": "data:image/png;base64,iQ=="},
}), up.requests[2].body)
}
func TestImageEmbeddingRefusesWhatTheVendorDocumentsItWillNotTake(t *testing.T) {
up := newUpstream(t)
gemini, ok := AsImageEmbedder(imageEmbedder(t, up, "gemini", "gemini-embedding-2", "/v1beta", nil))

View File

@ -118,6 +118,10 @@ func answer(path string, body map[string]any) string {
case strings.HasSuffix(path, "/embeddings/multimodal"):
n := itemLen(body["input"].([]any)[0])
return fmt.Sprintf(`{"data":{"embedding":[%d],"object":"embedding"}}`, n)
case body["messages"] != nil:
turn := body["messages"].([]any)[0].(map[string]any)
n := itemLen(turn["content"].([]any)[0])
return fmt.Sprintf(`{"data":[{"index":0,"embedding":[%d]}]}`, n)
default:
input := body["input"].([]any)
for i := len(input) - 1; i >= 0; i-- {

View File

@ -1,8 +1,11 @@
{
"aliyun/ASR/qwen3-asr-flash": "bb6104c02ce1013fd954de79c4998fd5b9e96e8b91cbbe083c9bc02a23660d7a",
"aliyun/ASR/snapshot": "error: catalog: aliyun does not serve \"\" through a protocol this build implements: Alibaba's other recognition models take a public file URL or run as asynchronous tasks; only qwen3-asr-flash accepts the audio in the request",
"aliyun/Embedding/multimodal-embedding-v1": "50cf14ae80fda14d123809d757833c44f08f7dc3b7f5d2d5d647d7ad66a8449b",
"aliyun/Embedding/multimodal-embeddingsnapshot": "abd8e6e33132030c401670448cbc649f51c200bd5b0042157d3ecbabb093fd4d",
"aliyun/Embedding/qwen2.5-vl-embedding": "277b713c73d600e661cab1a118b1f7ed6b936e4c48843b1cfb1a049385ddafff",
"aliyun/Embedding/qwen2.5-vl-embeddingsnapshot": "f957cb1c3bdba7cfe010c2c2f58a5148e422c629a4486321296a35ee4b7790f0",
"aliyun/Embedding/qwen3-vl-embedding": "80e3b7307df275d470f6c174f3bee891439fbe410d116720c7deba09bbde97af",
"aliyun/Embedding/qwen3-vl-embeddingsnapshot": "3d6dba847c93c3ea41c64a18dd31ec521c37426a05d94968bfea90c951544127",
"aliyun/Embedding/qwen3.7-text-embedding": "1747d31bcb369f32605000f3f2cc59406a4f7ddc5be6123c80e4e06f5d7fc2e7",
"aliyun/Embedding/qwen3.7-text-embedding-flash": "14ce61f8e4aea6394b43f6383bd2ec3d1398abd0a8d666374c4300fb86845cd5",
@ -10,7 +13,7 @@
"aliyun/Embedding/text-embedding-v2": "b3eb2440097a26c93dea795bdf9b9d1ea4d4a11f1e20a034e2bb146d7226883f",
"aliyun/Embedding/text-embedding-v3": "ef7acfe7150c048a9c46d5e8e2bc14d890df0721f877d76c41e6a2ed275bb7cc",
"aliyun/Embedding/text-embedding-v4": "acd6a3536ee4f98330fb02e39c9f1952dbb1eb578d70a82c64278355ca44bc36",
"aliyun/Embedding/tongyi-embedding-vision-flash": "fbcbf6d25d1078a137fa0d35dac9da104985a8da9b797bf161ec3b72a7775f02",
"aliyun/Embedding/tongyi-embedding-vision-flash": "bff51fd7a6cdb822b6b7463e436d68f953c1599796814dd0d9181db15c4fc4ec",
"aliyun/Embedding/tongyi-embedding-vision-plus": "fd37a8b75ef6f1c54dbbba9975e75f2668690d479f42beda61804f1be5b3c6bd",
"aliyun/Embedding/tongyi-embedding-visionsnapshot": "37e1d6878a0ec196c49339fc5ba48615f12453b7f065c6ac0e50e272b76ac219",
"aliyun/KnowledgeQA/deepseek-r1": "735322a596dda4a914b7e53041643eab61bd90b51e18e0cffd36dcc807b6fd16",
@ -98,6 +101,7 @@
"gemini/KnowledgeQA/gemini-3.7-flash": "570b30391590c3fa380e98e8323bf2187635d18db427eae3d75f9e63def26c8d",
"gemini/KnowledgeQA/gemini-3.8-flash": "6d40802ed8d3b310e1e91c186f769a4f8a298138538853cf623ad43e59c29140",
"gemini/KnowledgeQA/gemini-3snapshot": "c5931f2304774b718637f47540e78b66cbf855b4a4189c9fc0cea2040a12951d",
"generic/Embedding/snapshotqwen3-vl-embeddingsnapshot": "5b7394b420438753d8a7c4ad58af94ceab910f1e3497c0ccef357ca357f96626",
"generic/KnowledgeQA/gpt-5snapshot": "3e3712264316db008c1465a2d7c395072d85942d0352196e456342cef0b14eb0",
"generic/KnowledgeQA/o1snapshot": "52340886b4160a0cb9342ae33a7c81304c9dad5d22c2d1c1ce073f89860a30b1",
"generic/KnowledgeQA/o3snapshot": "782e836769e56c43d9c954ab2d563df7d2f476ae4fb4d588bb8bf3c88ae42ce6",
@ -144,15 +148,15 @@
"mimo/KnowledgeQA/mimo-v2.5": "a53a5bf889aa110e4460fed294a3230758ddaa49a6db60e8b0408fe183a60ed9",
"mimo/KnowledgeQA/mimo-v2.5-pro": "3c38342327d6286db15a7548f651c2780d8b0998b2671d05f187781eb6a23262",
"mimo/KnowledgeQA/mimo-v2.5-pro-ultraspeed": "a47aba2614020595d63accdc1c3c6058cfebdb3198df58936fb95e28c9046092",
"minimax/ASR/asr-1.0": "3466d2d1953d8165ba7677de85f36fa933118f5dc902bf39fdfa9158b0ec0eab",
"minimax/KnowledgeQA/MiniMax-M2": "145995deb36b85b3f0933c4aed6273b97cbacf92891cae224949dae906d251cd",
"minimax/KnowledgeQA/MiniMax-M2.1": "b745d75ad3095e2fdc25d22d7f723c30078b9bbb155c4d0f7d1bc64c5dbfbb59",
"minimax/KnowledgeQA/MiniMax-M2.1-highspeed": "6f3d05e123ef452d9ed0b40456ccbe08c552ca2408f225dce40115add1b23fd5",
"minimax/KnowledgeQA/MiniMax-M2.5": "163e53955eb8a46362e4581513644be9c86c39ff744819f777ae4f2bd81bf81d",
"minimax/KnowledgeQA/MiniMax-M2.5-highspeed": "c1731e761db65503eaa72acdba28c95a1eaae15882d813534f7831b3256022cb",
"minimax/KnowledgeQA/MiniMax-M2.7": "2efbb13fcb0efd7c36987ad5d12d1d6c1d80f2e0b5518abeaac8dc603ff44050",
"minimax/KnowledgeQA/MiniMax-M2.7-highspeed": "7f7518fa498dc12eeec38d8d73daae4f1a43a207d07b9d2369948aaca347c5e8",
"minimax/KnowledgeQA/MiniMax-M3": "e0ab669e2f427ed88ad4d5defec501fa835fa6e32922f8942be5216d5076b92b",
"minimax/ASR/asr-1.0": "1c51f1e261087dc295ca455f3f580ebd4e2af19b4dc520635729415f49ef4432",
"minimax/KnowledgeQA/MiniMax-M2": "43bf0fa0d1325b5c52f046109c730d6c7549cb4a31696cae7119a68480802fe9",
"minimax/KnowledgeQA/MiniMax-M2.1": "bd74e5923d0d12f7cd5d27cdc9ce7a03aea561fb932fbc2df7b9527f30a0b27e",
"minimax/KnowledgeQA/MiniMax-M2.1-highspeed": "dcc02614ff25e6ec426f81e8c5bc8c1fb54fe15fc3c61bc952158cdc5605b7a1",
"minimax/KnowledgeQA/MiniMax-M2.5": "12e81e74ad422b52aab382584e0e329d28f5e6f97f9fa517a16c63b448f46156",
"minimax/KnowledgeQA/MiniMax-M2.5-highspeed": "6b87c598fcfcb674dbe600ef19f30b518af8b6c6ba959d84ecbd02542eee3c12",
"minimax/KnowledgeQA/MiniMax-M2.7": "3f6c046bb99022ee88f3c99f3f9e7e75ecfabaad6f97a601352297ec33ffadde",
"minimax/KnowledgeQA/MiniMax-M2.7-highspeed": "bc4a3116243b4dd2a4b764db9fa8155f8262ab4a05a34cc1e8b6bcc2e17cde72",
"minimax/KnowledgeQA/MiniMax-M3": "153a02aafdcdfbd115c5e1fd6b5f03b145dfe22e13cf2641dc9f52722ac4ca98",
"modelscope/Embedding/Qwen/Qwen3-Embedding-0.6B": "a4515af27268a360fe7ee62eb4c4785bc5e3079f65e0264acc99ba974b9a777b",
"modelscope/Embedding/Qwen/Qwen3-Embedding-8B": "a76960f74528a100af92aad4def93306da5eb819ba8fc916693197d779f0ebd2",
"modelscope/KnowledgeQA/Qwen/Qwen3-235B-A22B-Instruct-2507": "cbd8290913b5b1a584aaa9cd20ccb6478b67c62a73d9aa3f9f9c5aee0baf7d80",

View File

@ -120,6 +120,11 @@ func newGenericProvider() *Definition {
SendEncodingFormat: api.Ptr(true),
DimensionsField: api.Ptr("dimensions"),
AcceptsTruncatePromptTokens: api.Ptr(true),
// Used only when the row declares image input. vLLM, the most
// common server behind this vendor, takes images only as a
// chat conversation in `messages`; an SGLang row switches to
// {"image_format": "object", "image_field": "image"}.
ImageFormat: api.Ptr(api.EmbeddingImageMessages),
},
Rerank: api.RerankCompat{
// Any OpenAI-compatible endpoint an operator points here is most

View File

@ -15,9 +15,10 @@
// - the only credential is a bearer API key: the GroupId that the legacy
// /v1/text/chatcompletion_v2 endpoint required is not used here, so this
// vendor needs no ExtraFields;
// - the mainland endpoint is https://api.minimaxi.com/v1 and the global one
// - the mainland endpoint is https://api.minimax.cn/v1 and the global one
// https://api.minimax.io/v1; keys are region-bound (mainland keys to
// minimaxi.com, global keys to minimax.io).
// minimax.cn, global keys to minimax.io). Mainland endpoint reference:
// https://platform.minimax.cn/docs/api-reference/text-openai-api.
//
// Second protocol: an Anthropic Messages facade lives at
// https://api.minimax.io/anthropic (mainland form in MinimaxAnthropicBaseURL), and
@ -40,10 +41,8 @@
// unverified: max output caps. The model table publishes context windows but
// no per-model output cap, so the max_output_tokens in models.json are
// carried over unconfirmed;
// unverified: only the global https://api.minimax.io/anthropic form of the
// Anthropic facade is spelled out in the docs; the mainland
// https://api.minimaxi.com/anthropic that MinimaxAnthropicBaseURL holds is inferred
// from the region split of the /v1 hosts.
// The mainland Anthropic facade is documented at
// https://platform.minimax.cn/docs/api-reference/text-anthropic-api.
package providers
import (
@ -60,13 +59,13 @@ var minimaxIcon []byte
const MinimaxID = "minimax"
// MinimaxBaseURL is the mainland-China OpenAI-compatible endpoint.
const MinimaxBaseURL = "https://api.minimaxi.com/v1"
const MinimaxBaseURL = "https://api.minimax.cn/v1"
// MinimaxGlobalBaseURL is the international endpoint.
const MinimaxGlobalBaseURL = "https://api.minimax.io/v1"
// MinimaxAnthropicBaseURL is the Anthropic Messages compatibility facade.
const MinimaxAnthropicBaseURL = "https://api.minimaxi.com/anthropic"
const MinimaxAnthropicBaseURL = "https://api.minimax.cn/anthropic"
func newMinimaxProvider() *Definition {
return &Definition{
@ -79,13 +78,14 @@ func newMinimaxProvider() *Definition {
"zh-CN": "MiniMax-M3, MiniMax-M2.7, MiniMax-M2.7-highspeed 等。国际版 " +
MinimaxGlobalBaseURL + ";Anthropic 兼容接口 " + MinimaxAnthropicBaseURL,
},
Website: "https://platform.minimaxi.com",
Website: "https://platform.minimax.cn",
Icon: minimaxIcon,
API: api.APIOpenAICompletions,
Order: 16,
RequiresAuth: true,
Auth: AuthBearer,
URLPatterns: []string{"minimax.io", "minimaxi.com"},
// Retain inference for saved rows using the previous mainland domain.
URLPatterns: []string{"minimax.cn", "minimax.io", "minimaxi.com"},
DefaultBaseURLs: map[types.ModelType]string{
types.ModelTypeKnowledgeQA: MinimaxBaseURL,
},

View File

@ -22,6 +22,65 @@ var expectedIDs = []string{
"jina", "nvidia", "novita", "gpustack",
}
func TestMinimaxOfficialDefaultURLs(t *testing.T) {
v, ok := modelruntime.Get(providers.MinimaxID)
if !ok {
t.Fatal("MiniMax provider missing")
}
for _, kind := range []types.ModelType{types.ModelTypeKnowledgeQA, types.ModelTypeASR} {
if got := v.GetDefaultURL(kind); got != "https://api.minimax.cn/v1" {
t.Errorf("%s default = %q, want current mainland endpoint", kind, got)
}
}
if providers.MinimaxAnthropicBaseURL != "https://api.minimax.cn/anthropic" {
t.Errorf("unexpected mainland Anthropic URL: %s", providers.MinimaxAnthropicBaseURL)
}
if v.Website != "https://platform.minimax.cn" {
t.Errorf("unexpected mainland website: %s", v.Website)
}
for _, base := range []string{
"https://api.minimax.cn/v1", "https://api.minimax.cn/anthropic",
"https://api.minimaxi.com/v1", "https://api.minimaxi.com/anthropic",
"https://api.minimax.io/v1", "https://api.minimax.io/anthropic",
} {
if got := modelruntime.DetectByURL(base); got != providers.MinimaxID {
t.Errorf("DetectByURL(%q) = %q", base, got)
}
resolved, err := modelruntime.Resolve(modelruntime.Ref{BaseURL: base, Model: "MiniMax-M2.7"})
if err != nil {
t.Fatal(err)
}
if resolved.BaseURL != base {
t.Errorf("explicit address was rewritten: %q -> %q", base, resolved.BaseURL)
}
wantAPI := api.APIOpenAICompletions
if strings.HasSuffix(base, "/anthropic") {
wantAPI = api.APIAnthropicMessages
}
if resolved.API != wantAPI {
t.Errorf("%s resolved API = %s, want %s", base, resolved.API, wantAPI)
}
}
}
func TestBuiltinDefaultURLsAcrossModelTypes(t *testing.T) {
for _, vendor := range modelruntime.List() {
for _, kind := range vendor.ModelTypes {
base := vendor.GetDefaultURL(kind)
// These connections have no hosted service to prefill.
if vendor.ID == "generic" || vendor.ID == "huggingface_tei" {
if base != "" {
t.Errorf("%s/%s should require an operator URL", vendor.ID, kind)
}
continue
}
if !strings.HasPrefix(base, "https://") && !strings.HasPrefix(base, "http://") {
t.Errorf("%s/%s has no absolute default URL: %q", vendor.ID, kind, base)
}
}
}
}
func TestAllVendorsRegistered(t *testing.T) {
if len(expectedIDs) != 29 {
t.Fatalf("expected 29 vendor ids in the spec, got %d", len(expectedIDs))

View File

@ -487,6 +487,11 @@ func resolveEmbeddings(
protocol = settings.API
}
settings.API = protocol
if !settings.ImageFormat.Known() {
return nil, fmt.Errorf(
"catalog: unknown embedding image_format %q on %s/%s (expected %q or %q)",
settings.ImageFormat, vendor.ID, spec.ID, api.EmbeddingImageObject, api.EmbeddingImageMessages)
}
out := &Resolved{
Vendor: configcopy.Clone(vendor.Definition),

View File

@ -30,7 +30,7 @@ const (
// imageLoadConcurrency bounds image reads from storage.
imageLoadConcurrency = 4
// MinImageKeepScore is the least vector similarity that keeps a pictorial
// image hit over a text reranker. It sits above the 0.1 image recall
// image hit over a text reranker. It sits above the 0.2 image recall
// threshold: recall only lets an image compete, keeping it needs more.
// Empirical, on the cosine scale most engines report.
MinImageKeepScore = 0.25

View File

@ -0,0 +1,115 @@
package reranking
import (
"context"
"fmt"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
)
// Observe selection in the shared implementation: agent tools and quick answers
// must expose the same decisions, including early exits and model-error fallback.
func recordSelection(ctx context.Context, input []*types.SearchResult, res *Result, opts Options,
scoredAsImage map[int]bool,
) {
out := summarizeSelection(input, res, opts, scoredAsImage)
logger.GetLogger(ctx).WithFields(logger.Fields(out)).Info("rerank selection")
_, span := langfuse.GetManager().StartSpan(ctx, langfuse.SpanOptions{Name: "rerank.selection"})
span.Finish(out, nil, nil)
}
func summarizeSelection(input []*types.SearchResult, res *Result, opts Options,
scoredAsImage map[int]bool,
) map[string]interface{} {
// Candidates retain input pointers. Final rows are copies, so use the
// returned input indices to distinguish retained and rejected inputs.
candidates := make(map[*types.SearchResult]int, len(res.Candidates))
for i, r := range res.Candidates {
candidates[r] = i
}
returned := make(map[int]*types.SearchResult, len(res.Indices))
for i, index := range res.Indices {
returned[index] = res.Results[i]
}
modelScores := make(map[int]float64, len(res.ModelScores))
for _, score := range res.ModelScores {
modelScores[score.Index] = score.RelevanceScore
}
passed := make(map[string]bool, len(res.Scored))
for _, r := range res.Scored {
passed[r.ID] = true
}
inputTypes, candidateTypes, outputTypes := map[string]int{}, map[string]int{}, map[string]int{}
for _, r := range res.Candidates {
candidateTypes[r.ChunkType]++
}
for _, r := range res.Results {
if r != nil {
outputTypes[r.ChunkType]++
}
}
images := make([]map[string]interface{}, 0)
decisions := map[string]int{}
imageCount := 0
for i, r := range input {
if r == nil {
continue
}
inputTypes[r.ChunkType]++
if !isImageVectorHit(r) {
continue
}
imageCount++
pos, candidate := candidates[r]
score, hasScore := modelScores[pos]
decision := "candidate_cap_or_empty_passage"
if kept, ok := returned[i]; ok {
decision = "kept"
if res.Diagnostics.Outcome == types.RerankOutcomeModelError {
decision = "model_error_fallback"
} else if kept.Metadata[types.MetadataKeptBy] == types.KeptByImageVector {
decision = "image_vector_fallback"
}
} else if candidate {
switch {
case !hasScore:
decision = "no_model_score"
case passed[r.ID]:
decision = "top_k_mmr_cut"
default:
decision = "rerank_rejected"
}
}
decisions[decision]++
if len(images) >= 10 {
continue
}
row := map[string]interface{}{
"input_rank": i + 1, "chunk_id": r.ID, "knowledge_id": r.KnowledgeID,
"vector_score": fmt.Sprintf("%.4f", r.VectorScore),
"retrieval_score": fmt.Sprintf("%.4f", r.Score), "decision": decision,
}
if candidate && hasScore {
row["model_score"] = fmt.Sprintf("%.4f", score)
row["scored_as"] = "text"
if scoredAsImage[pos] {
row["scored_as"] = "image"
}
}
images = append(images, row)
}
return map[string]interface{}{
"input_count": len(input), "candidate_count": len(res.Candidates), "output_count": len(res.Results),
"input_chunk_types": inputTypes, "candidate_chunk_types": candidateTypes, "output_chunk_types": outputTypes,
"outcome": res.Diagnostics.Outcome, "images_scored": res.Diagnostics.ImagesScored,
"images_kept_by_vector": res.Diagnostics.ImagesKept, "image_decision_counts": decisions,
"image_samples": images, "image_samples_truncated": imageCount - len(images),
// Keyword ordering and explicit scopes may use -Inf. Strings are JSON safe.
"threshold": fmt.Sprint(opts.Threshold), "effective_threshold": fmt.Sprint(res.Diagnostics.EffectiveThreshold),
"fallback_min_score": fmt.Sprint(opts.FallbackMinScore), "image_keep_score": fmt.Sprint(opts.ImageKeepScore),
"max_candidates": opts.MaxCandidates, "top_k": opts.TopK,
"top_model_score": fmt.Sprint(res.Diagnostics.TopScore),
}
}

View File

@ -0,0 +1,55 @@
package reranking
import (
"context"
"encoding/json"
"errors"
"math"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestSelectionDiagnosticsExplainRejectedAndProtectedImages(t *testing.T) {
input := []*types.SearchResult{
imageHit("weak", "private image caption", 0.21, 0),
imageHit("protected", "another caption", 0.4, 0),
}
opts := Options{Threshold: 0.3, FallbackMinScore: 0.15, ImageKeepScore: 0.25}
res := Rerank(context.Background(), &textScorer{}, "q", input, opts)
out := summarizeSelection(input, res, opts, nil)
require.Equal(t, map[string]int{"rerank_rejected": 1, "image_vector_fallback": 1}, out["image_decision_counts"])
require.Equal(t, map[string]int{"image_vector": 2}, out["input_chunk_types"])
require.Equal(t, map[string]int{"image_vector": 1}, out["output_chunk_types"])
rows := out["image_samples"].([]map[string]interface{})
require.Equal(t, "0.2100", rows[0]["vector_score"])
require.Equal(t, "0.0000", rows[0]["model_score"])
require.Equal(t, "text", rows[0]["scored_as"])
b, err := json.Marshal(out)
require.NoError(t, err)
require.NotContains(t, string(b), "private image caption")
}
func TestSelectionDiagnosticsExplainModelErrorAndAllowNonfiniteThresholds(t *testing.T) {
input := []*types.SearchResult{imageHit("image", "caption", 0.3, 0)}
opts := Options{Threshold: math.Inf(-1), FallbackMinScore: math.Inf(-1)}
res := Rerank(context.Background(), &textScorer{err: errors.New("offline")}, "q", input, opts)
out := summarizeSelection(input, res, opts, nil)
require.Equal(t, map[string]int{"model_error_fallback": 1}, out["image_decision_counts"])
require.Equal(t, "-Inf", out["threshold"])
_, err := json.Marshal(out)
require.NoError(t, err)
}
func TestSelectionDiagnosticsDistinguishTopKFromModelRejection(t *testing.T) {
input := []*types.SearchResult{
imageHit("first", "first", 0.4, 0), imageHit("second", "second", 0.3, 0),
}
opts := Options{Threshold: 0.3, TopK: 1}
model := &textScorer{scores: map[string]float64{"first": 0.9, "second": 0.8}}
res := Rerank(context.Background(), model, "q", input, opts)
out := summarizeSelection(input, res, opts, map[int]bool{0: true})
require.Equal(t, map[string]int{"kept": 1, "top_k_mmr_cut": 1}, out["image_decision_counts"])
require.Equal(t, "image", out["image_samples"].([]map[string]interface{})[0]["scored_as"])
}

View File

@ -122,6 +122,9 @@ func Rerank(
},
}
var scoredAsImage map[int]bool
defer func() { recordSelection(ctx, results, res, opts, scoredAsImage) }()
keep := topByScore(results, opts.MaxCandidates)
candidateIdx := make([]int, 0, len(results))
for i, r := range results {

View File

@ -96,6 +96,7 @@ func SummarizeSearchResults(results []*types.SearchResult, limit int) map[string
"composite_score": fmt.Sprintf("%.4f", sr.Score),
"match_type": sr.MatchType,
"chunk_type": sr.ChunkType,
"vector_score": fmt.Sprintf("%.4f", sr.VectorScore),
"preview": TruncateRunes(sr.Content, 160),
}
if sr.Metadata != nil {
@ -160,6 +161,8 @@ func summarizeIndexHits(hits []*types.IndexWithScore, limit int) []map[string]in
"knowledge_base_id": hit.KnowledgeBaseID,
"score": fmt.Sprintf("%.4f", hit.Score),
"match_type": hit.MatchType,
"source_type": hit.SourceType,
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
"preview": TruncateRunes(hit.Content, 160),
})
}
@ -253,6 +256,8 @@ func summarizeRetrieveHits(
"knowledge_base_id": hit.KnowledgeBaseID,
"score": fmt.Sprintf("%.4f", hit.Score),
"match_type": hit.MatchType,
"source_type": hit.SourceType,
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
"engine": string(ref.engine),
"retriever": string(ref.retriever),
"preview": TruncateRunes(hit.Content, 160),
@ -287,6 +292,8 @@ func SummarizePassagePreviews(
"knowledge_title": sr.KnowledgeTitle,
"retrieval_score": fmt.Sprintf("%.4f", sr.Score),
"match_type": sr.MatchType,
"chunk_type": sr.ChunkType,
"vector_score": fmt.Sprintf("%.4f", sr.VectorScore),
"preview": TruncateRunes(passages[i], 160),
})
}

View File

@ -90,3 +90,21 @@ func TestSummarizeSearchResults_sortsByScore(t *testing.T) {
t.Fatalf("unexpected hits: %#v", hits)
}
}
func TestRetrievalPreviewsPreserveImageSourceAndRawVectorScore(t *testing.T) {
out := SummarizeRetrieveOutput([]*types.RetrieveResult{{
RetrieverType: types.VectorRetrieverType,
Results: []*types.IndexWithScore{{
ChunkID: "image", SourceType: types.ImageSourceType, Score: 0.7, VectorScore: 0.2688,
}},
}})
hit := out["top_hits"].([]map[string]interface{})[0]
if hit["source_type"] != types.ImageSourceType || hit["vector_score"] != "0.2688" || hit["score"] != "0.7000" {
t.Fatalf("image identity/raw score lost: %#v", hit)
}
groups := out["top_hits_by_retriever"].([]map[string]interface{})
groupHit := groups[0]["top_hits"].([]map[string]interface{})[0]
if groupHit["source_type"] != types.ImageSourceType || groupHit["vector_score"] != "0.2688" {
t.Fatalf("per-retriever image identity/raw score lost: %#v", groupHit)
}
}

View File

@ -15,6 +15,9 @@ type KnowledgeProcessOverrides struct {
// pipeline switch for this one document. nil keeps the knowledge base's
// setting.
ImageAttrsEnabled *bool `json:"image_attrs_enabled,omitempty"`
// ImageVectorEnabled overrides image vector indexing for this document.
// It does not change the KB-wide retrieval setting or embedding model.
ImageVectorEnabled *bool `json:"image_vector_enabled,omitempty"`
// ImageActions overrides rows of the attribute->work table for this document
// (merged per action key on top of the knowledge base's table).
// nil keeps the knowledge base's setting.
@ -36,6 +39,7 @@ type EffectiveProcessConfig struct {
QuestionGenerationConfig QuestionGenerationConfig
GraphEnabled bool
ImageAttrsEnabled bool
ImageVectorEnabled bool
ImageActions ImageActionsConfig
ExtractConfig ExtractConfig
}

View File

@ -536,6 +536,9 @@ type ImageMultimodalPayload struct {
// in the payload rather than being re-read from the knowledge base at handle
// time.
ImageAttrsEnabled bool `json:"image_attrs_enabled,omitempty"`
// ImageVectorEnabled is resolved at enqueue time. nil preserves the KB
// fallback for tasks queued before per-document indexing overrides existed.
ImageVectorEnabled *bool `json:"image_vector_enabled,omitempty"`
// ImageActions is the attribute→work table, resolved from the KB config at
// enqueue time. A task without it — one already in flight when attribute
// observation shipped — falls back to the conservative policy: run OCR.

View File

@ -89,6 +89,9 @@
| 需要附带厂商私有参数,如阿里云联网搜索 | `{"extra_body": {"enable_search": true}}` |
| 多轮对话回传思考内容时报错 | `{"replay_reasoning_content": false}` |
| 自建向量服务一次请求只接受 16 条 | `{"max_batch_size": 16}` |
| vLLM 部署的多模态向量模型(VLM2Vec、GME 等) | 打开「图片输入」即可,自定义厂商默认按 vLLM 的 `messages` 格式发图片;模板需要指令时加 `{"image_prompt": "Represent the given image."}`;文本也必须经过对话模板才和图片同空间时加 `{"text_as_messages": true}` |
| vLLM 部署的 Qwen3-VL-Embedding | 无需填写:自定义厂商下模型名含 `qwen3-vl-embedding` 时已内置图片输入与 `{"text_as_messages": true, "extra_body": {"add_special_tokens": true}}`。文本走普通 `input` 会绕过对话模板,与图片不在同一空间;不加 `add_special_tokens` 时末尾缺少模型取向量用的 `<\|endoftext\|>`,与[模型卡](https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B)结果不符。已用该模型建立的索引需重建向量 |
| SGLang 部署的多模态向量模型 | 打开「图片输入」,并填 `{"image_format": "object", "image_field": "image"}` |
| 向量或重排服务响应慢,需要更长的超时 | `{"request_timeout_seconds": 120}` |
| 自建重排服务返回的是未归一化分数(logit) | `{"score_scale": "logit"}` |
@ -191,7 +194,10 @@
| `accepts_truncate_prompt_tokens` | bool | false | 服务是否支持 vLLM 的 `truncate_prompt_tokens` |
| `request_timeout_seconds` | int | 60 | 单次请求超时(秒) |
| `extra_body` | object | 空 | 附加到每次请求的字段 |
| `image_field` | string | 空 | 仅 `openai-embeddings`:`input` 里承载图片的对象键名(Jina 为 `image`)。留空表示该端点不收图片 |
| `image_field` | string | 空 | 仅 `openai-embeddings`:`input` 里承载图片的对象键名(Jina、SGLang 为 `image`)。留空且 `image_format` 不是 `messages` 时表示该端点不收图片 |
| `image_format` | string | 空(自定义厂商为 `messages`) | 仅 `openai-embeddings`:图片的发送方式。`object` 把 `{image_field: 图片}` 放进 `input`;`messages` 把图片作为一轮对话放进 `messages`(vLLM 的多模态向量接口),每张图一个请求。留空时有 `image_field` 即按 `object` 处理 |
| `image_prompt` | string | 空 | 仅 `messages` 格式:随图片一起发送的文字,部分模型的对话模板需要(如 VLM2Vec 的 `Represent the given image.`) |
| `text_as_messages` | bool | false | 仅 `openai-embeddings`:文本也按 `messages` 格式发送,每条一个请求。适用于必须经过对话模板、文本和图片才在同一向量空间的模型(自定义厂商下的 Qwen3-VL-Embedding 已内置);已有索引的模型改动后需要重建向量 |
| `max_image_batch_size` | int | 1 | 单次请求的最大图片数,超出自动分批 |
| `max_image_bytes` | int | 0(不限) | 单张图片的字节上限,超出的图片直接拒绝、不发请求 |
| `image_mime_types` | array | 空(不限) | 厂商接受的图片格式,如 `["image/png", "image/jpeg"]` |