fix(models): align configuration and multimodal embedding workflows (#4080)
* fix(models): align configuration defaults and multimodal workflows * feat(image): enhance image retrieval diagnostics and filtering - Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results. - Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores. - Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording. - Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling. * feat(embeddings): enhance image handling with new formats and message support - Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats. - Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows. - Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements. - Added tests to validate the new image message handling and ensure correct request formatting for different scenarios. - Updated documentation to reflect changes in image input handling and configuration options. * feat(similarity): add embedding similarity mode for enhanced document scoring - Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query. - Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type. - Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity. - Added tests to validate the functionality of the new similarity mode in embedding models. - Updated localization files to include new strings related to similarity mode in multiple languages. * feat(embeddings): add Qwen3-VL-Embedding support for multimodal input - Introduced Qwen3-VL-Embedding model with support for both text and image inputs. - Updated model configuration files to include compatibility settings for the new embedding type. - Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs. - Updated documentation to reflect the new model's capabilities and usage instructions. * test(models): wrap embedding assertions to satisfy lint * test(opensearch): isolate HTTP transports between parallel testsmain
parent
40eb34bc84
commit
c3f077ed18
|
|
@ -235,6 +235,8 @@ export interface ModelDebugOptions {
|
|||
thinking?: boolean
|
||||
// Graded thinking level; takes precedence over the boolean when set.
|
||||
reasoning_effort?: ReasoningEffortLevel | string
|
||||
// Embedding only: score `documents` (and the file) against `input` as a query.
|
||||
similarity?: boolean
|
||||
}
|
||||
|
||||
export interface ModelDebugResult {
|
||||
|
|
|
|||
|
|
@ -76,33 +76,40 @@
|
|||
<template v-if="selectedModel">
|
||||
<section class="setting-drawer__section">
|
||||
<h4 class="setting-drawer__section-title">{{ $t('modelSettings.debug.groupInput') }}</h4>
|
||||
<div v-if="selectedModel.type !== 'ASR'" class="form-item">
|
||||
<div v-if="hasImageMode || isEmbedding" class="form-item">
|
||||
<t-radio-group v-model="inputMode" @change="clearFile">
|
||||
<t-radio-button value="text">{{ isEmbedding ? $t('modelSettings.debug.embeddingInput') : $t('modelSettings.debug.documents') }}</t-radio-button>
|
||||
<t-radio-button v-if="hasImageMode" value="image">{{ $t('modelSettings.debug.imageFile') }}</t-radio-button>
|
||||
<t-radio-button v-if="isEmbedding" value="similarity">{{ $t('modelSettings.debug.similarityMode') }}</t-radio-button>
|
||||
</t-radio-group>
|
||||
</div>
|
||||
<div v-if="selectedModel.type !== 'ASR' && !(isEmbedding && useImageMode)" class="form-item">
|
||||
<label class="form-label">{{ inputLabel }}</label>
|
||||
<t-textarea
|
||||
v-model="input"
|
||||
:placeholder="inputPlaceholder"
|
||||
:autosize="{ minRows: 4, maxRows: 8 }"
|
||||
:autosize="{ minRows: useSimilarityMode ? 2 : 4, maxRows: 8 }"
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div v-if="selectedModel.type === 'Rerank'" class="form-item">
|
||||
<label class="form-label">{{ $t('modelSettings.debug.documents') }}</label>
|
||||
<div v-if="(selectedModel.type === 'Rerank' && !useImageMode) || useSimilarityMode" class="form-item">
|
||||
<label class="form-label">{{ useSimilarityMode ? $t('modelSettings.debug.similarityCandidates') : $t('modelSettings.debug.documents') }}</label>
|
||||
<t-textarea
|
||||
v-model="documentsText"
|
||||
:placeholder="$t('modelSettings.debug.documentsPlaceholder')"
|
||||
:placeholder="useSimilarityMode ? $t('modelSettings.debug.similarityCandidatesPlaceholder') : $t('modelSettings.debug.documentsPlaceholder')"
|
||||
:autosize="{ minRows: 4, maxRows: 8 }"
|
||||
/>
|
||||
<p class="form-desc">{{ $t('modelSettings.debug.documentsHint') }}</p>
|
||||
<p class="form-desc">{{ useSimilarityMode ? $t('modelSettings.debug.similarityHint') : $t('modelSettings.debug.documentsHint') }}</p>
|
||||
</div>
|
||||
|
||||
<div v-if="needsFile" class="form-item">
|
||||
<label class="form-label">{{ fileLabel }}</label>
|
||||
<div v-if="showFilePicker" class="form-item">
|
||||
<label class="form-label">{{ useSimilarityMode ? $t('modelSettings.debug.similarityImage') : fileLabel }}</label>
|
||||
<div class="file-picker">
|
||||
<input
|
||||
ref="fileInputRef"
|
||||
class="file-picker__input"
|
||||
type="file"
|
||||
:accept="selectedModel.type === 'VLLM' ? 'image/*' : 'audio/*'"
|
||||
:accept="selectedModel.type === 'ASR' ? 'audio/*' : 'image/*'"
|
||||
@change="onNativeFileChange"
|
||||
>
|
||||
<t-button variant="outline" size="small" @click="fileInputRef?.click()">
|
||||
|
|
@ -110,7 +117,11 @@
|
|||
{{ $t('modelSettings.debug.chooseFile') }}
|
||||
</t-button>
|
||||
</div>
|
||||
<p v-if="file" class="form-desc">{{ file.name }} · {{ formatBytes(file.size) }}</p>
|
||||
<div v-if="file" class="selected-file">
|
||||
<img v-if="imagePreview" :src="imagePreview" :alt="file.name" class="image-preview">
|
||||
<p class="form-desc">{{ file.name }} · {{ formatBytes(file.size) }}</p>
|
||||
<t-button variant="text" size="small" @click="clearFile">{{ $t('common.clear') }}</t-button>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
|
|
@ -196,6 +207,23 @@
|
|||
|
||||
<p v-if="result.error" class="result-error">{{ result.error }}</p>
|
||||
|
||||
<div v-if="similarityResults.length > 0" class="similarity-list">
|
||||
<div v-for="(item, rank) in similarityResults" :key="item.index" class="similarity-row">
|
||||
<span class="similarity-row__rank">{{ rank + 1 }}</span>
|
||||
<span class="similarity-row__content">
|
||||
<template v-if="item.kind === 'image'">
|
||||
<img v-if="imagePreview" :src="imagePreview" :alt="file?.name" class="similarity-row__image">
|
||||
{{ $t('modelSettings.debug.similarityImageCandidate') }}
|
||||
</template>
|
||||
<template v-else>{{ item.content }}</template>
|
||||
</span>
|
||||
<span class="similarity-row__bar">
|
||||
<span :style="{ width: `${Math.max(0, item.similarity) * 100}%` }" />
|
||||
</span>
|
||||
<span class="similarity-row__score">{{ item.similarity.toFixed(4) }}</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<t-tabs v-model="resultTab" class="result-tabs">
|
||||
<t-tab-panel value="response" :label="$t('modelSettings.debug.rawResponse')" />
|
||||
<t-tab-panel value="request" :label="$t('modelSettings.debug.requestPreview')" />
|
||||
|
|
@ -254,6 +282,8 @@ const selectedModelId = ref('')
|
|||
const input = ref('')
|
||||
const documentsText = ref('')
|
||||
const file = ref<File | null>(null)
|
||||
const inputMode = ref<'text' | 'image' | 'similarity'>('text')
|
||||
const imagePreview = ref('')
|
||||
const fileInputRef = ref<HTMLInputElement | null>(null)
|
||||
const reasoningEffort = ref<ReasoningLevel>('off')
|
||||
const temperature = ref(0.7)
|
||||
|
|
@ -277,14 +307,29 @@ const isChat = computed(() => selectedModel.value?.type === 'KnowledgeQA')
|
|||
// (remote chat models carry `capabilities` computed by the backend).
|
||||
const supportsThinking = computed(() => modelCanThink(selectedModel.value?.capabilities))
|
||||
const reasoningOptions = computed(() => supportedLevels(selectedModel.value?.capabilities))
|
||||
const needsFile = computed(() => ['VLLM', 'ASR'].includes(selectedModel.value?.type || ''))
|
||||
const acceptsImages = computed(() => {
|
||||
const model = selectedModel.value
|
||||
if (!model || model.type === 'ASR') return false
|
||||
return model.type === 'VLLM'
|
||||
|| model.capabilities?.input?.includes('image') === true
|
||||
|| (model.type === 'KnowledgeQA' && model.parameters.supports_vision === true)
|
||||
})
|
||||
const hasImageMode = computed(() => acceptsImages.value && ['Embedding', 'Rerank'].includes(selectedModel.value?.type || ''))
|
||||
const useImageMode = computed(() => hasImageMode.value && inputMode.value === 'image')
|
||||
const isEmbedding = computed(() => selectedModel.value?.type === 'Embedding')
|
||||
const useSimilarityMode = computed(() => isEmbedding.value && inputMode.value === 'similarity')
|
||||
// In similarity mode the image is an optional extra candidate.
|
||||
const showFilePicker = computed(() => selectedModel.value?.type === 'ASR'
|
||||
|| (acceptsImages.value && (!hasImageMode.value || useImageMode.value || useSimilarityMode.value)))
|
||||
const needsFile = computed(() => useImageMode.value || ['VLLM', 'ASR'].includes(selectedModel.value?.type || ''))
|
||||
const documents = computed(() => documentsText.value.split('\n').map(item => item.trim()).filter(Boolean))
|
||||
const canRun = computed(() => {
|
||||
if (!selectedModel.value) return false
|
||||
if (useSimilarityMode.value) return !!input.value.trim() && (documents.value.length > 0 || !!file.value)
|
||||
if (needsFile.value && !file.value) return false
|
||||
if (selectedModel.value.type === 'ASR') return true
|
||||
if (selectedModel.value.type === 'Rerank') return !!input.value.trim() && documents.value.length > 0
|
||||
return !!input.value.trim()
|
||||
if (['ASR', 'VLLM'].includes(selectedModel.value.type) || (selectedModel.value.type === 'Embedding' && useImageMode.value)) return true
|
||||
if (selectedModel.value.type === 'Rerank') return !!input.value.trim() && (useImageMode.value ? !!file.value : documents.value.length > 0)
|
||||
return !!input.value.trim() || (isChat.value && acceptsImages.value && !!file.value)
|
||||
})
|
||||
|
||||
const allModelTypeOptions = computed(() => {
|
||||
|
|
@ -319,6 +364,7 @@ const vendorLabel = (model: ModelConfig) => {
|
|||
}
|
||||
|
||||
const inputLabel = computed(() => {
|
||||
if (useSimilarityMode.value) return t('modelSettings.debug.similarityQuery')
|
||||
if (selectedModel.value?.type === 'Embedding') return t('modelSettings.debug.embeddingInput')
|
||||
if (selectedModel.value?.type === 'VLLM') return t('modelSettings.debug.vlmPrompt')
|
||||
if (selectedModel.value?.type === 'Rerank') return t('modelSettings.debug.query')
|
||||
|
|
@ -326,15 +372,16 @@ const inputLabel = computed(() => {
|
|||
})
|
||||
|
||||
const inputPlaceholder = computed(() => {
|
||||
if (useSimilarityMode.value) return t('modelSettings.debug.similarityQueryPlaceholder')
|
||||
if (selectedModel.value?.type === 'Embedding') return t('modelSettings.debug.embeddingPlaceholder')
|
||||
if (selectedModel.value?.type === 'VLLM') return t('modelSettings.debug.vlmPromptPlaceholder')
|
||||
return t('modelSettings.debug.queryPlaceholder')
|
||||
})
|
||||
|
||||
const fileLabel = computed(() =>
|
||||
selectedModel.value?.type === 'VLLM'
|
||||
? t('modelSettings.debug.imageFile')
|
||||
: t('modelSettings.debug.audioFile'),
|
||||
selectedModel.value?.type === 'ASR'
|
||||
? t('modelSettings.debug.audioFile')
|
||||
: t('modelSettings.debug.imageFile'),
|
||||
)
|
||||
|
||||
const formattedResult = computed(() => {
|
||||
|
|
@ -345,6 +392,19 @@ const formattedResult = computed(() => {
|
|||
return JSON.stringify(value, null, 2)
|
||||
})
|
||||
|
||||
interface SimilarityResult {
|
||||
index: number
|
||||
kind: 'text' | 'image'
|
||||
content?: string
|
||||
similarity: number
|
||||
}
|
||||
|
||||
const similarityResults = computed<SimilarityResult[]>(() => {
|
||||
const response = result.value?.raw_response as { results?: unknown } | null | undefined
|
||||
if (!isEmbedding.value || !result.value?.ok || !response || !Array.isArray(response.results)) return []
|
||||
return response.results as SimilarityResult[]
|
||||
})
|
||||
|
||||
const OBSERVATION_LABELS: Record<string, string> = {
|
||||
api: 'modelSettings.debug.metrics.api',
|
||||
thinking_format: 'modelSettings.debug.metrics.thinkingFormat',
|
||||
|
|
@ -418,11 +478,9 @@ watch(() => selectedModel.value?.id, () => {
|
|||
reasoningEffort.value = clampLevel(reasoningEffort.value, reasoningOptions.value)
|
||||
}, { immediate: true })
|
||||
|
||||
watch(() => selectedModel.value?.type, () => {
|
||||
file.value = null
|
||||
result.value = null
|
||||
history.value = []
|
||||
resultTab.value = 'response'
|
||||
watch(() => selectedModel.value?.id, () => {
|
||||
inputMode.value = 'text'
|
||||
clearFile()
|
||||
})
|
||||
|
||||
const resetResult = () => {
|
||||
|
|
@ -441,6 +499,17 @@ const selectModelType = (type: DebugModelType) => {
|
|||
resetResult()
|
||||
}
|
||||
|
||||
const clearFile = () => {
|
||||
file.value = null
|
||||
if (fileInputRef.value) fileInputRef.value.value = ''
|
||||
resetResult()
|
||||
}
|
||||
|
||||
watch(file, value => {
|
||||
if (imagePreview.value) URL.revokeObjectURL(imagePreview.value)
|
||||
imagePreview.value = value && selectedModel.value?.type !== 'ASR' ? URL.createObjectURL(value) : ''
|
||||
})
|
||||
|
||||
const onNativeFileChange = (event: Event) => {
|
||||
const target = event.target as HTMLInputElement
|
||||
const selectedFile = target.files?.[0] || null
|
||||
|
|
@ -473,9 +542,9 @@ const runDebug = async () => {
|
|||
try {
|
||||
const level: ReasoningLevel | null = supportsThinking.value ? reasoningEffort.value : null
|
||||
const nextResult = await debugModel(selectedModel.value.id, {
|
||||
input: input.value.trim(),
|
||||
documents: documents.value,
|
||||
file: file.value,
|
||||
input: isEmbedding.value && useImageMode.value ? '' : input.value.trim(),
|
||||
documents: (selectedModel.value.type === 'Rerank' && !useImageMode.value) || useSimilarityMode.value ? documents.value : [],
|
||||
file: showFilePicker.value ? file.value : null,
|
||||
options: isChat.value ? {
|
||||
system_prompt: systemPrompt.value.trim() || undefined,
|
||||
temperature: temperature.value,
|
||||
|
|
@ -483,7 +552,7 @@ const runDebug = async () => {
|
|||
max_tokens: maxTokens.value,
|
||||
// reasoning_effort is authoritative; the boolean keeps older backends working.
|
||||
...(level ? { reasoning_effort: level, thinking: levelEnablesThinking(level) } : { thinking: false }),
|
||||
} : {},
|
||||
} : useSimilarityMode.value ? { similarity: true } : {},
|
||||
})
|
||||
result.value = nextResult
|
||||
history.value.unshift({
|
||||
|
|
@ -506,6 +575,7 @@ const copyResult = async () => {
|
|||
}
|
||||
|
||||
onBeforeUnmount(() => {
|
||||
if (imagePreview.value) URL.revokeObjectURL(imagePreview.value)
|
||||
if (document.activeElement instanceof HTMLElement) {
|
||||
document.activeElement.blur()
|
||||
}
|
||||
|
|
@ -632,6 +702,20 @@ onBeforeUnmount(() => {
|
|||
}
|
||||
}
|
||||
|
||||
.selected-file {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-top: 8px;
|
||||
}
|
||||
|
||||
.image-preview {
|
||||
width: 80px;
|
||||
height: 80px;
|
||||
object-fit: contain;
|
||||
border-radius: var(--app-radius-sm);
|
||||
}
|
||||
|
||||
.history-list {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
|
|
@ -720,6 +804,65 @@ onBeforeUnmount(() => {
|
|||
font-size: var(--app-text-sm);
|
||||
}
|
||||
|
||||
.similarity-list {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 4px;
|
||||
margin-top: 10px;
|
||||
}
|
||||
|
||||
.similarity-row {
|
||||
display: grid;
|
||||
grid-template-columns: 20px minmax(0, 1fr) 80px 56px;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 6px 8px;
|
||||
border-radius: var(--app-radius-sm);
|
||||
background: var(--td-bg-color-secondarycontainer);
|
||||
font-size: var(--app-text-sm);
|
||||
|
||||
&__rank {
|
||||
color: var(--td-text-color-placeholder);
|
||||
text-align: right;
|
||||
}
|
||||
|
||||
&__content {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
overflow: hidden;
|
||||
color: var(--td-text-color-primary);
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
&__image {
|
||||
width: 24px;
|
||||
height: 24px;
|
||||
object-fit: cover;
|
||||
border-radius: var(--app-radius-xs);
|
||||
}
|
||||
|
||||
&__bar {
|
||||
height: 6px;
|
||||
overflow: hidden;
|
||||
border-radius: 3px;
|
||||
background: var(--td-component-stroke);
|
||||
|
||||
span {
|
||||
display: block;
|
||||
height: 100%;
|
||||
background: var(--td-brand-color);
|
||||
}
|
||||
}
|
||||
|
||||
&__score {
|
||||
color: var(--td-text-color-secondary);
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||||
text-align: right;
|
||||
}
|
||||
}
|
||||
|
||||
.result-error {
|
||||
margin: 10px 0 0;
|
||||
color: var(--td-error-color);
|
||||
|
|
|
|||
|
|
@ -87,10 +87,11 @@
|
|||
<span class="source-option__label">{{ $t('model.editor.sourceRemote') }}</span>
|
||||
</button>
|
||||
<button
|
||||
v-if="activeModelType !== 'rerank'"
|
||||
type="button"
|
||||
class="source-option"
|
||||
:class="{ 'is-active': formData.source === 'local', 'is-disabled': ollamaServiceStatus === false || activeModelType === 'rerank' }"
|
||||
:disabled="ollamaServiceStatus === false || activeModelType === 'rerank'"
|
||||
:class="{ 'is-active': formData.source === 'local', 'is-disabled': ollamaServiceStatus === false }"
|
||||
:disabled="ollamaServiceStatus === false"
|
||||
role="radio"
|
||||
:aria-checked="formData.source === 'local'"
|
||||
@click="formData.source = 'local'"
|
||||
|
|
@ -100,14 +101,8 @@
|
|||
</button>
|
||||
</div>
|
||||
|
||||
<!-- ReRank模型不支持Ollama的提示信息 -->
|
||||
<div v-if="activeModelType === 'rerank'" class="ollama-unavailable-tip rerank-tip">
|
||||
<t-icon name="info-circle-filled" class="tip-icon info" />
|
||||
<span class="tip-text">{{ $t('model.editor.ollamaNotSupportRerank') }}</span>
|
||||
</div>
|
||||
|
||||
<!-- Ollama不可用时的提示信息 -->
|
||||
<div v-else-if="shouldShowOllamaUnavailableTip(formData.source, activeModelType, ollamaServiceStatus)"
|
||||
<div v-if="shouldShowOllamaUnavailableTip(formData.source, activeModelType, ollamaServiceStatus)"
|
||||
class="ollama-unavailable-tip">
|
||||
<t-icon name="error-circle-filled" class="tip-icon" />
|
||||
<span class="tip-text">{{ $t('model.editor.ollamaUnavailable') }}</span>
|
||||
|
|
@ -1394,6 +1389,8 @@ const getModalDescription = () => {
|
|||
|
||||
// 获取模型名称占位符
|
||||
const getModelNamePlaceholder = () => {
|
||||
const catalogModel = catalogEntries.value[0]
|
||||
if (formData.value.source === 'remote' && catalogModel) return catalogModel.id
|
||||
if (activeModelType.value === 'vllm') {
|
||||
return formData.value.source === 'local'
|
||||
? t('model.editor.modelNamePlaceholder.localVllm')
|
||||
|
|
@ -1461,52 +1458,29 @@ const goToOllamaSettings = async () => {
|
|||
const lastOpenedModelId = ref<string | null>(null)
|
||||
|
||||
const selectModelType = async (type: EditorModelType) => {
|
||||
if (isEdit.value || draftModelType.value === type) return
|
||||
if (saving.value || isEdit.value || draftModelType.value === type) return
|
||||
draftModelType.value = type
|
||||
|
||||
if (type === 'rerank') {
|
||||
formData.value.source = 'remote'
|
||||
}
|
||||
if (type !== 'embedding') {
|
||||
formData.value.dimension = undefined
|
||||
formData.value.supportsDimensionOverride = false
|
||||
dimensionChecked.value = false
|
||||
dimensionSuccess.value = false
|
||||
dimensionMessage.value = ''
|
||||
}
|
||||
if (type !== 'chat') {
|
||||
formData.value.supportsVision = false
|
||||
}
|
||||
remoteChecked.value = false
|
||||
remoteAvailable.value = false
|
||||
remoteMessage.value = ''
|
||||
|
||||
// A different model type starts a new connection, including credentials
|
||||
// and advanced overrides. Clear the old draft before waiting for its catalog.
|
||||
resetForm()
|
||||
const draft = formData.value
|
||||
await loadProviders()
|
||||
const supported = providerOptions.value.some(p => p.value === formData.value.provider)
|
||||
if (!supported) {
|
||||
formData.value.provider = 'generic'
|
||||
formData.value.baseUrl = ''
|
||||
// 只丢厂商相关的 extra_config;协议 / 远端模型名是用户对这次接入的选择,
|
||||
// 与厂商无关,和 handleProviderChange 保持同一套规则。
|
||||
formData.value.extraConfig = keepVendorNeutralExtraConfig()
|
||||
formData.value.appSecret = ''
|
||||
} else {
|
||||
handleProviderChange(formData.value.provider || 'generic')
|
||||
if (props.visible && !isEdit.value && formData.value === draft && activeModelType.value === type) {
|
||||
applyExtraFieldDefaults()
|
||||
}
|
||||
}
|
||||
|
||||
// 监听 visible 变化,初始化表单
|
||||
watch(() => props.visible, (val) => {
|
||||
if (val) {
|
||||
const typeChanged = draftModelType.value !== props.modelType
|
||||
// Sync the entry type before loading its catalog; the previous draft may
|
||||
// belong to another settings tab (including the initial chat default).
|
||||
draftModelType.value = props.modelType
|
||||
|
||||
// 检查Ollama服务状态
|
||||
checkOllamaServiceStatus()
|
||||
|
||||
// 从 API 加载 Model Provider 列表(编辑已有行时顺便补齐额外字段默认值)。
|
||||
// Catalogs can be published by another administrator while this page is
|
||||
// open, so refresh this type without dropping other types' cached lists.
|
||||
loadProviders(true).then(() => {
|
||||
if (props.visible && !isEdit.value) applyExtraFieldDefaults()
|
||||
})
|
||||
advancedOpen.value = false
|
||||
|
||||
// 每次打开都清理上一次遗留的校验/检测结果,避免编辑别的模型时
|
||||
|
|
@ -1521,7 +1495,6 @@ watch(() => props.visible, (val) => {
|
|||
dimensionMessage.value = ''
|
||||
|
||||
const currentId = props.modelData?.id ?? null
|
||||
draftModelType.value = props.modelType
|
||||
|
||||
hydratingForm.value = true
|
||||
try {
|
||||
|
|
@ -1549,11 +1522,11 @@ watch(() => props.visible, (val) => {
|
|||
: [],
|
||||
}
|
||||
legacyThinkingControlLoaded.value = !!props.modelData.thinkingControl
|
||||
} else if (lastOpenedModelId.value !== null || !formData.value.id) {
|
||||
// 上次是编辑某个模型,或第一次新增 → 重置成空白
|
||||
} else if (typeChanged || lastOpenedModelId.value !== null || !formData.value.id) {
|
||||
// 切换类型、上次是编辑某个模型,或第一次新增 → 重置成空白
|
||||
resetForm()
|
||||
}
|
||||
// 否则:连续两次"新增"打开(中间是点遮罩/ESC 关闭的)→ 保留上次填写
|
||||
// 同一类型连续两次"新增"打开(中间是点遮罩/ESC 关闭的)→ 保留上次填写
|
||||
|
||||
lastOpenedModelId.value = currentId
|
||||
|
||||
|
|
@ -1571,6 +1544,16 @@ watch(() => props.visible, (val) => {
|
|||
hydratingForm.value = false
|
||||
})
|
||||
}
|
||||
|
||||
// Catalogs can be published while this page is open. Refresh this type,
|
||||
// but do not apply defaults to a different draft if the request finishes late.
|
||||
const draft = formData.value
|
||||
const type = activeModelType.value
|
||||
loadProviders(true).then(() => {
|
||||
if (props.visible && !isEdit.value && formData.value === draft && activeModelType.value === type) {
|
||||
applyExtraFieldDefaults()
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
|
|
@ -1621,6 +1604,17 @@ const resetForm = () => {
|
|||
customHeaders: [],
|
||||
appSecret: '',
|
||||
}
|
||||
catalogFilled.value = {}
|
||||
advancedOpen.value = false
|
||||
saveError.value = ''
|
||||
searchKeyword.value = ''
|
||||
invalidateConnectionTest(false)
|
||||
resolveRevision++
|
||||
if (resolveTimer) {
|
||||
clearTimeout(resolveTimer)
|
||||
resolveTimer = null
|
||||
}
|
||||
resolving.value = false
|
||||
resolved.value = null
|
||||
resolveFailed.value = false
|
||||
resolveError.value = ''
|
||||
|
|
@ -1632,6 +1626,7 @@ const resetForm = () => {
|
|||
dimensionChecked.value = false
|
||||
dimensionSuccess.value = false
|
||||
dimensionMessage.value = ''
|
||||
nextTick(() => formRef.value?.clearValidate?.())
|
||||
}
|
||||
|
||||
// 处理厂商选择变化 (自动填充默认 URL)
|
||||
|
|
@ -1676,13 +1671,11 @@ const resetModelSelectionForVendor = () => {
|
|||
|
||||
const handleProviderChange = (value: string) => {
|
||||
const provider = providerOptions.value.find(opt => opt.value === value)
|
||||
if (provider?.defaultUrls) {
|
||||
// 根据当前模型类型获取对应的默认 URL
|
||||
const defaultUrl = provider.defaultUrls[activeModelType.value]
|
||||
if (defaultUrl) {
|
||||
formData.value.baseUrl = defaultUrl
|
||||
}
|
||||
}
|
||||
// Older catalogs omit inherited type URLs. Mirror GetDefaultURL's chat
|
||||
// fallback, preserving an explicitly empty URL and never the previous vendor's.
|
||||
formData.value.baseUrl = provider?.defaultUrls?.[activeModelType.value]
|
||||
?? provider?.defaultUrls?.chat
|
||||
?? ''
|
||||
// 重置校验状态:它描述的是上一家厂商的连通性,跟新厂商无关。放在 defaultUrls
|
||||
// 判断之外,否则切到没有默认地址的厂商时会留着一条"连接正常"的旧结论。
|
||||
remoteChecked.value = false
|
||||
|
|
@ -2930,17 +2923,6 @@ const handleCancel = () => {
|
|||
line-height: 1.5;
|
||||
}
|
||||
|
||||
// ReRank提示使用主题绿色风格,与主页面保持一致
|
||||
&.rerank-tip {
|
||||
background: var(--td-success-color-light);
|
||||
border: 1px solid var(--td-success-color-focus);
|
||||
border-left: 3px solid var(--td-brand-color);
|
||||
|
||||
.tip-text {
|
||||
color: var(--td-success-color);
|
||||
}
|
||||
}
|
||||
|
||||
:deep(.tip-link) {
|
||||
color: var(--td-brand-color);
|
||||
font-size: var(--app-text-md);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,177 @@
|
|||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { createRequire } from 'node:module'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { runInNewContext } from 'node:vm'
|
||||
import test from 'node:test'
|
||||
import { compileScript, parse } from '@vue/compiler-sfc'
|
||||
import ts from 'typescript'
|
||||
import { createRenderer, h, nextTick, reactive } from 'vue'
|
||||
|
||||
const require = createRequire(import.meta.url)
|
||||
const filename = fileURLToPath(new URL('./ModelDebugDrawer.vue', import.meta.url))
|
||||
const { descriptor } = parse(readFileSync(filename, 'utf8'), { filename })
|
||||
const script = compileScript(descriptor, { id: 'model-debug-images' }).content
|
||||
.replace('__expose();', '')
|
||||
.replace('return __returned__', '__expose(__returned__); return __returned__')
|
||||
const compiled = ts.transpileModule(script, {
|
||||
compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022 },
|
||||
}).outputText
|
||||
|
||||
async function fixture(type: string, images = true, legacyVision = false) {
|
||||
const requests: any[] = []
|
||||
const revoked: string[] = []
|
||||
const props = reactive({ visible: false, models: [
|
||||
{ id: 'image-model', name: 'model', type, source: 'remote', parameters: { supports_vision: legacyVision }, capabilities: { input: images ? ['text', 'image'] : ['text'], thinking_levels: [] } },
|
||||
{ id: 'text-model', name: 'text', type, source: 'remote', parameters: {}, capabilities: { input: ['text'], thinking_levels: [] } },
|
||||
] })
|
||||
const exports: any = {}
|
||||
runInNewContext(compiled, {
|
||||
exports, console, document: { activeElement: null }, HTMLElement: class {},
|
||||
URL: { createObjectURL: () => 'blob:preview', revokeObjectURL: (url: string) => revoked.push(url) },
|
||||
require(name: string) {
|
||||
if (name === 'vue') return require('vue')
|
||||
if (name === 'vue-i18n') return { useI18n: () => ({ t: (key: string) => key, locale: { value: 'en-US' } }) }
|
||||
if (name === '@/api/model') return { debugModel: async (id: string, data: any) => { requests.push({ id, ...data }); return { ok: true, elapsed_ms: 1, raw_response: [], observations: {} } } }
|
||||
if (name === '@/stores/modelProviders') return { useModelProvidersStore: () => ({ ensureLoaded: async () => {}, labelFor: () => '' }) }
|
||||
if (name === '@/utils') return { fileSizeVerification: () => false }
|
||||
if (name === '@/utils/reasoningEffort') return require('../utils/reasoningEffort.ts')
|
||||
if (name === '@/utils/contextWindow') return require('../utils/contextWindow.ts')
|
||||
if (name === 'tdesign-vue-next') return { MessagePlugin: { error() {} } }
|
||||
return { default: {} }
|
||||
},
|
||||
})
|
||||
exports.default.render = () => null
|
||||
const renderer = createRenderer<any, any>({
|
||||
createElement: () => ({}), createText: () => ({}), createComment: () => ({}),
|
||||
insert() {}, remove() {}, setElementText() {}, setText() {}, patchProp() {},
|
||||
parentNode: () => null, nextSibling: () => null,
|
||||
})
|
||||
let vm: any
|
||||
const app = renderer.createApp({ setup: () => () => h(exports.default, { ...props, ref: (value: any) => { vm = value } }) })
|
||||
app.mount({})
|
||||
props.visible = true
|
||||
await nextTick()
|
||||
return { vm, requests, revoked, dispose: () => app.unmount() }
|
||||
}
|
||||
const file = { name: 'cat.png', size: 100, type: 'image/png' }
|
||||
|
||||
for (const type of ['Embedding', 'Rerank']) {
|
||||
test(`${type} image mode sends the upload without stale text documents`, async () => {
|
||||
const f = await fixture(type)
|
||||
try {
|
||||
assert.equal(f.vm.hasImageMode, true)
|
||||
assert.equal(f.vm.showFilePicker, false)
|
||||
f.vm.input = 'a cat'
|
||||
f.vm.documentsText = 'old candidate'
|
||||
f.vm.inputMode = 'image'
|
||||
assert.equal(f.vm.canRun, false)
|
||||
f.vm.onNativeFileChange({ target: { files: [file] } })
|
||||
await nextTick()
|
||||
assert.equal(f.vm.imagePreview, 'blob:preview')
|
||||
assert.equal(f.vm.canRun, true)
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[0].file.name, file.name)
|
||||
assert.equal(f.requests[0].input, type === 'Embedding' ? '' : 'a cat')
|
||||
assert.equal(f.requests[0].documents.length, 0)
|
||||
f.vm.clearFile()
|
||||
await nextTick()
|
||||
assert.equal(f.vm.canRun, false)
|
||||
assert.deepEqual(f.revoked, ['blob:preview'])
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
}
|
||||
|
||||
test('embedding similarity sends the query, candidates and an optional image', async () => {
|
||||
const f = await fixture('Embedding')
|
||||
try {
|
||||
f.vm.inputMode = 'similarity'
|
||||
await nextTick()
|
||||
assert.equal(f.vm.useSimilarityMode, true)
|
||||
assert.equal(f.vm.showFilePicker, true)
|
||||
assert.equal(f.vm.canRun, false)
|
||||
f.vm.input = 'a dog'
|
||||
assert.equal(f.vm.canRun, false, 'a query alone has nothing to compare against')
|
||||
f.vm.documentsText = 'a puppy\n\na car'
|
||||
assert.equal(f.vm.canRun, true)
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[0].input, 'a dog')
|
||||
assert.deepEqual([...f.requests[0].documents], ['a puppy', 'a car'])
|
||||
assert.equal(f.requests[0].options.similarity, true)
|
||||
assert.equal(f.requests[0].file, null)
|
||||
|
||||
f.vm.documentsText = ''
|
||||
f.vm.onNativeFileChange({ target: { files: [file] } })
|
||||
assert.equal(f.vm.canRun, true, 'an image is a candidate on its own')
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[1].file.name, file.name)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
|
||||
test('embedding similarity is offered on text-only models without an image picker', async () => {
|
||||
const f = await fixture('Embedding', false)
|
||||
try {
|
||||
assert.equal(f.vm.hasImageMode, false)
|
||||
f.vm.inputMode = 'similarity'
|
||||
await nextTick()
|
||||
assert.equal(f.vm.showFilePicker, false)
|
||||
f.vm.input = 'a dog'
|
||||
f.vm.documentsText = 'a puppy'
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[0].file, null)
|
||||
assert.equal(f.requests[0].options.similarity, true)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
|
||||
test('image-capable chat sends text and an optional image, including image-only input', async () => {
|
||||
const f = await fixture('KnowledgeQA')
|
||||
try {
|
||||
assert.equal(f.vm.showFilePicker, true)
|
||||
assert.equal(f.vm.hasImageMode, false)
|
||||
f.vm.onNativeFileChange({ target: { files: [file] } })
|
||||
assert.equal(f.vm.canRun, true)
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[0].input, '')
|
||||
assert.equal(f.requests[0].file.name, file.name)
|
||||
f.vm.input = 'describe'
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[1].input, 'describe')
|
||||
assert.equal(f.requests[1].file.name, file.name)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
|
||||
for (const type of ['KnowledgeQA', 'Embedding', 'Rerank']) {
|
||||
test(`${type} clears the file when switching to a text-only model of the same type`, async () => {
|
||||
const f = await fixture(type)
|
||||
try {
|
||||
f.vm.inputMode = 'image'
|
||||
f.vm.onNativeFileChange({ target: { files: [file] } })
|
||||
await nextTick()
|
||||
f.vm.selectedModelId = 'text-model'
|
||||
await nextTick()
|
||||
assert.equal(f.vm.file, null)
|
||||
assert.equal(f.vm.inputMode, 'text')
|
||||
assert.equal(f.vm.showFilePicker, false)
|
||||
assert.equal(f.vm.imagePreview, '')
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
}
|
||||
|
||||
test('legacy vision flag enables chat image uploads', async () => {
|
||||
const f = await fixture('KnowledgeQA', false, true)
|
||||
try { assert.equal(f.vm.showFilePicker, true) } finally { f.dispose() }
|
||||
})
|
||||
|
||||
for (const type of ['VLLM', 'ASR']) {
|
||||
test(`${type} still requires its file`, async () => {
|
||||
const f = await fixture(type, false)
|
||||
try {
|
||||
assert.equal(f.vm.showFilePicker, true)
|
||||
assert.equal(f.vm.canRun, false)
|
||||
f.vm.onNativeFileChange({ target: { files: [file] } })
|
||||
assert.equal(f.vm.canRun, true)
|
||||
await f.vm.runDebug()
|
||||
assert.equal(f.requests[0].file.name, file.name)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
}
|
||||
|
|
@ -28,10 +28,13 @@ function deferred<T>() {
|
|||
type Result = { available: boolean; message?: string; dimension?: number }
|
||||
async function fixture(options: {
|
||||
type?: string
|
||||
initialType?: string
|
||||
edit?: boolean
|
||||
check?: (payload: any) => Promise<Result>
|
||||
save?: (payload: any) => Promise<void>
|
||||
providers?: any[]
|
||||
providersByType?: Record<string, any[]>
|
||||
loadProviders?: (type?: string) => Promise<void>
|
||||
resolve?: (params: any) => Promise<any>
|
||||
} = {}) {
|
||||
const requests: any[] = []
|
||||
|
|
@ -39,9 +42,18 @@ async function fixture(options: {
|
|||
const resolves: any[] = []
|
||||
const providers: any[] = options.providers || []
|
||||
const catalogEvents: string[] = []
|
||||
const catalogRequests: Array<{ type?: string; force?: boolean }> = []
|
||||
const loadedProviders = reactive<Record<string, any[]>>({})
|
||||
const providersStore = {
|
||||
ensureLoaded: async (_type?: string, force?: boolean) => { catalogEvents.push(force ? 'refresh' : 'load'); return providers },
|
||||
providersFor: () => providers,
|
||||
ensureLoaded: async (type?: string, force?: boolean) => {
|
||||
catalogEvents.push(force ? 'refresh' : 'load')
|
||||
catalogRequests.push({ type, force })
|
||||
const entries = options.providersByType?.[type || ''] ?? providers
|
||||
await options.loadProviders?.(type)
|
||||
loadedProviders[type || ''] = entries
|
||||
return entries
|
||||
},
|
||||
providersFor: (type?: string) => options.providersByType ? loadedProviders[type || ''] || [] : providers,
|
||||
isLoading: () => false,
|
||||
providerById: (id: string) => providers.find((p) => p.value === id),
|
||||
iconFor: () => '',
|
||||
|
|
@ -52,7 +64,7 @@ async function fixture(options: {
|
|||
const visibility: boolean[] = []
|
||||
const props = reactive<{ visible: boolean; modelType: string; modelData: any }>({
|
||||
visible: false,
|
||||
modelType: options.type || 'chat',
|
||||
modelType: options.initialType || options.type || 'chat',
|
||||
modelData: options.edit ? {
|
||||
id: 'saved-model', modelName: 'old-model', name: '', source: 'remote',
|
||||
baseUrl: 'https://example.com/v1', provider: 'generic', isDefault: false,
|
||||
|
|
@ -109,15 +121,178 @@ async function fixture(options: {
|
|||
}),
|
||||
})
|
||||
app.mount({})
|
||||
props.modelType = options.type || 'chat'
|
||||
props.visible = true
|
||||
await nextTick()
|
||||
await nextTick()
|
||||
Object.assign(vm.formData, { modelName: 'draft-model', baseUrl: 'https://example.com/v1', apiKey: options.edit ? '' : 'draft-key' })
|
||||
vm.formRef = { validate: async () => true }
|
||||
await nextTick()
|
||||
return { vm, props, requests, saves, toasts, visibility, resolves, catalogEvents, close: () => app.unmount() }
|
||||
return { vm, props, requests, saves, toasts, visibility, resolves, catalogEvents, catalogRequests, close: () => app.unmount() }
|
||||
}
|
||||
|
||||
for (const type of ['embedding', 'rerank', 'vllm', 'asr']) {
|
||||
test(`${type}: opening from its settings tab loads the matching providers and models`, async () => {
|
||||
const provider = {
|
||||
value: 'vendor', label: 'Vendor', modelTypes: [type],
|
||||
models: [{ id: `${type}-model`, type: type === 'vllm' ? 'chat' : type }],
|
||||
}
|
||||
const f = await fixture({ initialType: 'chat', type, providersByType: { [type]: [provider] } })
|
||||
try {
|
||||
assert.deepEqual(f.catalogRequests, [{ type, force: true }])
|
||||
assert.equal(f.vm.activeModelType, type)
|
||||
assert.deepEqual(Array.from(f.vm.providerOptions, (p: any) => p.value), ['vendor'])
|
||||
f.vm.formData.provider = 'vendor'
|
||||
f.vm.formData.modelName = ''
|
||||
assert.deepEqual(Array.from(f.vm.catalogModelOptions, (m: any) => m.value), [`${type}-model`])
|
||||
} finally { f.close() }
|
||||
})
|
||||
}
|
||||
|
||||
test('reopening after changing the draft type refreshes the newly selected settings tab', async () => {
|
||||
const f = await fixture()
|
||||
try {
|
||||
await f.vm.selectModelType('rerank')
|
||||
f.props.visible = false
|
||||
await nextTick()
|
||||
f.catalogRequests.length = 0
|
||||
f.props.modelType = 'embedding'
|
||||
f.props.visible = true
|
||||
await nextTick()
|
||||
assert.deepEqual(f.catalogRequests, [{ type: 'embedding', force: true }])
|
||||
assert.equal(f.vm.activeModelType, 'embedding')
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
for (const type of ['chat', 'embedding', 'rerank', 'vllm', 'asr']) {
|
||||
test(`${type}: changing type resets the whole connection form and its results`, async () => {
|
||||
const f = await fixture({ type: type === 'chat' ? 'rerank' : 'chat', providers: [{
|
||||
value: 'vendor', modelTypes: ['chat', 'embedding', 'rerank', 'vllm', 'asr'], models: [],
|
||||
}] })
|
||||
try {
|
||||
Object.assign(f.vm.formData, {
|
||||
source: 'local', provider: 'vendor', modelName: 'old-model', displayName: 'Old display',
|
||||
baseUrl: 'https://example.com/rerank', apiKey: 'old-key', appSecret: 'old-secret',
|
||||
customHeaders: [{ key: 'X-Old', value: 'old' }], extraConfig: { api: 'old-api', remote_model_name: 'old-name' },
|
||||
dimension: 1024, supportsDimensionOverride: true, supportsVision: true, interfaceType: 'ollama',
|
||||
contextWindow: 64000, maxOutputTokens: 4096, maxConcurrency: 8, isDefault: true,
|
||||
thinkingControl: 'old-thinking', spec: { context_window: 64000 }, specCompat: '{"old":true}',
|
||||
})
|
||||
await nextTick()
|
||||
f.vm.remoteChecked = true
|
||||
f.vm.remoteAvailable = true
|
||||
f.vm.remoteMessage = 'old result'
|
||||
f.vm.dimensionChecked = true
|
||||
f.vm.dimensionSuccess = true
|
||||
f.vm.dimensionMessage = 'old dimension'
|
||||
f.vm.modelChecked = true
|
||||
f.vm.modelAvailable = true
|
||||
f.vm.catalogFilled = { contextWindow: 64000 }
|
||||
f.vm.advancedOpen = true
|
||||
f.vm.saveError = 'old error'
|
||||
let cleared = 0
|
||||
f.vm.formRef = { clearValidate: () => { cleared++ } }
|
||||
|
||||
await f.vm.selectModelType(type)
|
||||
await nextTick()
|
||||
assert.equal(f.vm.activeModelType, type)
|
||||
assert.equal(f.vm.formData.source, 'remote')
|
||||
assert.equal(f.vm.formData.provider, 'generic')
|
||||
for (const field of ['modelName', 'displayName', 'baseUrl', 'apiKey', 'appSecret', 'thinkingControl', 'specCompat']) {
|
||||
assert.equal(f.vm.formData[field], '', field)
|
||||
}
|
||||
for (const field of ['dimension', 'interfaceType', 'contextWindow', 'maxOutputTokens', 'maxConcurrency']) {
|
||||
assert.equal(f.vm.formData[field], undefined, field)
|
||||
}
|
||||
for (const field of ['supportsDimensionOverride', 'supportsVision', 'isDefault']) {
|
||||
assert.equal(f.vm.formData[field], false, field)
|
||||
}
|
||||
assert.equal(f.vm.formData.spec, null)
|
||||
assert.equal(f.vm.formData.customHeaders.length, 0)
|
||||
assert.equal(Object.keys(f.vm.formData.extraConfig).length, 0)
|
||||
assert.equal(Object.keys(f.vm.catalogFilled).length, 0)
|
||||
for (const field of ['remoteChecked', 'remoteAvailable', 'remoteStale', 'dimensionChecked', 'dimensionSuccess', 'modelChecked', 'modelAvailable', 'advancedOpen']) {
|
||||
assert.equal(f.vm[field], false, field)
|
||||
}
|
||||
for (const field of ['remoteMessage', 'dimensionMessage', 'saveError']) assert.equal(f.vm[field], '', field)
|
||||
assert.ok(cleared > 0)
|
||||
} finally { f.close() }
|
||||
})
|
||||
}
|
||||
|
||||
test('a delayed type catalog cannot rewrite a newer connection draft', async () => {
|
||||
const pending = deferred<void>()
|
||||
const f = await fixture({
|
||||
loadProviders: type => type === 'embedding' ? pending.promise : Promise.resolve(),
|
||||
providersByType: { asr: [{ value: 'vendor', defaultUrls: { asr: 'https://default.example.com' }, models: [] }] },
|
||||
})
|
||||
try {
|
||||
const earlier = f.vm.selectModelType('embedding')
|
||||
assert.equal(f.vm.formData.apiKey, '', 'reset must not wait for the catalog')
|
||||
await f.vm.selectModelType('asr')
|
||||
f.vm.formData.provider = 'vendor'
|
||||
f.vm.formData.baseUrl = 'https://custom.example.com'
|
||||
f.vm.formData.extraConfig = { region: 'custom' }
|
||||
pending.resolve()
|
||||
await earlier
|
||||
assert.equal(f.vm.activeModelType, 'asr')
|
||||
assert.equal(f.vm.formData.baseUrl, 'https://custom.example.com')
|
||||
assert.equal(f.vm.formData.extraConfig.region, 'custom')
|
||||
} finally { pending.resolve(); f.close() }
|
||||
})
|
||||
|
||||
test('same-type reopening keeps the draft, but another settings tab starts a fresh form', async () => {
|
||||
const f = await fixture()
|
||||
try {
|
||||
f.props.visible = false
|
||||
await nextTick()
|
||||
f.props.visible = true
|
||||
await nextTick()
|
||||
assert.equal(f.vm.formData.modelName, 'draft-model')
|
||||
f.props.visible = false
|
||||
await nextTick()
|
||||
f.props.modelType = 'embedding'
|
||||
f.props.visible = true
|
||||
await nextTick()
|
||||
assert.equal(f.vm.formData.modelName, '')
|
||||
assert.equal(f.vm.formData.apiKey, '')
|
||||
assert.equal(f.vm.formData.baseUrl, '')
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
test('type reset discards pending connection and catalog diagnosis results', async () => {
|
||||
const connection = deferred<Result>()
|
||||
const diagnosis = deferred<any>()
|
||||
const f = await fixture({ check: () => connection.promise, resolve: () => diagnosis.promise })
|
||||
try {
|
||||
const checking = f.vm.checkRemoteAPI()
|
||||
const resolving = f.vm.runResolve()
|
||||
await f.vm.selectModelType('asr')
|
||||
connection.resolve({ available: true })
|
||||
diagnosis.resolve({ cataloged: true, model: { id: 'old-model' } })
|
||||
await Promise.all([checking, resolving])
|
||||
assert.equal(f.vm.remoteChecked, false)
|
||||
assert.equal(f.vm.remoteStale, false)
|
||||
assert.equal(f.vm.checking, false)
|
||||
assert.equal(f.vm.resolved, null)
|
||||
assert.equal(f.vm.resolving, false)
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
test('selecting the current type, editing, or saving does not reset the draft', async () => {
|
||||
for (const state of ['same-type', 'editing', 'saving']) {
|
||||
const f = await fixture({ edit: state === 'editing' })
|
||||
try {
|
||||
const original = JSON.stringify(f.vm.formData)
|
||||
const requests = f.catalogRequests.length
|
||||
if (state === 'saving') f.vm.saving = true
|
||||
await f.vm.selectModelType(state === 'same-type' ? 'chat' : 'asr')
|
||||
assert.equal(JSON.stringify(f.vm.formData), original, state)
|
||||
assert.equal(f.catalogRequests.length, requests, state)
|
||||
} finally { f.close() }
|
||||
}
|
||||
})
|
||||
|
||||
for (const type of ['chat', 'embedding', 'rerank', 'vllm', 'asr']) {
|
||||
test(`${type}: connection tests use unsaved form values without saving`, async () => {
|
||||
const f = await fixture({ type })
|
||||
|
|
@ -791,6 +966,54 @@ test('switching vendor clears a connection result that described the old one', a
|
|||
} finally { f.close() }
|
||||
})
|
||||
|
||||
test('ASR vendor changes replace the previous URL and show the selected catalog model as the hint', async () => {
|
||||
const f = await fixture({ type: 'asr', providers: [
|
||||
{ value: 'openai', defaultUrls: { asr: 'https://api.openai.com/v1' }, models: [{ id: 'whisper-1', type: 'asr' }] },
|
||||
{ value: 'zhipu', defaultUrls: { asr: 'https://open.bigmodel.cn/api/paas/v4' }, models: [{ id: 'glm-asr-2512', type: 'asr' }] },
|
||||
{ value: 'generic', models: [] },
|
||||
] })
|
||||
try {
|
||||
for (const [vendor, url, hint] of [
|
||||
['openai', 'https://api.openai.com/v1', 'whisper-1'],
|
||||
['zhipu', 'https://open.bigmodel.cn/api/paas/v4', 'glm-asr-2512'],
|
||||
['generic', '', 'model.editor.modelNamePlaceholder.remoteAsr'],
|
||||
]) {
|
||||
f.vm.formData.provider = vendor
|
||||
f.vm.handleProviderChange(vendor)
|
||||
assert.equal(f.vm.formData.baseUrl, url, vendor)
|
||||
assert.equal(f.vm.getModelNamePlaceholder(), hint, vendor)
|
||||
}
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
test('ASR inherits the vendor chat base URL from older catalog responses', async () => {
|
||||
const f = await fixture({ type: 'asr', providers: [
|
||||
{ value: 'mimo', defaultUrls: { chat: 'https://api.xiaomimimo.com/v1' }, models: [{ id: 'mimo-v2.5-asr', type: 'asr' }] },
|
||||
{ value: 'zhipu', defaultUrls: { chat: 'https://open.bigmodel.cn/api/paas/v4' }, models: [] },
|
||||
{ value: 'custom', defaultUrls: { chat: 'https://chat.example.com', asr: 'https://speech.example.com' }, models: [] },
|
||||
] })
|
||||
try {
|
||||
for (const [vendor, url] of [
|
||||
['mimo', 'https://api.xiaomimimo.com/v1'],
|
||||
['zhipu', 'https://open.bigmodel.cn/api/paas/v4'],
|
||||
['custom', 'https://speech.example.com'],
|
||||
]) {
|
||||
f.vm.formData.provider = vendor
|
||||
f.vm.handleProviderChange(vendor)
|
||||
assert.equal(f.vm.formData.baseUrl, url, vendor)
|
||||
}
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
test('an explicitly empty type URL clears the old endpoint without inheriting the chat URL', async () => {
|
||||
const f = await fixture({ type: 'asr', providers: [{ value: 'custom', defaultUrls: { chat: 'https://chat.example.com', asr: '' } }] })
|
||||
try {
|
||||
f.vm.formData.provider = 'custom'
|
||||
f.vm.handleProviderChange('custom')
|
||||
assert.equal(f.vm.formData.baseUrl, '')
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
||||
// A VLM entry is a chat model that accepts images, so the backend — which
|
||||
// already scopes the provider list to the requested model type — returns it
|
||||
// typed "chat". Re-filtering on that type in the editor emptied the picker
|
||||
|
|
@ -844,7 +1067,7 @@ for (const [type, modelName] of [['chat', 'chat'], ['embedding', 'embedding'], [
|
|||
})
|
||||
}
|
||||
|
||||
test('switching WeKnora Cloud model types updates the managed choices', async () => {
|
||||
test('reselecting WeKnora Cloud after a type reset offers the new managed choices', async () => {
|
||||
const f = await fixture({ providers: [cloudProvider] })
|
||||
try {
|
||||
f.vm.formData.provider = 'weknoracloud'
|
||||
|
|
@ -853,6 +1076,9 @@ test('switching WeKnora Cloud model types updates the managed choices', async ()
|
|||
f.vm.formData.modelName = 'chat'
|
||||
await f.vm.selectModelType('vllm')
|
||||
assert.equal(f.vm.formData.modelName, '')
|
||||
assert.equal(f.vm.formData.provider, 'generic')
|
||||
f.vm.formData.provider = 'weknoracloud'
|
||||
f.vm.handleProviderChange('weknoracloud')
|
||||
assert.deepEqual(Array.from(f.vm.catalogModelOptions, (o: any) => o.value), ['vlm'])
|
||||
} finally { f.close() }
|
||||
})
|
||||
|
|
|
|||
|
|
@ -1009,6 +1009,11 @@ export default {
|
|||
chunkLoadFailed: 'Failed to load chunks'
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "Image indexing embedding model",
|
||||
imageIndexInherited: "Image indexing uses the embedding model assigned to this knowledge base. Change it in knowledge base settings.",
|
||||
imageVectorIndex: "Image vector indexing for this import",
|
||||
imageVectorIndexHint: "Generate image vectors for this upload or reparse without changing knowledge base defaults. Retrieval still follows the knowledge base image vector search setting.",
|
||||
imageVectorRequiresIndex: "Enable vector indexing in knowledge base settings first.",
|
||||
documentSummary: "Document summary",
|
||||
documentSummaryDescription: "Choose whether to automatically summarize documents in this import.",
|
||||
generateSummary: "Generate document summaries",
|
||||
|
|
@ -5508,6 +5513,14 @@ export default {
|
|||
documentsPlaceholder: 'Enter one candidate document per line',
|
||||
documentsHint: 'Each non-empty line is sent as a separate document to the ReRank model',
|
||||
imageFile: 'Image file',
|
||||
similarityMode: 'Similarity',
|
||||
similarityQuery: 'Query',
|
||||
similarityQueryPlaceholder: 'Enter a query, e.g. a dog running on grass',
|
||||
similarityCandidates: 'Candidates',
|
||||
similarityCandidatesPlaceholder: 'One candidate text per line',
|
||||
similarityHint: 'The query is embedded as a search query and the candidates as documents, as in retrieval; results are sorted by cosine similarity, highest first',
|
||||
similarityImage: 'Candidate image (optional)',
|
||||
similarityImageCandidate: 'Image',
|
||||
audioFile: 'Audio file',
|
||||
chooseFile: 'Choose file',
|
||||
parameters: 'Request parameters',
|
||||
|
|
|
|||
|
|
@ -1009,6 +1009,11 @@ export default {
|
|||
chunkLoadFailed: 'チャンクの読み込みに失敗しました'
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "画像インデックス用 Embedding モデル",
|
||||
imageIndexInherited: "画像インデックスにはナレッジベースの Embedding モデルを使用します。変更はナレッジベース設定で行ってください。",
|
||||
imageVectorIndex: "今回の画像ベクトル索引",
|
||||
imageVectorIndexHint: "今回のアップロードまたは再解析で画像ベクトルを生成します。ナレッジベースの既定設定は変更しません。検索への利用は画像ベクトル検索設定に従います。",
|
||||
imageVectorRequiresIndex: "先にナレッジベース設定でベクトル索引を有効にしてください。",
|
||||
documentSummary: "ドキュメントの要約",
|
||||
documentSummaryDescription: "今回インポートするドキュメントの要約を自動生成するか選択します。",
|
||||
generateSummary: "ドキュメントの要約を生成",
|
||||
|
|
@ -5508,6 +5513,14 @@ export default {
|
|||
documentsPlaceholder: '候補ドキュメントを1行に1件ずつ入力してください',
|
||||
documentsHint: '空でない各行が個別のドキュメントとしてリランクモデルに送信されます',
|
||||
imageFile: '画像ファイル',
|
||||
similarityMode: '類似度',
|
||||
similarityQuery: 'クエリ',
|
||||
similarityQueryPlaceholder: 'クエリを入力(例:芝生を走る犬)',
|
||||
similarityCandidates: '候補テキスト',
|
||||
similarityCandidatesPlaceholder: '候補テキストを1行に1件ずつ入力してください',
|
||||
similarityHint: 'クエリは検索クエリとして、候補はドキュメントとしてベクトル化されます(実際の検索と同じ)。結果はコサイン類似度の高い順に並びます',
|
||||
similarityImage: '候補画像(任意)',
|
||||
similarityImageCandidate: '画像',
|
||||
audioFile: '音声ファイル',
|
||||
chooseFile: 'ファイルを選択',
|
||||
parameters: 'リクエストパラメータ',
|
||||
|
|
|
|||
|
|
@ -2592,6 +2592,14 @@ export default {
|
|||
documentsPlaceholder: '후보 문서를 한 줄에 하나씩 입력하세요',
|
||||
documentsHint: '비어 있지 않은 각 줄은 ReRank 모델에 별도 문서로 전송됩니다',
|
||||
imageFile: '이미지 파일',
|
||||
similarityMode: '유사도',
|
||||
similarityQuery: '쿼리',
|
||||
similarityQueryPlaceholder: '쿼리를 입력하세요. 예: 잔디 위를 달리는 개',
|
||||
similarityCandidates: '후보 텍스트',
|
||||
similarityCandidatesPlaceholder: '한 줄에 하나의 후보 텍스트를 입력하세요',
|
||||
similarityHint: '쿼리는 검색 쿼리로, 후보는 문서로 벡터화됩니다(실제 검색과 동일). 결과는 코사인 유사도가 높은 순으로 정렬됩니다',
|
||||
similarityImage: '후보 이미지(선택)',
|
||||
similarityImageCandidate: '이미지',
|
||||
audioFile: '오디오 파일',
|
||||
chooseFile: '파일 선택',
|
||||
parameters: '요청 매개변수',
|
||||
|
|
@ -7009,6 +7017,11 @@ export default {
|
|||
}
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "이미지 인덱싱 Embedding 모델",
|
||||
imageIndexInherited: "이미지 인덱싱에는 지식 베이스에 연결된 Embedding 모델을 사용합니다. 변경하려면 지식 베이스 설정으로 이동하세요.",
|
||||
imageVectorIndex: "이번 가져오기의 이미지 벡터 인덱싱",
|
||||
imageVectorIndexHint: "이번 업로드 또는 재분석에서 이미지 벡터를 생성하며 지식 베이스 기본 설정은 변경하지 않습니다. 검색 사용 여부는 지식 베이스의 이미지 벡터 검색 설정을 따릅니다.",
|
||||
imageVectorRequiresIndex: "먼저 지식 베이스 설정에서 벡터 인덱싱을 활성화하세요.",
|
||||
documentSummary: "문서 요약",
|
||||
documentSummaryDescription: "이번에 가져오는 문서의 요약을 자동으로 생성할지 선택합니다.",
|
||||
generateSummary: "문서 요약 생성",
|
||||
|
|
|
|||
|
|
@ -2592,6 +2592,14 @@ export default {
|
|||
documentsPlaceholder: 'Введите один документ на строку',
|
||||
documentsHint: 'Каждая непустая строка отправляется в модель ReRank как отдельный документ',
|
||||
imageFile: 'Файл изображения',
|
||||
similarityMode: 'Сходство',
|
||||
similarityQuery: 'Запрос',
|
||||
similarityQueryPlaceholder: 'Введите запрос, например: собака бежит по траве',
|
||||
similarityCandidates: 'Кандидаты',
|
||||
similarityCandidatesPlaceholder: 'По одному тексту-кандидату в строке',
|
||||
similarityHint: 'Запрос векторизуется как поисковый запрос, кандидаты — как документы, как при реальном поиске; результаты отсортированы по косинусному сходству по убыванию',
|
||||
similarityImage: 'Изображение-кандидат (необязательно)',
|
||||
similarityImageCandidate: 'Изображение',
|
||||
audioFile: 'Аудиофайл',
|
||||
chooseFile: 'Выбрать файл',
|
||||
parameters: 'Параметры запроса',
|
||||
|
|
@ -7009,6 +7017,11 @@ export default {
|
|||
}
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "Модель эмбеддингов для индексации изображений",
|
||||
imageIndexInherited: "Индексация изображений использует модель эмбеддингов базы знаний. Изменить её можно в настройках базы знаний.",
|
||||
imageVectorIndex: "Индексация изображений для этого импорта",
|
||||
imageVectorIndexHint: "Создавать векторы изображений при этой загрузке или повторном разборе без изменения настроек базы знаний. Поиск по ним по-прежнему зависит от настройки поиска изображений базы знаний.",
|
||||
imageVectorRequiresIndex: "Сначала включите векторную индексацию в настройках базы знаний.",
|
||||
documentSummary: "Сводка документа",
|
||||
documentSummaryDescription: "Выберите, создавать ли сводки документов при этом импорте.",
|
||||
generateSummary: "Создавать сводки документов",
|
||||
|
|
|
|||
|
|
@ -2594,6 +2594,14 @@ export default {
|
|||
documentsPlaceholder: '每行输入一个候选文档',
|
||||
documentsHint: '每个非空行会作为一个独立文档发送给 ReRank 模型',
|
||||
imageFile: '图片文件',
|
||||
similarityMode: '相似度',
|
||||
similarityQuery: '查询文本',
|
||||
similarityQueryPlaceholder: '输入一条查询,例如:草地上奔跑的狗',
|
||||
similarityCandidates: '候选文本',
|
||||
similarityCandidatesPlaceholder: '每行输入一个候选文本',
|
||||
similarityHint: '查询按检索查询方式向量化,候选按文档方式向量化,与线上检索一致;结果按余弦相似度从高到低排列',
|
||||
similarityImage: '候选图片(可选)',
|
||||
similarityImageCandidate: '图片',
|
||||
audioFile: '音频文件',
|
||||
chooseFile: '选择文件',
|
||||
parameters: '请求参数',
|
||||
|
|
@ -7011,6 +7019,11 @@ export default {
|
|||
}
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "图像索引 Embedding 模型",
|
||||
imageIndexInherited: "图像索引使用知识库绑定的 Embedding 模型;如需更换,请前往知识库设置。",
|
||||
imageVectorIndex: "本次图片向量索引",
|
||||
imageVectorIndexHint: "为本次上传或重新解析的图片生成向量,不改变知识库默认配置。是否参与检索仍由知识库的“图片向量检索”开关决定。",
|
||||
imageVectorRequiresIndex: "请先在知识库设置中开启向量索引。",
|
||||
documentSummary: "文档摘要",
|
||||
documentSummaryDescription: "为本次导入的文档选择是否自动生成摘要。",
|
||||
generateSummary: "生成文档摘要",
|
||||
|
|
|
|||
|
|
@ -2594,6 +2594,14 @@ export default {
|
|||
documentsPlaceholder: '每行輸入一個候選文件',
|
||||
documentsHint: '每個非空行會作為一個獨立文件傳送給 ReRank 模型',
|
||||
imageFile: '圖片檔案',
|
||||
similarityMode: '相似度',
|
||||
similarityQuery: '查詢文字',
|
||||
similarityQueryPlaceholder: '輸入一條查詢,例如:草地上奔跑的狗',
|
||||
similarityCandidates: '候選文字',
|
||||
similarityCandidatesPlaceholder: '每行輸入一個候選文字',
|
||||
similarityHint: '查詢按檢索查詢方式向量化,候選按文件方式向量化,與線上檢索一致;結果按餘弦相似度由高到低排列',
|
||||
similarityImage: '候選圖片(可選)',
|
||||
similarityImageCandidate: '圖片',
|
||||
audioFile: '音訊檔案',
|
||||
chooseFile: '選擇檔案',
|
||||
parameters: '請求參數',
|
||||
|
|
@ -7011,6 +7019,11 @@ export default {
|
|||
}
|
||||
},
|
||||
uploadConfirm: {
|
||||
imageEmbeddingModel: "圖像索引 Embedding 模型",
|
||||
imageIndexInherited: "圖像索引使用知識庫綁定的 Embedding 模型;如需更換,請前往知識庫設定。",
|
||||
imageVectorIndex: "本次圖片向量索引",
|
||||
imageVectorIndexHint: "為本次上傳或重新解析的圖片產生向量,不改變知識庫預設設定。是否參與檢索仍由知識庫的「圖片向量檢索」開關決定。",
|
||||
imageVectorRequiresIndex: "請先在知識庫設定中開啟向量索引。",
|
||||
documentSummary: "文件摘要",
|
||||
documentSummaryDescription: "為本次匯入的文件選擇是否自動生成摘要。",
|
||||
generateSummary: "生成文件摘要",
|
||||
|
|
|
|||
|
|
@ -74,6 +74,8 @@ export interface KnowledgeProcessOverrides {
|
|||
// Per-upload override of the knowledge base's attribute-observed image
|
||||
// pipeline switch.
|
||||
image_attrs_enabled?: boolean
|
||||
// Controls indexing for this document, without changing KB retrieval.
|
||||
image_vector_enabled?: boolean
|
||||
// Per-task attribute -> work table; merged per action key on top of the
|
||||
// knowledge base's table.
|
||||
image_actions?: ImageActionsConfig
|
||||
|
|
|
|||
|
|
@ -421,6 +421,35 @@
|
|||
/>
|
||||
</div>
|
||||
</div>
|
||||
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
|
||||
<div class="setting-info">
|
||||
<label>{{ t('uploadConfirm.imageEmbeddingModel') }}</label>
|
||||
<p class="desc">{{ t('uploadConfirm.imageIndexInherited') }}</p>
|
||||
<p v-if="uiState.imageVectorEnabled && !embeddingTakesImages" class="warn">
|
||||
{{ t('knowledgeEditor.advanced.multimodal.imageVectorModelUnsupported') }}
|
||||
</p>
|
||||
</div>
|
||||
<div class="setting-control">
|
||||
<ModelSelector
|
||||
model-type="Embedding"
|
||||
:selected-model-id="kbInfo?.embedding_model_id || ''"
|
||||
:all-models="allModels"
|
||||
disabled
|
||||
:placeholder="t('uploadConfirm.notSet')"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
|
||||
<div class="setting-info">
|
||||
<label>{{ t('uploadConfirm.imageVectorIndex') }}</label>
|
||||
<p class="desc">{{ t('uploadConfirm.imageVectorIndexHint') }}</p>
|
||||
<p v-if="!kbInfo?.indexing_strategy?.vector_enabled" class="warn">{{ t('uploadConfirm.imageVectorRequiresIndex') }}</p>
|
||||
</div>
|
||||
<div class="setting-control">
|
||||
<t-switch v-model="uiState.imageVectorEnabled" :disabled="!kbInfo?.indexing_strategy?.vector_enabled" size="medium" />
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div v-if="uiState.multimodalConfig.enabled" class="setting-row">
|
||||
<div class="setting-info">
|
||||
<label>{{ t('knowledgeEditor.advanced.multimodal.imageAttrsLabel') }}</label>
|
||||
|
|
@ -705,6 +734,7 @@ interface UploadUIState {
|
|||
chunkingConfig: ChunkingUIConfig
|
||||
multimodalConfig: { enabled: boolean; vllmModelId: string; descriptionLanguage?: string; customInstructions?: string }
|
||||
imageAttrsEnabled: boolean
|
||||
imageVectorEnabled: boolean
|
||||
imageActions: ImageActionsConfig
|
||||
asrConfig: { enabled: boolean; modelId: string; language: string }
|
||||
questionGenerationConfig: { enabled: boolean; questionCount: number; customInstructions?: string }
|
||||
|
|
@ -775,6 +805,11 @@ const editorResources = useEditorResourcesStore()
|
|||
const uiStore = useUIStore()
|
||||
|
||||
const allModels = ref<any[]>([])
|
||||
const embeddingTakesImages = computed(() => {
|
||||
const model = allModels.value.find(m => m.id === props.kbInfo?.embedding_model_id)
|
||||
// A shared model may be absent from the model list; the backend resolves it.
|
||||
return !model || !!model.capabilities?.input?.includes('image')
|
||||
})
|
||||
const localFiles = ref<File[]>([])
|
||||
const localUrls = ref<string[]>([])
|
||||
const availableTags = ref<Array<{ id: string; name: string }>>([])
|
||||
|
|
@ -1197,6 +1232,7 @@ function createDefaultUIState(): UploadUIState {
|
|||
},
|
||||
multimodalConfig: { enabled: false, vllmModelId: '', descriptionLanguage: '', customInstructions: '' },
|
||||
imageAttrsEnabled: false,
|
||||
imageVectorEnabled: false,
|
||||
imageActions: mergeImageActions(),
|
||||
asrConfig: { enabled: false, modelId: '', language: '' },
|
||||
questionGenerationConfig: { enabled: true, questionCount: 3, customInstructions: '' },
|
||||
|
|
@ -1242,6 +1278,7 @@ function initFromKbInfo(kb: any) {
|
|||
},
|
||||
// 默认跟随知识库的图片属性观察设置;用户可对本次任务单独覆盖。
|
||||
imageAttrsEnabled: !!kb.image_processing_config?.image_attrs_enabled,
|
||||
imageVectorEnabled: !!kb.image_processing_config?.image_vector_enabled,
|
||||
imageActions: mergeImageActions(kb.image_processing_config?.image_actions),
|
||||
asrConfig: {
|
||||
enabled: !!kb.asr_config?.enabled,
|
||||
|
|
@ -1292,6 +1329,7 @@ function buildProcessOverrides(): KnowledgeProcessOverrides {
|
|||
},
|
||||
enable_multimodel: state.multimodalConfig.enabled,
|
||||
image_attrs_enabled: state.imageAttrsEnabled,
|
||||
image_vector_enabled: state.imageVectorEnabled,
|
||||
image_actions: {
|
||||
ocr: {
|
||||
// The KB's own conditions (mergeImageActions keeps a custom list), so
|
||||
|
|
@ -1357,6 +1395,7 @@ function applyOverridesToState(o?: KnowledgeProcessOverrides | null) {
|
|||
if (o.parser_engine_rules) s.chunkingConfig.parserEngineRules = o.parser_engine_rules
|
||||
if (o.enable_multimodel != null) s.multimodalConfig.enabled = o.enable_multimodel
|
||||
if (o.image_attrs_enabled != null) s.imageAttrsEnabled = o.image_attrs_enabled
|
||||
if (o.image_vector_enabled != null) s.imageVectorEnabled = o.image_vector_enabled
|
||||
if (o.image_actions) s.imageActions = mergeImageActions(o.image_actions)
|
||||
if (o.vlm_config) {
|
||||
if (o.vlm_config.enabled != null) s.multimodalConfig.enabled = o.vlm_config.enabled
|
||||
|
|
|
|||
|
|
@ -0,0 +1,90 @@
|
|||
import assert from 'node:assert/strict'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { createRequire } from 'node:module'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { runInNewContext } from 'node:vm'
|
||||
import test from 'node:test'
|
||||
import { compileScript, parse } from '@vue/compiler-sfc'
|
||||
import ts from 'typescript'
|
||||
import { createRenderer, h } from 'vue'
|
||||
|
||||
const require = createRequire(import.meta.url)
|
||||
const filename = fileURLToPath(new URL('./UploadConfirmDialog.vue', import.meta.url))
|
||||
const { descriptor } = parse(readFileSync(filename, 'utf8'), { filename })
|
||||
const script = compileScript(descriptor, { id: 'upload-image-vector-test' }).content
|
||||
.replace('__expose();', '')
|
||||
.replace('return __returned__', '__expose(__returned__); return __returned__')
|
||||
const compiled = ts.transpileModule(script, {
|
||||
compilerOptions: { module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2022 },
|
||||
}).outputText
|
||||
|
||||
function fixture(kbInfo: any) {
|
||||
const exports: any = {}
|
||||
runInNewContext(compiled, {
|
||||
exports, console,
|
||||
require(name: string) {
|
||||
if (name === 'vue') return require('vue')
|
||||
if (name === 'vue-i18n') return { useI18n: () => ({ t: (key: string) => key, te: () => false }) }
|
||||
if (name === '@/stores/chatResources') return { useChatResourcesStore: () => ({}) }
|
||||
if (name === '@/stores/editorResources') return { useEditorResourcesStore: () => ({}) }
|
||||
if (name === '@/stores/ui') return { useUIStore: () => ({}) }
|
||||
if (name === '@/api/knowledge-base') return { mergeImageActions: () => ({ ocr: { on: [], on_unobserved: true } }) }
|
||||
return { default: {} }
|
||||
},
|
||||
})
|
||||
exports.default.render = () => null
|
||||
const renderer = createRenderer<any, any>({
|
||||
createElement: () => ({}), createText: () => ({}), createComment: () => ({}),
|
||||
insert() {}, remove() {}, setElementText() {}, setText() {}, patchProp() {},
|
||||
parentNode: () => null, nextSibling: () => null,
|
||||
})
|
||||
let vm: any
|
||||
const app = renderer.createApp({ setup: () => () => h(exports.default, { visible: false, kbInfo, ref: (value: any) => { vm = value } }) })
|
||||
app.mount({})
|
||||
vm.initFromKbInfo(kbInfo)
|
||||
return { vm, dispose: () => app.unmount() }
|
||||
}
|
||||
|
||||
for (const enabled of [false, true]) {
|
||||
test(`upload inherits ${enabled} and submits the explicit opposite without changing the KB`, () => {
|
||||
const kb = { embedding_model_id: 'shared-embedding', image_processing_config: { image_vector_enabled: enabled } }
|
||||
const f = fixture(kb)
|
||||
try {
|
||||
assert.equal(f.vm.uiState.imageVectorEnabled, enabled)
|
||||
f.vm.uiState.imageVectorEnabled = !enabled
|
||||
const payload = JSON.parse(JSON.stringify(f.vm.buildProcessOverrides()))
|
||||
assert.equal(payload.image_vector_enabled, !enabled)
|
||||
assert.equal(kb.image_processing_config.image_vector_enabled, enabled)
|
||||
assert.equal('embedding_model_id' in payload, false)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
test(`reparse restores an explicit ${enabled} over the KB default`, () => {
|
||||
const f = fixture({ image_processing_config: { image_vector_enabled: !enabled } })
|
||||
try {
|
||||
f.vm.applyOverridesToState({ image_vector_enabled: enabled })
|
||||
assert.equal(f.vm.uiState.imageVectorEnabled, enabled)
|
||||
assert.equal(f.vm.buildProcessOverrides().image_vector_enabled, enabled)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
}
|
||||
|
||||
test('old documents inherit the KB value and old KBs default off', () => {
|
||||
const f = fixture({ image_processing_config: { image_vector_enabled: true } })
|
||||
try {
|
||||
f.vm.applyOverridesToState({ summary_enabled: false })
|
||||
assert.equal(f.vm.uiState.imageVectorEnabled, true)
|
||||
f.vm.initFromKbInfo({})
|
||||
assert.equal(f.vm.uiState.imageVectorEnabled, false)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
|
||||
test('image capability warning uses the KB embedding model and tolerates shared models absent from the list', () => {
|
||||
const f = fixture({ embedding_model_id: 'embedding' })
|
||||
try {
|
||||
assert.equal(f.vm.embeddingTakesImages, true)
|
||||
f.vm.allModels = [{ id: 'embedding', capabilities: { input: ['text'] } }]
|
||||
assert.equal(f.vm.embeddingTakesImages, false)
|
||||
f.vm.allModels = [{ id: 'embedding', capabilities: { input: ['text', 'image'] } }]
|
||||
assert.equal(f.vm.embeddingTakesImages, true)
|
||||
} finally { f.dispose() }
|
||||
})
|
||||
|
|
@ -9,6 +9,7 @@ import (
|
|||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/http/httptrace"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
|
@ -31,9 +32,14 @@ import (
|
|||
|
||||
func newTestClient(t *testing.T, url string) *osapi.Client {
|
||||
t.Helper()
|
||||
// httptest.Server.Close touches http.DefaultTransport. Give each client
|
||||
// its own pool so parallel server cleanup cannot interrupt its requests.
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
t.Cleanup(transport.CloseIdleConnections)
|
||||
c, err := osapi.NewClient(osapi.Config{
|
||||
Client: opensearch.Config{
|
||||
Addresses: []string{url},
|
||||
Transport: transport,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
|
|
@ -42,6 +48,32 @@ func newTestClient(t *testing.T, url string) *osapi.Client {
|
|||
return c
|
||||
}
|
||||
|
||||
// Closing any httptest server also closes idle connections on the global
|
||||
// transport. A sibling test must not be able to disrupt this client's pool.
|
||||
func TestNewTestClient_IsolatesTransportFromOtherServers(t *testing.T) {
|
||||
t.Parallel()
|
||||
ts := httptest.NewServer(versionHandler("opensearch", "3.3.2"))
|
||||
defer ts.Close()
|
||||
client := newTestClient(t, ts.URL)
|
||||
if err := probeVersion(context.Background(), client); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
sibling := httptest.NewServer(http.NotFoundHandler())
|
||||
sibling.Close()
|
||||
|
||||
var reused bool
|
||||
ctx := httptrace.WithClientTrace(context.Background(), &httptrace.ClientTrace{
|
||||
GotConn: func(info httptrace.GotConnInfo) { reused = info.Reused },
|
||||
})
|
||||
if err := probeVersion(ctx, client); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !reused {
|
||||
t.Fatal("closing another test server discarded this client's idle connection")
|
||||
}
|
||||
}
|
||||
|
||||
// newTestRepo builds a Repository directly (bypassing NewRepository's probe
|
||||
// calls) for unit tests that exercise post-construction behavior. The
|
||||
// caller supplies the http handler that simulates the cluster.
|
||||
|
|
|
|||
|
|
@ -691,8 +691,8 @@ func (s *ImageMultimodalService) indexChunks(
|
|||
// indexImageVector embeds the image itself with a multimodal embedding model
|
||||
// and stores the vector under an image_vector chunk, so a query can find the
|
||||
// image by what it shows even where the caption left that out. It returns a
|
||||
// short status for the trace, "" when the knowledge base has not opted in
|
||||
// (ImageProcessingConfig.ImageVectorEnabled) or the model takes no images.
|
||||
// short status for the trace, "" when image indexing is disabled for this
|
||||
// document (falling back to the KB setting) or the model takes no images.
|
||||
//
|
||||
// The caption and OCR chunks are already indexed, so a failure here only
|
||||
// loses the extra recall and never fails the task.
|
||||
|
|
@ -700,9 +700,13 @@ func (s *ImageMultimodalService) indexImageVector(
|
|||
ctx context.Context, kb *types.KnowledgeBase, payload types.ImageMultimodalPayload, img []byte,
|
||||
chunks []*types.Chunk, model embedding.Embedder, engine *retriever.CompositeRetrieveEngine,
|
||||
) string {
|
||||
// Off unless the knowledge base asked for it: a model that takes images
|
||||
// is often chosen for text alone, and this costs a call per image.
|
||||
if !kb.IsImageVectorEnabled() {
|
||||
// The task's explicit choice wins over the KB default, including false.
|
||||
// A vector index is still required; old tasks inherit the KB setting.
|
||||
enabled := kb.IsImageVectorEnabled()
|
||||
if payload.ImageVectorEnabled != nil {
|
||||
enabled = kb.IsVectorEnabled() && *payload.ImageVectorEnabled
|
||||
}
|
||||
if !enabled {
|
||||
return ""
|
||||
}
|
||||
imageModel, ok := embedding.AsImageEmbedder(model)
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package service
|
|||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"image"
|
||||
"image/png"
|
||||
"testing"
|
||||
|
|
@ -239,3 +240,65 @@ func TestIndexImageVectorNeedsTheKnowledgeBaseToOptIn(t *testing.T) {
|
|||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Follow the saved per-document choice through metadata, config resolution,
|
||||
// queue serialization and the worker, without changing KB retrieval defaults.
|
||||
func TestImageVectorUploadOverrideReachesWorker(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
kbEnabled, vectorEnabled bool
|
||||
override *bool
|
||||
want bool
|
||||
}{
|
||||
{"enable one upload", false, true, processConfigBoolPtr(true), true},
|
||||
{"disable one upload", true, true, processConfigBoolPtr(false), false},
|
||||
{"inherit enabled", true, true, nil, true},
|
||||
{"inherit disabled", false, true, nil, false},
|
||||
{"no vector index", true, false, processConfigBoolPtr(true), false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
knowledge, kb, images, chunks := slotReleaseFixture(1)
|
||||
knowledge.ID = "image-vector-" + tc.name
|
||||
useLiteCounter(t, knowledge.ID)
|
||||
kb.IndexingStrategy.VectorEnabled = tc.vectorEnabled
|
||||
kb.ImageProcessingConfig.ImageVectorEnabled = tc.kbEnabled
|
||||
require.NoError(t, knowledge.SetProcessOverrides(&types.KnowledgeProcessOverrides{
|
||||
ImageVectorEnabled: tc.override,
|
||||
}))
|
||||
overrides, err := knowledge.ProcessOverrides()
|
||||
require.NoError(t, err)
|
||||
eff := ResolveProcessConfig(kb, overrides)
|
||||
wantSetting := tc.kbEnabled
|
||||
if tc.override != nil {
|
||||
wantSetting = *tc.override
|
||||
}
|
||||
require.Equal(t, wantSetting, eff.ImageVectorEnabled)
|
||||
enqueuer := &metadataUpdateTaskEnqueuer{}
|
||||
enqueueService := &knowledgeService{task: enqueuer}
|
||||
require.NoError(t, enqueueService.enqueueImageMultimodalTasks(
|
||||
context.Background(), knowledge, kb, images, chunks, nil,
|
||||
))
|
||||
require.Len(t, enqueuer.tasks, 1)
|
||||
var payload types.ImageMultimodalPayload
|
||||
require.NoError(t, json.Unmarshal(enqueuer.tasks[0].Payload(), &payload))
|
||||
require.NotNil(t, payload.ImageVectorEnabled, "explicit false must survive JSON serialization")
|
||||
require.Equal(t, wantSetting, *payload.ImageVectorEnabled)
|
||||
// The queued choice survives a later change to the KB default.
|
||||
kb.ImageProcessingConfig.ImageVectorEnabled = !tc.kbEnabled
|
||||
svc, repo, recorder, engine := imageVectorFixture(t)
|
||||
model := &imageModel{dims: 3}
|
||||
status := svc.indexImageVector(
|
||||
context.Background(), kb, payload, testPNG(t), multimodalChunks(), model, engine,
|
||||
)
|
||||
if tc.want {
|
||||
require.Equal(t, "indexed", status)
|
||||
require.Len(t, model.images, 1)
|
||||
require.Len(t, recorder.rows, 1)
|
||||
} else {
|
||||
require.Empty(t, status)
|
||||
require.Empty(t, model.images)
|
||||
require.Empty(t, repo.chunks)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4443,20 +4443,21 @@ func (s *knowledgeService) enqueueImageMultimodalTasks(
|
|||
}
|
||||
|
||||
payload := types.ImageMultimodalPayload{
|
||||
TenantID: knowledge.TenantID,
|
||||
KnowledgeID: knowledge.ID,
|
||||
KnowledgeBaseID: kb.ID,
|
||||
ChunkID: chunkID,
|
||||
ImageURL: img.ServingURL,
|
||||
EnableOCR: true,
|
||||
EnableCaption: true,
|
||||
ImageAttrsEnabled: eff.ImageAttrsEnabled,
|
||||
ImageActions: eff.ImageActions,
|
||||
Language: lang,
|
||||
ImageSourceType: metadata["image_source_type"],
|
||||
Attempt: attempt,
|
||||
ImageIndex: idx,
|
||||
SourceLocators: img.SourceLocators,
|
||||
TenantID: knowledge.TenantID,
|
||||
KnowledgeID: knowledge.ID,
|
||||
KnowledgeBaseID: kb.ID,
|
||||
ChunkID: chunkID,
|
||||
ImageURL: img.ServingURL,
|
||||
EnableOCR: true,
|
||||
EnableCaption: true,
|
||||
ImageAttrsEnabled: eff.ImageAttrsEnabled,
|
||||
ImageVectorEnabled: &eff.ImageVectorEnabled,
|
||||
ImageActions: eff.ImageActions,
|
||||
Language: lang,
|
||||
ImageSourceType: metadata["image_source_type"],
|
||||
Attempt: attempt,
|
||||
ImageIndex: idx,
|
||||
SourceLocators: img.SourceLocators,
|
||||
}
|
||||
|
||||
langfuse.InjectTracing(ctx, &payload)
|
||||
|
|
|
|||
|
|
@ -49,6 +49,7 @@ func ResolveProcessConfig(kb *types.KnowledgeBase, overrides *types.KnowledgePro
|
|||
GraphEnabled: kb.IsGraphEnabled(),
|
||||
ExtractConfig: derefExtractConfig(kb.ExtractConfig),
|
||||
ImageAttrsEnabled: imageCfg.ImageAttrsEnabled,
|
||||
ImageVectorEnabled: imageCfg.ImageVectorEnabled,
|
||||
ImageActions: types.ResolveImageActions(imageCfg.ImageActions),
|
||||
}
|
||||
if overrides == nil {
|
||||
|
|
@ -93,6 +94,9 @@ func ResolveProcessConfig(kb *types.KnowledgeBase, overrides *types.KnowledgePro
|
|||
if overrides.ImageAttrsEnabled != nil {
|
||||
eff.ImageAttrsEnabled = *overrides.ImageAttrsEnabled
|
||||
}
|
||||
if overrides.ImageVectorEnabled != nil {
|
||||
eff.ImageVectorEnabled = *overrides.ImageVectorEnabled
|
||||
}
|
||||
if overrides.ImageActions != nil {
|
||||
base := eff.ImageActions
|
||||
// Same rule as types.MergeImageActions: the OCR clause is a unit keyed
|
||||
|
|
|
|||
|
|
@ -135,6 +135,8 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
|
|||
return nil, err
|
||||
}
|
||||
|
||||
recordImageRecallDecision(ctx, "candidate_cut", nil, summarizeImageCandidateCut(chunks, params.MatchCount))
|
||||
|
||||
// Truncate to the primary-match cap. MatchCount is guaranteed positive by
|
||||
// the normalization at the top of this function; the slice bound below
|
||||
// depends on that.
|
||||
|
|
|
|||
|
|
@ -141,12 +141,19 @@ func (s *knowledgeBaseService) retrieveFromStores(
|
|||
// (refillPastDroppedImages).
|
||||
func retrieveGroup(ctx context.Context, g *storeGroup) ([]*types.RetrieveResult, error) {
|
||||
params := paramsWithTopK(g)
|
||||
res, err := g.Engine.Retrieve(ctx, params)
|
||||
res, err := retrieveObservedImagePool(ctx, g, params, "initial")
|
||||
res, err = retainPartialResults(ctx, g, res, err)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
refillPastDroppedImages(ctx, g, params, res)
|
||||
for _, rr := range res {
|
||||
if rr != nil {
|
||||
out := summarizeImagePool(rr, g)
|
||||
out["retriever"], out["kb_ids"] = rr.RetrieverType, g.KBIDs
|
||||
recordImageRecallDecision(ctx, "filter", nil, out)
|
||||
}
|
||||
}
|
||||
filterImageHits(res, g)
|
||||
return res, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,10 +15,9 @@ import (
|
|||
// passage saying the same thing (the modality gap). So they get their own
|
||||
// threshold, and vector retrieval widens its pool to leave them room.
|
||||
const (
|
||||
// imageVectorThreshold is the most an image hit is asked to score. It is
|
||||
// empirical, a ceiling below the text defaults (0.15–0.2); a lower text
|
||||
// threshold, or none, still wins.
|
||||
imageVectorThreshold = 0.1
|
||||
// imageVectorThreshold caps the image recall threshold at an empirical
|
||||
// value; a lower text threshold, or none, still wins.
|
||||
imageVectorThreshold = 0.2
|
||||
// imageRecallWidening is how much larger the vector pool is when image
|
||||
// hits compete in it: one half again, so text keeps close to the pool
|
||||
// it had before images arrived.
|
||||
|
|
@ -135,27 +134,38 @@ func filterImageHits(results []*types.RetrieveResult, g *storeGroup) {
|
|||
}
|
||||
|
||||
func keepHit(hit *types.IndexWithScore, retriever types.RetrieverType, g *storeGroup) bool {
|
||||
if hit == nil {
|
||||
return false
|
||||
}
|
||||
if retriever == types.VectorRetrieverType {
|
||||
// Recorded here, before normalization and fusion rewrite Score.
|
||||
if hit != nil && retriever == types.VectorRetrieverType {
|
||||
// Preserve the raw score before normalization and fusion.
|
||||
hit.VectorScore = hit.Score
|
||||
}
|
||||
return imageHitDropReason(hit, retriever, g) == ""
|
||||
}
|
||||
|
||||
// imageHitDropReason is shared by filtering and diagnostics so the trace uses
|
||||
// the actual admission rules. It does not mutate scores or candidate order.
|
||||
func imageHitDropReason(hit *types.IndexWithScore, retriever types.RetrieverType, g *storeGroup) string {
|
||||
if hit == nil {
|
||||
return "nil_hit"
|
||||
}
|
||||
image := hit.SourceType == types.ImageSourceType
|
||||
switch retriever {
|
||||
case types.KeywordsRetrieverType:
|
||||
return !image
|
||||
if image {
|
||||
return "duplicate_image_keyword"
|
||||
}
|
||||
case types.VectorRetrieverType:
|
||||
if image {
|
||||
return !staleImage(hit, g) && hit.Score >= imageThreshold(g.VectorThreshold)
|
||||
if staleImage(hit, g) {
|
||||
return "image_recall_disabled"
|
||||
}
|
||||
if !(hit.Score >= imageThreshold(g.VectorThreshold)) {
|
||||
return "below_image_threshold"
|
||||
}
|
||||
} else if g.imageRecall() && !(hit.Score >= g.VectorThreshold) {
|
||||
return "below_text_threshold"
|
||||
}
|
||||
if !g.imageRecall() {
|
||||
return true
|
||||
}
|
||||
return hit.Score >= g.VectorThreshold
|
||||
}
|
||||
return true
|
||||
return ""
|
||||
}
|
||||
|
||||
// staleImage reports whether a vector hit is an image row of a KB that does
|
||||
|
|
@ -259,7 +269,7 @@ func refillPool(ctx context.Context, g *storeGroup, retriever types.RetrieverTyp
|
|||
// refillSearch runs one refill search and reports whether its answer is
|
||||
// complete; refillPool keeps the pool it has otherwise.
|
||||
func refillSearch(ctx context.Context, g *storeGroup, p types.RetrieveParams) (*types.RetrieveResult, bool) {
|
||||
more, err := g.Engine.Retrieve(ctx, []types.RetrieveParams{p})
|
||||
more, err := retrieveObservedImagePool(ctx, g, []types.RetrieveParams{p}, "refill")
|
||||
if err == nil && len(more) == 1 && more[0] != nil && more[0].Error != nil {
|
||||
err = more[0].Error
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,140 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"slices"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
const imageRecallSampleLimit = 10
|
||||
|
||||
// Every engine call (including refills) is observed before application filters
|
||||
// mutate its rows. This adds no database/model requests. Missing rows cannot be
|
||||
// attributed to the threshold vs the engine TopK from returned rows alone.
|
||||
func retrieveObservedImagePool(ctx context.Context, g *storeGroup, params []types.RetrieveParams, phase string) (
|
||||
[]*types.RetrieveResult, error,
|
||||
) {
|
||||
res, err := g.Engine.Retrieve(ctx, params)
|
||||
for _, rr := range res {
|
||||
if rr == nil {
|
||||
continue
|
||||
}
|
||||
out := summarizeImagePool(rr, g)
|
||||
requests := make([]map[string]interface{}, 0, len(params))
|
||||
for _, p := range params {
|
||||
if p.RetrieverType == rr.RetrieverType {
|
||||
requests = append(requests, map[string]interface{}{
|
||||
"top_k": p.TopK, "threshold": p.Threshold, "knowledge_type": p.KnowledgeType,
|
||||
})
|
||||
}
|
||||
}
|
||||
imageKBs := make([]string, 0, len(g.ImageKBIDs))
|
||||
for id := range g.ImageKBIDs {
|
||||
imageKBs = append(imageKBs, id)
|
||||
}
|
||||
slices.Sort(imageKBs)
|
||||
if len(requests) == 1 {
|
||||
out["pool_limit_reached"] = len(rr.Results) >= requests[0]["top_k"].(int)
|
||||
}
|
||||
out["phase"], out["requests"] = phase, requests
|
||||
out["engine"], out["retriever"] = rr.RetrieverEngineType, rr.RetrieverType
|
||||
out["kb_ids"], out["image_kb_ids"] = g.KBIDs, imageKBs
|
||||
out["result_complete"] = err == nil && rr.Error == nil
|
||||
out["unreturned_images"] = "unknown: engine threshold, top_k, or no eligible indexed images"
|
||||
recordImageRecallDecision(ctx, "engine_pool", nil, out)
|
||||
}
|
||||
return res, err
|
||||
}
|
||||
|
||||
func summarizeImagePool(rr *types.RetrieveResult, g *storeGroup) map[string]interface{} {
|
||||
reasons := map[string]int{}
|
||||
images := make([]map[string]interface{}, 0)
|
||||
imageCount, keptImages := 0, 0
|
||||
var minScore, maxScore float64
|
||||
hasScore := false
|
||||
for i, hit := range rr.Results {
|
||||
reason := imageHitDropReason(hit, rr.RetrieverType, g)
|
||||
if reason == "" {
|
||||
reason = "kept"
|
||||
}
|
||||
reasons[reason]++
|
||||
if hit == nil {
|
||||
continue
|
||||
}
|
||||
if !hasScore || hit.Score < minScore {
|
||||
minScore = hit.Score
|
||||
}
|
||||
if !hasScore || hit.Score > maxScore {
|
||||
maxScore = hit.Score
|
||||
}
|
||||
hasScore = true
|
||||
if hit.SourceType != types.ImageSourceType {
|
||||
continue
|
||||
}
|
||||
imageCount++
|
||||
if reason == "kept" {
|
||||
keptImages++
|
||||
}
|
||||
if len(images) < imageRecallSampleLimit {
|
||||
images = append(images, map[string]interface{}{
|
||||
"pool_rank": i + 1, "chunk_id": hit.ChunkID, "knowledge_id": hit.KnowledgeID,
|
||||
"raw_score": fmt.Sprintf("%.4f", hit.Score), "filter_decision": reason,
|
||||
})
|
||||
}
|
||||
}
|
||||
out := map[string]interface{}{
|
||||
"returned_count": len(rr.Results), "returned_image_count": imageCount,
|
||||
"image_count_passing_filter": keptImages, "filter_decision_counts": reasons,
|
||||
"has_returned_scores": hasScore,
|
||||
"returned_min_score": fmt.Sprint(minScore), "returned_max_score": fmt.Sprint(maxScore),
|
||||
"image_recall_enabled": g.imageRecall(),
|
||||
"image_samples": images, "image_samples_truncated": imageCount - len(images),
|
||||
}
|
||||
if g.imageRecall() {
|
||||
out["text_threshold"] = g.VectorThreshold
|
||||
out["image_threshold"] = imageThreshold(g.VectorThreshold)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// The fusion score cut is separate from the raw vector threshold. Retain image
|
||||
// samples even when none appears in the generic top-hit preview.
|
||||
func summarizeImageCandidateCut(hits []*types.IndexWithScore, limit int) map[string]interface{} {
|
||||
images := make([]map[string]interface{}, 0)
|
||||
before, after := 0, 0
|
||||
for i, hit := range hits {
|
||||
if hit == nil || hit.SourceType != types.ImageSourceType {
|
||||
continue
|
||||
}
|
||||
before++
|
||||
decision := "match_count_cut"
|
||||
if i < limit {
|
||||
after++
|
||||
decision = "kept"
|
||||
}
|
||||
if len(images) < imageRecallSampleLimit {
|
||||
images = append(images, map[string]interface{}{
|
||||
"rank": i + 1, "chunk_id": hit.ChunkID,
|
||||
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
|
||||
"fusion_score": fmt.Sprintf("%.4f", hit.Score), "decision": decision,
|
||||
})
|
||||
}
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"input_count": len(hits), "match_count": limit, "output_count": min(len(hits), limit),
|
||||
"input_image_count": before, "output_image_count": after, "dropped_image_count": before - after,
|
||||
"image_samples": images, "image_samples_truncated": before - len(images),
|
||||
}
|
||||
}
|
||||
|
||||
func recordImageRecallDecision(ctx context.Context, stage string, input, output map[string]interface{}) {
|
||||
logger.GetLogger(ctx).WithFields(logger.Fields(output)).Info("image recall " + stage)
|
||||
_, span := langfuse.GetManager().StartSpan(ctx, langfuse.SpanOptions{
|
||||
Name: "retrieval.image_" + stage, Input: input,
|
||||
})
|
||||
span.Finish(output, nil, nil)
|
||||
}
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestImagePoolDiagnosticsUseActualFilterRulesWithoutMutatingHits(t *testing.T) {
|
||||
g := &storeGroup{ImageKBIDs: recalls("kb"), VectorThreshold: 0.5}
|
||||
rr := &types.RetrieveResult{RetrieverType: types.VectorRetrieverType, Results: []*types.IndexWithScore{
|
||||
hit("text-low", 0.4, types.ChunkSourceType),
|
||||
hit("image-low", 0.199, types.ImageSourceType),
|
||||
hit("image-boundary", 0.2, types.ImageSourceType),
|
||||
kbHit("disabled", "image-disabled", 0.9, types.ImageSourceType),
|
||||
}}
|
||||
out := summarizeImagePool(rr, g)
|
||||
require.Equal(t, 3, out["returned_image_count"])
|
||||
require.Equal(t, 1, out["image_count_passing_filter"])
|
||||
require.Equal(t, map[string]int{
|
||||
"below_text_threshold": 1, "below_image_threshold": 1, "kept": 1, "image_recall_disabled": 1,
|
||||
}, out["filter_decision_counts"])
|
||||
require.Len(t, rr.Results, 4)
|
||||
require.Zero(t, rr.Results[2].VectorScore, "observability must not alter raw scores")
|
||||
filterImageHits([]*types.RetrieveResult{rr}, g)
|
||||
require.Equal(t, []string{"image-boundary"}, ids(rr.Results))
|
||||
require.Equal(t, 0.2, rr.Results[0].VectorScore)
|
||||
}
|
||||
|
||||
func TestImagePoolDiagnosticsCoverKeywordExclusionAndBoundSamples(t *testing.T) {
|
||||
g := &storeGroup{ImageKBIDs: recalls("kb")}
|
||||
rr := &types.RetrieveResult{RetrieverType: types.KeywordsRetrieverType}
|
||||
for i := 0; i < 13; i++ {
|
||||
h := hit(fmt.Sprint(i), 10, types.ImageSourceType)
|
||||
h.Content = "private image description"
|
||||
rr.Results = append(rr.Results, h)
|
||||
}
|
||||
out := summarizeImagePool(rr, g)
|
||||
require.Equal(t, 13, out["returned_image_count"])
|
||||
require.Equal(t, 0, out["image_count_passing_filter"])
|
||||
require.Equal(t, 3, out["image_samples_truncated"])
|
||||
require.Equal(t, map[string]int{"duplicate_image_keyword": 13}, out["filter_decision_counts"])
|
||||
b, err := json.Marshal(out)
|
||||
require.NoError(t, err)
|
||||
require.NotContains(t, string(b), "private image description")
|
||||
}
|
||||
|
||||
func TestImageCandidateCutDiagnosticsDistinguishFusionCutFromVectorThreshold(t *testing.T) {
|
||||
image := hit("image", 0.3, types.ImageSourceType)
|
||||
image.VectorScore = 0.2688
|
||||
hits := []*types.IndexWithScore{hit("text", 0.9, types.ChunkSourceType), image}
|
||||
out := summarizeImageCandidateCut(hits, 1)
|
||||
require.Equal(t, 1, out["input_image_count"])
|
||||
require.Equal(t, 0, out["output_image_count"])
|
||||
rows := out["image_samples"].([]map[string]interface{})
|
||||
require.Equal(t, "match_count_cut", rows[0]["decision"])
|
||||
require.Equal(t, "0.2688", rows[0]["vector_score"])
|
||||
require.Equal(t, "0.3000", rows[0]["fusion_score"])
|
||||
require.Equal(t, 2, rows[0]["rank"])
|
||||
require.Len(t, hits, 2)
|
||||
require.Equal(t, 1, summarizeImageCandidateCut(hits, 2)["output_image_count"])
|
||||
}
|
||||
|
|
@ -81,6 +81,8 @@ func TestFilterImageHitsHoldsEachKindToItsOwnThreshold(t *testing.T) {
|
|||
vector := &types.RetrieveResult{RetrieverType: types.VectorRetrieverType, Results: []*types.IndexWithScore{
|
||||
hit("text-strong", 0.6, types.ChunkSourceType),
|
||||
hit("image-strong", 0.25, types.ImageSourceType),
|
||||
hit("image-boundary", 0.2, types.ImageSourceType),
|
||||
hit("image-below-boundary", 0.199, types.ImageSourceType),
|
||||
hit("text-weak", 0.2, types.ChunkSourceType), // only reached because the query threshold was lowered
|
||||
hit("image-weak", 0.05, types.ImageSourceType),
|
||||
}}
|
||||
|
|
@ -89,7 +91,7 @@ func TestFilterImageHitsHoldsEachKindToItsOwnThreshold(t *testing.T) {
|
|||
hit("image-kw", 0.9, types.ImageSourceType),
|
||||
}}
|
||||
filterImageHits([]*types.RetrieveResult{vector, keyword}, g)
|
||||
assert.Equal(t, []string{"text-strong", "image-strong"}, ids(vector.Results))
|
||||
assert.Equal(t, []string{"text-strong", "image-strong", "image-boundary"}, ids(vector.Results))
|
||||
assert.Equal(t, []string{"text-kw"}, ids(keyword.Results),
|
||||
"an image row's Content is the caption, already indexed as text")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -194,9 +194,8 @@ type ModelResponse struct {
|
|||
// caller is a system administrator. See MCPServiceResponse.Credentials.
|
||||
Credentials map[string]CredentialFieldMetadata `json:"credentials,omitempty"`
|
||||
// Capabilities is the catalog view of the model. Chat models get the
|
||||
// protocol, thinking levels and context window; embedding models get the
|
||||
// input modalities, which decide whether images can be embedded in the
|
||||
// same space as text.
|
||||
// protocol, thinking levels and context window; embedding and rerank
|
||||
// models get input modalities, including support for image input.
|
||||
Capabilities *modelruntime.Capabilities `json:"capabilities,omitempty"`
|
||||
}
|
||||
|
||||
|
|
@ -311,7 +310,7 @@ func NewModelResponse(ctx context.Context, m *types.Model) *ModelResponse {
|
|||
}
|
||||
var caps *modelruntime.Capabilities
|
||||
switch m.Type {
|
||||
case types.ModelTypeKnowledgeQA, types.ModelTypeVLLM, types.ModelTypeEmbedding:
|
||||
case types.ModelTypeKnowledgeQA, types.ModelTypeVLLM, types.ModelTypeEmbedding, types.ModelTypeRerank:
|
||||
if resolved, err := modelruntime.Resolve(modelruntime.Ref{
|
||||
Provider: m.Parameters.Provider, Model: m.Name, BaseURL: m.Parameters.BaseURL,
|
||||
ModelType: m.Type, Extra: m.Parameters.ExtraConfig, Override: m.Parameters.Spec,
|
||||
|
|
|
|||
|
|
@ -16,6 +16,9 @@ import (
|
|||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/models/api"
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
"github.com/Tencent/WeKnora/internal/models/embedding"
|
||||
"github.com/Tencent/WeKnora/internal/models/imageprep"
|
||||
"github.com/Tencent/WeKnora/internal/models/rerank"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
secutils "github.com/Tencent/WeKnora/internal/utils"
|
||||
|
|
@ -293,6 +296,9 @@ type ModelDebugOptions struct {
|
|||
Thinking *bool `json:"thinking,omitempty"`
|
||||
// ReasoningEffort is the graded level; when set it overrides Thinking.
|
||||
ReasoningEffort string `json:"reasoning_effort,omitempty"`
|
||||
// Similarity makes an embedding test score the documents (and an
|
||||
// optional image) against the input as a search query.
|
||||
Similarity bool `json:"similarity,omitempty"`
|
||||
}
|
||||
|
||||
func parseModelDebugOptions(raw string) (ModelDebugOptions, error) {
|
||||
|
|
@ -433,6 +439,18 @@ func consumeModelDebugChatStream(stream <-chan types.StreamResponse) (*modelDebu
|
|||
return result, nil
|
||||
}
|
||||
|
||||
// Model metadata and the legacy vision switch both allow image chat tests.
|
||||
func modelDebugChatAcceptsImages(model *types.Model) bool {
|
||||
if model.Parameters.SupportsVision {
|
||||
return true
|
||||
}
|
||||
if model.Source == types.ModelSourceLocal {
|
||||
return false
|
||||
}
|
||||
resolved, err := chat.Resolve(chat.ConfigFromModel(model, "", ""))
|
||||
return err == nil && resolved.Spec.AcceptsImages()
|
||||
}
|
||||
|
||||
// DebugModel executes a saved model through the same service constructors used
|
||||
// by production calls and returns the complete normalized response. Credentials
|
||||
// stay server-side; the request preview contains only non-secret fields.
|
||||
|
|
@ -516,13 +534,26 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
|
|||
fileSize = int64(len(fileBytes))
|
||||
}
|
||||
|
||||
var image api.EmbedImage
|
||||
if fileName != "" && model.Type != types.ModelTypeASR {
|
||||
image, err = imageprep.Prepare(fileBytes, imageprep.Limits{})
|
||||
if err != nil {
|
||||
_ = c.Error(errors.NewBadRequestError(err.Error()))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
requestPreview := modelDebugRequestPreview(model, input, documents, opts, fileName, fileSize)
|
||||
observations := gin.H{}
|
||||
|
||||
switch model.Type {
|
||||
case types.ModelTypeKnowledgeQA:
|
||||
if strings.TrimSpace(input) == "" {
|
||||
c.Error(errors.NewBadRequestError("query cannot be empty"))
|
||||
if len(fileBytes) > 0 && !modelDebugChatAcceptsImages(model) {
|
||||
_ = c.Error(errors.NewBadRequestError("chat model does not accept images"))
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(input) == "" && len(fileBytes) == 0 {
|
||||
_ = c.Error(errors.NewBadRequestError("query cannot be empty"))
|
||||
return
|
||||
}
|
||||
instance, callErr := h.service.GetChatModel(ctx, id)
|
||||
|
|
@ -534,7 +565,17 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
|
|||
if strings.TrimSpace(opts.SystemPrompt) != "" {
|
||||
messages = append(messages, chat.Message{Role: "system", Content: opts.SystemPrompt})
|
||||
}
|
||||
messages = append(messages, chat.Message{Role: "user", Content: input})
|
||||
message := chat.Message{Role: "user", Content: input}
|
||||
if len(fileBytes) > 0 {
|
||||
message.Content = ""
|
||||
if strings.TrimSpace(input) != "" {
|
||||
message.MultiContent = append(message.MultiContent, chat.MessageContentPart{Type: "text", Text: input})
|
||||
}
|
||||
message.MultiContent = append(message.MultiContent, chat.MessageContentPart{
|
||||
Type: "image_url", ImageURL: &chat.ImageURL{URL: image.DataURI()},
|
||||
})
|
||||
}
|
||||
messages = append(messages, message)
|
||||
chatOpts := &chat.ChatOptions{}
|
||||
if opts.Temperature != nil {
|
||||
chatOpts.Temperature = *opts.Temperature
|
||||
|
|
@ -575,8 +616,12 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
|
|||
}
|
||||
writeModelDebugResult(c, started, requestPreview, resp, callErr, observations)
|
||||
case types.ModelTypeEmbedding:
|
||||
if strings.TrimSpace(input) == "" {
|
||||
c.Error(errors.NewBadRequestError("input cannot be empty"))
|
||||
if opts.Similarity {
|
||||
h.debugEmbeddingSimilarity(c, id, input, documents, fileBytes, started, requestPreview, observations)
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(input) == "" && len(fileBytes) == 0 {
|
||||
_ = c.Error(errors.NewBadRequestError("input cannot be empty"))
|
||||
return
|
||||
}
|
||||
instance, callErr := h.service.GetEmbeddingModel(ctx, id)
|
||||
|
|
@ -584,12 +629,35 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
|
|||
writeModelDebugResult(c, started, requestPreview, nil, callErr, observations)
|
||||
return
|
||||
}
|
||||
if len(fileBytes) > 0 {
|
||||
if strings.TrimSpace(input) != "" {
|
||||
_ = c.Error(errors.NewBadRequestError("test either text or an image for embedding"))
|
||||
return
|
||||
}
|
||||
imageModel, ok := embedding.AsImageEmbedder(instance)
|
||||
if !ok {
|
||||
_ = c.Error(errors.NewBadRequestError(embedding.ErrImagesUnsupported.Error()))
|
||||
return
|
||||
}
|
||||
prepared, prepErr := imageprep.Prepare(fileBytes, imageModel.ImageLimits())
|
||||
if prepErr != nil {
|
||||
_ = c.Error(errors.NewBadRequestError(prepErr.Error()))
|
||||
return
|
||||
}
|
||||
vectors, callErr := imageModel.BatchEmbedImages(ctx, []embedding.Image{prepared})
|
||||
observations["result_count"] = len(vectors)
|
||||
if len(vectors) > 0 {
|
||||
observations["dimension"] = len(vectors[0])
|
||||
}
|
||||
writeModelDebugResult(c, started, requestPreview, vectors, callErr, observations)
|
||||
return
|
||||
}
|
||||
vector, callErr := instance.Embed(ctx, input)
|
||||
observations["dimension"] = len(vector)
|
||||
writeModelDebugResult(c, started, requestPreview, vector, callErr, observations)
|
||||
case types.ModelTypeRerank:
|
||||
if strings.TrimSpace(input) == "" || len(documents) == 0 {
|
||||
c.Error(errors.NewBadRequestError("query and documents cannot be empty"))
|
||||
if strings.TrimSpace(input) == "" || (len(documents) == 0 && len(fileBytes) == 0) {
|
||||
_ = c.Error(errors.NewBadRequestError("query and documents cannot be empty"))
|
||||
return
|
||||
}
|
||||
instance, callErr := h.service.GetRerankModel(ctx, id)
|
||||
|
|
@ -597,6 +665,26 @@ func (h *ModelHandler) DebugModel(c *gin.Context) {
|
|||
writeModelDebugResult(c, started, requestPreview, nil, callErr, observations)
|
||||
return
|
||||
}
|
||||
if len(fileBytes) > 0 {
|
||||
if len(documents) > 0 {
|
||||
_ = c.Error(errors.NewBadRequestError("test either text documents or an image for reranking"))
|
||||
return
|
||||
}
|
||||
imageModel, ok := rerank.AsImageReranker(instance)
|
||||
if !ok {
|
||||
_ = c.Error(errors.NewBadRequestError(rerank.ErrImagesUnsupported.Error()))
|
||||
return
|
||||
}
|
||||
prepared, prepErr := imageprep.Prepare(fileBytes, imageModel.ImageLimits())
|
||||
if prepErr != nil {
|
||||
_ = c.Error(errors.NewBadRequestError(prepErr.Error()))
|
||||
return
|
||||
}
|
||||
results, callErr := imageModel.RerankImages(ctx, input, []rerank.Image{prepared})
|
||||
observations["result_count"] = len(results)
|
||||
writeModelDebugResult(c, started, requestPreview, results, callErr, observations)
|
||||
return
|
||||
}
|
||||
results, callErr := instance.Rerank(ctx, input, documents)
|
||||
observations["result_count"] = len(results)
|
||||
writeModelDebugResult(c, started, requestPreview, results, callErr, observations)
|
||||
|
|
|
|||
|
|
@ -104,6 +104,9 @@ func providerDTO(v *modelruntime.Provider, modelType types.ModelType, includeMod
|
|||
modelTypes := make([]string, 0, len(v.ModelTypes))
|
||||
for _, mt := range v.ModelTypes {
|
||||
modelTypes = append(modelTypes, modelTypeToFrontend(mt))
|
||||
// Use the same fallback as runtime resolution. Several ASR vendors
|
||||
// share their chat base URL without declaring a separate ASR entry.
|
||||
defaultURLs[modelTypeToFrontend(mt)] = v.GetDefaultURL(mt)
|
||||
}
|
||||
dto := ModelProviderDTO{
|
||||
Value: v.ID,
|
||||
|
|
|
|||
|
|
@ -8,10 +8,74 @@ import (
|
|||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/models/api"
|
||||
"github.com/Tencent/WeKnora/internal/models/providers"
|
||||
modelruntime "github.com/Tencent/WeKnora/internal/models/runtime"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
func TestAliyunMultimodalEmbeddingsAreSelectable(t *testing.T) {
|
||||
vendor, ok := modelruntime.Get(providers.AliyunID)
|
||||
if !ok {
|
||||
t.Fatal("Aliyun provider is not registered")
|
||||
}
|
||||
entries := providerDTO(vendor, types.ModelTypeEmbedding, true).Models
|
||||
for id, dimension := range map[string]int{
|
||||
"qwen3-vl-embedding": 2560,
|
||||
"qwen2.5-vl-embedding": 1024,
|
||||
"tongyi-embedding-vision-flash": 768,
|
||||
"multimodal-embedding-v1": 1024,
|
||||
} {
|
||||
t.Run(id, func(t *testing.T) {
|
||||
var found bool
|
||||
for _, entry := range entries {
|
||||
if entry.ID == id {
|
||||
found = true
|
||||
if entry.Dimension != dimension || entry.Type != "embedding" {
|
||||
t.Errorf("unexpected picker metadata: %+v", entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Fatalf("%s is missing from the model picker", id)
|
||||
}
|
||||
resolved, err := modelruntime.Resolve(modelruntime.Ref{
|
||||
Provider: providers.AliyunID, Model: id, ModelType: types.ModelTypeEmbedding,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resolved.EmbeddingAPI != api.EmbeddingDashScope {
|
||||
t.Errorf("model must keep the native DashScope protocol, got %s", resolved.EmbeddingAPI)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderDTOIncludesEffectiveDefaultURLs(t *testing.T) {
|
||||
for _, vendor := range modelruntime.List() {
|
||||
t.Run(vendor.ID, func(t *testing.T) {
|
||||
got := providerDTO(vendor, types.ModelTypeASR, true)
|
||||
for _, kind := range vendor.ModelTypes {
|
||||
want := vendor.GetDefaultURL(kind)
|
||||
if got.DefaultURLs[modelTypeToFrontend(kind)] != want {
|
||||
t.Errorf("%s default URL = %q, want runtime default %q",
|
||||
kind, got.DefaultURLs[modelTypeToFrontend(kind)], want)
|
||||
}
|
||||
}
|
||||
if vendor.ID == providers.ZhipuID {
|
||||
if got.DefaultURLs["asr"] != providers.ZhipuBaseURL {
|
||||
t.Errorf("Zhipu ASR must prefill its own endpoint, got %q", got.DefaultURLs["asr"])
|
||||
}
|
||||
if len(got.Models) != 1 || got.Models[0].ID != "glm-asr-2512" {
|
||||
t.Errorf("unexpected Zhipu ASR models: %+v", got.Models)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// listProvidersAs drives the real handler with one tenant role.
|
||||
func listProvidersAs(t *testing.T, role types.TenantRole) []map[string]any {
|
||||
t.Helper()
|
||||
|
|
|
|||
|
|
@ -0,0 +1,382 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"image"
|
||||
"image/png"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/handler/dto"
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
"github.com/Tencent/WeKnora/internal/models/embedding"
|
||||
"github.com/Tencent/WeKnora/internal/models/imageprep"
|
||||
"github.com/Tencent/WeKnora/internal/models/rerank"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/gin-gonic/gin"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type debugImageService struct {
|
||||
interfaces.ModelService
|
||||
row *types.Model
|
||||
embed *debugImageEmbedder
|
||||
similarity *similarityEmbedder
|
||||
rank *debugImageReranker
|
||||
chat *debugImageChat
|
||||
}
|
||||
|
||||
func (s *debugImageService) GetModelByID(context.Context, string) (*types.Model, error) {
|
||||
return s.row, nil
|
||||
}
|
||||
|
||||
func (s *debugImageService) GetEmbeddingModel(context.Context, string) (embedding.Embedder, error) {
|
||||
if s.similarity != nil {
|
||||
return s.similarity, nil
|
||||
}
|
||||
return s.embed, nil
|
||||
}
|
||||
|
||||
func (s *debugImageService) GetRerankModel(context.Context, string) (rerank.Reranker, error) {
|
||||
return s.rank, nil
|
||||
}
|
||||
|
||||
func (s *debugImageService) GetChatModel(context.Context, string) (chat.Chat, error) {
|
||||
return s.chat, nil
|
||||
}
|
||||
|
||||
type debugImageEmbedder struct {
|
||||
embedding.Embedder
|
||||
accepts bool
|
||||
images []embedding.Image
|
||||
text string
|
||||
}
|
||||
|
||||
func (e *debugImageEmbedder) AcceptsImages() bool { return e.accepts }
|
||||
func (e *debugImageEmbedder) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
|
||||
func (e *debugImageEmbedder) BatchEmbedImages(_ context.Context, images []embedding.Image) ([][]float32, error) {
|
||||
e.images = images
|
||||
return [][]float32{{1, 2, 3}}, nil
|
||||
}
|
||||
|
||||
func (e *debugImageEmbedder) Embed(_ context.Context, text string) ([]float32, error) {
|
||||
e.text = text
|
||||
return []float32{1, 2, 3}, nil
|
||||
}
|
||||
|
||||
type debugImageReranker struct {
|
||||
rerank.Reranker
|
||||
accepts bool
|
||||
images []rerank.Image
|
||||
query string
|
||||
}
|
||||
|
||||
func (r *debugImageReranker) AcceptsImages() bool { return r.accepts }
|
||||
func (r *debugImageReranker) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
|
||||
func (r *debugImageReranker) RerankImages(
|
||||
_ context.Context, query string, images []rerank.Image,
|
||||
) ([]rerank.RankResult, error) {
|
||||
r.query = query
|
||||
r.images = images
|
||||
return []rerank.RankResult{{Index: 0, RelevanceScore: 0.9}}, nil
|
||||
}
|
||||
|
||||
type debugChatBase interface{ chat.Chat }
|
||||
|
||||
type debugImageChat struct {
|
||||
debugChatBase
|
||||
messages []chat.Message
|
||||
}
|
||||
|
||||
func (m *debugImageChat) ChatStream(
|
||||
_ context.Context, messages []chat.Message, _ *chat.ChatOptions,
|
||||
) (<-chan types.StreamResponse, error) {
|
||||
m.messages = messages
|
||||
ch := make(chan types.StreamResponse, 1)
|
||||
ch <- types.StreamResponse{ResponseType: types.ResponseTypeAnswer, Content: "an image"}
|
||||
close(ch)
|
||||
return ch, nil
|
||||
}
|
||||
|
||||
func runImageDebug(
|
||||
t *testing.T, svc *debugImageService, input string, data []byte, documents string,
|
||||
) (map[string]any, string) {
|
||||
t.Helper()
|
||||
var body bytes.Buffer
|
||||
w := multipart.NewWriter(&body)
|
||||
require.NoError(t, w.WriteField("input", input))
|
||||
require.NoError(t, w.WriteField("documents", documents))
|
||||
if data != nil {
|
||||
f, err := w.CreateFormFile("file", "test.png")
|
||||
require.NoError(t, err)
|
||||
_, err = f.Write(data)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
require.NoError(t, w.Close())
|
||||
req := httptest.NewRequest(http.MethodPost, "/models/test/debug", &body)
|
||||
req.Header.Set("Content-Type", w.FormDataContentType())
|
||||
recorder := httptest.NewRecorder()
|
||||
c, _ := gin.CreateTestContext(recorder)
|
||||
c.Request = req
|
||||
c.Params = gin.Params{{Key: "id", Value: "test"}}
|
||||
(&ModelHandler{service: svc}).DebugModel(c)
|
||||
if len(c.Errors) > 0 {
|
||||
return nil, c.Errors.Last().Error()
|
||||
}
|
||||
require.Equal(t, http.StatusOK, recorder.Code)
|
||||
var result struct {
|
||||
Data map[string]any `json:"data"`
|
||||
}
|
||||
require.NoError(t, json.Unmarshal(recorder.Body.Bytes(), &result))
|
||||
return result.Data, ""
|
||||
}
|
||||
|
||||
func debugPNG(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
var b bytes.Buffer
|
||||
require.NoError(t, png.Encode(&b, image.NewRGBA(image.Rect(0, 0, 2, 2))))
|
||||
return b.Bytes()
|
||||
}
|
||||
|
||||
func TestModelDebugImageEmbedding(t *testing.T) {
|
||||
data := debugPNG(t)
|
||||
for _, tc := range []struct {
|
||||
name, input string
|
||||
file []byte
|
||||
accepts bool
|
||||
wantErr string
|
||||
}{
|
||||
{"image only", "", data, true, ""},
|
||||
{"text only", "hello", nil, true, ""},
|
||||
{"unsupported image", "", data, false, "does not accept images"},
|
||||
{"invalid upload", "", []byte("not an image"), true, "not an image"},
|
||||
{"no input", "", nil, true, "input cannot be empty"},
|
||||
{"mixed input", "hello", data, true, "either text or an image"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
embed := &debugImageEmbedder{accepts: tc.accepts}
|
||||
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeEmbedding}, embed: embed}
|
||||
result, errText := runImageDebug(t, svc, tc.input, tc.file, "")
|
||||
if tc.wantErr != "" {
|
||||
require.Contains(t, errText, tc.wantErr)
|
||||
require.Empty(t, embed.images)
|
||||
require.Empty(t, embed.text)
|
||||
return
|
||||
}
|
||||
require.Empty(t, errText)
|
||||
require.Equal(t, true, result["ok"])
|
||||
require.Equal(t, float64(3), result["observations"].(map[string]any)["dimension"])
|
||||
if tc.file != nil {
|
||||
require.Len(t, embed.images, 1)
|
||||
require.Equal(t, data, embed.images[0].Data)
|
||||
require.Equal(t, "image/png", embed.images[0].MIMEType)
|
||||
} else {
|
||||
require.Equal(t, "hello", embed.text)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// similarityEmbedder places each text on a fixed axis so the expected cosine
|
||||
// is known, and records which side every call was embedded as.
|
||||
type similarityEmbedder struct {
|
||||
embedding.Embedder
|
||||
accepts bool
|
||||
imageWidth int
|
||||
queries []string
|
||||
documents []string
|
||||
images int
|
||||
}
|
||||
|
||||
var similarityAxes = map[string][]float32{
|
||||
"dog": {1, 0, 0}, "puppy": {0.9, 0.1, 0}, "car": {0, 1, 0}, "image": {0.6, 0.8, 0},
|
||||
}
|
||||
|
||||
func (e *similarityEmbedder) Embed(ctx context.Context, text string) ([]float32, error) {
|
||||
if types.IsEmbedQuery(ctx) {
|
||||
e.queries = append(e.queries, text)
|
||||
}
|
||||
return similarityAxes[text], nil
|
||||
}
|
||||
|
||||
func (e *similarityEmbedder) BatchEmbed(ctx context.Context, texts []string) ([][]float32, error) {
|
||||
if !types.IsEmbedQuery(ctx) {
|
||||
e.documents = append(e.documents, texts...)
|
||||
}
|
||||
out := make([][]float32, len(texts))
|
||||
for i, text := range texts {
|
||||
out[i] = similarityAxes[text]
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (e *similarityEmbedder) AcceptsImages() bool { return e.accepts }
|
||||
func (e *similarityEmbedder) ImageLimits() imageprep.Limits { return imageprep.Limits{} }
|
||||
func (e *similarityEmbedder) BatchEmbedImages(context.Context, []embedding.Image) ([][]float32, error) {
|
||||
e.images++
|
||||
if e.imageWidth > 0 {
|
||||
return [][]float32{make([]float32, e.imageWidth)}, nil
|
||||
}
|
||||
return [][]float32{similarityAxes["image"]}, nil
|
||||
}
|
||||
|
||||
func runSimilarityDebug(
|
||||
t *testing.T, embed *similarityEmbedder, query, documents string, file []byte,
|
||||
) (map[string]any, string) {
|
||||
t.Helper()
|
||||
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeEmbedding}}
|
||||
svc.similarity = embed
|
||||
var body bytes.Buffer
|
||||
w := multipart.NewWriter(&body)
|
||||
require.NoError(t, w.WriteField("input", query))
|
||||
require.NoError(t, w.WriteField("documents", documents))
|
||||
require.NoError(t, w.WriteField("options", `{"similarity":true}`))
|
||||
if file != nil {
|
||||
f, err := w.CreateFormFile("file", "test.png")
|
||||
require.NoError(t, err)
|
||||
_, err = f.Write(file)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
require.NoError(t, w.Close())
|
||||
req := httptest.NewRequest(http.MethodPost, "/models/test/debug", &body)
|
||||
req.Header.Set("Content-Type", w.FormDataContentType())
|
||||
recorder := httptest.NewRecorder()
|
||||
c, _ := gin.CreateTestContext(recorder)
|
||||
c.Request = req
|
||||
c.Params = gin.Params{{Key: "id", Value: "test"}}
|
||||
(&ModelHandler{service: svc}).DebugModel(c)
|
||||
if len(c.Errors) > 0 {
|
||||
return nil, c.Errors.Last().Error()
|
||||
}
|
||||
var result struct {
|
||||
Data map[string]any `json:"data"`
|
||||
}
|
||||
require.NoError(t, json.Unmarshal(recorder.Body.Bytes(), &result))
|
||||
return result.Data, ""
|
||||
}
|
||||
|
||||
func TestModelDebugEmbeddingSimilarity(t *testing.T) {
|
||||
embed := &similarityEmbedder{accepts: true}
|
||||
result, errText := runSimilarityDebug(t, embed, "dog", `["car","puppy"]`, debugPNG(t))
|
||||
require.Empty(t, errText)
|
||||
require.Equal(t, true, result["ok"], result["error"])
|
||||
require.Equal(t, []string{"dog"}, embed.queries, "the query is embedded as a search query")
|
||||
require.Equal(t, []string{"car", "puppy"}, embed.documents, "candidates are embedded as content")
|
||||
require.Equal(t, 1, embed.images)
|
||||
|
||||
results := result["raw_response"].(map[string]any)["results"].([]any)
|
||||
require.Len(t, results, 3)
|
||||
var order []string
|
||||
for _, r := range results {
|
||||
item := r.(map[string]any)
|
||||
content, _ := item["content"].(string)
|
||||
order = append(order, item["kind"].(string)+":"+content)
|
||||
}
|
||||
require.Equal(t, []string{"text:puppy", "image:", "text:car"}, order, "sorted by cosine, highest first")
|
||||
require.InDelta(t, 0.6, results[1].(map[string]any)["similarity"], 1e-6)
|
||||
require.Equal(t, float64(2), results[1].(map[string]any)["index"], "the image follows the documents")
|
||||
require.Equal(t, float64(3), result["observations"].(map[string]any)["result_count"])
|
||||
}
|
||||
|
||||
func TestModelDebugEmbeddingSimilarityRefusals(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, query, documents string
|
||||
file bool
|
||||
embed *similarityEmbedder
|
||||
wantErr string
|
||||
}{
|
||||
{"no query", "", `["car"]`, false, &similarityEmbedder{}, "at least one candidate"},
|
||||
{"no candidate", "dog", "", false, &similarityEmbedder{}, "at least one candidate"},
|
||||
{"image on a text model", "dog", "", true, &similarityEmbedder{}, "does not accept images"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
var file []byte
|
||||
if tc.file {
|
||||
file = debugPNG(t)
|
||||
}
|
||||
_, errText := runSimilarityDebug(t, tc.embed, tc.query, tc.documents, file)
|
||||
require.Contains(t, errText, tc.wantErr)
|
||||
require.Empty(t, tc.embed.queries, "nothing is sent for a refused test")
|
||||
})
|
||||
}
|
||||
|
||||
// An image vector of another width is a failed call, not a cosine of 0.
|
||||
result, errText := runSimilarityDebug(t, &similarityEmbedder{accepts: true, imageWidth: 5}, "dog", "", debugPNG(t))
|
||||
require.Empty(t, errText)
|
||||
require.Equal(t, false, result["ok"])
|
||||
require.Contains(t, result["error"], "5 dimensions, the query 3")
|
||||
}
|
||||
|
||||
func TestModelDebugImageChat(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, input string
|
||||
vision bool
|
||||
wantErr bool
|
||||
}{
|
||||
{"gpt-4o", "describe", false, false},
|
||||
{"gpt-4o", "", false, false},
|
||||
{"unknown-model", "describe", true, false},
|
||||
{"unknown-model", "describe", false, true},
|
||||
} {
|
||||
t.Run(tc.name+tc.input, func(t *testing.T) {
|
||||
model := &debugImageChat{}
|
||||
svc := &debugImageService{
|
||||
row: &types.Model{
|
||||
Type: types.ModelTypeKnowledgeQA, Source: types.ModelSourceRemote, Name: tc.name,
|
||||
Parameters: types.ModelParameters{Provider: "openai", SupportsVision: tc.vision},
|
||||
},
|
||||
chat: model,
|
||||
}
|
||||
result, errText := runImageDebug(t, svc, tc.input, debugPNG(t), "")
|
||||
if tc.wantErr {
|
||||
require.Contains(t, errText, "does not accept images")
|
||||
require.Empty(t, model.messages)
|
||||
return
|
||||
}
|
||||
require.Empty(t, errText)
|
||||
require.Equal(t, true, result["ok"])
|
||||
require.Len(t, model.messages, 1)
|
||||
parts := model.messages[0].MultiContent
|
||||
require.True(t, len(parts) > 0)
|
||||
require.Contains(t, parts[len(parts)-1].ImageURL.URL, "data:image/png;base64,")
|
||||
if tc.input != "" {
|
||||
require.Equal(t, tc.input, parts[0].Text)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestModelDebugImageRerank(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
accepts bool
|
||||
documents, wantErr string
|
||||
}{{true, "", ""}, {false, "", "does not accept images"}, {true, `["text"]`, "either text documents or an image"}} {
|
||||
rank := &debugImageReranker{accepts: tc.accepts}
|
||||
svc := &debugImageService{row: &types.Model{Type: types.ModelTypeRerank}, rank: rank}
|
||||
result, errText := runImageDebug(t, svc, "a cat", debugPNG(t), tc.documents)
|
||||
if tc.wantErr != "" {
|
||||
require.Contains(t, errText, tc.wantErr)
|
||||
require.Empty(t, rank.images)
|
||||
continue
|
||||
}
|
||||
require.Empty(t, errText)
|
||||
require.Equal(t, true, result["ok"])
|
||||
require.Len(t, rank.images, 1)
|
||||
require.Equal(t, "a cat", rank.query)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRerankResponseIncludesImageCapabilities(t *testing.T) {
|
||||
row := &types.Model{
|
||||
Type: types.ModelTypeRerank, Source: types.ModelSourceRemote,
|
||||
Name: "nvidia/llama-nemotron-rerank-vl-1b-v2", Parameters: types.ModelParameters{Provider: "nvidia"},
|
||||
}
|
||||
response := dto.NewModelResponse(context.Background(), row)
|
||||
require.NotNil(t, response.Capabilities)
|
||||
require.Contains(t, response.Capabilities.Input, "image")
|
||||
}
|
||||
|
|
@ -0,0 +1,110 @@
|
|||
package handler
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/models/embedding"
|
||||
"github.com/Tencent/WeKnora/internal/models/imageprep"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/gin-gonic/gin"
|
||||
)
|
||||
|
||||
// embeddingSimilarity is one scored candidate of an embedding similarity test.
|
||||
type embeddingSimilarity struct {
|
||||
// Index is the candidate's position in the request: the documents in
|
||||
// order, then the image.
|
||||
Index int `json:"index"`
|
||||
Kind string `json:"kind"`
|
||||
Content string `json:"content,omitempty"`
|
||||
Similarity float64 `json:"similarity"`
|
||||
}
|
||||
|
||||
// debugEmbeddingSimilarity embeds the input as a search query and every
|
||||
// candidate as indexed content, the two sides retrieval uses, and returns
|
||||
// the candidates by cosine similarity.
|
||||
func (h *ModelHandler) debugEmbeddingSimilarity(
|
||||
c *gin.Context, id, query string, documents []string, fileBytes []byte,
|
||||
started time.Time, requestPreview, observations gin.H,
|
||||
) {
|
||||
ctx := c.Request.Context()
|
||||
if strings.TrimSpace(query) == "" || (len(documents) == 0 && len(fileBytes) == 0) {
|
||||
_ = c.Error(errors.NewBadRequestError("query and at least one candidate are required"))
|
||||
return
|
||||
}
|
||||
instance, err := h.service.GetEmbeddingModel(ctx, id)
|
||||
if err != nil {
|
||||
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
|
||||
return
|
||||
}
|
||||
var (
|
||||
imageModel embedding.ImageEmbedder
|
||||
prepared embedding.Image
|
||||
)
|
||||
if len(fileBytes) > 0 {
|
||||
var ok bool
|
||||
if imageModel, ok = embedding.AsImageEmbedder(instance); !ok {
|
||||
_ = c.Error(errors.NewBadRequestError(embedding.ErrImagesUnsupported.Error()))
|
||||
return
|
||||
}
|
||||
if prepared, err = imageprep.Prepare(fileBytes, imageModel.ImageLimits()); err != nil {
|
||||
_ = c.Error(errors.NewBadRequestError(err.Error()))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
queryVector, err := instance.Embed(types.WithEmbedQuery(ctx), query)
|
||||
if err != nil {
|
||||
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
|
||||
return
|
||||
}
|
||||
observations["dimension"] = len(queryVector)
|
||||
|
||||
results := make([]embeddingSimilarity, 0, len(documents)+1)
|
||||
score := func(index int, kind, content string, vector []float32) error {
|
||||
// A width that differs from the query's means the two sides are not
|
||||
// in one space; a cosine of 0 would hide that.
|
||||
if len(vector) != len(queryVector) {
|
||||
return fmt.Errorf("%s candidate %d has %d dimensions, the query %d",
|
||||
kind, index, len(vector), len(queryVector))
|
||||
}
|
||||
results = append(results, embeddingSimilarity{
|
||||
Index: index, Kind: kind, Content: content,
|
||||
Similarity: types.CosineSimilarity(queryVector, vector),
|
||||
})
|
||||
return nil
|
||||
}
|
||||
if len(documents) > 0 {
|
||||
vectors, err := instance.BatchEmbed(ctx, documents)
|
||||
if err == nil && len(vectors) != len(documents) {
|
||||
err = fmt.Errorf("%d vectors for %d documents", len(vectors), len(documents))
|
||||
}
|
||||
for i := 0; err == nil && i < len(documents); i++ {
|
||||
err = score(i, "text", documents[i], vectors[i])
|
||||
}
|
||||
if err != nil {
|
||||
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
|
||||
return
|
||||
}
|
||||
}
|
||||
if imageModel != nil {
|
||||
vectors, err := imageModel.BatchEmbedImages(ctx, []embedding.Image{prepared})
|
||||
if err == nil && len(vectors) != 1 {
|
||||
err = fmt.Errorf("%d vectors for 1 image", len(vectors))
|
||||
}
|
||||
if err == nil {
|
||||
err = score(len(documents), "image", "", vectors[0])
|
||||
}
|
||||
if err != nil {
|
||||
writeModelDebugResult(c, started, requestPreview, nil, err, observations)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
sort.SliceStable(results, func(i, j int) bool { return results[i].Similarity > results[j].Similarity })
|
||||
observations["result_count"] = len(results)
|
||||
writeModelDebugResult(c, started, requestPreview, gin.H{"results": results}, nil, observations)
|
||||
}
|
||||
|
|
@ -132,6 +132,7 @@ func TestURLResolution(t *testing.T) {
|
|||
"https://api.anthropic.com": "https://api.anthropic.com/v1/messages",
|
||||
"https://api.anthropic.com/v1": "https://api.anthropic.com/v1/messages",
|
||||
"https://api.minimaxi.com/anthropic": "https://api.minimaxi.com/anthropic/v1/messages",
|
||||
"https://api.minimax.cn/anthropic": "https://api.minimax.cn/anthropic/v1/messages",
|
||||
"https://open.bigmodel.cn/api/anthropic": "https://open.bigmodel.cn/api/anthropic/v1/messages",
|
||||
"https://proxy.example.com/v1/messages": "https://proxy.example.com/v1/messages",
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,13 +19,36 @@ type EmbeddingsCompat struct {
|
|||
MaxInputChars *int `json:"max_input_chars,omitempty"`
|
||||
// AcceptsTruncatePromptTokens marks a vLLM-class runtime, the only kind
|
||||
// that implements the `truncate_prompt_tokens` extension.
|
||||
AcceptsTruncatePromptTokens *bool `json:"accepts_truncate_prompt_tokens,omitempty"`
|
||||
RequestTimeout *int `json:"request_timeout_seconds,omitempty"`
|
||||
ExtraBody map[string]any `json:"extra_body,omitempty"`
|
||||
ImageField *string `json:"image_field,omitempty"`
|
||||
MaxImageBatchSize *int `json:"max_image_batch_size,omitempty"`
|
||||
MaxImageBytes *int `json:"max_image_bytes,omitempty"`
|
||||
ImageMIMETypes []string `json:"image_mime_types,omitempty"`
|
||||
AcceptsTruncatePromptTokens *bool `json:"accepts_truncate_prompt_tokens,omitempty"`
|
||||
RequestTimeout *int `json:"request_timeout_seconds,omitempty"`
|
||||
ExtraBody map[string]any `json:"extra_body,omitempty"`
|
||||
ImageField *string `json:"image_field,omitempty"`
|
||||
MaxImageBatchSize *int `json:"max_image_batch_size,omitempty"`
|
||||
MaxImageBytes *int `json:"max_image_bytes,omitempty"`
|
||||
ImageMIMETypes []string `json:"image_mime_types,omitempty"`
|
||||
ImageFormat *EmbeddingImageFormat `json:"image_format,omitempty"`
|
||||
ImagePrompt *string `json:"image_prompt,omitempty"`
|
||||
TextAsMessages *bool `json:"text_as_messages,omitempty"`
|
||||
}
|
||||
|
||||
// EmbeddingImageFormat is how an image travels on the OpenAI embedding
|
||||
// shape, which has no standard image input of its own.
|
||||
type EmbeddingImageFormat string
|
||||
|
||||
const (
|
||||
// EmbeddingImageObject puts {ImageField: data URI} in `input`, as Jina
|
||||
// and SGLang document.
|
||||
EmbeddingImageObject EmbeddingImageFormat = "object"
|
||||
// EmbeddingImageMessages sends a chat conversation in `messages` instead
|
||||
// of `input`, which is how vLLM serves multimodal embedding models
|
||||
// (https://docs.vllm.ai/en/latest/models/pooling_models.html). One
|
||||
// conversation yields one vector, so every image is its own request.
|
||||
EmbeddingImageMessages EmbeddingImageFormat = "messages"
|
||||
)
|
||||
|
||||
// Known reports whether f is a format this build can send; empty is known.
|
||||
func (f EmbeddingImageFormat) Known() bool {
|
||||
return f == "" || f == EmbeddingImageObject || f == EmbeddingImageMessages
|
||||
}
|
||||
|
||||
// EmbeddingsSettings is the resolved (fully defaulted) form.
|
||||
|
|
@ -76,6 +99,30 @@ type EmbeddingsSettings struct {
|
|||
// has no standard image input, so an endpoint that names none takes no
|
||||
// images), plus the documented per-request and per-image limits.
|
||||
ImageInput
|
||||
// ImageFormat chooses between ImageField's object in `input` and a chat
|
||||
// conversation in `messages`. Empty means object when ImageField is set.
|
||||
ImageFormat EmbeddingImageFormat `json:",omitempty"`
|
||||
// ImagePrompt is the text sent beside each image in the messages format;
|
||||
// some chat templates (VLM2Vec's) expect an instruction there. Empty
|
||||
// sends the image alone.
|
||||
ImagePrompt string `json:",omitempty"`
|
||||
// TextAsMessages sends texts in the messages format too, one per
|
||||
// request. A model whose chat template frames its inputs only places a
|
||||
// text and an image in the same space when both pass through it; a
|
||||
// plain `input` string skips the template.
|
||||
TextAsMessages bool `json:",omitempty"`
|
||||
}
|
||||
|
||||
// OpenAIImageFormat is the effective image format on the OpenAI shape, empty
|
||||
// when the endpoint takes no images there.
|
||||
func (s EmbeddingsSettings) OpenAIImageFormat() EmbeddingImageFormat {
|
||||
switch {
|
||||
case s.ImageFormat == EmbeddingImageMessages:
|
||||
return EmbeddingImageMessages
|
||||
case s.ImageField != "":
|
||||
return EmbeddingImageObject
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// BatchLimits renders the documented ceilings for SplitBatches.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,9 @@
|
|||
// - `encoding_format` is absent from Zhipu's reference;
|
||||
// - the parameter that separates a search query from an indexed document is
|
||||
// `input_type` on NIM, `task` on Jina and `taskType` on Gemini, and most
|
||||
// vendors have none.
|
||||
// vendors have none;
|
||||
// - images have no standard place: Jina and SGLang take an object in
|
||||
// `input`, vLLM takes a chat conversation in `messages`.
|
||||
//
|
||||
// This package contains no vendor names.
|
||||
package openaiembeddings
|
||||
|
|
@ -75,7 +77,7 @@ type response struct {
|
|||
// BuildRequestBody is the golden-test entry point: it returns the exact JSON
|
||||
// object that would be sent.
|
||||
func (c *Client) BuildRequestBody(texts []string, kind api.EmbedInputType) map[string]any {
|
||||
return c.body(texts, kind)
|
||||
return c.body("input", texts, kind)
|
||||
}
|
||||
|
||||
// BuildImageRequestBody is the golden-test entry point for images. The shape
|
||||
|
|
@ -87,14 +89,35 @@ func (c *Client) BuildImageRequestBody(images []api.EmbedImage, kind api.EmbedIn
|
|||
for _, img := range images {
|
||||
input = append(input, map[string]any{c.cfg.Settings.ImageField: img.DataURI()})
|
||||
}
|
||||
return c.body(input, kind)
|
||||
return c.body("input", input, kind)
|
||||
}
|
||||
|
||||
func (c *Client) body(input any, kind api.EmbedInputType) map[string]any {
|
||||
// BuildImageMessageBody is the golden-test entry point for the messages
|
||||
// format: one user turn carrying the image, then the configured prompt, as
|
||||
// in vLLM's multimodal embedding example.
|
||||
func (c *Client) BuildImageMessageBody(img api.EmbedImage, kind api.EmbedInputType) map[string]any {
|
||||
content := []any{map[string]any{"type": "image_url", "image_url": map[string]any{"url": img.DataURI()}}}
|
||||
if prompt := c.cfg.Settings.ImagePrompt; prompt != "" {
|
||||
content = append(content, map[string]any{"type": "text", "text": prompt})
|
||||
}
|
||||
return c.messageBody(content, kind)
|
||||
}
|
||||
|
||||
// BuildTextMessageBody is the golden-test entry point for a text sent in the
|
||||
// messages format.
|
||||
func (c *Client) BuildTextMessageBody(text string, kind api.EmbedInputType) map[string]any {
|
||||
return c.messageBody([]any{map[string]any{"type": "text", "text": text}}, kind)
|
||||
}
|
||||
|
||||
func (c *Client) messageBody(content []any, kind api.EmbedInputType) map[string]any {
|
||||
return c.body("messages", []any{map[string]any{"role": "user", "content": content}}, kind)
|
||||
}
|
||||
|
||||
func (c *Client) body(key string, input any, kind api.EmbedInputType) map[string]any {
|
||||
s := c.cfg.Settings
|
||||
body := map[string]any{
|
||||
"model": c.cfg.Endpoint.Model,
|
||||
"input": input,
|
||||
key: input,
|
||||
}
|
||||
if s.SendEncodingFormat {
|
||||
body["encoding_format"] = "float"
|
||||
|
|
@ -127,22 +150,46 @@ func (c *Client) body(input any, kind api.EmbedInputType) map[string]any {
|
|||
func (c *Client) Embed(
|
||||
ctx context.Context, texts []string, kind api.EmbedInputType,
|
||||
) ([][]float32, error) {
|
||||
if c.cfg.Settings.TextAsMessages {
|
||||
return c.postEach(ctx, len(texts), func(i int) map[string]any {
|
||||
return c.BuildTextMessageBody(texts[i], kind)
|
||||
})
|
||||
}
|
||||
return c.post(ctx, c.BuildRequestBody(texts, kind), len(texts))
|
||||
}
|
||||
|
||||
// AcceptsImages reports whether the vendor names an image field. Without
|
||||
// one a plain OpenAI-compatible server would read the objects as malformed
|
||||
// text input.
|
||||
func (c *Client) AcceptsImages() bool { return c.cfg.Settings.ImageField != "" }
|
||||
// AcceptsImages reports whether the vendor declares an image format. Without
|
||||
// one a plain OpenAI-compatible server would read image objects as
|
||||
// malformed text input.
|
||||
func (c *Client) AcceptsImages() bool { return c.cfg.Settings.OpenAIImageFormat() != "" }
|
||||
|
||||
// EmbedImages vectorizes one batch of images, in the order it was given.
|
||||
func (c *Client) EmbedImages(
|
||||
ctx context.Context, images []api.EmbedImage, kind api.EmbedInputType,
|
||||
) ([][]float32, error) {
|
||||
if !c.AcceptsImages() {
|
||||
return nil, fmt.Errorf("this embedding endpoint declares no image input")
|
||||
switch c.cfg.Settings.OpenAIImageFormat() {
|
||||
case api.EmbeddingImageObject:
|
||||
return c.post(ctx, c.BuildImageRequestBody(images, kind), len(images))
|
||||
case api.EmbeddingImageMessages:
|
||||
return c.postEach(ctx, len(images), func(i int) map[string]any {
|
||||
return c.BuildImageMessageBody(images[i], kind)
|
||||
})
|
||||
}
|
||||
return c.post(ctx, c.BuildImageRequestBody(images, kind), len(images))
|
||||
return nil, fmt.Errorf("this embedding endpoint declares no image input")
|
||||
}
|
||||
|
||||
// postEach sends one request per input, for the messages format where a
|
||||
// conversation yields exactly one vector.
|
||||
func (c *Client) postEach(ctx context.Context, n int, body func(int) map[string]any) ([][]float32, error) {
|
||||
out := make([][]float32, n)
|
||||
for i := range n {
|
||||
vectors, err := c.post(ctx, body(i), 1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[i] = vectors[0]
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func (c *Client) post(ctx context.Context, body map[string]any, want int) ([][]float32, error) {
|
||||
|
|
|
|||
|
|
@ -3,8 +3,10 @@ package openaiembeddings
|
|||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/models/api"
|
||||
|
|
@ -122,6 +124,92 @@ func TestTruncatePromptTokensReachesOnlyVLLMClassRuntimes(t *testing.T) {
|
|||
assert.Equal(t, 512, vllm.BuildRequestBody([]string{"a"}, api.EmbedDocument)["truncate_prompt_tokens"])
|
||||
}
|
||||
|
||||
// vLLM's multimodal embedding example sends one user turn in `messages`:
|
||||
// the image part, then the instruction its chat template expects. The rest
|
||||
// of the body is the same as for `input`.
|
||||
func TestVLLMImageMessageMatchesItsExample(t *testing.T) {
|
||||
c := newClient("http://vllm.internal/v1", api.EmbeddingsSettings{
|
||||
SendEncodingFormat: true, AcceptsTruncatePromptTokens: true, TruncatePromptTokens: 512,
|
||||
ImageFormat: api.EmbeddingImageMessages, ImagePrompt: "Represent the given image.",
|
||||
}, 0)
|
||||
require.True(t, c.AcceptsImages())
|
||||
|
||||
body := c.BuildImageMessageBody(api.EmbedImage{Data: []byte{0x89}, MIMEType: "image/png"}, api.EmbedDocument)
|
||||
assert.Equal(t, map[string]any{
|
||||
"model": "m", "encoding_format": "float", "truncate_prompt_tokens": 512,
|
||||
"messages": []any{map[string]any{"role": "user", "content": []any{
|
||||
map[string]any{"type": "image_url", "image_url": map[string]any{"url": "data:image/png;base64,iQ=="}},
|
||||
map[string]any{"type": "text", "text": "Represent the given image."},
|
||||
}}},
|
||||
}, body)
|
||||
|
||||
bare := newClient("http://vllm.internal/v1", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0)
|
||||
content := bare.BuildImageMessageBody(
|
||||
api.EmbedImage{Data: []byte{0x89}, MIMEType: "image/png"}, api.EmbedDocument,
|
||||
)["messages"].([]any)[0].(map[string]any)["content"]
|
||||
assert.Len(t, content, 1, "no prompt configured, so the image goes alone")
|
||||
}
|
||||
|
||||
// The object format needs a field name; without one there is no image input.
|
||||
func TestImageFormatDecidesWhetherImagesAreAccepted(t *testing.T) {
|
||||
assert.False(t, newClient("u", api.EmbeddingsSettings{}, 0).AcceptsImages())
|
||||
assert.False(t, newClient("u", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageObject}, 0).AcceptsImages())
|
||||
assert.True(t, newClient("u", api.EmbeddingsSettings{
|
||||
ImageInput: api.ImageInput{ImageField: "image"},
|
||||
}, 0).AcceptsImages())
|
||||
assert.True(t, newClient("u", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0).AcceptsImages())
|
||||
}
|
||||
|
||||
// A conversation yields one vector, so a batch is one request per item, and
|
||||
// texts follow images into `messages` only when the row asks.
|
||||
func TestMessagesFormatSendsOneRequestPerItem(t *testing.T) {
|
||||
t.Setenv("SSRF_WHITELIST", "127.0.0.1")
|
||||
var bodies []map[string]any
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
var body map[string]any
|
||||
_ = json.NewDecoder(r.Body).Decode(&body)
|
||||
bodies = append(bodies, body)
|
||||
if input, ok := body["input"].([]any); ok {
|
||||
parts := make([]string, len(input))
|
||||
for i := range input {
|
||||
parts[i] = fmt.Sprintf(`{"embedding":[0],"index":%d}`, i)
|
||||
}
|
||||
_, _ = fmt.Fprintf(w, `{"data":[%s]}`, strings.Join(parts, ","))
|
||||
return
|
||||
}
|
||||
_, _ = fmt.Fprintf(w, `{"data":[{"embedding":[%d],"index":0}]}`, len(bodies))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
c := newClient(server.URL+"/v1", api.EmbeddingsSettings{ImageFormat: api.EmbeddingImageMessages}, 0)
|
||||
got, err := c.EmbedImages(context.Background(), []api.EmbedImage{
|
||||
{Data: []byte{1}, MIMEType: "image/png"}, {Data: []byte{2}, MIMEType: "image/png"},
|
||||
}, api.EmbedDocument)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, [][]float32{{1}, {2}}, got)
|
||||
require.Len(t, bodies, 2)
|
||||
assert.Contains(t, bodies[0], "messages")
|
||||
assert.NotContains(t, bodies[0], "input")
|
||||
|
||||
bodies = nil
|
||||
_, err = c.Embed(context.Background(), []string{"a", "b"}, api.EmbedQuery)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, bodies, 1, "texts stay in one `input` request unless the row opts in")
|
||||
assert.Equal(t, []any{"a", "b"}, bodies[0]["input"])
|
||||
|
||||
bodies = nil
|
||||
c = newClient(server.URL+"/v1", api.EmbeddingsSettings{
|
||||
ImageFormat: api.EmbeddingImageMessages, TextAsMessages: true,
|
||||
}, 0)
|
||||
got, err = c.Embed(context.Background(), []string{"a", "b"}, api.EmbedQuery)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, [][]float32{{1}, {2}}, got)
|
||||
require.Len(t, bodies, 2)
|
||||
assert.Equal(t, []any{map[string]any{"role": "user", "content": []any{
|
||||
map[string]any{"type": "text", "text": "a"},
|
||||
}}}, bodies[0]["messages"])
|
||||
}
|
||||
|
||||
func TestDecodesArkTextResponse(t *testing.T) {
|
||||
server, path, body := serve(t, arkTextResponse)
|
||||
defer server.Close()
|
||||
|
|
|
|||
|
|
@ -816,6 +816,24 @@
|
|||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tongyi-embedding-vision-flash",
|
||||
"name": "tongyi-embedding-vision-flash",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 768,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "",
|
||||
"max_image_batch_size": 8,
|
||||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "tongyi-embedding-vision-flash (fixed 768 dimensions)",
|
||||
"match": "tongyi-embedding-vision-flash",
|
||||
|
|
@ -851,6 +869,25 @@
|
|||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "qwen3-vl-embedding",
|
||||
"name": "qwen3-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 2560,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "dimension",
|
||||
"max_batch_size": 20,
|
||||
"max_image_batch_size": 10,
|
||||
"max_image_bytes": 10000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "qwen3-vl-embedding (independent vectors by default)",
|
||||
"match": "qwen3-vl-embedding*",
|
||||
|
|
@ -869,6 +906,25 @@
|
|||
"max_image_bytes": 10000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "qwen2.5-vl-embedding",
|
||||
"name": "qwen2.5-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 1024,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "dimension",
|
||||
"max_batch_size": 1,
|
||||
"max_image_batch_size": 1,
|
||||
"max_image_bytes": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "qwen2.5-vl-embedding (fused vectors only)",
|
||||
"match": "qwen2.5-vl-embedding*",
|
||||
|
|
@ -887,6 +943,25 @@
|
|||
"max_image_bytes": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "multimodal-embedding-v1",
|
||||
"name": "multimodal-embedding-v1",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 1024,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "",
|
||||
"max_batch_size": 20,
|
||||
"max_image_batch_size": 1,
|
||||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "multimodal-embedding-v1",
|
||||
"match": "multimodal-embedding*",
|
||||
|
|
@ -1979,6 +2054,22 @@
|
|||
"xhigh": "high",
|
||||
"max": "high"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Qwen3-VL-Embedding (self-hosted)",
|
||||
"match": "*qwen3-vl-embedding*",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"source": "https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B",
|
||||
"compat": {
|
||||
"text_as_messages": true,
|
||||
"extra_body": {
|
||||
"add_special_tokens": true
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"gpustack": [],
|
||||
|
|
|
|||
|
|
@ -362,6 +362,16 @@
|
|||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "tongyi-embedding-vision-flash",
|
||||
"type": "Embedding",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "",
|
||||
"max_image_batch_size": 8,
|
||||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "tongyi-embedding-vision-flash",
|
||||
"type": "Embedding",
|
||||
|
|
@ -383,6 +393,17 @@
|
|||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "qwen3-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "dimension",
|
||||
"max_batch_size": 20,
|
||||
"max_image_batch_size": 10,
|
||||
"max_image_bytes": 10000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "qwen3-vl-embedding*",
|
||||
"type": "Embedding",
|
||||
|
|
@ -394,6 +415,17 @@
|
|||
"max_image_bytes": 10000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "qwen2.5-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "dimension",
|
||||
"max_batch_size": 1,
|
||||
"max_image_batch_size": 1,
|
||||
"max_image_bytes": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "qwen2.5-vl-embedding*",
|
||||
"type": "Embedding",
|
||||
|
|
@ -405,6 +437,17 @@
|
|||
"max_image_bytes": 5000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "multimodal-embedding-v1",
|
||||
"type": "Embedding",
|
||||
"compat": {
|
||||
"api": "dashscope-embeddings",
|
||||
"dimensions_field": "",
|
||||
"max_batch_size": 20,
|
||||
"max_image_batch_size": 1,
|
||||
"max_image_bytes": 3000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "multimodal-embedding*",
|
||||
"type": "Embedding",
|
||||
|
|
@ -927,6 +970,16 @@
|
|||
"xhigh": "high",
|
||||
"max": "high"
|
||||
}
|
||||
},
|
||||
{
|
||||
"match": "*qwen3-vl-embedding*",
|
||||
"type": "Embedding",
|
||||
"compat": {
|
||||
"text_as_messages": true,
|
||||
"extra_body": {
|
||||
"add_special_tokens": true
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"hunyuan": [
|
||||
|
|
|
|||
|
|
@ -592,6 +592,18 @@
|
|||
"dimension": 1152,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"id": "tongyi-embedding-vision-flash",
|
||||
"name": "tongyi-embedding-vision-flash",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 768,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"name": "tongyi-embedding-vision-flash (fixed 768 dimensions)",
|
||||
"match": "tongyi-embedding-vision-flash",
|
||||
|
|
@ -614,6 +626,18 @@
|
|||
],
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"id": "qwen3-vl-embedding",
|
||||
"name": "qwen3-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 2560,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"name": "qwen3-vl-embedding (independent vectors by default)",
|
||||
"match": "qwen3-vl-embedding*",
|
||||
|
|
@ -625,6 +649,18 @@
|
|||
],
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"id": "qwen2.5-vl-embedding",
|
||||
"name": "qwen2.5-vl-embedding",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 1024,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"name": "qwen2.5-vl-embedding (fused vectors only)",
|
||||
"match": "qwen2.5-vl-embedding*",
|
||||
|
|
@ -636,6 +672,18 @@
|
|||
],
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"id": "multimodal-embedding-v1",
|
||||
"name": "multimodal-embedding-v1",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image",
|
||||
"video"
|
||||
],
|
||||
"dimension": 1024,
|
||||
"source": "https://help.aliyun.com/zh/model-studio/multimodal-embedding-api-reference"
|
||||
},
|
||||
{
|
||||
"name": "multimodal-embedding-v1",
|
||||
"match": "multimodal-embedding*",
|
||||
|
|
@ -1342,6 +1390,16 @@
|
|||
"match": "o4*",
|
||||
"reasoning": true,
|
||||
"source": "https://platform.openai.com/docs/guides/reasoning"
|
||||
},
|
||||
{
|
||||
"name": "Qwen3-VL-Embedding (self-hosted)",
|
||||
"match": "*qwen3-vl-embedding*",
|
||||
"type": "Embedding",
|
||||
"input": [
|
||||
"text",
|
||||
"image"
|
||||
],
|
||||
"source": "https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B"
|
||||
}
|
||||
],
|
||||
"gpustack": [],
|
||||
|
|
|
|||
|
|
@ -105,11 +105,26 @@ func TestImageEmbeddingWireFormatPerVendor(t *testing.T) {
|
|||
"input": []any{map[string]any{"image": uri}},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "a generic (vLLM) row sends each image as a messages conversation", provider: "generic",
|
||||
model: "TIGER-Lab/VLM2Vec-Full", base: "/v1", images: 2, wantRequests: 2,
|
||||
wantPath: "/v1/embeddings",
|
||||
wantBody: map[string]any{
|
||||
"model": "TIGER-Lab/VLM2Vec-Full", "encoding_format": "float", "truncate_prompt_tokens": float64(511),
|
||||
"messages": []any{map[string]any{"role": "user", "content": []any{
|
||||
map[string]any{"type": "image_url", "image_url": map[string]any{"url": uri}},
|
||||
}}},
|
||||
},
|
||||
},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
up := newUpstream(t)
|
||||
ie, ok := AsImageEmbedder(imageEmbedder(t, up, tc.provider, tc.model, tc.base, nil))
|
||||
var spec *types.ModelSpecOverride
|
||||
if tc.provider == "generic" {
|
||||
spec = &types.ModelSpecOverride{Input: []string{"text", "image"}}
|
||||
}
|
||||
ie, ok := AsImageEmbedder(imageEmbedder(t, up, tc.provider, tc.model, tc.base, spec))
|
||||
require.True(t, ok, "the catalog lists this model as taking images")
|
||||
|
||||
images := pngs(tc.images)
|
||||
|
|
@ -137,9 +152,15 @@ func TestImageEmbeddingNeedsBothTheModelAndTheEndpoint(t *testing.T) {
|
|||
}{
|
||||
{"text model on a vendor with an image field", "jina", "jina-embeddings-v3", nil, false},
|
||||
{"text model on the OpenAI shape", "openai", "text-embedding-3-small", nil, false},
|
||||
// A generic OpenAI-compatible server has no image input to send to,
|
||||
// whatever the row declares.
|
||||
{"declared, but the endpoint names no image field", "generic", "my-clip", declare, false},
|
||||
{"generic endpoint, image input not declared", "generic", "my-clip", nil, false},
|
||||
{"declared on a generic (vLLM) endpoint", "generic", "my-clip", declare, true},
|
||||
{
|
||||
"declared, but the row switched to an object format without a field", "generic", "my-clip",
|
||||
&types.ModelSpecOverride{
|
||||
Input: []string{"text", "image"}, Compat: map[string]any{"image_format": "object"},
|
||||
},
|
||||
false,
|
||||
},
|
||||
{"declared on a vendor that names one", "jina", "my-jina-omni", declare, true},
|
||||
{
|
||||
"the row narrows a multimodal catalog entry", "volcengine", "doubao-embedding-vision-251215",
|
||||
|
|
@ -163,6 +184,37 @@ func TestImageEmbeddingNeedsBothTheModelAndTheEndpoint(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// Qwen3-VL-Embedding on vLLM matches its model card only when text and
|
||||
// image both pass the chat template and the prompt ends in <|endoftext|>,
|
||||
// the token its last-token pooling reads. A plain `input` string skips the
|
||||
// template and lands in a different space from the images.
|
||||
func TestSelfHostedQwen3VLEmbeddingSendsTextAndImagesThroughTheTemplate(t *testing.T) {
|
||||
up := newUpstream(t)
|
||||
e := imageEmbedder(t, up, "generic", "Qwen/Qwen3-VL-Embedding-8B", "/v1", nil)
|
||||
ie, ok := AsImageEmbedder(e)
|
||||
require.True(t, ok, "the catalog declares image input without a row override")
|
||||
|
||||
got, err := e.BatchEmbed(types.WithEmbedQuery(context.Background()), []string{"ab", "c"})
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, [][]float32{{2}, {1}}, got)
|
||||
_, err = ie.BatchEmbedImages(context.Background(), pngs(1))
|
||||
require.NoError(t, err)
|
||||
|
||||
message := func(part map[string]any) map[string]any {
|
||||
return map[string]any{
|
||||
"model": "Qwen/Qwen3-VL-Embedding-8B", "encoding_format": "float",
|
||||
"truncate_prompt_tokens": float64(511), "add_special_tokens": true,
|
||||
"messages": []any{map[string]any{"role": "user", "content": []any{part}}},
|
||||
}
|
||||
}
|
||||
require.Len(t, up.requests, 3, "the messages format carries one input per request")
|
||||
assert.Equal(t, message(map[string]any{"type": "text", "text": "ab"}), up.requests[0].body)
|
||||
assert.Equal(t, message(map[string]any{"type": "text", "text": "c"}), up.requests[1].body)
|
||||
assert.Equal(t, message(map[string]any{
|
||||
"type": "image_url", "image_url": map[string]any{"url": "data:image/png;base64,iQ=="},
|
||||
}), up.requests[2].body)
|
||||
}
|
||||
|
||||
func TestImageEmbeddingRefusesWhatTheVendorDocumentsItWillNotTake(t *testing.T) {
|
||||
up := newUpstream(t)
|
||||
gemini, ok := AsImageEmbedder(imageEmbedder(t, up, "gemini", "gemini-embedding-2", "/v1beta", nil))
|
||||
|
|
|
|||
|
|
@ -118,6 +118,10 @@ func answer(path string, body map[string]any) string {
|
|||
case strings.HasSuffix(path, "/embeddings/multimodal"):
|
||||
n := itemLen(body["input"].([]any)[0])
|
||||
return fmt.Sprintf(`{"data":{"embedding":[%d],"object":"embedding"}}`, n)
|
||||
case body["messages"] != nil:
|
||||
turn := body["messages"].([]any)[0].(map[string]any)
|
||||
n := itemLen(turn["content"].([]any)[0])
|
||||
return fmt.Sprintf(`{"data":[{"index":0,"embedding":[%d]}]}`, n)
|
||||
default:
|
||||
input := body["input"].([]any)
|
||||
for i := len(input) - 1; i >= 0; i-- {
|
||||
|
|
|
|||
|
|
@ -1,8 +1,11 @@
|
|||
{
|
||||
"aliyun/ASR/qwen3-asr-flash": "bb6104c02ce1013fd954de79c4998fd5b9e96e8b91cbbe083c9bc02a23660d7a",
|
||||
"aliyun/ASR/snapshot": "error: catalog: aliyun does not serve \"\" through a protocol this build implements: Alibaba's other recognition models take a public file URL or run as asynchronous tasks; only qwen3-asr-flash accepts the audio in the request",
|
||||
"aliyun/Embedding/multimodal-embedding-v1": "50cf14ae80fda14d123809d757833c44f08f7dc3b7f5d2d5d647d7ad66a8449b",
|
||||
"aliyun/Embedding/multimodal-embeddingsnapshot": "abd8e6e33132030c401670448cbc649f51c200bd5b0042157d3ecbabb093fd4d",
|
||||
"aliyun/Embedding/qwen2.5-vl-embedding": "277b713c73d600e661cab1a118b1f7ed6b936e4c48843b1cfb1a049385ddafff",
|
||||
"aliyun/Embedding/qwen2.5-vl-embeddingsnapshot": "f957cb1c3bdba7cfe010c2c2f58a5148e422c629a4486321296a35ee4b7790f0",
|
||||
"aliyun/Embedding/qwen3-vl-embedding": "80e3b7307df275d470f6c174f3bee891439fbe410d116720c7deba09bbde97af",
|
||||
"aliyun/Embedding/qwen3-vl-embeddingsnapshot": "3d6dba847c93c3ea41c64a18dd31ec521c37426a05d94968bfea90c951544127",
|
||||
"aliyun/Embedding/qwen3.7-text-embedding": "1747d31bcb369f32605000f3f2cc59406a4f7ddc5be6123c80e4e06f5d7fc2e7",
|
||||
"aliyun/Embedding/qwen3.7-text-embedding-flash": "14ce61f8e4aea6394b43f6383bd2ec3d1398abd0a8d666374c4300fb86845cd5",
|
||||
|
|
@ -10,7 +13,7 @@
|
|||
"aliyun/Embedding/text-embedding-v2": "b3eb2440097a26c93dea795bdf9b9d1ea4d4a11f1e20a034e2bb146d7226883f",
|
||||
"aliyun/Embedding/text-embedding-v3": "ef7acfe7150c048a9c46d5e8e2bc14d890df0721f877d76c41e6a2ed275bb7cc",
|
||||
"aliyun/Embedding/text-embedding-v4": "acd6a3536ee4f98330fb02e39c9f1952dbb1eb578d70a82c64278355ca44bc36",
|
||||
"aliyun/Embedding/tongyi-embedding-vision-flash": "fbcbf6d25d1078a137fa0d35dac9da104985a8da9b797bf161ec3b72a7775f02",
|
||||
"aliyun/Embedding/tongyi-embedding-vision-flash": "bff51fd7a6cdb822b6b7463e436d68f953c1599796814dd0d9181db15c4fc4ec",
|
||||
"aliyun/Embedding/tongyi-embedding-vision-plus": "fd37a8b75ef6f1c54dbbba9975e75f2668690d479f42beda61804f1be5b3c6bd",
|
||||
"aliyun/Embedding/tongyi-embedding-visionsnapshot": "37e1d6878a0ec196c49339fc5ba48615f12453b7f065c6ac0e50e272b76ac219",
|
||||
"aliyun/KnowledgeQA/deepseek-r1": "735322a596dda4a914b7e53041643eab61bd90b51e18e0cffd36dcc807b6fd16",
|
||||
|
|
@ -98,6 +101,7 @@
|
|||
"gemini/KnowledgeQA/gemini-3.7-flash": "570b30391590c3fa380e98e8323bf2187635d18db427eae3d75f9e63def26c8d",
|
||||
"gemini/KnowledgeQA/gemini-3.8-flash": "6d40802ed8d3b310e1e91c186f769a4f8a298138538853cf623ad43e59c29140",
|
||||
"gemini/KnowledgeQA/gemini-3snapshot": "c5931f2304774b718637f47540e78b66cbf855b4a4189c9fc0cea2040a12951d",
|
||||
"generic/Embedding/snapshotqwen3-vl-embeddingsnapshot": "5b7394b420438753d8a7c4ad58af94ceab910f1e3497c0ccef357ca357f96626",
|
||||
"generic/KnowledgeQA/gpt-5snapshot": "3e3712264316db008c1465a2d7c395072d85942d0352196e456342cef0b14eb0",
|
||||
"generic/KnowledgeQA/o1snapshot": "52340886b4160a0cb9342ae33a7c81304c9dad5d22c2d1c1ce073f89860a30b1",
|
||||
"generic/KnowledgeQA/o3snapshot": "782e836769e56c43d9c954ab2d563df7d2f476ae4fb4d588bb8bf3c88ae42ce6",
|
||||
|
|
@ -144,15 +148,15 @@
|
|||
"mimo/KnowledgeQA/mimo-v2.5": "a53a5bf889aa110e4460fed294a3230758ddaa49a6db60e8b0408fe183a60ed9",
|
||||
"mimo/KnowledgeQA/mimo-v2.5-pro": "3c38342327d6286db15a7548f651c2780d8b0998b2671d05f187781eb6a23262",
|
||||
"mimo/KnowledgeQA/mimo-v2.5-pro-ultraspeed": "a47aba2614020595d63accdc1c3c6058cfebdb3198df58936fb95e28c9046092",
|
||||
"minimax/ASR/asr-1.0": "3466d2d1953d8165ba7677de85f36fa933118f5dc902bf39fdfa9158b0ec0eab",
|
||||
"minimax/KnowledgeQA/MiniMax-M2": "145995deb36b85b3f0933c4aed6273b97cbacf92891cae224949dae906d251cd",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.1": "b745d75ad3095e2fdc25d22d7f723c30078b9bbb155c4d0f7d1bc64c5dbfbb59",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.1-highspeed": "6f3d05e123ef452d9ed0b40456ccbe08c552ca2408f225dce40115add1b23fd5",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.5": "163e53955eb8a46362e4581513644be9c86c39ff744819f777ae4f2bd81bf81d",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.5-highspeed": "c1731e761db65503eaa72acdba28c95a1eaae15882d813534f7831b3256022cb",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.7": "2efbb13fcb0efd7c36987ad5d12d1d6c1d80f2e0b5518abeaac8dc603ff44050",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.7-highspeed": "7f7518fa498dc12eeec38d8d73daae4f1a43a207d07b9d2369948aaca347c5e8",
|
||||
"minimax/KnowledgeQA/MiniMax-M3": "e0ab669e2f427ed88ad4d5defec501fa835fa6e32922f8942be5216d5076b92b",
|
||||
"minimax/ASR/asr-1.0": "1c51f1e261087dc295ca455f3f580ebd4e2af19b4dc520635729415f49ef4432",
|
||||
"minimax/KnowledgeQA/MiniMax-M2": "43bf0fa0d1325b5c52f046109c730d6c7549cb4a31696cae7119a68480802fe9",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.1": "bd74e5923d0d12f7cd5d27cdc9ce7a03aea561fb932fbc2df7b9527f30a0b27e",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.1-highspeed": "dcc02614ff25e6ec426f81e8c5bc8c1fb54fe15fc3c61bc952158cdc5605b7a1",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.5": "12e81e74ad422b52aab382584e0e329d28f5e6f97f9fa517a16c63b448f46156",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.5-highspeed": "6b87c598fcfcb674dbe600ef19f30b518af8b6c6ba959d84ecbd02542eee3c12",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.7": "3f6c046bb99022ee88f3c99f3f9e7e75ecfabaad6f97a601352297ec33ffadde",
|
||||
"minimax/KnowledgeQA/MiniMax-M2.7-highspeed": "bc4a3116243b4dd2a4b764db9fa8155f8262ab4a05a34cc1e8b6bcc2e17cde72",
|
||||
"minimax/KnowledgeQA/MiniMax-M3": "153a02aafdcdfbd115c5e1fd6b5f03b145dfe22e13cf2641dc9f52722ac4ca98",
|
||||
"modelscope/Embedding/Qwen/Qwen3-Embedding-0.6B": "a4515af27268a360fe7ee62eb4c4785bc5e3079f65e0264acc99ba974b9a777b",
|
||||
"modelscope/Embedding/Qwen/Qwen3-Embedding-8B": "a76960f74528a100af92aad4def93306da5eb819ba8fc916693197d779f0ebd2",
|
||||
"modelscope/KnowledgeQA/Qwen/Qwen3-235B-A22B-Instruct-2507": "cbd8290913b5b1a584aaa9cd20ccb6478b67c62a73d9aa3f9f9c5aee0baf7d80",
|
||||
|
|
|
|||
|
|
@ -120,6 +120,11 @@ func newGenericProvider() *Definition {
|
|||
SendEncodingFormat: api.Ptr(true),
|
||||
DimensionsField: api.Ptr("dimensions"),
|
||||
AcceptsTruncatePromptTokens: api.Ptr(true),
|
||||
// Used only when the row declares image input. vLLM, the most
|
||||
// common server behind this vendor, takes images only as a
|
||||
// chat conversation in `messages`; an SGLang row switches to
|
||||
// {"image_format": "object", "image_field": "image"}.
|
||||
ImageFormat: api.Ptr(api.EmbeddingImageMessages),
|
||||
},
|
||||
Rerank: api.RerankCompat{
|
||||
// Any OpenAI-compatible endpoint an operator points here is most
|
||||
|
|
|
|||
|
|
@ -15,9 +15,10 @@
|
|||
// - the only credential is a bearer API key: the GroupId that the legacy
|
||||
// /v1/text/chatcompletion_v2 endpoint required is not used here, so this
|
||||
// vendor needs no ExtraFields;
|
||||
// - the mainland endpoint is https://api.minimaxi.com/v1 and the global one
|
||||
// - the mainland endpoint is https://api.minimax.cn/v1 and the global one
|
||||
// https://api.minimax.io/v1; keys are region-bound (mainland keys to
|
||||
// minimaxi.com, global keys to minimax.io).
|
||||
// minimax.cn, global keys to minimax.io). Mainland endpoint reference:
|
||||
// https://platform.minimax.cn/docs/api-reference/text-openai-api.
|
||||
//
|
||||
// Second protocol: an Anthropic Messages facade lives at
|
||||
// https://api.minimax.io/anthropic (mainland form in MinimaxAnthropicBaseURL), and
|
||||
|
|
@ -40,10 +41,8 @@
|
|||
// unverified: max output caps. The model table publishes context windows but
|
||||
// no per-model output cap, so the max_output_tokens in models.json are
|
||||
// carried over unconfirmed;
|
||||
// unverified: only the global https://api.minimax.io/anthropic form of the
|
||||
// Anthropic facade is spelled out in the docs; the mainland
|
||||
// https://api.minimaxi.com/anthropic that MinimaxAnthropicBaseURL holds is inferred
|
||||
// from the region split of the /v1 hosts.
|
||||
// The mainland Anthropic facade is documented at
|
||||
// https://platform.minimax.cn/docs/api-reference/text-anthropic-api.
|
||||
package providers
|
||||
|
||||
import (
|
||||
|
|
@ -60,13 +59,13 @@ var minimaxIcon []byte
|
|||
const MinimaxID = "minimax"
|
||||
|
||||
// MinimaxBaseURL is the mainland-China OpenAI-compatible endpoint.
|
||||
const MinimaxBaseURL = "https://api.minimaxi.com/v1"
|
||||
const MinimaxBaseURL = "https://api.minimax.cn/v1"
|
||||
|
||||
// MinimaxGlobalBaseURL is the international endpoint.
|
||||
const MinimaxGlobalBaseURL = "https://api.minimax.io/v1"
|
||||
|
||||
// MinimaxAnthropicBaseURL is the Anthropic Messages compatibility facade.
|
||||
const MinimaxAnthropicBaseURL = "https://api.minimaxi.com/anthropic"
|
||||
const MinimaxAnthropicBaseURL = "https://api.minimax.cn/anthropic"
|
||||
|
||||
func newMinimaxProvider() *Definition {
|
||||
return &Definition{
|
||||
|
|
@ -79,13 +78,14 @@ func newMinimaxProvider() *Definition {
|
|||
"zh-CN": "MiniMax-M3, MiniMax-M2.7, MiniMax-M2.7-highspeed 等。国际版 " +
|
||||
MinimaxGlobalBaseURL + ";Anthropic 兼容接口 " + MinimaxAnthropicBaseURL,
|
||||
},
|
||||
Website: "https://platform.minimaxi.com",
|
||||
Website: "https://platform.minimax.cn",
|
||||
Icon: minimaxIcon,
|
||||
API: api.APIOpenAICompletions,
|
||||
Order: 16,
|
||||
RequiresAuth: true,
|
||||
Auth: AuthBearer,
|
||||
URLPatterns: []string{"minimax.io", "minimaxi.com"},
|
||||
// Retain inference for saved rows using the previous mainland domain.
|
||||
URLPatterns: []string{"minimax.cn", "minimax.io", "minimaxi.com"},
|
||||
DefaultBaseURLs: map[types.ModelType]string{
|
||||
types.ModelTypeKnowledgeQA: MinimaxBaseURL,
|
||||
},
|
||||
|
|
|
|||
|
|
@ -22,6 +22,65 @@ var expectedIDs = []string{
|
|||
"jina", "nvidia", "novita", "gpustack",
|
||||
}
|
||||
|
||||
func TestMinimaxOfficialDefaultURLs(t *testing.T) {
|
||||
v, ok := modelruntime.Get(providers.MinimaxID)
|
||||
if !ok {
|
||||
t.Fatal("MiniMax provider missing")
|
||||
}
|
||||
for _, kind := range []types.ModelType{types.ModelTypeKnowledgeQA, types.ModelTypeASR} {
|
||||
if got := v.GetDefaultURL(kind); got != "https://api.minimax.cn/v1" {
|
||||
t.Errorf("%s default = %q, want current mainland endpoint", kind, got)
|
||||
}
|
||||
}
|
||||
if providers.MinimaxAnthropicBaseURL != "https://api.minimax.cn/anthropic" {
|
||||
t.Errorf("unexpected mainland Anthropic URL: %s", providers.MinimaxAnthropicBaseURL)
|
||||
}
|
||||
if v.Website != "https://platform.minimax.cn" {
|
||||
t.Errorf("unexpected mainland website: %s", v.Website)
|
||||
}
|
||||
for _, base := range []string{
|
||||
"https://api.minimax.cn/v1", "https://api.minimax.cn/anthropic",
|
||||
"https://api.minimaxi.com/v1", "https://api.minimaxi.com/anthropic",
|
||||
"https://api.minimax.io/v1", "https://api.minimax.io/anthropic",
|
||||
} {
|
||||
if got := modelruntime.DetectByURL(base); got != providers.MinimaxID {
|
||||
t.Errorf("DetectByURL(%q) = %q", base, got)
|
||||
}
|
||||
resolved, err := modelruntime.Resolve(modelruntime.Ref{BaseURL: base, Model: "MiniMax-M2.7"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resolved.BaseURL != base {
|
||||
t.Errorf("explicit address was rewritten: %q -> %q", base, resolved.BaseURL)
|
||||
}
|
||||
wantAPI := api.APIOpenAICompletions
|
||||
if strings.HasSuffix(base, "/anthropic") {
|
||||
wantAPI = api.APIAnthropicMessages
|
||||
}
|
||||
if resolved.API != wantAPI {
|
||||
t.Errorf("%s resolved API = %s, want %s", base, resolved.API, wantAPI)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuiltinDefaultURLsAcrossModelTypes(t *testing.T) {
|
||||
for _, vendor := range modelruntime.List() {
|
||||
for _, kind := range vendor.ModelTypes {
|
||||
base := vendor.GetDefaultURL(kind)
|
||||
// These connections have no hosted service to prefill.
|
||||
if vendor.ID == "generic" || vendor.ID == "huggingface_tei" {
|
||||
if base != "" {
|
||||
t.Errorf("%s/%s should require an operator URL", vendor.ID, kind)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if !strings.HasPrefix(base, "https://") && !strings.HasPrefix(base, "http://") {
|
||||
t.Errorf("%s/%s has no absolute default URL: %q", vendor.ID, kind, base)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllVendorsRegistered(t *testing.T) {
|
||||
if len(expectedIDs) != 29 {
|
||||
t.Fatalf("expected 29 vendor ids in the spec, got %d", len(expectedIDs))
|
||||
|
|
|
|||
|
|
@ -487,6 +487,11 @@ func resolveEmbeddings(
|
|||
protocol = settings.API
|
||||
}
|
||||
settings.API = protocol
|
||||
if !settings.ImageFormat.Known() {
|
||||
return nil, fmt.Errorf(
|
||||
"catalog: unknown embedding image_format %q on %s/%s (expected %q or %q)",
|
||||
settings.ImageFormat, vendor.ID, spec.ID, api.EmbeddingImageObject, api.EmbeddingImageMessages)
|
||||
}
|
||||
|
||||
out := &Resolved{
|
||||
Vendor: configcopy.Clone(vendor.Definition),
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ const (
|
|||
// imageLoadConcurrency bounds image reads from storage.
|
||||
imageLoadConcurrency = 4
|
||||
// MinImageKeepScore is the least vector similarity that keeps a pictorial
|
||||
// image hit over a text reranker. It sits above the 0.1 image recall
|
||||
// image hit over a text reranker. It sits above the 0.2 image recall
|
||||
// threshold: recall only lets an image compete, keeping it needs more.
|
||||
// Empirical, on the cosine scale most engines report.
|
||||
MinImageKeepScore = 0.25
|
||||
|
|
|
|||
|
|
@ -0,0 +1,115 @@
|
|||
package reranking
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// Observe selection in the shared implementation: agent tools and quick answers
|
||||
// must expose the same decisions, including early exits and model-error fallback.
|
||||
func recordSelection(ctx context.Context, input []*types.SearchResult, res *Result, opts Options,
|
||||
scoredAsImage map[int]bool,
|
||||
) {
|
||||
out := summarizeSelection(input, res, opts, scoredAsImage)
|
||||
logger.GetLogger(ctx).WithFields(logger.Fields(out)).Info("rerank selection")
|
||||
_, span := langfuse.GetManager().StartSpan(ctx, langfuse.SpanOptions{Name: "rerank.selection"})
|
||||
span.Finish(out, nil, nil)
|
||||
}
|
||||
|
||||
func summarizeSelection(input []*types.SearchResult, res *Result, opts Options,
|
||||
scoredAsImage map[int]bool,
|
||||
) map[string]interface{} {
|
||||
// Candidates retain input pointers. Final rows are copies, so use the
|
||||
// returned input indices to distinguish retained and rejected inputs.
|
||||
candidates := make(map[*types.SearchResult]int, len(res.Candidates))
|
||||
for i, r := range res.Candidates {
|
||||
candidates[r] = i
|
||||
}
|
||||
returned := make(map[int]*types.SearchResult, len(res.Indices))
|
||||
for i, index := range res.Indices {
|
||||
returned[index] = res.Results[i]
|
||||
}
|
||||
modelScores := make(map[int]float64, len(res.ModelScores))
|
||||
for _, score := range res.ModelScores {
|
||||
modelScores[score.Index] = score.RelevanceScore
|
||||
}
|
||||
passed := make(map[string]bool, len(res.Scored))
|
||||
for _, r := range res.Scored {
|
||||
passed[r.ID] = true
|
||||
}
|
||||
inputTypes, candidateTypes, outputTypes := map[string]int{}, map[string]int{}, map[string]int{}
|
||||
for _, r := range res.Candidates {
|
||||
candidateTypes[r.ChunkType]++
|
||||
}
|
||||
for _, r := range res.Results {
|
||||
if r != nil {
|
||||
outputTypes[r.ChunkType]++
|
||||
}
|
||||
}
|
||||
images := make([]map[string]interface{}, 0)
|
||||
decisions := map[string]int{}
|
||||
imageCount := 0
|
||||
for i, r := range input {
|
||||
if r == nil {
|
||||
continue
|
||||
}
|
||||
inputTypes[r.ChunkType]++
|
||||
if !isImageVectorHit(r) {
|
||||
continue
|
||||
}
|
||||
imageCount++
|
||||
pos, candidate := candidates[r]
|
||||
score, hasScore := modelScores[pos]
|
||||
decision := "candidate_cap_or_empty_passage"
|
||||
if kept, ok := returned[i]; ok {
|
||||
decision = "kept"
|
||||
if res.Diagnostics.Outcome == types.RerankOutcomeModelError {
|
||||
decision = "model_error_fallback"
|
||||
} else if kept.Metadata[types.MetadataKeptBy] == types.KeptByImageVector {
|
||||
decision = "image_vector_fallback"
|
||||
}
|
||||
} else if candidate {
|
||||
switch {
|
||||
case !hasScore:
|
||||
decision = "no_model_score"
|
||||
case passed[r.ID]:
|
||||
decision = "top_k_mmr_cut"
|
||||
default:
|
||||
decision = "rerank_rejected"
|
||||
}
|
||||
}
|
||||
decisions[decision]++
|
||||
if len(images) >= 10 {
|
||||
continue
|
||||
}
|
||||
row := map[string]interface{}{
|
||||
"input_rank": i + 1, "chunk_id": r.ID, "knowledge_id": r.KnowledgeID,
|
||||
"vector_score": fmt.Sprintf("%.4f", r.VectorScore),
|
||||
"retrieval_score": fmt.Sprintf("%.4f", r.Score), "decision": decision,
|
||||
}
|
||||
if candidate && hasScore {
|
||||
row["model_score"] = fmt.Sprintf("%.4f", score)
|
||||
row["scored_as"] = "text"
|
||||
if scoredAsImage[pos] {
|
||||
row["scored_as"] = "image"
|
||||
}
|
||||
}
|
||||
images = append(images, row)
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"input_count": len(input), "candidate_count": len(res.Candidates), "output_count": len(res.Results),
|
||||
"input_chunk_types": inputTypes, "candidate_chunk_types": candidateTypes, "output_chunk_types": outputTypes,
|
||||
"outcome": res.Diagnostics.Outcome, "images_scored": res.Diagnostics.ImagesScored,
|
||||
"images_kept_by_vector": res.Diagnostics.ImagesKept, "image_decision_counts": decisions,
|
||||
"image_samples": images, "image_samples_truncated": imageCount - len(images),
|
||||
// Keyword ordering and explicit scopes may use -Inf. Strings are JSON safe.
|
||||
"threshold": fmt.Sprint(opts.Threshold), "effective_threshold": fmt.Sprint(res.Diagnostics.EffectiveThreshold),
|
||||
"fallback_min_score": fmt.Sprint(opts.FallbackMinScore), "image_keep_score": fmt.Sprint(opts.ImageKeepScore),
|
||||
"max_candidates": opts.MaxCandidates, "top_k": opts.TopK,
|
||||
"top_model_score": fmt.Sprint(res.Diagnostics.TopScore),
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
package reranking
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestSelectionDiagnosticsExplainRejectedAndProtectedImages(t *testing.T) {
|
||||
input := []*types.SearchResult{
|
||||
imageHit("weak", "private image caption", 0.21, 0),
|
||||
imageHit("protected", "another caption", 0.4, 0),
|
||||
}
|
||||
opts := Options{Threshold: 0.3, FallbackMinScore: 0.15, ImageKeepScore: 0.25}
|
||||
res := Rerank(context.Background(), &textScorer{}, "q", input, opts)
|
||||
out := summarizeSelection(input, res, opts, nil)
|
||||
require.Equal(t, map[string]int{"rerank_rejected": 1, "image_vector_fallback": 1}, out["image_decision_counts"])
|
||||
require.Equal(t, map[string]int{"image_vector": 2}, out["input_chunk_types"])
|
||||
require.Equal(t, map[string]int{"image_vector": 1}, out["output_chunk_types"])
|
||||
rows := out["image_samples"].([]map[string]interface{})
|
||||
require.Equal(t, "0.2100", rows[0]["vector_score"])
|
||||
require.Equal(t, "0.0000", rows[0]["model_score"])
|
||||
require.Equal(t, "text", rows[0]["scored_as"])
|
||||
b, err := json.Marshal(out)
|
||||
require.NoError(t, err)
|
||||
require.NotContains(t, string(b), "private image caption")
|
||||
}
|
||||
|
||||
func TestSelectionDiagnosticsExplainModelErrorAndAllowNonfiniteThresholds(t *testing.T) {
|
||||
input := []*types.SearchResult{imageHit("image", "caption", 0.3, 0)}
|
||||
opts := Options{Threshold: math.Inf(-1), FallbackMinScore: math.Inf(-1)}
|
||||
res := Rerank(context.Background(), &textScorer{err: errors.New("offline")}, "q", input, opts)
|
||||
out := summarizeSelection(input, res, opts, nil)
|
||||
require.Equal(t, map[string]int{"model_error_fallback": 1}, out["image_decision_counts"])
|
||||
require.Equal(t, "-Inf", out["threshold"])
|
||||
_, err := json.Marshal(out)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func TestSelectionDiagnosticsDistinguishTopKFromModelRejection(t *testing.T) {
|
||||
input := []*types.SearchResult{
|
||||
imageHit("first", "first", 0.4, 0), imageHit("second", "second", 0.3, 0),
|
||||
}
|
||||
opts := Options{Threshold: 0.3, TopK: 1}
|
||||
model := &textScorer{scores: map[string]float64{"first": 0.9, "second": 0.8}}
|
||||
res := Rerank(context.Background(), model, "q", input, opts)
|
||||
out := summarizeSelection(input, res, opts, map[int]bool{0: true})
|
||||
require.Equal(t, map[string]int{"kept": 1, "top_k_mmr_cut": 1}, out["image_decision_counts"])
|
||||
require.Equal(t, "image", out["image_samples"].([]map[string]interface{})[0]["scored_as"])
|
||||
}
|
||||
|
|
@ -122,6 +122,9 @@ func Rerank(
|
|||
},
|
||||
}
|
||||
|
||||
var scoredAsImage map[int]bool
|
||||
defer func() { recordSelection(ctx, results, res, opts, scoredAsImage) }()
|
||||
|
||||
keep := topByScore(results, opts.MaxCandidates)
|
||||
candidateIdx := make([]int, 0, len(results))
|
||||
for i, r := range results {
|
||||
|
|
|
|||
|
|
@ -96,6 +96,7 @@ func SummarizeSearchResults(results []*types.SearchResult, limit int) map[string
|
|||
"composite_score": fmt.Sprintf("%.4f", sr.Score),
|
||||
"match_type": sr.MatchType,
|
||||
"chunk_type": sr.ChunkType,
|
||||
"vector_score": fmt.Sprintf("%.4f", sr.VectorScore),
|
||||
"preview": TruncateRunes(sr.Content, 160),
|
||||
}
|
||||
if sr.Metadata != nil {
|
||||
|
|
@ -160,6 +161,8 @@ func summarizeIndexHits(hits []*types.IndexWithScore, limit int) []map[string]in
|
|||
"knowledge_base_id": hit.KnowledgeBaseID,
|
||||
"score": fmt.Sprintf("%.4f", hit.Score),
|
||||
"match_type": hit.MatchType,
|
||||
"source_type": hit.SourceType,
|
||||
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
|
||||
"preview": TruncateRunes(hit.Content, 160),
|
||||
})
|
||||
}
|
||||
|
|
@ -253,6 +256,8 @@ func summarizeRetrieveHits(
|
|||
"knowledge_base_id": hit.KnowledgeBaseID,
|
||||
"score": fmt.Sprintf("%.4f", hit.Score),
|
||||
"match_type": hit.MatchType,
|
||||
"source_type": hit.SourceType,
|
||||
"vector_score": fmt.Sprintf("%.4f", hit.VectorScore),
|
||||
"engine": string(ref.engine),
|
||||
"retriever": string(ref.retriever),
|
||||
"preview": TruncateRunes(hit.Content, 160),
|
||||
|
|
@ -287,6 +292,8 @@ func SummarizePassagePreviews(
|
|||
"knowledge_title": sr.KnowledgeTitle,
|
||||
"retrieval_score": fmt.Sprintf("%.4f", sr.Score),
|
||||
"match_type": sr.MatchType,
|
||||
"chunk_type": sr.ChunkType,
|
||||
"vector_score": fmt.Sprintf("%.4f", sr.VectorScore),
|
||||
"preview": TruncateRunes(passages[i], 160),
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -90,3 +90,21 @@ func TestSummarizeSearchResults_sortsByScore(t *testing.T) {
|
|||
t.Fatalf("unexpected hits: %#v", hits)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrievalPreviewsPreserveImageSourceAndRawVectorScore(t *testing.T) {
|
||||
out := SummarizeRetrieveOutput([]*types.RetrieveResult{{
|
||||
RetrieverType: types.VectorRetrieverType,
|
||||
Results: []*types.IndexWithScore{{
|
||||
ChunkID: "image", SourceType: types.ImageSourceType, Score: 0.7, VectorScore: 0.2688,
|
||||
}},
|
||||
}})
|
||||
hit := out["top_hits"].([]map[string]interface{})[0]
|
||||
if hit["source_type"] != types.ImageSourceType || hit["vector_score"] != "0.2688" || hit["score"] != "0.7000" {
|
||||
t.Fatalf("image identity/raw score lost: %#v", hit)
|
||||
}
|
||||
groups := out["top_hits_by_retriever"].([]map[string]interface{})
|
||||
groupHit := groups[0]["top_hits"].([]map[string]interface{})[0]
|
||||
if groupHit["source_type"] != types.ImageSourceType || groupHit["vector_score"] != "0.2688" {
|
||||
t.Fatalf("per-retriever image identity/raw score lost: %#v", groupHit)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,6 +15,9 @@ type KnowledgeProcessOverrides struct {
|
|||
// pipeline switch for this one document. nil keeps the knowledge base's
|
||||
// setting.
|
||||
ImageAttrsEnabled *bool `json:"image_attrs_enabled,omitempty"`
|
||||
// ImageVectorEnabled overrides image vector indexing for this document.
|
||||
// It does not change the KB-wide retrieval setting or embedding model.
|
||||
ImageVectorEnabled *bool `json:"image_vector_enabled,omitempty"`
|
||||
// ImageActions overrides rows of the attribute->work table for this document
|
||||
// (merged per action key on top of the knowledge base's table).
|
||||
// nil keeps the knowledge base's setting.
|
||||
|
|
@ -36,6 +39,7 @@ type EffectiveProcessConfig struct {
|
|||
QuestionGenerationConfig QuestionGenerationConfig
|
||||
GraphEnabled bool
|
||||
ImageAttrsEnabled bool
|
||||
ImageVectorEnabled bool
|
||||
ImageActions ImageActionsConfig
|
||||
ExtractConfig ExtractConfig
|
||||
}
|
||||
|
|
|
|||
|
|
@ -536,6 +536,9 @@ type ImageMultimodalPayload struct {
|
|||
// in the payload rather than being re-read from the knowledge base at handle
|
||||
// time.
|
||||
ImageAttrsEnabled bool `json:"image_attrs_enabled,omitempty"`
|
||||
// ImageVectorEnabled is resolved at enqueue time. nil preserves the KB
|
||||
// fallback for tasks queued before per-document indexing overrides existed.
|
||||
ImageVectorEnabled *bool `json:"image_vector_enabled,omitempty"`
|
||||
// ImageActions is the attribute→work table, resolved from the KB config at
|
||||
// enqueue time. A task without it — one already in flight when attribute
|
||||
// observation shipped — falls back to the conservative policy: run OCR.
|
||||
|
|
|
|||
|
|
@ -89,6 +89,9 @@
|
|||
| 需要附带厂商私有参数,如阿里云联网搜索 | `{"extra_body": {"enable_search": true}}` |
|
||||
| 多轮对话回传思考内容时报错 | `{"replay_reasoning_content": false}` |
|
||||
| 自建向量服务一次请求只接受 16 条 | `{"max_batch_size": 16}` |
|
||||
| vLLM 部署的多模态向量模型(VLM2Vec、GME 等) | 打开「图片输入」即可,自定义厂商默认按 vLLM 的 `messages` 格式发图片;模板需要指令时加 `{"image_prompt": "Represent the given image."}`;文本也必须经过对话模板才和图片同空间时加 `{"text_as_messages": true}` |
|
||||
| vLLM 部署的 Qwen3-VL-Embedding | 无需填写:自定义厂商下模型名含 `qwen3-vl-embedding` 时已内置图片输入与 `{"text_as_messages": true, "extra_body": {"add_special_tokens": true}}`。文本走普通 `input` 会绕过对话模板,与图片不在同一空间;不加 `add_special_tokens` 时末尾缺少模型取向量用的 `<\|endoftext\|>`,与[模型卡](https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B)结果不符。已用该模型建立的索引需重建向量 |
|
||||
| SGLang 部署的多模态向量模型 | 打开「图片输入」,并填 `{"image_format": "object", "image_field": "image"}` |
|
||||
| 向量或重排服务响应慢,需要更长的超时 | `{"request_timeout_seconds": 120}` |
|
||||
| 自建重排服务返回的是未归一化分数(logit) | `{"score_scale": "logit"}` |
|
||||
|
||||
|
|
@ -191,7 +194,10 @@
|
|||
| `accepts_truncate_prompt_tokens` | bool | false | 服务是否支持 vLLM 的 `truncate_prompt_tokens` |
|
||||
| `request_timeout_seconds` | int | 60 | 单次请求超时(秒) |
|
||||
| `extra_body` | object | 空 | 附加到每次请求的字段 |
|
||||
| `image_field` | string | 空 | 仅 `openai-embeddings`:`input` 里承载图片的对象键名(Jina 为 `image`)。留空表示该端点不收图片 |
|
||||
| `image_field` | string | 空 | 仅 `openai-embeddings`:`input` 里承载图片的对象键名(Jina、SGLang 为 `image`)。留空且 `image_format` 不是 `messages` 时表示该端点不收图片 |
|
||||
| `image_format` | string | 空(自定义厂商为 `messages`) | 仅 `openai-embeddings`:图片的发送方式。`object` 把 `{image_field: 图片}` 放进 `input`;`messages` 把图片作为一轮对话放进 `messages`(vLLM 的多模态向量接口),每张图一个请求。留空时有 `image_field` 即按 `object` 处理 |
|
||||
| `image_prompt` | string | 空 | 仅 `messages` 格式:随图片一起发送的文字,部分模型的对话模板需要(如 VLM2Vec 的 `Represent the given image.`) |
|
||||
| `text_as_messages` | bool | false | 仅 `openai-embeddings`:文本也按 `messages` 格式发送,每条一个请求。适用于必须经过对话模板、文本和图片才在同一向量空间的模型(自定义厂商下的 Qwen3-VL-Embedding 已内置);已有索引的模型改动后需要重建向量 |
|
||||
| `max_image_batch_size` | int | 1 | 单次请求的最大图片数,超出自动分批 |
|
||||
| `max_image_bytes` | int | 0(不限) | 单张图片的字节上限,超出的图片直接拒绝、不发请求 |
|
||||
| `image_mime_types` | array | 空(不限) | 厂商接受的图片格式,如 `["image/png", "image/jpeg"]` |
|
||||
|
|
|
|||
Loading…
Reference in New Issue