E2B reports a running template build as in_progress. Mapping that status
to building lets catalog refresh reconcile the current build instead of
surfacing the raw status.
* fix(models): align configuration defaults and multimodal workflows
* feat(image): enhance image retrieval diagnostics and filtering
- Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results.
- Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores.
- Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording.
- Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling.
* feat(embeddings): enhance image handling with new formats and message support
- Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats.
- Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows.
- Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements.
- Added tests to validate the new image message handling and ensure correct request formatting for different scenarios.
- Updated documentation to reflect changes in image input handling and configuration options.
* feat(similarity): add embedding similarity mode for enhanced document scoring
- Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query.
- Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type.
- Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity.
- Added tests to validate the functionality of the new similarity mode in embedding models.
- Updated localization files to include new strings related to similarity mode in multiple languages.
* feat(embeddings): add Qwen3-VL-Embedding support for multimodal input
- Introduced Qwen3-VL-Embedding model with support for both text and image inputs.
- Updated model configuration files to include compatibility settings for the new embedding type.
- Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs.
- Updated documentation to reflect the new model's capabilities and usage instructions.
* test(models): wrap embedding assertions to satisfy lint
* test(opensearch): isolate HTTP transports between parallel tests
* fix(agent): align web_search count limit with the configured maximum
count 的参数描述写死了 "at most 20",而实际生效的上限来自 agent 配置
(web_search_max_results,常见值 7)。模型按描述填 8/20 就会被 Execute 里的
"count must be between 1 and N" 硬拒——那一轮连 provider 都没被调用,而且在
trace 里与真正的故障同为 span error,无法区分。
- count 的静态描述去掉写死的 20;NewWebSearchTool 再按实际生效的 maxResults
动态改写 schema 描述(解析失败原样返回,不影响工具可用性)
- count 超范围由"整轮失败"改为"截断到上限",输出里附一句纠正提示,日志降为 Warn。
同类标量 search_knowledge.limit / search_memory.limit 走的也是截断;且工具本来
就会静默返回少于 count 条(结果被 URL 校验过滤时 break),同一字段不该两套策略
行为变更:count 传 0/负数/超过上限不再返回 Success=false,而是按上限执行。
TestSearchOptionsAndExplicitPageContent 原先把 count 与非法过滤值并列断言
"必须失败且不得调用 provider",该断言已相应调整,新行为由
TestAgentWebSearchClampsCountToConfiguredMaximum 覆盖。
测试:go test ./internal/agent/tools/... -count=1
* fix(agent): clamp only an over-maximum web_search count
count is documented as 1..maximum, so a value below 1 is a malformed
argument rather than a near miss: reject it before the provider runs and
restore the previous message ("count must be between 1 and N"). Only a
count above the configured maximum is clamped, which keeps the change
scoped to the near-miss case it was meant to fix.
The correction now travels in Data as well as Output. web_search enables
sourceOutput, so once there is at least one row the model reads the
rendered retrieval block, which is rebuilt from Data["results"]; the note
written to Output never reached it. annotateWebSearchNotes restates the
clamp as <count_clamped> inside that block, the same way
annotateSearchNotes surfaces <omitted> and <subset>. The zero-result early
return carries the note too, instead of returning before it is built.
Also documents the clamping in website-docs and moves the added comments
back to English, keeping only the ones that describe what the code does.
* fix(websearch): return results for non-webpage Metaso scopes
Metaso 把结果放在与 scope 同名的键下(webpage→webpages、scholar→scholars…),
而 Search() 只读 response.Webpages,导致 extra_config.scope 配成 document /
scholar / podcast / video / image 时永远返回 0 条。validMetasoScopes 与
website-docs/03-features/11-web-search.md 都把 6 个 scope 列为可用,属于
"文档承诺了、代码没实现"。
- 新增 itemsForScope:按 scope 选取对应结果数组,未知 scope 回退到首个非空数组并告警
- 新增 location():依次回退 link → url → imageUrl。image 条目只有 imageUrl,
只认 link 会让这些条目在 agent 侧的 URL 校验里被全部丢弃
- parseMetasoDate 补 "2006年01月02日"、"2006/01/02" 两种实测格式
- metasoWebpage 只建模真正会被读取的字段:多声明一个类型不符的键,整次搜索就会因
反序列化失败而返回 0 条。补 TestMetasoProviderToleratesUnmodelledFields 锁死这一点
- 日志补 scope= / array= 维度,便于区分"provider 没结果"与"读错了数组"
- 文档补充结果数组键名与 scope 的对应关系
测试:go test ./internal/infrastructure/web_search/... -count=1
* fix(websearch): scope Metaso result arrays only
Follow-up to the review. This change now does exactly one thing: read the
result array whose key matches the configured scope, and log which array
was read.
Removed — all of it added by the original commit, none of it supported by
a captured response:
- the `url` and `imageUrl` fields on metasoWebpage, and the location()
helper. Neither key appears in the document, scholar or image
responses, and a real image entry carries no `link` at all.
- the fallback to the first non-empty array when the target array is
empty. Real responses key each array strictly by scope, so for any
scope that passed validation this branch cannot be reached.
- the unknown-scope warning. NewMetasoProvider rejects unknown scopes
before a provider is ever constructed.
- the "2006年01月02日" and "2006/01/02" date layouts. Neither format
appears in any captured response, so parseMetasoDate is back to main.
The accompanying TestMetasoParseDate was introduced with them and is
removed too.
Tests: the TestMetasoProviderSearch fixture now returns `scholars` for a
scholar-scoped request — the previous fixture answered `webpages`, which
is what let the bug through. The document and scholar cases in
TestMetasoProviderScopeArrays are built from sanitised real responses,
and two new cases pin that `url`/`imageUrl` stay undeclared and that
image scope returns entries with an empty URL. Comments are English
again, matching the rest of the file.
The image scope limitation is unchanged from main and is now documented:
image entries carry no `link`, so the provider produces an empty URL and
the agent drops the result.
* style(websearch): drop added comments and revert doc paragraph
The PR is scoped to reading the result array that matches the scope, so the
extra doc comment blocks and the website-docs paragraph are noise. Remove the
12 comment lines added to metaso.go and the 29 added to metaso_test.go, and
restore website-docs/03-features/11-web-search.md to main (zero diff).
No behaviour change.
Pass the app APT mirror to all build stages and expose optional Playwright download hosts for docreader through the build script and Docker Compose. Document the mirror settings and WebKit-specific host precedence.
* fix(sandbox): omit env overrides when creating Aliyun E2B sandboxes from snapshots
Aliyun Agent Sandbox rejects create-from-snapshot requests that resend env vars or function config. Drop those fields for Aliyun hosts, and retry once if another endpoint returns the same rejection.
* fix(sandbox): keep the snapshot rejection fixture within the line limit
---------
Co-authored-by: hailongzhao <hailongzhao@tencent.com>
* feat(models): expose embedding input modalities
Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.
- Catalog: declare input modalities for the Doubao and Tongyi vision
embeddings, the Aliyun multimodal / qwen-vl embedding families and
jina-embeddings-v4, as listed in each vendor's documentation. The
migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
the catalog and writes spec.input only when the row differs from it;
picking another catalog model or switching vendor drops the
declaration. Model cards show the image badge for such models.
* feat(embedding): embed images with multimodal embedding models
Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.
- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
object store is usually unreachable from the vendor) and an
ImageEmbedder implemented by all four protocols, each in the image
shape its reference documents: Ark image_url, DashScope contents[].image,
Gemini inlineData parts, and on the OpenAI shape an object keyed by the
vendor's image_field (Jina: "image"). A plain OpenAI-compatible
endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
max_image_bytes and image_mime_types, filled from vendor docs for the
DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
AsImageEmbedder, so existing Embedder implementations and fakes are
untouched. An embedder takes images only when the catalog (or the
row's spec.input) lists image AND the endpoint can carry one. Images
are checked against the documented limits before any request, then
split at the documented images per request. The concurrency, debug
and Langfuse decorators forward it; logs and traces carry the MIME
type and size, never the bytes.
The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.
* feat(knowledge): index images by their own vector and recall them
When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.
Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
the image (embedding.PrepareImage converts and shrinks it to the
vendor's documented limits) and stores it as a new image_vector chunk:
Content is the caption (OCR text when there is none) for reranking and
the answer context; its index row carries the image's vector with
SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
so a second row would overwrite the caption's text vector, and copy,
move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
through each engine's normal BatchIndex, keeping row order (the
composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
syncChunkIndex only syncs its enabled flag, updateChunkVector skips
it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
backends with a column default were storing them as enabled.
Retrieval
- A text query scores lower against an image than against text, so when
the model embeds images the document vector search is widened by half
(capped at 500) and run at min(VectorThreshold, 0.1); each store
group's hits are then held to their own threshold before score
normalization: text at VectorThreshold, images at min(VectorThreshold,
0.1). Keyword hits on image rows are dropped, their Content being the
already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
context is attached, and the caption is not repeated from ImageInfo.
Adds golang.org/x/image for scaling and WebP/BMP decoding.
* fix(models): report no thinking levels for embedding, rerank and ASR models
Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.
* fix(compaction): never read a cancelled turn's stream as a summary
streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.
Check the context before each receive and again when the stream closes.
* feat(rerank): score images with multimodal rerank models
Rerank models that read images can now be sent an image document and
score it against a text query on the same scale as text, which is what a
recall made by an image's own vector needs: a text reranker only sees the
caption, and rejects an image for what the caption left out.
- api: ImageReranker, implemented by the Cohere and NIM shapes. Image
documents go in requests of their own, so text requests keep their
exact shape. NIM sends image-only passages without text, as base64 data
URLs (its VLM reranking reference requires both). The Cohere shape sends
{"<image_field>": data URI} documents and asks for no echo of them.
- The image settings embeddings already had (image_field,
max_image_batch_size, max_image_bytes, image_mime_types) move into a
shared ImageInput embedded in both settings; resolved embedding settings
serialize exactly as before.
- Catalog: jina-reranker-m0 and nvidia/llama-nemotron-rerank-vl-1b-v2 take
text and image. Jina names "image" documents; its base64 form is not
shown in any reranker example, so the data URI its embeddings reference
documents is used and marked unverified. NVIDIA lists its six formats
and 512 passages per request.
- rerank: an optional ImageReranker reached through AsImageReranker, gated
on the model's catalog input and the endpoint; images are checked against
the documented limits before any request, scores are normalized like
text, and the debug and Langfuse decorators forward it without logging
image bytes.
- Image preparation (format conversion, shrink to fit) moves from the
embedding package to internal/models/imageprep, since rerank needs it
too.
* feat(retrieval): keep a vector hit's own similarity through fusion
Hybrid fusion replaces a hit's Score with a rank-based RRF score. That is
right for ordering, but it erases the evidence reranking needs to judge an
image found by its own vector: how close the vector actually was. Record
the engine's similarity on each vector hit (IndexWithScore.VectorScore)
before normalization and fusion rewrite Score, and carry it onto the
search result (SearchResult.VectorScore, not serialized).
* feat(reranking): judge image hits by their image, not only their caption
An image found by its own vector was reranked on its caption and OCR text,
so a text reranker rejected it for exactly what the caption left out.
- With a reranker that reads images and a LoadImage callback, image_vector
hits are scored by their image (prepared to the model's limits) on the
same scale as text, and pass or fail the same threshold. A hit whose
image cannot be read, prepared or scored falls back to its text.
- With a text reranker, its verdict stands for images that are mostly
text (OCR of 200 characters or more). A pictorial hit it rejects is kept
when its vector similarity (VectorScore) reaches Options.ImageKeepScore:
at most two, strongest vector first, appended after the ranked results
and marked kept_by=image_vector. This also returns an image when no text
passes at all.
- Diagnostics report images_scored and images_kept.
* feat(rerank): read image hits' images in every rerank path
The chat pipeline, the agent search_knowledge tool and the hybrid-search
API now pass the rerank stage a way to read an image hit's image and an
image keep score, so a multimodal reranker scores images by what they show
and a text reranker can no longer drop a pictorial image its caption
undersells.
- KnowledgeBaseService.ReadChunkImage reads the first image of a result's
image_info from the storage of its knowledge base (the knowledge base's
tenant, so shared knowledge bases read from their owner's storage),
capped at 32 MB. Plain http(s) images go through the SSRF-safe
downloader.
- The backend resolution the image multimodal task used moves into a
shared knowledgeImageReader, so ingestion and reranking find an image in
the same place.
- reranking.ImageKeepScoreFor: an image is kept over a text reranker only
when its vector was at least as close as a text hit had to be, and never
below 0.25 (empirical).
* feat(chat): show a vision model the images its answer rests on
An image recalled by its own vector reached the answer as its caption only,
so a vision model could not answer from what the caption left out.
- Knowledge QA: INTO_CHAT_MESSAGE reads, for a model that supports vision,
the images of the contexts that rest on an image vector match (at most
three distinct ones, prepared to what every vision API takes: PNG, JPEG,
WebP or GIF within 5 MB), attaches them after the user's own images, and
names the contexts they belong to.
- Agent: search_knowledge attaches the same images when the agent's model
supports vision (the agent describes tool images to models that cannot
see them, and these already carry captions). Stored history drops them;
they are read again from storage.
- An image hit and its caption hit read the same once merged, and
de-duplication keeps either. The kept copy now inherits the image
(image_vector_match), in the pipeline's exact and partial-overlap
de-duplication and in the agent tool's.
- Results the rerank stage kept outside its top-k (kept_by) no longer fall
to the later top-k cuts: the pipeline's FilterTopK and the agent tool's
MMR and limit let them ride along after the ranked results.
* docs(website): describe how image hits are reranked and shown to vision models
* Fix image evidence preservation and model source mapping
* fix(knowledge): claim cloned and released document images in the catalog
#3493 binds images at parse time (bindStoredImages), which covers ingestion
but not the two paths that move image ownership:
Clone. A copied image is written as a new object, yet nothing claimed it for
the destination knowledge, so the KB file proxy answered 403 and cloned
documents rendered without images. The copy now goes through SaveBytes into
the servable exports/ namespace (CopyFile wrote to the knowledge-scoped
upload layout the proxy rejects), is claimed by the destination knowledge,
and CloneChunk / FAQ clone bind the handles left in chunk content and
image_info.
Delete. Release only looked at multimodal ImageInfo, which stays empty for
documents whose images live in markdown alone, so those objects outlived the
knowledge that owned them. The release set is now the union of ImageInfo URLs
and the bindings recorded for the knowledge being removed, with source_file
excluded so reparse and cleanup keep the original document.
The parse-path claim from the earlier attempt is dropped: ProcessDocument
already gets it from bindStoredImages, and the removed test only string
matched that call in the source. Every claim path stays best-effort,
including the clone path, which used to fail the whole clone and delete the
fresh copy when a single Bind errored.
* fix(knowledge): keep markdown-only images out of the pre-reindex cleanup
cleanupKnowledgeResources runs before a knowledge is re-indexed -- a manual
update, a reparse, a move with reparse -- not before it is removed. Unioning
in the catalog bindings there released a solely owned markdown-only
attachment, deleteExtractedImages deleted the file, and the catalog file
service marked the resource deleted. The re-claim that runs after the cleanup
resolves through a state=active lookup, so it could never bind that handle
again: the republished body was left pointing at a deleted image.
Cleanup collects ImageInfo URLs alone again. The paths that do remove a
knowledge entry (executeKnowledgeDelete, ProcessKBDelete) keep the union, which
is what lets markdown-only images be released when the knowledge actually goes
away.
Reported by @lyingbug in review on #3752.
---------
Co-authored-by: tayadwx <tayadwx@users.noreply.github.com>
* feat(models): expose embedding input modalities
Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.
- Catalog: declare input modalities for the Doubao and Tongyi vision
embeddings, the Aliyun multimodal / qwen-vl embedding families and
jina-embeddings-v4, as listed in each vendor's documentation. The
migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
the catalog and writes spec.input only when the row differs from it;
picking another catalog model or switching vendor drops the
declaration. Model cards show the image badge for such models.
* feat(embedding): embed images with multimodal embedding models
Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.
- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
object store is usually unreachable from the vendor) and an
ImageEmbedder implemented by all four protocols, each in the image
shape its reference documents: Ark image_url, DashScope contents[].image,
Gemini inlineData parts, and on the OpenAI shape an object keyed by the
vendor's image_field (Jina: "image"). A plain OpenAI-compatible
endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
max_image_bytes and image_mime_types, filled from vendor docs for the
DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
AsImageEmbedder, so existing Embedder implementations and fakes are
untouched. An embedder takes images only when the catalog (or the
row's spec.input) lists image AND the endpoint can carry one. Images
are checked against the documented limits before any request, then
split at the documented images per request. The concurrency, debug
and Langfuse decorators forward it; logs and traces carry the MIME
type and size, never the bytes.
The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.
* feat(knowledge): index images by their own vector and recall them
When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.
Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
the image (embedding.PrepareImage converts and shrinks it to the
vendor's documented limits) and stores it as a new image_vector chunk:
Content is the caption (OCR text when there is none) for reranking and
the answer context; its index row carries the image's vector with
SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
so a second row would overwrite the caption's text vector, and copy,
move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
through each engine's normal BatchIndex, keeping row order (the
composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
syncChunkIndex only syncs its enabled flag, updateChunkVector skips
it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
backends with a column default were storing them as enabled.
Retrieval
- A text query scores lower against an image than against text, so when
the model embeds images the document vector search is widened by half
(capped at 500) and run at min(VectorThreshold, 0.1); each store
group's hits are then held to their own threshold before score
normalization: text at VectorThreshold, images at min(VectorThreshold,
0.1). Keyword hits on image rows are dropped, their Content being the
already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
context is attached, and the caption is not repeated from ImageInfo.
Adds golang.org/x/image for scaling and WebP/BMP decoding.
* fix(models): report no thinking levels for embedding, rerank and ASR models
Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.
* fix(compaction): never read a cancelled turn's stream as a summary
streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.
Check the context before each receive and again when the stream closes.
* feat(knowledge): put image vectors behind a knowledge-base switch, off by default
Image vector indexing and recall used to turn on whenever the knowledge
base's embedding model declared image input. A model that takes images is
often chosen for text alone, so an upgrade would have added an embedding
call per image, widened vector recall 1.5x and lowered the image threshold
to 0.1 for those knowledge bases without anyone asking.
ImageProcessingConfig gains image_vector_enabled (default false; stored
JSON without the key reads as off, so existing knowledge bases stay off
with no migration). Both sides now need the switch and the model:
- indexImageVector returns before touching the model unless the KB opted
in (KnowledgeBase.IsImageVectorEnabled, which also needs vector indexing).
- applyImageRecall resolves the embedding model only when some KB in the
search opted in, and records the opted-in KBs per store group
(storeGroup.ImageKBIDs). Only those groups widen the pool and lower the
threshold; image hits of any other KB, including ones indexed before the
switch was turned off, are dropped.
The KB editor shows the switch in the image processing section, with a
warning when the selected embedding model does not declare image input;
docs describe the switch, its cost and what turning it off does.
* fix(knowledge): leave image vectors behind when cloning into a KB that is off
CloneChunk copied every chunk of the source document and then CopyIndices
for all of them, so copying a document from a knowledge base with image
vectors on into one with the switch off (same model and store) still
created image_vector chunks and their index rows, bypassing the switch
that indexImageVector honours.
Image vector chunks are now dropped from the copy when the target has not
opted in. Every engine's CopyIndices copies only the rows of chunks in the
mapping, so their index rows stay behind too.
* fix(search): refill the vector pool past stale image vectors
Image hits of a knowledge base whose switch is off were dropped only after
the engine had ranked and cut its results at TopK. Image vectors indexed
before the switch was turned off could fill that pool and push out valid
text hits: with MatchCount 5 the pool is 50, and 50 stale image rows
ranked above the text left the search empty.
No engine can filter rows by source type (Postgres and SQLite do not even
take ExcludeChunkIDs), so the exclusion cannot move into the query. When
the document vector pool comes back full and holds stale image rows, that
search alone runs again on a pool twice as large, until it holds TopK rows
that are not stale, the index runs out, or the pool reaches
maxRetrievalPoolSize (at most four extra calls from a pool of 50). The pool
is then cut back to TopK such rows, matching an engine-side exclusion
unless more than maxRetrievalPoolSize-TopK stale rows rank above the text.
* feat(i18n): translate the image vector switch and image input keys into zh-TW
The zh-TW locale arrived from main after this branch added five keys: the
knowledge base's image vector switch (label, description, unsupported-model
warning) and the embedding model's image input flag (label, description).
* fix(knowledge): leave image vectors behind when moving into a KB that is off
A reuse_vectors move relocates a document's chunks and index rows in place:
MoveKnowledgeIndices rebinds the rows by document ID and
MoveChunksByKnowledgeID rebinds the chunks. Moving a document from a
knowledge base with image vectors on into one with the switch off therefore
carried its image_vector chunks and their vectors along, the same bypass the
clone path had.
When the target has not opted in, the image_vector chunks are now deleted
during the move, their index rows first and then the chunks, so a retry
finds the chunks again rather than leaving orphaned rows. A target that
opted in still receives them unchanged.
* fix(retriever): keep image vectors out of keyword-only engines
BatchIndexVectors handed image rows to every engine of the composite,
including one that serves keyword retrieval alone (Elasticsearch v7 next to
a vector store). There the row holds nothing but the image's caption, which
the caption's own chunk already indexes, and retrieval discards every
keyword hit on it.
Image rows now go only to engines that serve vector retrieval. An engine
that serves both keeps one row for both kinds, so its keyword search still
meets them; the search side handles that.
* fix(search): refill the keyword pool past image rows
Every engine that serves both kinds of retrieval stores one row per index
entry and runs its keyword search over that row's content, so an image
vector row, whose content is its caption, is a keyword hit wherever the
caption's own chunk is. filterImageHits drops all of them, but only after
the engine has cut its ranking at TopK, so they crowded out text and caption
hits; a keyword ranking that put the image rows first came back empty.
The keyword pool is now refilled the way the vector pool is refilled past
stale image vectors: when it came back full and held image rows, the
keyword search runs again on a pool twice as large, up to
maxRetrievalPoolSize, and is cut back to TopK hits that are kept.
Leaving the rows out of the keyword index instead would mean writing them
without searchable content (Milvus and Tencent VectorDB derive a BM25
sparse vector from it, and how they take an empty one is unverified) or a
source-type filter in every engine's keyword query (ten engines, and
filterable fields that existing collections lack).
* revert: let image vectors move with the document into a KB that is off
This reverts commit e259977fe915f7d0bc30c4b02bc7af04da1c8dd4.
Deleting the image vector index rows by chunk ID before the move is not
reliable: the Weaviate and Elasticsearch v8 delete wrappers ignore partial
failures, and OpenSearch rejects more than 1000 IDs with ErrBatchTooLarge.
A partial delete followed by the chunk delete leaves index rows whose chunks
are gone, which nothing ever cleans up.
Without it, the image_vector chunks and their index rows move together
with the document. While the target's switch is off they are filtered at
query time (the pool refill keeps them from crowding out text), they are
removed with the document on delete, reparse or knowledge base delete, both
chunks and index rows by knowledge ID, and they become active again if the
target turns the switch on. The clone path keeps dropping them, since there
the rows are copied rather than moved.
* fix(search): keep the image row refill from losing hits it already had
refillPool replaced a pool with the answer to each larger search. A refill
that answered only in part (RetrieveResult.Error, swallowed into a warning
by retainPartialResults, as when some collections of the store did not
answer) could hold fewer text rows than the complete smaller pool, or none:
40 stale image rows above 10 text came back from a partial refill as the
40 image rows alone, and the search lost all 10 text hits. A refill that
failed outright failed the whole store group, discarding a first round that
had succeeded.
A refill is now best effort. A failed or incomplete refill search stops
the refill and leaves the last complete pool in place, cut back to TopK
kept rows as before, so a refill can no longer leave a search worse off
than it was without it. Merging the partial answer into the pool was not
done: its rows would need deduplicating and its ranking is cut at a
different TopK over a different set of collections, while keeping the
previous pool is exact.
* docs(knowledge): say what happens to image vectors in a KB that is off and what the refill costs
The image vector section now says that with the switch off, image vectors
that were indexed before it was turned off, or that came along when the
document was moved in from a KB that has it on, are kept but filtered at
query time, are removed when the document is deleted or reparsed or the
KB is deleted, and are recalled again once the switch is turned on; a copy
into such a KB leaves them behind.
It also states the cost of keeping dropped image rows from crowding text
out of the pools: the vector and keyword pools are searched again with
TopK doubled up to the 500 candidate cap, which adds queries for store
groups that hold such rows, and text can still be missed when more than
about 500 candidates are image rows. A failed or partial refill keeps the
previous complete pool.
* fix(search): log unexpected refill result sets, and describe the refill stop condition precisely
#3956 (Hugging Face TEI) and #3957 (Pinecone) added two rerank-only
vendors, but the READMEs, the models page, the extension-points page and
the homepage still said 27, and the vendor table had no rows for them.
The v0.8.2 release notes keep their historical 27.
Assistant history messages that carry tool_calls but empty text previously
omitted the content field entirely on the wire. Strict local OpenAI-compatible
engines (vLLM / Xinference serving Qwen-family Jinja chat templates) treat the
missing field as undefined/null and crash while rendering the next turn,
surfacing as "I'm sorry, I was unable to generate a response".
Always serialize "content": "" for assistant tool-call turns. The empty
string is accepted by every standard OpenAI-compatible endpoint.
Fixes#1487
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
The rerank limits row said every limit splits the request automatically.
Only max_documents and max_request_chars do; an over-long query or
document (max_query_chars / max_document_chars, or a single document
that does not fit the request budget beside the query) fails the call.
Taken from the documentation part of #3757; max_documents keeps
0 = unlimited.
Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
#3616 (Seafile) and #3916 (zh-TW) were merged one after the other; each
was green on its own, but zh-TW had no translations for the 21 Seafile
keys, so the locale key audit fails on main.
* fix(datasource): omit last_sync_cursor from the data source list response
The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.
* feat(datasource): add Seafile connector with fingerprint-based incremental sync
Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.
- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils
* feat(datasource): add Seafile frontend connector and docs
Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.
- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
updates, and Seafile in the README data source lists
---------
Co-authored-by: zaqqmm <q-zhang10@qq.com>
* fix(chat_pipeline): treat non-JSON model refusals as skipped graph extraction
When a chunk carries nothing extractable (e.g. a table-of-contents page),
the LLM correctly answers with a prose refusal such as "抱歉…". parseOutput
then fed that prose to json.Unmarshal, failing with a confusing
"invalid character 'æ'" (the first byte of a CJK character), marking the
chunk task GRAPH_EXTRACT_FAILED and letting asynq replay the identical
doomed LLM call up to the retry limit.
Classify responses that contain no JSON structure at all (no '{' or '[')
as errModelDeclined and complete the extraction with an empty graph so the
task succeeds as an explicit skip. Malformed or truncated JSON (structure
present) still fails and retries as before.
Fixes#3600
* fix(chat_pipeline): keep empty model responses retriable in graph extraction
An empty response is usually transient (gateway returning 200 with no
content, a reasoning model spending its whole budget on reasoning,
content filtering), and a retry often succeeds. Only non-empty prose
with no JSON structure is treated as a declined extraction.
* fix(graph): skip prose refusals only in the async chunk task, keep truncation retriable
ParseGraph no longer turns a prose answer into an empty graph. It returns
an error wrapping ErrModelDeclined, and ChunkExtractService.Handle is the
only caller that treats it as a skip (span output skipped=model_declined).
The /initialization "try extract" endpoint and query-side entity extraction
report the refusal as a failure again, so the graph settings form no longer
gets its few-shot examples replaced with an empty result.
Extractor.Extract drops the ErrModelDeclined marker when finish_reason is
"length": prose with no JSON because the output budget ran out is
truncation, and asynq should retry it.
* fix(graph): retry every truncation spelling, not only finish_reason=length
Extract kept a prose answer retriable only when FinishReason was exactly
"length". max_tokens / max_output_tokens (and case/space variants that
IsLengthFinishReason already accepts), a stream that broke before the stop
event ("incomplete") and an adapter that reports no reason were still read
as a refusal, so the async chunk task skipped the chunk for good.
Only a reason that proves the model stopped on its own now counts as a
refusal. Ollama's non-streaming Chat also forwards done_reason, so a
num_predict cut-off is reported as "length" instead of an empty string.
* fix(graph): only treat a response with no JSON anywhere as a refusal
The refusal check looked at the first extracted fence, so a prose fence
followed by a JSON fence was skipped as a refusal instead of failing as a
retriable parse error.
---------
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(kb): let update API set vlm_config
UpdateKnowledgeBaseRequest only carried name/description/config, so the
PUT /knowledge-bases/:id endpoint could never change a knowledge base's
multimodal (VLM) config, even though CreateKnowledgeBase accepts a
top-level vlm_config. Thread an optional vlm_config through the handler
and service so the update path mirrors create (nil = no change).
Fixes#1723
* test(kb): pass the vlm_config argument in tests added on main
* fix(kb): accept only model-managed vlm_config fields on update
The update API copied the request's VLMConfig wholesale, so a caller
could set the legacy ModelName/BaseURL/APIKey/InterfaceType fields and
have the VLM client call BaseURL directly, outside model management and
its SSRF checks. Only Enabled/ModelID/DescriptionLanguage/
CustomInstructions are taken now; legacy fields keep their stored value.
A referenced model must exist and be a VLLM model (400 otherwise), and
disabling clears ModelID, matching the initialization config endpoint.
* fix(kb): validate and gate vlm_config on the update API
- Enforce the 4000-character custom_instructions cap
(types.ValidateKnowledgeBasePromptInstructions), as create and
/initialization/config already do.
- vlm_config is a KB setting: require kbSettingsAccess (owner workspace
or admin share) like /initialization/config, so share editors get 403.
- Stored legacy model_name/base_url keep IsEnabled() true on their own,
so disabling returned 200 but left VLM on, and managed -> disabled fell
back to the old endpoint. Clear them when the request enables a managed
model_id, or disables VLM without echoing back the stored
model_name+base_url (GET-then-PUT keeps legacy on). Request legacy
fields are only compared, never written. Enabling without model_id on a
non-legacy KB is now a 400 instead of a silent no-op.
- Document the vlm_config semantics in the KB API reference.
* fix(kb): keep the legacy VLM endpoint only for pure legacy knowledge bases
CreateKnowledgeBase stores vlm_config as sent, so a managed model_id can
sit next to legacy model_name/base_url; resolveVLM then uses the managed
model. The update's keep-legacy paths (disable echoing the stored legacy
fields, enabled=true without model_id) cleared model_id but kept the
legacy fields, so IsEnabled() stayed true and image calls moved to the
stored base_url.
Those paths now apply only when no model_id is stored. On a mixed config
a disable clears everything, and enabled=true without model_id is a 400,
the same as on any managed knowledge base.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(docreader): preserve PPTX fallback image associations
* fix(docreader): visit PPTX shapes in MarkItDown's position order
MarkItDown emits pictures sorted by (top, left) per slide and per group, not
in z-order, so two same-named pictures on one slide could swap images. Sort
the same way when collecting placeholder sources.
* fix(docreader): skip unresolvable PPTX picture relationships instead of failing the document
A picture with no a:blip, an r:embed that is dangling or points at an
external relationship, or a media part python-pptx dropped at load time
made _picture_sources raise, so the whole PPTX (slide text included)
failed to parse. Record None for that picture to keep the queue position
and leave its placeholder untouched; other pictures still attach.
* fix(docreader): also skip PPTX pictures whose blipFill is missing
python-pptx raises InvalidXmlError (a PythonPptxError, not a ValueError)
when a p:pic has no p:blipFill; catch it so the picture is skipped
instead of failing the whole document.
---------
Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* feat(rerank): support Hugging Face TEI protocol
* style(rerank): satisfy TEI lint checks
* fix(rerank): cap Hugging Face TEI batches at 32 documents
TEI rejects requests above --max-client-batch-size, which defaults to 32.
Without a declared ceiling the shared batching layer sends the whole
candidate set in one request, so a default TEI deployment fails rerank
whenever hybrid retrieval returns more than 32 candidates.
* fix(rerank): send truncate and merge extra_body in the TEI client
TEI before 1.9 starts with --auto-truncate=false and rejects the whole
/rerank call when one passage is longer than the model's input limit, so
retrieval silently falls back to the original order. Ask for truncation
explicitly, as the NIM client does with truncate=END.
The TEI client also ignored compat extra_body, unlike the Cohere, NIM and
DashScope clients. Merge it the same way: query, texts and raw_scores stay
owned by the client, truncate is a default that extra_body may override.
---------
Co-authored-by: bigdu332 <bigdu332@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(model): treat 400 as reachable in chat connection test again
checkChatModelConnection probes with max_tokens=1 and is meant to treat a
400 as "endpoint reachable, auth OK". It detected that by matching the
go-openai wording "status code: 400". Since #3470 the error comes from
api.HTTPError ("API request failed with status 400: ..."), so the branch
never matched and the probe reported a failure.
This shows up with reasoning models behind LiteLLM: the 1-token budget
cannot finish a message, the gateway answers 400, and Test Connection
fails although the saved model chats fine.
Match the status code via errors.As on *api.HTTPError instead of the
message text.
* test(handler): drop unused request parameter in fake chat gateway
* fix(model): report a 400 that carries an auth error as auth failure
Gemini's native API answers an invalid key with 400 INVALID_ARGUMENT /
API_KEY_INVALID. Since the connection check now treats any typed 400 as
reachable, that showed "连接正常,模型可用" for a bad key. Keep the 400 pass
only when the body does not describe a credential problem, and let
classifyConnectionError recognise those bodies so the message says
认证失败 instead of a bare 连接失败.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* md预览链接 a 增加target
* fix(security): keep target on links in document preview sanitizer
documentPreviewDomPurifyConfig spread the base config and then replaced
ADD_ATTR with the image attributes only, dropping the 'target' entry
this PR adds, so links in document previews still lost target="_blank".
Extend the base ADD_ATTR instead, and lock both configs with tests.
* fix(security): drop author-supplied target before sanitizing links
ADD_ATTR now allows target globally so the hook-added target="_blank"
survives, but that also kept author values on elements the hook never
rewrites (SVG <a>, <area>, protocol-relative / uppercase / leading-space
hrefs in sanitizeHTML and document preview), e.g. target="_blank"
rel="opener" or target="_top". Strip target in beforeSanitizeElements
so only the hook's _blank, always paired with rel="noopener noreferrer",
can appear.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(knowledge): delete written chunks when the storage quota check fails
processChunks writes the chunks before the storage quota check. When the
check fails, or the tenant cannot be re-read for it, the knowledge is marked
failed and the function returned without removing those chunks, so they
stayed active (deleted_at IS NULL, is_enabled = true) under a failed,
disabled knowledge. The BatchIndex failure path already deletes them; do the
same in both quota branches. Nothing has been indexed at that point, so only
the chunk rows need removing.
Fixes#3648
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(knowledge): keep the processChunks doc comment on processChunks
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(cli): preserve directory structure in doc upload --recursive
`doc upload --recursive` passed an empty customFileName to
CreateKnowledgeFromFile, so every file landed in the knowledge base root
and the directory tree it was asked to walk was lost. The web UI's folder
upload does not have this problem -- it passes the browser-supplied
relative path, which the server splits into folder_path + file_name via
types.SplitKnowledgeRelativePath.
Pass the path relative to the walked directory instead. Files directly
under that directory still send "" so the request stays byte-identical to
the previous behaviour for the common flat-directory case; only nested
files gain a folder prefix.
- knowledgeRelativePath() converts to forward slashes (the server splits on
"/", so Windows backslashes would not separate) and returns "" for paths
outside the walked root rather than leaking a "../" chain into the folder
tree.
- The test double previously discarded customFileName with `_,` -- which is
why this could regress with the suite green. It now records the value and
two tests cover the nested and flat cases.
* fix(cli): reject only the parent component in knowledgeRelativePath
Review feedback on the recursive-upload folder path:
- `strings.HasPrefix(rel, "..")` also rejected children whose name merely
starts with dots, e.g. `..drafts/report.pdf`. Replaced with a
component-wise check (`rel == ".."` or `".."+separator` prefix).
- Root-level files must send ""; the guard now returns "" for
`filepath.Dir(rel) == "."` explicitly rather than relying on the
accidental prefix match to encode that rule.
Covered by a `..drafts/report.pdf` case in
TestUploadRecursive_PreservesFolderStructure.
* test(cli): expect an empty name for root files in knowledgeRelativePath
knowledgeRelativePath returns "" for files directly under the walked
root, so the test must expect that. Also wrap an over-long line.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(dataset): allow selecting the QA answer model
* test(dataset): expect gpt-5.6-sol as the default QA model
The default moved from gpt-4o-2024-05-13 to gpt-5.6-sol on main (#3965).
---------
Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(qdrant): 复制索引改用服务端返回的游标,重复游标报错并给分页加跳数上限
CopyIndices 用"上一页最后一个点的 id"当游标(repository.go:987)。Qdrant 的
scroll offset 是**包含**语义 —— proto 里写的是 "Start with this ID",实测下一页
正是从该 id 本身开始返回,所以每翻一页都会把上一页的最后一个点再读一次、再写一份
到目标库。服务端本来就把下一页该用的游标放在 ScrollResponse.next_page_offset
("Use this offset for the next query")里,而客户端只在 ScrollAndOffset() 中返回它,
本函数调用的 Scroll() 恰好把它丢掉了;终止条件也只能退化成"这一页比 64 短"。
在真实 Qdrant 1.19.1 上实测:150 个源 chunk 复制出 152 个目标点,chunk-63 与
chunk-126 各被复制两次(每个页边界一个),totalCopied 同步虚高。源库只要超过一页
(>64 个 chunk)就会命中,不需要服务端有任何异常行为。
本次修改:
- 改用 q.client.ScrollAndOffset,游标直接取服务端返回的 next_page_offset;终止
条件改为"服务端没有下一页"(next_page_offset == nil),不再用短页推断读完。
- 记录用过的游标,重复即返回 "copy indices made no progress at cursor ...",
检测点在整页处理之前,重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页);触顶打印 Warn 并返回错误,
而不是把这一轮复制当成成功。
- 修复后同一份数据在真实 Qdrant 上测得 150 个目标点、0 重复。
回归用例用假 gRPC 服务端(拦截 unary RPC,对齐 repository_test.go 的
newInterceptedQdrantClient)复刻真实服务端的游标语义:150 个源 chunk 必须各复制
一次(修复前 152 个点、2 个重复);游标恒定不动的源端必须报 no progress
(修复前 5 秒写入 3127040 个点仍未返回);一直推进但永不结束的源端在第 10000 页
被拦下。
* fix(qdrant): drop the CopyIndices page cap
10000 pages of 64 points capped a copy at 640k points and failed the
whole clone beyond that. The scroll walks every point of the source
knowledge base (a single-knowledge clone included), so a cap derived from
the chunk map would cut off small clones out of large bases too.
next_page_offset only moves forward, so the repeated-cursor check already
stops a walk that cannot finish, and the task context bounds the rest.
---------
Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(markdown): 单个波浪号不再被解析成删除线
GFM 会把区间标记(如 2020~2035)里的单个 `~` 误解析成删除线。
覆盖 marked 的 del tokenizer:`~~...~~` 仍走默认删除线规则,
单个 `~` 则按字面保留;并新增单测覆盖时间/区间场景。
* fix(markdown): register the literal single-tilde tokenizer once
doc-content.vue and WikiBrowser.vue called marked.use() for the del
tokenizer override inside <script setup>, so every mount wrapped the
global marked tokenizer one more time. Move the extension into
utils/markedLiteralTilde.ts and register it on the global marked behind
a once-only guard; the preview Marked instance reuses the same
extension object. Add a test that repeated registration does not
re-wrap tokenizer.del.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(data-analysis): skip style-only worksheets before DuckDB loading
* chore: retrigger CI for XLSX sheet fix
* fix(data-analysis): keep all sheets when the worksheet preflight fails
The style-only sheet preflight is best-effort: if it cannot read the
workbook parts, log a warning and load the sheets DuckDB enumerated
instead of failing the whole workbook. Context cancellation still
aborts the load.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>