Commit Graph

3406 Commits (6e2e5d8bb07cd9c8db360c03b80ca47ea2e81fd5)

Author SHA1 Message Date
hahahahahayesyeseys 6e2e5d8bb0
fix(frontend): follow configured upload limit in hints (#4110) 2026-10-10 11:50:30 +08:00
lyingbug 0c71fff2a3
fix(sandbox): treat E2B in_progress template builds as building (#4108)
E2B reports a running template build as in_progress. Mapping that status
to building lets catalog refresh reconcile the current build instead of
surfacing the raw status.
2026-10-10 11:28:44 +08:00
lyingbug c3f077ed18
fix(models): align configuration and multimodal embedding workflows (#4080)
* fix(models): align configuration defaults and multimodal workflows

* feat(image): enhance image retrieval diagnostics and filtering

- Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results.
- Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores.
- Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording.
- Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling.

* feat(embeddings): enhance image handling with new formats and message support

- Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats.
- Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows.
- Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements.
- Added tests to validate the new image message handling and ensure correct request formatting for different scenarios.
- Updated documentation to reflect changes in image input handling and configuration options.

* feat(similarity): add embedding similarity mode for enhanced document scoring

- Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query.
- Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type.
- Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity.
- Added tests to validate the functionality of the new similarity mode in embedding models.
- Updated localization files to include new strings related to similarity mode in multiple languages.

* feat(embeddings): add Qwen3-VL-Embedding support for multimodal input

- Introduced Qwen3-VL-Embedding model with support for both text and image inputs.
- Updated model configuration files to include compatibility settings for the new embedding type.
- Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs.
- Updated documentation to reflect the new model's capabilities and usage instructions.

* test(models): wrap embedding assertions to satisfy lint

* test(opensearch): isolate HTTP transports between parallel tests
2026-10-10 11:06:00 +08:00
Jundong Zhang 40eb34bc84
fix(agent): align web_search count limit with the configured maximum (#4011)
* fix(agent): align web_search count limit with the configured maximum

count 的参数描述写死了 "at most 20",而实际生效的上限来自 agent 配置
(web_search_max_results,常见值 7)。模型按描述填 8/20 就会被 Execute 里的
"count must be between 1 and N" 硬拒——那一轮连 provider 都没被调用,而且在
trace 里与真正的故障同为 span error,无法区分。

- count 的静态描述去掉写死的 20;NewWebSearchTool 再按实际生效的 maxResults
  动态改写 schema 描述(解析失败原样返回,不影响工具可用性)
- count 超范围由"整轮失败"改为"截断到上限",输出里附一句纠正提示,日志降为 Warn。
  同类标量 search_knowledge.limit / search_memory.limit 走的也是截断;且工具本来
  就会静默返回少于 count 条(结果被 URL 校验过滤时 break),同一字段不该两套策略

行为变更:count 传 0/负数/超过上限不再返回 Success=false,而是按上限执行。
TestSearchOptionsAndExplicitPageContent 原先把 count 与非法过滤值并列断言
"必须失败且不得调用 provider",该断言已相应调整,新行为由
TestAgentWebSearchClampsCountToConfiguredMaximum 覆盖。

测试:go test ./internal/agent/tools/... -count=1

* fix(agent): clamp only an over-maximum web_search count

count is documented as 1..maximum, so a value below 1 is a malformed
argument rather than a near miss: reject it before the provider runs and
restore the previous message ("count must be between 1 and N"). Only a
count above the configured maximum is clamped, which keeps the change
scoped to the near-miss case it was meant to fix.

The correction now travels in Data as well as Output. web_search enables
sourceOutput, so once there is at least one row the model reads the
rendered retrieval block, which is rebuilt from Data["results"]; the note
written to Output never reached it. annotateWebSearchNotes restates the
clamp as <count_clamped> inside that block, the same way
annotateSearchNotes surfaces <omitted> and <subset>. The zero-result early
return carries the note too, instead of returning before it is built.

Also documents the clamping in website-docs and moves the added comments
back to English, keeping only the ones that describe what the code does.
2026-10-10 10:27:52 +08:00
Jundong Zhang d074ec7b0b
fix(websearch): return results for non-webpage Metaso scopes (#4010)
* fix(websearch): return results for non-webpage Metaso scopes

Metaso 把结果放在与 scope 同名的键下(webpage→webpages、scholar→scholars…),
而 Search() 只读 response.Webpages,导致 extra_config.scope 配成 document /
scholar / podcast / video / image 时永远返回 0 条。validMetasoScopes 与
website-docs/03-features/11-web-search.md 都把 6 个 scope 列为可用,属于
"文档承诺了、代码没实现"。

- 新增 itemsForScope:按 scope 选取对应结果数组,未知 scope 回退到首个非空数组并告警
- 新增 location():依次回退 link → url → imageUrl。image 条目只有 imageUrl,
  只认 link 会让这些条目在 agent 侧的 URL 校验里被全部丢弃
- parseMetasoDate 补 "2006年01月02日"、"2006/01/02" 两种实测格式
- metasoWebpage 只建模真正会被读取的字段:多声明一个类型不符的键,整次搜索就会因
  反序列化失败而返回 0 条。补 TestMetasoProviderToleratesUnmodelledFields 锁死这一点
- 日志补 scope= / array= 维度,便于区分"provider 没结果"与"读错了数组"
- 文档补充结果数组键名与 scope 的对应关系

测试:go test ./internal/infrastructure/web_search/... -count=1

* fix(websearch): scope Metaso result arrays only

Follow-up to the review. This change now does exactly one thing: read the
result array whose key matches the configured scope, and log which array
was read.

Removed — all of it added by the original commit, none of it supported by
a captured response:

- the `url` and `imageUrl` fields on metasoWebpage, and the location()
  helper. Neither key appears in the document, scholar or image
  responses, and a real image entry carries no `link` at all.
- the fallback to the first non-empty array when the target array is
  empty. Real responses key each array strictly by scope, so for any
  scope that passed validation this branch cannot be reached.
- the unknown-scope warning. NewMetasoProvider rejects unknown scopes
  before a provider is ever constructed.
- the "2006年01月02日" and "2006/01/02" date layouts. Neither format
  appears in any captured response, so parseMetasoDate is back to main.
  The accompanying TestMetasoParseDate was introduced with them and is
  removed too.

Tests: the TestMetasoProviderSearch fixture now returns `scholars` for a
scholar-scoped request — the previous fixture answered `webpages`, which
is what let the bug through. The document and scholar cases in
TestMetasoProviderScopeArrays are built from sanitised real responses,
and two new cases pin that `url`/`imageUrl` stay undeclared and that
image scope returns entries with an empty URL. Comments are English
again, matching the rest of the file.

The image scope limitation is unchanged from main and is now documented:
image entries carry no `link`, so the provider produces an empty URL and
the agent drops the result.

* style(websearch): drop added comments and revert doc paragraph

The PR is scoped to reading the result array that matches the scope, so the
extra doc comment blocks and the website-docs paragraph are noise. Remove the
12 comment lines added to metaso.go and the 29 added to metaso_test.go, and
restore website-docs/03-features/11-web-search.md to main (zero diff).

No behaviour change.
2026-10-10 10:25:40 +08:00
LL_Ricky 602d0c388b
feat: support app and docreader build download mirrors (#4073)
Pass the app APT mirror to all build stages and expose optional Playwright download hosts for docreader through the build script and Docker Compose. Document the mirror settings and WebKit-specific host precedence.
2026-10-09 14:28:47 +08:00
nullptr-error 52617f6410
fix(sandbox): omit env overrides when creating Aliyun E2B sandboxes from snapshots (#4054)
* fix(sandbox): omit env overrides when creating Aliyun E2B sandboxes from snapshots

Aliyun Agent Sandbox rejects create-from-snapshot requests that resend env vars or function config. Drop those fields for Aliyun hosts, and retry once if another endpoint returns the same rejection.

* fix(sandbox): keep the snapshot rejection fixture within the line limit

---------

Co-authored-by: hailongzhao <hailongzhao@tencent.com>
2026-10-09 14:24:54 +08:00
lyingbug 65f88f4121
feat: judge image hits by their image in rerank and show them to vision models (#4013)
* feat(models): expose embedding input modalities

Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.

- Catalog: declare input modalities for the Doubao and Tongyi vision
  embeddings, the Aliyun multimodal / qwen-vl embedding families and
  jina-embeddings-v4, as listed in each vendor's documentation. The
  migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
  capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
  the catalog and writes spec.input only when the row differs from it;
  picking another catalog model or switching vendor drops the
  declaration. Model cards show the image badge for such models.

* feat(embedding): embed images with multimodal embedding models

Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.

- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
  object store is usually unreachable from the vendor) and an
  ImageEmbedder implemented by all four protocols, each in the image
  shape its reference documents: Ark image_url, DashScope contents[].image,
  Gemini inlineData parts, and on the OpenAI shape an object keyed by the
  vendor's image_field (Jina: "image"). A plain OpenAI-compatible
  endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
  max_image_bytes and image_mime_types, filled from vendor docs for the
  DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
  AsImageEmbedder, so existing Embedder implementations and fakes are
  untouched. An embedder takes images only when the catalog (or the
  row's spec.input) lists image AND the endpoint can carry one. Images
  are checked against the documented limits before any request, then
  split at the documented images per request. The concurrency, debug
  and Langfuse decorators forward it; logs and traces carry the MIME
  type and size, never the bytes.

The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.

* feat(knowledge): index images by their own vector and recall them

When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.

Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
  the image (embedding.PrepareImage converts and shrinks it to the
  vendor's documented limits) and stores it as a new image_vector chunk:
  Content is the caption (OCR text when there is none) for reranking and
  the answer context; its index row carries the image's vector with
  SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
  text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
  the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
  so a second row would overwrite the caption's text vector, and copy,
  move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
  through each engine's normal BatchIndex, keeping row order (the
  composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
  syncChunkIndex only syncs its enabled flag, updateChunkVector skips
  it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
  backends with a column default were storing them as enabled.

Retrieval
- A text query scores lower against an image than against text, so when
  the model embeds images the document vector search is widened by half
  (capped at 500) and run at min(VectorThreshold, 0.1); each store
  group's hits are then held to their own threshold before score
  normalization: text at VectorThreshold, images at min(VectorThreshold,
  0.1). Keyword hits on image rows are dropped, their Content being the
  already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
  context is attached, and the caption is not repeated from ImageInfo.

Adds golang.org/x/image for scaling and WebP/BMP decoding.

* fix(models): report no thinking levels for embedding, rerank and ASR models

Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.

* fix(compaction): never read a cancelled turn's stream as a summary

streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.

Check the context before each receive and again when the stream closes.

* feat(rerank): score images with multimodal rerank models

Rerank models that read images can now be sent an image document and
score it against a text query on the same scale as text, which is what a
recall made by an image's own vector needs: a text reranker only sees the
caption, and rejects an image for what the caption left out.

- api: ImageReranker, implemented by the Cohere and NIM shapes. Image
  documents go in requests of their own, so text requests keep their
  exact shape. NIM sends image-only passages without text, as base64 data
  URLs (its VLM reranking reference requires both). The Cohere shape sends
  {"<image_field>": data URI} documents and asks for no echo of them.
- The image settings embeddings already had (image_field,
  max_image_batch_size, max_image_bytes, image_mime_types) move into a
  shared ImageInput embedded in both settings; resolved embedding settings
  serialize exactly as before.
- Catalog: jina-reranker-m0 and nvidia/llama-nemotron-rerank-vl-1b-v2 take
  text and image. Jina names "image" documents; its base64 form is not
  shown in any reranker example, so the data URI its embeddings reference
  documents is used and marked unverified. NVIDIA lists its six formats
  and 512 passages per request.
- rerank: an optional ImageReranker reached through AsImageReranker, gated
  on the model's catalog input and the endpoint; images are checked against
  the documented limits before any request, scores are normalized like
  text, and the debug and Langfuse decorators forward it without logging
  image bytes.
- Image preparation (format conversion, shrink to fit) moves from the
  embedding package to internal/models/imageprep, since rerank needs it
  too.

* feat(retrieval): keep a vector hit's own similarity through fusion

Hybrid fusion replaces a hit's Score with a rank-based RRF score. That is
right for ordering, but it erases the evidence reranking needs to judge an
image found by its own vector: how close the vector actually was. Record
the engine's similarity on each vector hit (IndexWithScore.VectorScore)
before normalization and fusion rewrite Score, and carry it onto the
search result (SearchResult.VectorScore, not serialized).

* feat(reranking): judge image hits by their image, not only their caption

An image found by its own vector was reranked on its caption and OCR text,
so a text reranker rejected it for exactly what the caption left out.

- With a reranker that reads images and a LoadImage callback, image_vector
  hits are scored by their image (prepared to the model's limits) on the
  same scale as text, and pass or fail the same threshold. A hit whose
  image cannot be read, prepared or scored falls back to its text.
- With a text reranker, its verdict stands for images that are mostly
  text (OCR of 200 characters or more). A pictorial hit it rejects is kept
  when its vector similarity (VectorScore) reaches Options.ImageKeepScore:
  at most two, strongest vector first, appended after the ranked results
  and marked kept_by=image_vector. This also returns an image when no text
  passes at all.
- Diagnostics report images_scored and images_kept.

* feat(rerank): read image hits' images in every rerank path

The chat pipeline, the agent search_knowledge tool and the hybrid-search
API now pass the rerank stage a way to read an image hit's image and an
image keep score, so a multimodal reranker scores images by what they show
and a text reranker can no longer drop a pictorial image its caption
undersells.

- KnowledgeBaseService.ReadChunkImage reads the first image of a result's
  image_info from the storage of its knowledge base (the knowledge base's
  tenant, so shared knowledge bases read from their owner's storage),
  capped at 32 MB. Plain http(s) images go through the SSRF-safe
  downloader.
- The backend resolution the image multimodal task used moves into a
  shared knowledgeImageReader, so ingestion and reranking find an image in
  the same place.
- reranking.ImageKeepScoreFor: an image is kept over a text reranker only
  when its vector was at least as close as a text hit had to be, and never
  below 0.25 (empirical).

* feat(chat): show a vision model the images its answer rests on

An image recalled by its own vector reached the answer as its caption only,
so a vision model could not answer from what the caption left out.

- Knowledge QA: INTO_CHAT_MESSAGE reads, for a model that supports vision,
  the images of the contexts that rest on an image vector match (at most
  three distinct ones, prepared to what every vision API takes: PNG, JPEG,
  WebP or GIF within 5 MB), attaches them after the user's own images, and
  names the contexts they belong to.
- Agent: search_knowledge attaches the same images when the agent's model
  supports vision (the agent describes tool images to models that cannot
  see them, and these already carry captions). Stored history drops them;
  they are read again from storage.
- An image hit and its caption hit read the same once merged, and
  de-duplication keeps either. The kept copy now inherits the image
  (image_vector_match), in the pipeline's exact and partial-overlap
  de-duplication and in the agent tool's.
- Results the rerank stage kept outside its top-k (kept_by) no longer fall
  to the later top-k cuts: the pipeline's FilterTopK and the agent tool's
  MMR and limit let them ride along after the ranked results.

* docs(website): describe how image hits are reranked and shown to vision models

* Fix image evidence preservation and model source mapping
2026-10-09 13:01:48 +08:00
tayadwx a1450d382e
fix(knowledge): claim cloned and released document images in the resource catalog (#3752)
* fix(knowledge): claim cloned and released document images in the catalog

#3493 binds images at parse time (bindStoredImages), which covers ingestion
but not the two paths that move image ownership:

Clone. A copied image is written as a new object, yet nothing claimed it for
the destination knowledge, so the KB file proxy answered 403 and cloned
documents rendered without images. The copy now goes through SaveBytes into
the servable exports/ namespace (CopyFile wrote to the knowledge-scoped
upload layout the proxy rejects), is claimed by the destination knowledge,
and CloneChunk / FAQ clone bind the handles left in chunk content and
image_info.

Delete. Release only looked at multimodal ImageInfo, which stays empty for
documents whose images live in markdown alone, so those objects outlived the
knowledge that owned them. The release set is now the union of ImageInfo URLs
and the bindings recorded for the knowledge being removed, with source_file
excluded so reparse and cleanup keep the original document.

The parse-path claim from the earlier attempt is dropped: ProcessDocument
already gets it from bindStoredImages, and the removed test only string
matched that call in the source. Every claim path stays best-effort,
including the clone path, which used to fail the whole clone and delete the
fresh copy when a single Bind errored.

* fix(knowledge): keep markdown-only images out of the pre-reindex cleanup

cleanupKnowledgeResources runs before a knowledge is re-indexed -- a manual
update, a reparse, a move with reparse -- not before it is removed. Unioning
in the catalog bindings there released a solely owned markdown-only
attachment, deleteExtractedImages deleted the file, and the catalog file
service marked the resource deleted. The re-claim that runs after the cleanup
resolves through a state=active lookup, so it could never bind that handle
again: the republished body was left pointing at a deleted image.

Cleanup collects ImageInfo URLs alone again. The paths that do remove a
knowledge entry (executeKnowledgeDelete, ProcessKBDelete) keep the union, which
is what lets markdown-only images be released when the knowledge actually goes
away.

Reported by @lyingbug in review on #3752.

---------

Co-authored-by: tayadwx <tayadwx@users.noreply.github.com>
2026-10-09 13:01:26 +08:00
helloxjade 4243018ff1
fix(mcp): report advertised directory failures in connection tests (#3905)
* fix(mcp): report tool listing failures in connection tests

* fix(mcp): preserve discovery for unadvertised capabilities

* fix(mcp): resolve connection test lint findings
2026-10-09 12:01:27 +08:00
Linyao Du 3d01bebce3
fix(docreader): resolve EPUB images relative to their chapters (#3924)
* fix(docreader): resolve EPUB images relative to their chapters

* fix(docreader): preserve literal EPUB archive path characters

* fix(docreader): preserve literal percent-encoded EPUB image paths

---------

Co-authored-by: SuperSgdk <287576545+SuperSgdk@users.noreply.github.com>
2026-10-09 12:00:57 +08:00
helloxjade 09411106f0
fix(session): allow tenant admins to manage channel sessions (#3939)
* fix(session): allow tenant admins to manage channel sessions

* fix(session): restrict admin management to channel sessions
2026-10-09 12:00:07 +08:00
linus-liu-web 8319d59dd0
feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf) (#3786)
* feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf)

钉钉连接器此前只导入原生 .adoc 文档:知识库里的上传文件既不会出现在选择器里,
也不会被同步,而且没有任何提示。

- 节点判定拆成 isOnlineDocument / isBinaryDocument / binaryContentType,isDocument()
  变成"在线文档 ∪ 二进制文档",上传文件因此可被枚举与摄取;表格(axls)等其它原生
  类型仍不在其中,避免枚举出同步读不了的节点。
- 上传文件通过钉盘下载 API 逐字节传输,交给 WeKnora 自己的解析器:先用
  /v2.0/doc/dentries/{id}/queryDentryId 把节点 UUID 换成数字 spaceId/dentryId
  (存储 API 拒绝非数字 spaceId),再取预签名 URL 下载。
- 大小上限加在传输上而不是列表元数据上:node.size 与存储对象并不一致。
- 预签名 URL 短命且带版本,传输失败会用新签名重试。
- Validate 对上传文件"解析位置但不传 payload",凭证校验不必为每个候选文件下载
  数 MB;adoc 仍走 blocks API。
- 同步按类型分派:adoc 渲染 blocks,上传文件落原始字节并带上具体 MIME 类型与只含
  一个扩展名的文件名;下载失败与读取失败一样是可重试的失败项,不推进游标。

本提交只含 C(上传文件摄取)。不含:校验循环容错(A)、跳过项日志与可见性(B)、
原生表格 axls(D)、多维表/手工引用/前端(E)。

* test(datasource/dingtalk): 上传文件用例改用合成文档名与占位 ID

- 夹具、断言与文件名注释里的文档名统一换成合成名 Employee-Handbook.docx,
  用例语义不变:仍覆盖"名字已带扩展名时不重复追加"与 PK\x03\x04 字节断言。
- 去掉测试注释里"as verified against the live tenant"的表述,改为描述调用顺序。
- dentry 响应示例与用例中的数字 ID 换成明确占位符,不再出现可被误认的真实 ID。

* fix(datasource/dingtalk): 上传文件摄取改为数据源级开关,超限不再重试

评审意见(#3786):上传文件是本次新增的摄取路径,存量数据源升级后无需用户显式启用
就会在下一次同步(含增量)开始下载并向量化所选范围内的 docx/pptx/xlsx/pdf,可能
增加存储与 embedding 成本;同时单次下载里超出体积上限被当成可重试错误,没有
Content-Length 的超大文件会重签名再下满 3 次。

- 新增 `Settings.include_uploaded_files`(默认 false,未设置/非法值一律按 false)。
  读取走与本仓库语雀连接器同名的 settingBool(v, def),并新增 documentSettings /
  parseDocumentSettings 承载开关。
- 判定拆成"有没有读取路径"(isDocument)与"这个数据源收不收"(isIngestible):
  Validate 的探测、ListResources、全量同步与增量同步(共用 scanWorkspace)四处
  全部改用 isIngestible,关闭时行为与 main 一致。
- skipReason 对"能读但没开开关"的上传文件给出可操作的原因:说明
  include_uploaded_files 未启用,而不是伪装成类型不支持;媒体文件与真正没有
  摄取路径的类型保持原有说法。
- readLimited 改为返回哨兵错误 errDocumentTooLarge(同时包装上限字节数),
  fetchSigned 据此返回 retryable=false:体积是对象的属性,新签名只会拿到同样的
  字节。普通读错误仍然 retryable=true,仍会重签名重试。
- 前端数据源编辑弹窗(同步策略步骤)加对应开关,未触碰时不写回 settings;5 个
  语言包补齐文案。
- website-docs 钉钉一节写明支持范围、默认值、64 MiB 单文件上限与成本影响。

测试补 include_uploaded_files 的未设置/显式关闭/显式开启三种状态:校验、资源列表、
全量同步、增量同步四处各自断言,直接选择上传文件在关闭时仍是失败的资源;流式超限
用例断言签名查询与对象下载各只发生一次,另加"普通读错误仍重试"的用例。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:56:04 +08:00
linus-liu-web abc365f919
fix(wiki): 唯一写出口 UpdatePage 拦住丢表格行的机器重写(agent 整页写入不再静默改短页面) (#3793)
* fix(wiki): 在唯一写出口 UpdatePage 拦住丢表格行的机器重写

模型被要求把整页重新吐一遍时,markdown 表格是最常出错的地方:一百行的
证书台账回来只剩三十行,finish_reason=stop、没有任何报错,页面历史里就是
一次普通编辑。守卫因此加在 UpdatePage —— 所有调用方共用的唯一写出口,
而不是某一个调用方。

- 按「每行第一个非空单元格」的身份集合比较新旧内容(去掉强调符号、折叠
  空白、统一大小写):换行、重排、改日期/金额列都不算丢行,只有一行指向
  的实体整个消失才算;
- 同一身份出现两次按多重集比较,丢掉其中一行同样是丢;
- 表头行(`| --- |` 的上一行)不参与比较,模型改列名不算丢行。

有意的缩短放行:人在页面上删行(user)、回滚到更短的旧版本(revert),
以及调用方用 types.WithWikiShrinkAllowed 明确标注的写入 —— ingest 的
retract 轮(源文档被删除,被删文档名下的行本来就该消失)与 agent 的精确
文本替换(old_text 必须逐字命中现有内容,删除是有意为之)各自在自己的
写入上打这个标记。

agent 的整页写入拿到的是错误而不是静默成功:否则模型会以为改了一页、其实
一个字节都没动;它可以把那几行重新吐出来,或改用精确替换工具。

测试:身份比较、多重集、表头跳过、散文中孤立竖线不算表格;sqlite 上的
UpdatePage 拒绝写入且版本不推进、四类放行都能写进去、正常重写照常落库。
反向验证两组:去掉豁免 → 4 个放行用例 FAIL;让身份比较失效 → 6 个用例
FAIL(含两个 UpdatePage 用例);文件逐字节还原后全 PASS。

* fix(wiki): 拒写要向调用方明说,否则 trim 会删掉没落地的文档

UpdatePage 的守卫对非 agent 来源是**静默拒写**(返回现有页面 + nil
error)—— 页面本身没坏,这个选择是对的。但对 ingest 这条调用链来说,
"静默"本身就是个洞:

  reduceSlugUpdates 拿到 err == nil
    → 返回 changed=true, err=nil
    → 调用方那句 `if reduceErr != nil { collectUnapplied(updates) }` 走不到
    → trim 阶段认为这个 slug 已经成功合并
    → 把它背后的文档删掉
    → 而这些文档的贡献从来没写进页面,下一批也不会再回来

collectUnapplied 是"这些文档别删、下一批再来"的唯一入口,全仓只有三个
调用点(lockErr / !acquired / reduceErr != nil),没有一个能覆盖静默拒写。

修法不改 UpdatePage 的契约:拒写时它返回的是**未经改动的页面**,所以
调用方比较写入前后的 Version 就能判断,然后把 updateDeferred 一路带出去,
交给 collectUnapplied —— 与 #3617 的 updateDeferred 语义一致。

补的是 #3617 里这块语义;守卫本身仍在唯一写出口(不改架构)。
本提交只补连线,没有新增单测:这条路径要 asynq + chat model 的整套夹具,
而"静默拒写返回 nil"这一前提已由 TestUpdatePageRefusesARewriteThatDropsTableRows 覆盖。

* fix(wiki): 表格行守卫按去重身份集算比例,拒写改用显式信号

- agent 整页更新仍是任一身份缺失即拒写;ingest(未显式允许缩减)改为
  丢失超过旧内容唯一身份数 20% 才拒写,user/revert/显式允许缩减不变
- wikiWriteMissingRowIdentities 改为集合比较:同身份的两行合成一行不再
  算丢失;身份只取首个非空单元格,注释与日志都写明这个边界
- UpdatePage 的拒写统一返回 ErrWikiWriteDroppedTableRows;reduce 用
  errors.Is 区分"守卫拒写"与"成功但版本未变",不再拿版本号当信号,
  只有确实未落地的贡献才保留关联文档重试

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:55:28 +08:00
linus-liu-web cbfc0fed4a
fix(chat): 在模型上下文中标注 FILTER_TOP_K 的子集截断,避免把前 N 条当作全集作答 (#3758)
* fix(chat): 在模型上下文中标注 FILTER_TOP_K 的子集截断

FILTER_TOP_K 排序后直接 searchResult[:topK],模型只拿到 topK 条却按全集
语气作答(聚合类问题最明显:名单只被截取排名靠前的若干条,模型仍输出
"这些人是……"),属于完整但错误的答案。

本次只让截断对模型可见:真正丢结果时记录 RetrievalTruncation{Shown,
Candidates};prepareMessagesWithModelContext 把它作为 retrieval_shown /
retrieval_candidates 传给 modelcontext;annotateSearchNotes 在
<retrieval> 内追加 <subset shown="N" candidates="M">,说明这里只是排名
靠前的若干条、无法据此给出完整名单或精确计数。未截断时字段为 nil,不产
生任何标注。

提示落在 model context 视图(模型真正读到的那份)而不是
RenderedContexts:后者会被 prepareMessagesWithModelContext 整段替换成
<retrieval> 视图,写在那里的说明到不了模型。

不做日志:截断几乎每次检索都会触发,#3565 把 info 升 Warn 已被维护者以
"只会变噪音"否决;这里要解决的是模型看不到截断,不是运维看不到。

不动检索策略:不提高 topK、不改召回与重排,避免影响既有延迟和答案质量。

测试(夹具为合成数值):
- filter_top_k_test.go:截断时 Truncation={Shown:topK,Candidates:before},
  未截断(结果数 ≤ topK)时为 nil;
- modelcontext/omitted_results_test.go:带 retrieval_shown=5 /
  retrieval_candidates=48 时输出 <subset shown="5" candidates="48"> 且仍
  以 </retrieval> 结尾;shown==candidates 或计数缺失时不出现;
- references_test.go:Truncation 非 nil 时最终 model context 含
  <subset shown="3" candidates="48">,nil 时不含;另有端到端用例串起
  FILTER_TOP_K → INTO_CHAT_MESSAGE → prepareMessagesWithModelContext,确认
  提示到达模型消息而不是被替换掉。

* fix(chat): 截断提示改条件式并覆盖重排/合并的提前裁剪

评审意见(#3758):原提示无条件要求模型 "say the answer may be incomplete",
会让普通事实问答也带上免责声明;且重排阶段与 merge 回退已可能提前裁剪,
FILTER_TOP_K 不再截断时这条提示根本不生成。

- subset 提示改用评审给定的条件式措辞:只有问题在计数或要求完整列举时
  才说明结果范围,其他问题照常回答;N/M 填实际计数
- 明确 M 的语义:它是首次裁剪阶段(rerank / merge / filter_top_k)收到的
  候选片段数,不是全库匹配总数
- 重排阶段的阈值过滤、候选上限与 MMR top-k 裁剪,以及 merge 回退到
  RerankTopK 的裁剪,都经 ChatManage.RecordRetrievalCut 记录;首次裁剪为准,
  后续更窄的裁剪不覆盖候选池
- references 用实际渲染的知识片段数作为 N;补重排提前裁剪、merge 提前裁剪
  到模型消息的端到端回归测试

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:54:37 +08:00
hahahahahayesyeseys 419f17fc65
fix(wiki): resolve titles for unloaded backlinks (#3931)
* fix(wiki): resolve titles for unloaded backlinks

* fix(wiki): return backlink titles with page details

* docs(wiki): verify backlink titles in page detail schema
2026-10-09 10:50:52 +08:00
jacentsao 5d40f81ee4
fix(im): retry Yunzhijia attachment downloads (#3809) 2026-10-09 10:47:58 +08:00
linus-liu-web 04da26024c
fix(weaviate): CopyIndices 补游标防护,并修掉 GraphQL 报错时的 panic (#3881)
* fix(weaviate): 复制索引时游标不前进立即报错,并给分页加跳数上限

CopyIndices 的游标由每页最后一个对象的 id 反推(repository.go:763),循环的
退出条件只有"查询出错"和"这一页为空"。源端只要重复返回同一页,游标就永远
停在同一个 id 上:循环把这一页反复读出、再反复 upsert 进目标库,直到复制任务
撞上 2 小时超时;源端在该页之后的数据永远读不到,而目标库里同一批 chunk 已经
被写成成千上万份候选向量。同包的 move.go 早就记录了 after+where 在 Weaviate
上不可靠,并为此用 seen 集合拦住"游标不前进",CopyIndices 用的正是这个组合,
却没有那层保护。

本次修改:

- 记录每个用过的游标,重复即返回 "copy indices made no progress at cursor",
  不再把"游标不动"当成"读完了"。检测点放在整页解析之后、批量写入之前,所以
  重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页,按每页 64 条合 64 万行,与
  confluence 的 maxPaginationHops 同量级);触顶时打印 Warn 并返回错误,而不是
  把这轮复制当成成功——静默截断会留下一个看起来正常的残缺索引。
- additional["id"] 从无检查断言改为带检查断言,缺 id 或 id 为空时返回错误;
  原先这里会 panic。同包 move.go 的 move index ID 已是这种写法。

回归用例用假 Weaviate 服务端复现两种源端形态:恒定返回同一页的源端在第二次
查询后被拦下(GraphQL 查询 2 次、目标库只写 64 条,而修复前 5s 内同一页被重放
4484 次、写入 286912 条);一直推进但永不结束的源端在第 10000 页被拦下。

* fix(weaviate): 复制索引时 GraphQL 层报错不再被当成"源已读完",也不再 panic

Weaviate 对于"执行不了的查询"返回的是 HTTP 200 + errors 数组 + 没有 data。SDK 的
runGraphQLQuery 只在 HTTP 状态异常时返回 error(weaviate-go-client 的
graphql/gql.go:136-150),所以这种响应到 CopyIndices 手里时 err == nil,而
result.Data 是 nil。

原代码只检查 err,随后一行完成两件事:

    objects, ok := result.Data["Get"].(map[string]interface{})[collectionName].([]interface{})

这里逗号-ok 只作用于最后一个断言,第一个 `.(map[string]interface{})` 是单值断言;
result.Data 为 nil 时该表达式的值是 nil 接口,单值断言直接 panic。实测
(假服务端返回 {"errors":[{"message":"class Weknora_embeddings_64 does not exist"}]}):

    panic: interface conversion: models.JSONObject is nil, not map[string]interface {}
        .../weaviate.(*weaviateRepository).CopyIndices(...)
            repository.go:753

也就是说复制任务在这种响应下不是"静默少拷",而是直接 panic;即便不 panic,把
"没有 data"当成"这一页为空"也会让一次失败的查询以 Index copy completed 收尾。

本次修改:

- 在 err 检查之后补 `len(result.Errors) > 0` → 记录日志并返回
  "weaviate: copy indices query failed: <message>",与同文件 VectorRetrieve
  (:608-611)、KeywordsRetrieve (:677-680) 以及同包 move.go:32-34 的既有写法一致。
- 把那一行的两级断言拆成两次逗号-ok 断言,响应缺 Get 或 class 键时返回
  "weaviate: copy indices invalid response",不再 panic。

回归用例 TestCopyIndicesReportsAQueryError:假服务端对所有 Get 返回上述 errors
响应,断言 CopyIndices 返回带原始 message 的错误(修复前是 panic),且只发一次
查询、没有把任何东西当成复制成功。

* test(weaviate): 补一个"响应读不出 data 时不得 panic"的复制用例

上一个提交把 `result.Data["Get"].(map[string]interface{})` 这一处单值断言拆成了两次
逗号-ok 断言,但只验证了"带 errors 的响应"那条路径:那条路径在 Errors 检查处就返回
了,走不到断言。这里补 TestCopyIndicesRejectsAResponseWithoutData,让假服务端返回
HTTP 200 + `{"data":{}}`(没有 errors,也没有 Get 键),断言 CopyIndices 返回
invalid response 而不是 panic;去掉断言拆分、只保留 Errors 检查时该用例必红。

两个用例都用 recover 把 panic 转成明确的用例失败,避免一个 panic 直接终结整个测试
二进制、连带吞掉同包其它用例的结果。

* fix(retriever/weaviate): CopyIndices 按映射分块读源行,去掉会被服务端拒绝的游标

CopyIndices 用 where + after 遍历源知识库,而 Weaviate 的游标 API 不允许过滤条件
与 after 同时出现:ValidateCursor 直接返回 "where cannot be set with after and
limit parameters",且 client 第一页就会把 WithAfter("") 发成 after: "",所以这个
查询在真实服务端上一页都跑不通。该报错以 HTTP 200 + errors 返回,不检查 errors 的
代码会把它当成"源已读完",返回成功但一条都没复制(本 PR 已补的 errors 检查会把它
变成错误,但那是在暴露一个本就不合法的查询)。即使去掉 where,游标仍然是"跨请求
保持不变的结果顺序"上的位置,服务端不承诺维持该顺序,一旦漂移就会漏行且仍报成功。

改成按映射里的源 chunk ID 分块读:谓词 = knowledge_base_id 相等 AND chunk_id
ContainsAny(本批 64 个),不带 after。读到的行集合由谓词唯一确定,没有游标可漂。

同一个 chunk 会有正文向量 + 多个生成问题向量,64 个 ID 不等于最多 64 行,所以每批
查询显式带 limit(QUERY_MAXIMUM_RESULTS,v1.37.3 默认 10000)。该配置没有客户端
接口(/v1/meta 只有 version/hostname/modules),先按默认值查,被服务端以
"query maximum results exceeded ... QUERY_MAXIMUM_RESULTS 'N'" 拒绝时按 N 降级
重试。一批返回行数达到 limit 说明可能被截断,对该批对半拆分重查直到单个 chunk;
单个 chunk 仍然填满 limit 时明确报错(含 chunk ID 与服务端上限),绝不返回复制成功。
不带 limit 的 GraphQL Get 会落到 QUERY_DEFAULTS_LIMIT_GRAPHQL(100) 静默截断。

页数上限(maxCopyPaginationHops)与重复游标防护随游标一起去掉:读取范围由映射
限定,不存在走不完的分页。GraphQL errors 检查与 data["Get"]、_additional["id"]
的逗号-ok 解包保留。

测试:copy_test.go 的假服务改成按 chunk_id 过滤的真实形状,并像 v1.37.3 一样拒绝
where+after 组合(假服务会断言这种组合从未出现);补同一 chunk 多向量、批次触顶
拆分、单 chunk 超限报错、QUERY_MAXIMUM_RESULTS 降级四类用例。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:47:30 +08:00
诉说晚秋 b14bcbcd54
fix(notion): 保留正文抓取失败时的同步游标 (#3693) 2026-10-09 10:43:59 +08:00
icearia0219 005627b180 fix(agent): compare tool enum values without string coercion
Fixes Tencent/WeKnora#3946
2026-10-08 21:07:11 +08:00
clumsypsc 2a02807a61 fix(chunker): respect Markdown fences during heading detection 2026-10-08 21:06:59 +08:00
wizardchen 676a59307e fix(settings): improve modal sizing and integration page layout 2026-10-08 20:36:36 +08:00
lyingbug 0c5d28abb7
feat: index and recall images with multimodal embedding models (#3683)
* feat(models): expose embedding input modalities

Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.

- Catalog: declare input modalities for the Doubao and Tongyi vision
  embeddings, the Aliyun multimodal / qwen-vl embedding families and
  jina-embeddings-v4, as listed in each vendor's documentation. The
  migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
  capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
  the catalog and writes spec.input only when the row differs from it;
  picking another catalog model or switching vendor drops the
  declaration. Model cards show the image badge for such models.

* feat(embedding): embed images with multimodal embedding models

Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.

- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
  object store is usually unreachable from the vendor) and an
  ImageEmbedder implemented by all four protocols, each in the image
  shape its reference documents: Ark image_url, DashScope contents[].image,
  Gemini inlineData parts, and on the OpenAI shape an object keyed by the
  vendor's image_field (Jina: "image"). A plain OpenAI-compatible
  endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
  max_image_bytes and image_mime_types, filled from vendor docs for the
  DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
  AsImageEmbedder, so existing Embedder implementations and fakes are
  untouched. An embedder takes images only when the catalog (or the
  row's spec.input) lists image AND the endpoint can carry one. Images
  are checked against the documented limits before any request, then
  split at the documented images per request. The concurrency, debug
  and Langfuse decorators forward it; logs and traces carry the MIME
  type and size, never the bytes.

The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.

* feat(knowledge): index images by their own vector and recall them

When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.

Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
  the image (embedding.PrepareImage converts and shrinks it to the
  vendor's documented limits) and stores it as a new image_vector chunk:
  Content is the caption (OCR text when there is none) for reranking and
  the answer context; its index row carries the image's vector with
  SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
  text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
  the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
  so a second row would overwrite the caption's text vector, and copy,
  move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
  through each engine's normal BatchIndex, keeping row order (the
  composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
  syncChunkIndex only syncs its enabled flag, updateChunkVector skips
  it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
  backends with a column default were storing them as enabled.

Retrieval
- A text query scores lower against an image than against text, so when
  the model embeds images the document vector search is widened by half
  (capped at 500) and run at min(VectorThreshold, 0.1); each store
  group's hits are then held to their own threshold before score
  normalization: text at VectorThreshold, images at min(VectorThreshold,
  0.1). Keyword hits on image rows are dropped, their Content being the
  already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
  context is attached, and the caption is not repeated from ImageInfo.

Adds golang.org/x/image for scaling and WebP/BMP decoding.

* fix(models): report no thinking levels for embedding, rerank and ASR models

Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.

* fix(compaction): never read a cancelled turn's stream as a summary

streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.

Check the context before each receive and again when the stream closes.

* feat(knowledge): put image vectors behind a knowledge-base switch, off by default

Image vector indexing and recall used to turn on whenever the knowledge
base's embedding model declared image input. A model that takes images is
often chosen for text alone, so an upgrade would have added an embedding
call per image, widened vector recall 1.5x and lowered the image threshold
to 0.1 for those knowledge bases without anyone asking.

ImageProcessingConfig gains image_vector_enabled (default false; stored
JSON without the key reads as off, so existing knowledge bases stay off
with no migration). Both sides now need the switch and the model:

- indexImageVector returns before touching the model unless the KB opted
  in (KnowledgeBase.IsImageVectorEnabled, which also needs vector indexing).
- applyImageRecall resolves the embedding model only when some KB in the
  search opted in, and records the opted-in KBs per store group
  (storeGroup.ImageKBIDs). Only those groups widen the pool and lower the
  threshold; image hits of any other KB, including ones indexed before the
  switch was turned off, are dropped.

The KB editor shows the switch in the image processing section, with a
warning when the selected embedding model does not declare image input;
docs describe the switch, its cost and what turning it off does.

* fix(knowledge): leave image vectors behind when cloning into a KB that is off

CloneChunk copied every chunk of the source document and then CopyIndices
for all of them, so copying a document from a knowledge base with image
vectors on into one with the switch off (same model and store) still
created image_vector chunks and their index rows, bypassing the switch
that indexImageVector honours.

Image vector chunks are now dropped from the copy when the target has not
opted in. Every engine's CopyIndices copies only the rows of chunks in the
mapping, so their index rows stay behind too.

* fix(search): refill the vector pool past stale image vectors

Image hits of a knowledge base whose switch is off were dropped only after
the engine had ranked and cut its results at TopK. Image vectors indexed
before the switch was turned off could fill that pool and push out valid
text hits: with MatchCount 5 the pool is 50, and 50 stale image rows
ranked above the text left the search empty.

No engine can filter rows by source type (Postgres and SQLite do not even
take ExcludeChunkIDs), so the exclusion cannot move into the query. When
the document vector pool comes back full and holds stale image rows, that
search alone runs again on a pool twice as large, until it holds TopK rows
that are not stale, the index runs out, or the pool reaches
maxRetrievalPoolSize (at most four extra calls from a pool of 50). The pool
is then cut back to TopK such rows, matching an engine-side exclusion
unless more than maxRetrievalPoolSize-TopK stale rows rank above the text.

* feat(i18n): translate the image vector switch and image input keys into zh-TW

The zh-TW locale arrived from main after this branch added five keys: the
knowledge base's image vector switch (label, description, unsupported-model
warning) and the embedding model's image input flag (label, description).

* fix(knowledge): leave image vectors behind when moving into a KB that is off

A reuse_vectors move relocates a document's chunks and index rows in place:
MoveKnowledgeIndices rebinds the rows by document ID and
MoveChunksByKnowledgeID rebinds the chunks. Moving a document from a
knowledge base with image vectors on into one with the switch off therefore
carried its image_vector chunks and their vectors along, the same bypass the
clone path had.

When the target has not opted in, the image_vector chunks are now deleted
during the move, their index rows first and then the chunks, so a retry
finds the chunks again rather than leaving orphaned rows. A target that
opted in still receives them unchanged.

* fix(retriever): keep image vectors out of keyword-only engines

BatchIndexVectors handed image rows to every engine of the composite,
including one that serves keyword retrieval alone (Elasticsearch v7 next to
a vector store). There the row holds nothing but the image's caption, which
the caption's own chunk already indexes, and retrieval discards every
keyword hit on it.

Image rows now go only to engines that serve vector retrieval. An engine
that serves both keeps one row for both kinds, so its keyword search still
meets them; the search side handles that.

* fix(search): refill the keyword pool past image rows

Every engine that serves both kinds of retrieval stores one row per index
entry and runs its keyword search over that row's content, so an image
vector row, whose content is its caption, is a keyword hit wherever the
caption's own chunk is. filterImageHits drops all of them, but only after
the engine has cut its ranking at TopK, so they crowded out text and caption
hits; a keyword ranking that put the image rows first came back empty.

The keyword pool is now refilled the way the vector pool is refilled past
stale image vectors: when it came back full and held image rows, the
keyword search runs again on a pool twice as large, up to
maxRetrievalPoolSize, and is cut back to TopK hits that are kept.

Leaving the rows out of the keyword index instead would mean writing them
without searchable content (Milvus and Tencent VectorDB derive a BM25
sparse vector from it, and how they take an empty one is unverified) or a
source-type filter in every engine's keyword query (ten engines, and
filterable fields that existing collections lack).

* revert: let image vectors move with the document into a KB that is off

This reverts commit e259977fe915f7d0bc30c4b02bc7af04da1c8dd4.

Deleting the image vector index rows by chunk ID before the move is not
reliable: the Weaviate and Elasticsearch v8 delete wrappers ignore partial
failures, and OpenSearch rejects more than 1000 IDs with ErrBatchTooLarge.
A partial delete followed by the chunk delete leaves index rows whose chunks
are gone, which nothing ever cleans up.

Without it, the image_vector chunks and their index rows move together
with the document. While the target's switch is off they are filtered at
query time (the pool refill keeps them from crowding out text), they are
removed with the document on delete, reparse or knowledge base delete, both
chunks and index rows by knowledge ID, and they become active again if the
target turns the switch on. The clone path keeps dropping them, since there
the rows are copied rather than moved.

* fix(search): keep the image row refill from losing hits it already had

refillPool replaced a pool with the answer to each larger search. A refill
that answered only in part (RetrieveResult.Error, swallowed into a warning
by retainPartialResults, as when some collections of the store did not
answer) could hold fewer text rows than the complete smaller pool, or none:
40 stale image rows above 10 text came back from a partial refill as the
40 image rows alone, and the search lost all 10 text hits. A refill that
failed outright failed the whole store group, discarding a first round that
had succeeded.

A refill is now best effort. A failed or incomplete refill search stops
the refill and leaves the last complete pool in place, cut back to TopK
kept rows as before, so a refill can no longer leave a search worse off
than it was without it. Merging the partial answer into the pool was not
done: its rows would need deduplicating and its ranking is cut at a
different TopK over a different set of collections, while keeping the
previous pool is exact.

* docs(knowledge): say what happens to image vectors in a KB that is off and what the refill costs

The image vector section now says that with the switch off, image vectors
that were indexed before it was turned off, or that came along when the
document was moved in from a KB that has it on, are kept but filtered at
query time, are removed when the document is deleted or reparsed or the
KB is deleted, and are recalled again once the switch is turned on; a copy
into such a KB leaves them behind.

It also states the cost of keeping dropped image rows from crowding text
out of the pools: the vector and keyword pools are searched again with
TopK doubled up to the 500 candidate cap, which adds queries for store
groups that hold such rows, and text can still be missed when more than
about 500 candidates are image rows. A failed or partial refill keeps the
previous complete pool.

* fix(search): log unexpected refill result sets, and describe the refill stop condition precisely
2026-10-08 18:19:53 +08:00
lyingbug b7f3181066
docs: count 29 built-in model vendors and list Hugging Face TEI and Pinecone (#4048)
#3956 (Hugging Face TEI) and #3957 (Pinecone) added two rerank-only
vendors, but the READMEs, the models page, the extension-points page and
the homepage still said 27, and the vendor table had no rows for them.
The v0.8.2 release notes keep their historical 27.
2026-10-08 18:03:07 +08:00
ZW. 5a25e26142
feat(rerank): support Pinecone Inference protocol (#3957)
* feat(rerank): support Pinecone Inference protocol

* test(rerank): align Pinecone catalog baseline and lint

* test(models): expect 29 built-in vendors with Pinecone and TEI

---------

Co-authored-by: bigdu332 <bigdu332@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 17:54:15 +08:00
LuckVd ea7cf8d5d7
fix(models): emit explicit empty content for assistant tool-call turns (#3605)
Assistant history messages that carry tool_calls but empty text previously
omitted the content field entirely on the wire. Strict local OpenAI-compatible
engines (vLLM / Xinference serving Qwen-family Jinja chat templates) treat the
missing field as undefined/null and crash while rendering the next turn,
surfacing as "I'm sorry, I was unable to generate a response".

Always serialize "content": "" for assistant tool-call turns. The empty
string is accepted by every standard OpenAI-compatible endpoint.

Fixes #1487

Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
2026-10-08 17:43:05 +08:00
wizardchen a729d1fc8c docs(models): state which rerank limits split requests and which fail the call
The rerank limits row said every limit splits the request automatically.
Only max_documents and max_request_chars do; an over-long query or
document (max_query_chars / max_document_chars, or a single document
that does not fit the request budget beside the query) fails the call.
Taken from the documentation part of #3757; max_documents keeps
0 = unlimited.

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 17:03:41 +08:00
wizardchen 0de62b2f80 fix(chat): restore citation source navigation and soften highlights 2026-10-08 16:59:50 +08:00
wizardchen 1c528d83de fix(i18n): add the zh-TW Seafile strings missing after #3616 and #3916
#3616 (Seafile) and #3916 (zh-TW) were merged one after the other; each
was green on its own, but zh-TW had no translations for the 21 Seafile
keys, so the locale key audit fails on main.
2026-10-08 16:47:53 +08:00
Junhao Ye d2bc0df6a2
feat(i18n): add Traditional Chinese (zh-TW) locale support (#3916)
* i18n(zh-TW): add Traditional Chinese (Taiwan) locale bundle (6,547 keys)

機械產生自 zh-CN.ts:以 zhconv (zh-tw) 為字形基底,詞彙層以 tw-terms.tsv(364 條生效詞條)覆寫,保留 26 條 never 與 47 條保護詞。骨架等價、鍵序列、Node 真實載入、內嵌區塊 四道檢查皆通過(詳見審計目錄)。

* i18n(zh-TW): register locale bundle in i18n infrastructure (5 → 6)

- resolveDefaultLocale.ts: SUPPORTED_LOCALES = ['zh-CN','zh-TW','en-US','ru-RU','ko-KR','ja-JP']
- i18n/index.ts: 載入 zhTW 並掛入 messages['zh-TW']
- localeKeyAudit.ts: 加入 LOCALE_BUNDLES['zh-TW'] 與測試守門(LOCALE_ORDER、emptyBundles)
- localeKeyAudit.test.ts: 把 zh-TW 補進期望清單
- locales/{en-US,ja-JP,ko-KR,ru-RU,zh-CN}.ts: language 區塊補 zhTW 顯示名(讓語言切換器正確顯示繁中選項)

* i18n(zh-TW): fix normalizeEmbedLocale and add zh-TW embed block (514 keys)

原實作 startsWith('zh') 直接命中 zh-TW/zh-HK/zh-Hant 並回傳 zh-CN
→ 繁中使用者將永遠看到簡體字串。修正後:
  - 繁中變體 (zh-tw/zh-hk/zh-mo/zh-hant) 排在通用 zh 之前,優先回傳 zh-TW
  - 內嵌 zh-TW 區塊加入 514 個字串,骨架與 zh-CN 區塊逐字元相同(只換區塊名)

* i18n(zh-TW): wire zh-TW into UI language pickers and TDesign locale

- App.vue: 載入 tdesign-vue-next esm/zh_TW,tdLocaleMap['zh-TW'] = zhTWConfig
- Login.vue: 語言切換器新增 { value: 'zh-TW', label: '繁體中文', shortLabel: '繁中', flag: '繁' }
  (刻意不用旗標 emoji,改以「繁」字標示,語意清楚且無爭議)
- GeneralSettings.vue: 設定頁語言下拉新增 <t-option value='zh-TW' ...>
- AgentEmbedChannelPanel.vue: 代理嵌入頻道語言選單新增繁中
- IMChannelPanel.vue: IM 頻道語言選單新增繁中

* i18n(zh-TW): widen API union types and update tests for zh-TW

- api/agent/index.ts: IMChannel.locale 聯合型別補 'zh-TW'
- api/embed/index.ts: EmbedLocaleTag 聯合型別補 'zh-TW'
- i18n/embedLocale.test.ts: supported-list 期望補上 zh-TW;新增 7 條 normalizeEmbedLocale 斷言(覆蓋 zh-tw / zh-hk / zh-mo / zh-hant / zh-Hant-TW 等變體正確路由到 zh-TW)
- locales/workspaceTerminology.test.ts: workspace 詞彙測試也對 zh-TW 做同樣的禁用詞斷言(與其他 locale 同步)

* i18n(zh-TW): add zh-TW to backend supportedLocales whitelist

- internal/types/locale.go: supportedLocales["zh-TW"] = {}
  (NormalizeSupportedLocale 會靜默丟掉未列出的 tag,導致前端語言選擇器實際寫入空字串——
   此修正保證前端選 zh-TW 時 embed 預設語言與 IM 回覆語言會被後端接受)
- internal/types/locale_test.go: 期望清單補上 zh-TW
- internal/handler/mcp_usage_instructions.go: MCP 工具使用說明對照表新增 zh-TW → 'Traditional Chinese'

* fix(i18n): translate revoke as 撤銷 in zh-TW and allow zh-TW as the runtime default locale

The zh-CN → zh-TW conversion turned every 撤销 into 復原 (restore), so
revoke buttons for OAuth tokens, device pairing, invitations and system
admins read as the opposite action. Only the undo sense (此操作不可撤销)
keeps 復原.

docker-entrypoint.sh whitelists DEFAULT_LOCALE and did not accept zh-TW,
so DEFAULT_LOCALE=zh-TW silently fell back to Simplified Chinese. Also
list zh-TW in the IM locale error and the locale hints.

---------

Co-authored-by: junhao-ye <junhao-ye@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 16:18:17 +08:00
zaqqmm 19542f2a1d
feat(datasource): add Seafile data source connector (#3616)
* fix(datasource): omit last_sync_cursor from the data source list response

The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.

* feat(datasource): add Seafile connector with fingerprint-based incremental sync

Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.

- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils

* feat(datasource): add Seafile frontend connector and docs

Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.

- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
  updates, and Seafile in the README data source lists

---------

Co-authored-by: zaqqmm <q-zhang10@qq.com>
2026-10-08 16:18:09 +08:00
linus-liu-web bcbd65a459
fix(datasource/dingtalk): Validate 不再因为某个工作区读不到就整库判失败 (#3782)
* fix(datasource/dingtalk): 多工作区租户里一个工作区读不到就整库判失败

Validate 原来在第一个读不到的工作区或文档上直接 return err。租户的
listWorkspaces 会返回组织内的全部工作区,而 operator 通常只被授权其中
一部分,于是"知识库里混了多个工作区、其中一个没权限"会被判成"凭据
错误",整库都建不起来,哪怕另一个工作区完全可读。

改为在循环里累计 lastErr / workspaceFail / sawDocument:某个工作区列不出
节点、或某个文档读不到,都只记录原因并 continue,不再中断;循环结束后
只有"一个能读的都没有"才报错。一个文档都没暴露的租户仍然放行——没有
可读之物就无法证伪凭据,只有"存在却读不到"才算问题。

文档探测抽成 verifyDocument,后续接入上传文件、表格时在这里加分支。

验证:
- 容器内 go build ./internal/...(仅第三方 sqlite-vec cgo 因环境缺
  sqlite3.h 失败)+ go test ./internal/datasource/... 全部通过
- 反向验证:把 Validate 还原成"第一个失败即返回"后,
  TestValidateSkipsWorkspacesTheOperatorCannotRead 与
  TestValidateSurvivesWorkspaceListingFailure 立即失败

* test(datasource/dingtalk): 钉住 Validate 收紧路径并修正 verifyDocument 注释

Validate 改成继续遍历所有工作区后,多出一条比改动前更严的失败路径:
"工作区 1 只有文件夹(可读)+ 工作区 2 有一个本 operator 读不到的文档"
这种配置,改动前在第一个工作区末尾就 return nil,现在会报错。这条路径是
有意的——存在却读不到的文档意味着数据源同步不出任何东西,不是无关工作区
的权限噪音——但之前没有任何测试。

新增 TestValidateRejectsTenantWhoseOnlyDocumentIsUnreadable 固定该行为与
意图;并给既有的"全部文档都读不到"用例补上"错误必须带工作区名/文档名"的
断言(原先只 Contains("no permission"))。

顺带把 verifyDocument 的注释从"上传文件与表格目前不可摄取、永远到不了
这个函数"改成描述不变量:只有 isDocument 接受的原生类型才会进入该函数,
一个类型获得摄取路径时同时成为 document、获得同步读路径并在该函数获得
探测,上传文件与表格随各自的摄取路径一起接入。原措辞在表格/上传 PR 合并
后即失效。

验证:
- go test ./internal/datasource/connector/dingtalk/... -count=1 通过
- 反向验证:把 Validate 换回改动前的实现(首个工作区无文档即 return nil)
  后,新用例失败并报 "Validate must fail";恢复后通过

* fix(datasource/dingtalk): Validate 每个工作区只探 1 个文档,并给探测总数加上限

应用缺少文档读权限时每次探测都会失败,原先会把每个工作区根目录下的每个文档
都串行探一遍,大租户里一次校验就是几百次调用。改为探测失败后直接跳到下一个
工作区,并且最多探测 maxValidateProbes(5) 个文档后返回最后一个错误。

* fix(datasource/dingtalk): Validate 进入文件夹找文档,根目录只有文件夹的工作区不再被误判

根目录只有文件夹、文件夹里有可读文档的工作区,原先会被另一个读不到的工作区拖累,
整体判失败(main 上这种情况是通过的)。Validate 现在按 scanWorkspace 的走法对
每个工作区做广度优先查找,取第一个文档探测;子文件夹列举共用 20 次额度,额度用尽
仍有文件夹没看完时视为无法证伪,直接通过。

* fix(datasource/dingtalk): Validate 子文件夹列举失败不再当空目录,并按页限额、防环

- 非根文件夹列举失败时把本次校验记为无法确定(放行),不再当成空目录导致
  另一个工作区的不可读文档把整库判死;取消/超时错误直接返回,不再被当成
  凭据有效,也不会被之前记下的 inconclusive 吞掉。
- BFS 加 visited 集合,子节点 ID 等于祖先或工作区根时不再无限列举。
- 新增 listNodesPage 单页列举,Validate 的列举额度按实际 HTTP 分页计数:
  节点列举请求最多 len(workspaces) + maxValidateListings 次(另加客户端重试)。

* fix(datasource/dingtalk): Validate 先翻完同层分页再进子文件夹,重复 nextToken 时停下

子文件夹排在父目录下一页之前时,根目录第一页全是文件夹、文档在第二页的
工作区会把列举额度花在子文件夹上,第二页的文档永远探不到。现在每个文件夹
先翻完自己的分页再下钻,和 listNodes 给 scanWorkspace 的顺序一致;同一
文件夹重复返回 nextToken 时停下并按无法确定处理。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: 🏠 <linus@192.168.10.6>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 15:20:27 +08:00
wizardchen ae90546262 fix(frontend): avoid false unsaved changes warnings 2026-10-08 14:57:40 +08:00
LuckVd b5c8d90cc2
fix(chat_pipeline): treat non-JSON model refusals as skipped graph extraction (#3606)
* fix(chat_pipeline): treat non-JSON model refusals as skipped graph extraction

When a chunk carries nothing extractable (e.g. a table-of-contents page),
the LLM correctly answers with a prose refusal such as "抱歉…". parseOutput
then fed that prose to json.Unmarshal, failing with a confusing
"invalid character 'æ'" (the first byte of a CJK character), marking the
chunk task GRAPH_EXTRACT_FAILED and letting asynq replay the identical
doomed LLM call up to the retry limit.

Classify responses that contain no JSON structure at all (no '{' or '[')
as errModelDeclined and complete the extraction with an empty graph so the
task succeeds as an explicit skip. Malformed or truncated JSON (structure
present) still fails and retries as before.

Fixes #3600

* fix(chat_pipeline): keep empty model responses retriable in graph extraction

An empty response is usually transient (gateway returning 200 with no
content, a reasoning model spending its whole budget on reasoning,
content filtering), and a retry often succeeds. Only non-empty prose
with no JSON structure is treated as a declined extraction.

* fix(graph): skip prose refusals only in the async chunk task, keep truncation retriable

ParseGraph no longer turns a prose answer into an empty graph. It returns
an error wrapping ErrModelDeclined, and ChunkExtractService.Handle is the
only caller that treats it as a skip (span output skipped=model_declined).
The /initialization "try extract" endpoint and query-side entity extraction
report the refusal as a failure again, so the graph settings form no longer
gets its few-shot examples replaced with an empty result.

Extractor.Extract drops the ErrModelDeclined marker when finish_reason is
"length": prose with no JSON because the output budget ran out is
truncation, and asynq should retry it.

* fix(graph): retry every truncation spelling, not only finish_reason=length

Extract kept a prose answer retriable only when FinishReason was exactly
"length". max_tokens / max_output_tokens (and case/space variants that
IsLengthFinishReason already accepts), a stream that broke before the stop
event ("incomplete") and an adapter that reports no reason were still read
as a refusal, so the async chunk task skipped the chunk for good.

Only a reason that proves the model stopped on its own now counts as a
refusal. Ollama's non-streaming Chat also forwards done_reason, so a
num_predict cut-off is reported as "length" instead of an empty string.

* fix(graph): only treat a response with no JSON anywhere as a refusal

The refusal check looked at the first extracted fence, so a prose fence
followed by a JSON fence was skipped as a refusal instead of failing as a
retriable parse error.

---------

Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:57:11 +08:00
5quan 3fb6b9d631
fix(kb): let the update API set vlm_config (#3704)
* fix(kb): let update API set vlm_config

UpdateKnowledgeBaseRequest only carried name/description/config, so the
PUT /knowledge-bases/:id endpoint could never change a knowledge base's
multimodal (VLM) config, even though CreateKnowledgeBase accepts a
top-level vlm_config. Thread an optional vlm_config through the handler
and service so the update path mirrors create (nil = no change).

Fixes #1723

* test(kb): pass the vlm_config argument in tests added on main

* fix(kb): accept only model-managed vlm_config fields on update

The update API copied the request's VLMConfig wholesale, so a caller
could set the legacy ModelName/BaseURL/APIKey/InterfaceType fields and
have the VLM client call BaseURL directly, outside model management and
its SSRF checks. Only Enabled/ModelID/DescriptionLanguage/
CustomInstructions are taken now; legacy fields keep their stored value.
A referenced model must exist and be a VLLM model (400 otherwise), and
disabling clears ModelID, matching the initialization config endpoint.

* fix(kb): validate and gate vlm_config on the update API

- Enforce the 4000-character custom_instructions cap
  (types.ValidateKnowledgeBasePromptInstructions), as create and
  /initialization/config already do.
- vlm_config is a KB setting: require kbSettingsAccess (owner workspace
  or admin share) like /initialization/config, so share editors get 403.
- Stored legacy model_name/base_url keep IsEnabled() true on their own,
  so disabling returned 200 but left VLM on, and managed -> disabled fell
  back to the old endpoint. Clear them when the request enables a managed
  model_id, or disables VLM without echoing back the stored
  model_name+base_url (GET-then-PUT keeps legacy on). Request legacy
  fields are only compared, never written. Enabling without model_id on a
  non-legacy KB is now a 400 instead of a silent no-op.
- Document the vlm_config semantics in the KB API reference.

* fix(kb): keep the legacy VLM endpoint only for pure legacy knowledge bases

CreateKnowledgeBase stores vlm_config as sent, so a managed model_id can
sit next to legacy model_name/base_url; resolveVLM then uses the managed
model. The update's keep-legacy paths (disable echoing the stored legacy
fields, enabled=true without model_id) cleared model_id but kept the
legacy fields, so IsEnabled() stayed true and image calls moved to the
stored base_url.

Those paths now apply only when no model_id is stored. On a mixed config
a disable clears everything, and enabled=true without model_id is a 400,
the same as on any managed knowledge base.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:37:23 +08:00
LingYi bea0d7b689
fix(docreader): preserve PPTX fallback image associations (#3973)
* fix(docreader): preserve PPTX fallback image associations

* fix(docreader): visit PPTX shapes in MarkItDown's position order

MarkItDown emits pictures sorted by (top, left) per slide and per group, not
in z-order, so two same-named pictures on one slide could swap images. Sort
the same way when collecting placeholder sources.

* fix(docreader): skip unresolvable PPTX picture relationships instead of failing the document

A picture with no a:blip, an r:embed that is dangling or points at an
external relationship, or a media part python-pptx dropped at load time
made _picture_sources raise, so the whole PPTX (slide text included)
failed to parse. Record None for that picture to keep the queue position
and leave its placeholder untouched; other pictures still attach.

* fix(docreader): also skip PPTX pictures whose blipFill is missing

python-pptx raises InvalidXmlError (a PythonPptxError, not a ValueError)
when a p:pic has no p:blipFill; catch it so the picture is skipped
instead of failing the whole document.

---------

Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:37:07 +08:00
诉说晚秋 d84cdabac4
fix(rss): refresh titles and source links when article text is unchanged (#3824)
* fix(rss): 修复正文未变时标题和来源链接不同步

* fix(datasource/rss): 旧 h: 游标只升级不重建,原文抓取失败时保留旧游标

升级后第一次增量原先会把 feed 里每一条仍在的旧条目删掉重建(Wiki、手工标签、
知识 ID 都会丢),原文接口临时失败时还会把库里的全文降级成 feed 摘要。

现在对旧 h: 游标的条目:重新解析出的正文哈希与 h: 相同时,只把游标升级为 i:
指纹、不输出;原文抓取失败时沿用 h: 游标、不输出,留到下次重试;只有正文真的
变了才输出。

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:56 +08:00
ZW. 99b4c3a9b0
feat(rerank): support Hugging Face TEI protocol (#3956)
* feat(rerank): support Hugging Face TEI protocol

* style(rerank): satisfy TEI lint checks

* fix(rerank): cap Hugging Face TEI batches at 32 documents

TEI rejects requests above --max-client-batch-size, which defaults to 32.
Without a declared ceiling the shared batching layer sends the whole
candidate set in one request, so a default TEI deployment fails rerank
whenever hybrid retrieval returns more than 32 candidates.

* fix(rerank): send truncate and merge extra_body in the TEI client

TEI before 1.9 starts with --auto-truncate=false and rejects the whole
/rerank call when one passage is longer than the model's input limit, so
retrieval silently falls back to the original order. Ask for truncation
explicitly, as the NIM client does with truncate=END.

The TEI client also ignored compat extra_body, unlike the Cohere, NIM and
DashScope clients. Merge it the same way: query, texts and raw_scores stay
owned by the client, truncate is a default that extra_body may override.

---------

Co-authored-by: bigdu332 <bigdu332@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:49 +08:00
linus-liu-web 1826f46b12
fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891)
* fix(docparser): 云解析轮询上限接入配置,去掉两处硬编码超时

PaddleOCR-VL Cloud 与 WeKnoraCloud 两个 reader 的整批轮询上限仍是写死的:
paddleocr_vl_cloud_converter.go 的 paddleOCRVLCloudTimeout(600s)直接决定了
pollJob 的 deadline,reader 结构体没有任何 timeout 字段;weknoracloud_http_reader.go
的 pollTimeout 固定 20 分钟。

同族的另外两个 reader 早就可以配置:MinerU Cloud 走
WEKNORA_MINERU_CLOUD_TIMEOUT(mineru_cloud_converter.go),自建 PaddleOCR-VL 走
WEKNORA_PADDLEOCR_VL_TIMEOUT(paddleocr_vl_converter.go)。后果是运维把
WEKNORA_DOCREADER_CALL_TIMEOUT(默认 30m)调大也无效:WeKnoraCloud 的 20 分钟
比它还短,硬编码上限先触发,大体积扫描件固定在第 10/20 分钟失败。

改动:
- PaddleOCRVLCloudReader 增加 timeout / pollInterval 字段(与 MinerUCloudReader
  同形),构造函数用 requestTimeoutFromEnv 读 WEKNORA_PADDLEOCR_VL_CLOUD_TIMEOUT
  (默认仍是 600s),pollJob 用 c.timeout / c.pollInterval;超时文案对齐 MinerU
  的 "task timed out after %s (%d polls)"(旧的 "after %d polls" 是 27b0c84b 已经
  替 MinerU 换掉的写法)。
- WeKnoraCloudSignedDocumentReader 的 pollTimeout 改读
  WEKNORA_WEKNORACLOUD_TIMEOUT(默认仍是 20m)。
- .env.example 补两条说明,位置紧跟 WEKNORA_MINERU_CLOUD_TIMEOUT。

新增用例:两个 reader 的 env→超时值映射(含空/非法/0/负数回退默认)、超时真的会
结束一个厂商持续报 running 的任务(假服务器与假 Transport,带看门狗,回归时 5 秒
内失败而不是轮询 600 秒)。只回退接线(保留字段)时四条用例在 10 秒内全部变红:
PaddleOCR-VL Cloud 的 200ms 配置被忽略、WeKnoraCloud 读回 20m 而非配置值;
把两个源文件整体回退则用例引用的新配置面不存在,包测试无法编译。

* docs(docparser): document PaddleOCR-VL Cloud / WeKnoraCloud poll timeouts

Add the two new env vars to both website-docs timeout tables and pass them
through docker-compose.yml next to WEKNORA_MINERU_CLOUD_TIMEOUT.

* fix(docparser): WeKnoraCloud 轮询上限在调用方已有 deadline 时也生效

入库和临时文档调用 pollTaskResult 时 ctx 都带有 DocReader 调用超时,
原先的 !ok 条件让 WEKNORA_WEKNORACLOUD_TIMEOUT 在生产路径上被跳过。
改为始终叠加 WithTimeout,context 自动取内外较早的 deadline;
回归测试补上父 ctx 带更长 deadline 的场景。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:42 +08:00
Jang Dong Ik b5fd79edf2
fix(model): treat 400 as reachable in chat connection test again (#3929)
* fix(model): treat 400 as reachable in chat connection test again

checkChatModelConnection probes with max_tokens=1 and is meant to treat a
400 as "endpoint reachable, auth OK". It detected that by matching the
go-openai wording "status code: 400". Since #3470 the error comes from
api.HTTPError ("API request failed with status 400: ..."), so the branch
never matched and the probe reported a failure.

This shows up with reasoning models behind LiteLLM: the 1-token budget
cannot finish a message, the gateway answers 400, and Test Connection
fails although the saved model chats fine.

Match the status code via errors.As on *api.HTTPError instead of the
message text.

* test(handler): drop unused request parameter in fake chat gateway

* fix(model): report a 400 that carries an auth error as auth failure

Gemini's native API answers an invalid key with 400 INVALID_ARGUMENT /
API_KEY_INVALID. Since the connection check now treats any typed 400 as
reachable, that showed "连接正常,模型可用" for a bad key. Keep the 400 pass
only when the body does not describe a credential problem, and let
classifyConnectionError recognise those bodies so the message says
认证失败 instead of a bare 连接失败.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:35 +08:00
yetuge a75e8e9ac0
fix(datasource): 语雀/Notion/IMA/飞书连接器 API 响应体补体积上限 (#3985)
* fix(datasource): 语雀/Notion/IMA/飞书 API 响应体补体积上限

#3887 给 GitLab 客户端补了 readCapped 上限;语雀、Notion、IMA、飞书四个
连接器的 API 读取仍是裸 io.ReadAll,破损或恶意端点可让同步进程缓冲无界
响应体。各连接器比照 GitLab 同步同一上限(16 MiB)与显式超限报错,limit
做成字段以便测试下调。飞书 429/5xx 错误路径的 best-effort 读取刻意不动。

* fix(datasource): 响应体上限为零值时回退到默认 16 MiB

测试里直接用字面量构造的 Client 不经过 NewClient,jsonLimit 为 0,
readCapped 会拒绝任何响应,导致 feishu/core 11 个现有测试失败。
readCapped 遇到非正数 limit 时改用 maxJSONResponseBytes。

* fix(datasource): 响应体超限不再重试;补齐飞书 token 与下载错误分支的读取上限

- yuque/ima/feishu/notion 的 readCapped 改用哨兵错误 errResponseTooLarge,
  文案不变;yuque doRequest、ima callAPIAt、feishu DoRequest 遇到超限直接返回,
  不再按退避重试 3 次(同一请求必然再次超限,只会白耗限流额度)。
- 飞书 GetTenantAccessToken 改为 readCapped + json.Unmarshal,和其他 API 响应同一上限。
- 飞书 downloadRawBytes 的 429/5xx/非 200 错误预览只读 4 KiB,
  download failed 错误不再携带完整响应体。

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:29 +08:00
rill 9a2cd20063
fix(frontend): keep target="_blank" on sanitized markdown links (#3818)
* md预览链接 a 增加target

* fix(security): keep target on links in document preview sanitizer

documentPreviewDomPurifyConfig spread the base config and then replaced
ADD_ATTR with the image attributes only, dropping the 'target' entry
this PR adds, so links in document previews still lost target="_blank".
Extend the base ADD_ATTR instead, and lock both configs with tests.

* fix(security): drop author-supplied target before sanitizing links

ADD_ATTR now allows target globally so the hook-added target="_blank"
survives, but that also kept author values on elements the hook never
rewrites (SVG <a>, <area>, protocol-relative / uppercase / leading-space
hrefs in sanitizeHTML and document preview), e.g. target="_blank"
rel="opener" or target="_top". Strip target in beforeSanitizeElements
so only the hook's _blank, always paired with rel="noopener noreferrer",
can appear.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:22 +08:00
澍 11aa00f85f
fix(agent): preserve untouched numeric tool arguments during casting (#3936)
* fix(agent): preserve untouched numeric tool arguments during casting

Fixes Tencent/WeKnora#3934

* test(agent): wrap long schema literal to satisfy lll

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:14:18 +08:00
Jin cbfd66fd92
fix(knowledge): delete written chunks when the storage quota check fails (#3656)
* fix(knowledge): delete written chunks when the storage quota check fails

processChunks writes the chunks before the storage quota check. When the
check fails, or the tenant cannot be re-read for it, the knowledge is marked
failed and the function returned without removing those chunks, so they
stayed active (deleted_at IS NULL, is_enabled = true) under a failed,
disabled knowledge. The BatchIndex failure path already deletes them; do the
same in both quota branches. Nothing has been indexed at that point, so only
the chunk rows need removing.

Fixes #3648

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(knowledge): keep the processChunks doc comment on processChunks

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:14:10 +08:00
Wen.Vale de2dfd3034
fix(multimodal): bound remote image downloads by the file size limit (#3906)
* fix(multimodal): bound remote image downloads by the file size limit

* style(utils): wrap over-long lines in the DownloadBytes test

---------

Co-authored-by: Wen.Vale <224270013+SEVEN-us@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:14:04 +08:00
SIRIUS 49e262147c
fix(cli): preserve directory structure in doc upload --recursive (#3753)
* fix(cli): preserve directory structure in doc upload --recursive

`doc upload --recursive` passed an empty customFileName to
CreateKnowledgeFromFile, so every file landed in the knowledge base root
and the directory tree it was asked to walk was lost. The web UI's folder
upload does not have this problem -- it passes the browser-supplied
relative path, which the server splits into folder_path + file_name via
types.SplitKnowledgeRelativePath.

Pass the path relative to the walked directory instead. Files directly
under that directory still send "" so the request stays byte-identical to
the previous behaviour for the common flat-directory case; only nested
files gain a folder prefix.

- knowledgeRelativePath() converts to forward slashes (the server splits on
  "/", so Windows backslashes would not separate) and returns "" for paths
  outside the walked root rather than leaking a "../" chain into the folder
  tree.
- The test double previously discarded customFileName with `_,` -- which is
  why this could regress with the suite green. It now records the value and
  two tests cover the nested and flat cases.

* fix(cli): reject only the parent component in knowledgeRelativePath

Review feedback on the recursive-upload folder path:

- `strings.HasPrefix(rel, "..")` also rejected children whose name merely
  starts with dots, e.g. `..drafts/report.pdf`. Replaced with a
  component-wise check (`rel == ".."` or `".."+separator` prefix).
- Root-level files must send ""; the guard now returns "" for
  `filepath.Dir(rel) == "."` explicitly rather than relying on the
  accidental prefix match to encode that rule.

Covered by a `..drafts/report.pdf` case in
TestUploadRecursive_PreservesFolderStructure.

* test(cli): expect an empty name for root files in knowledgeRelativePath

knowledgeRelativePath returns "" for files directly under the walked
root, so the test must expect that. Also wrap an over-long line.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:13:57 +08:00
LingYi e6ecb9adcc
fix(dataset): allow selecting the QA answer model (#3944)
* fix(dataset): allow selecting the QA answer model

* test(dataset): expect gpt-5.6-sol as the default QA model

The default moved from gpt-4o-2024-05-13 to gpt-5.6-sol on main (#3965).

---------

Co-authored-by: LingYi-Liang <270316896+LingYi-Liang@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:49 +08:00
linus-liu-web 4eb23a8560
fix(qdrant): CopyIndices 用服务端返回的 next_page_offset 翻页,不再按页尾 id 回填 (#3882)
* fix(qdrant): 复制索引改用服务端返回的游标,重复游标报错并给分页加跳数上限

CopyIndices 用"上一页最后一个点的 id"当游标(repository.go:987)。Qdrant 的
scroll offset 是**包含**语义 —— proto 里写的是 "Start with this ID",实测下一页
正是从该 id 本身开始返回,所以每翻一页都会把上一页的最后一个点再读一次、再写一份
到目标库。服务端本来就把下一页该用的游标放在 ScrollResponse.next_page_offset
("Use this offset for the next query")里,而客户端只在 ScrollAndOffset() 中返回它,
本函数调用的 Scroll() 恰好把它丢掉了;终止条件也只能退化成"这一页比 64 短"。

在真实 Qdrant 1.19.1 上实测:150 个源 chunk 复制出 152 个目标点,chunk-63 与
chunk-126 各被复制两次(每个页边界一个),totalCopied 同步虚高。源库只要超过一页
(>64 个 chunk)就会命中,不需要服务端有任何异常行为。

本次修改:

- 改用 q.client.ScrollAndOffset,游标直接取服务端返回的 next_page_offset;终止
  条件改为"服务端没有下一页"(next_page_offset == nil),不再用短页推断读完。
- 记录用过的游标,重复即返回 "copy indices made no progress at cursor ...",
  检测点在整页处理之前,重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页);触顶打印 Warn 并返回错误,
  而不是把这一轮复制当成成功。
- 修复后同一份数据在真实 Qdrant 上测得 150 个目标点、0 重复。

回归用例用假 gRPC 服务端(拦截 unary RPC,对齐 repository_test.go 的
newInterceptedQdrantClient)复刻真实服务端的游标语义:150 个源 chunk 必须各复制
一次(修复前 152 个点、2 个重复);游标恒定不动的源端必须报 no progress
(修复前 5 秒写入 3127040 个点仍未返回);一直推进但永不结束的源端在第 10000 页
被拦下。

* fix(qdrant): drop the CopyIndices page cap

10000 pages of 64 points capped a copy at 640k points and failed the
whole clone beyond that. The scroll walks every point of the source
knowledge base (a single-knowledge clone included), so a cap derived from
the chunk map would cut off small clones out of large bases too.
next_page_offset only moves forward, so the repeated-cursor check already
stops a walk that cannot finish, and the task context bounds the rest.

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:43 +08:00
MLYoshi 4dd19b5356
fix(frontend): render single tilde literally instead of GFM strikethrough (#3963)
* fix(markdown): 单个波浪号不再被解析成删除线

GFM 会把区间标记(如 2020~2035)里的单个 `~` 误解析成删除线。
覆盖 marked 的 del tokenizer:`~~...~~` 仍走默认删除线规则,
单个 `~` 则按字面保留;并新增单测覆盖时间/区间场景。

* fix(markdown): register the literal single-tilde tokenizer once

doc-content.vue and WikiBrowser.vue called marked.use() for the del
tokenizer override inside <script setup>, so every mount wrapped the
global marked tokenizer one more time. Move the extension into
utils/markedLiteralTilde.ts and register it on the global marked behind
a once-only guard; the preview Marked instance reuses the same
extension object. Add a test that repeated registration does not
re-wrap tokenizer.del.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:36 +08:00
Perfecto 336f84c3be
fix(data-analysis): skip style-only worksheets before DuckDB loading (#3660)
* fix(data-analysis): skip style-only worksheets before DuckDB loading

* chore: retrigger CI for XLSX sheet fix

* fix(data-analysis): keep all sheets when the worksheet preflight fails

The style-only sheet preflight is best-effort: if it cannot read the
workbook parts, log a warning and load the sheets DuckDB enumerated
instead of failing the whole workbook. Context cancellation still
aborts the load.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:29 +08:00