WeKnora/website-docs
lyingbug 0c5d28abb7
feat: index and recall images with multimodal embedding models (#3683)
* feat(models): expose embedding input modalities

Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.

- Catalog: declare input modalities for the Doubao and Tongyi vision
  embeddings, the Aliyun multimodal / qwen-vl embedding families and
  jina-embeddings-v4, as listed in each vendor's documentation. The
  migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
  capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
  the catalog and writes spec.input only when the row differs from it;
  picking another catalog model or switching vendor drops the
  declaration. Model cards show the image badge for such models.

* feat(embedding): embed images with multimodal embedding models

Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.

- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
  object store is usually unreachable from the vendor) and an
  ImageEmbedder implemented by all four protocols, each in the image
  shape its reference documents: Ark image_url, DashScope contents[].image,
  Gemini inlineData parts, and on the OpenAI shape an object keyed by the
  vendor's image_field (Jina: "image"). A plain OpenAI-compatible
  endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
  max_image_bytes and image_mime_types, filled from vendor docs for the
  DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
  AsImageEmbedder, so existing Embedder implementations and fakes are
  untouched. An embedder takes images only when the catalog (or the
  row's spec.input) lists image AND the endpoint can carry one. Images
  are checked against the documented limits before any request, then
  split at the documented images per request. The concurrency, debug
  and Langfuse decorators forward it; logs and traces carry the MIME
  type and size, never the bytes.

The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.

* feat(knowledge): index images by their own vector and recall them

When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.

Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
  the image (embedding.PrepareImage converts and shrinks it to the
  vendor's documented limits) and stores it as a new image_vector chunk:
  Content is the caption (OCR text when there is none) for reranking and
  the answer context; its index row carries the image's vector with
  SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
  text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
  the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
  so a second row would overwrite the caption's text vector, and copy,
  move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
  through each engine's normal BatchIndex, keeping row order (the
  composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
  syncChunkIndex only syncs its enabled flag, updateChunkVector skips
  it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
  backends with a column default were storing them as enabled.

Retrieval
- A text query scores lower against an image than against text, so when
  the model embeds images the document vector search is widened by half
  (capped at 500) and run at min(VectorThreshold, 0.1); each store
  group's hits are then held to their own threshold before score
  normalization: text at VectorThreshold, images at min(VectorThreshold,
  0.1). Keyword hits on image rows are dropped, their Content being the
  already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
  context is attached, and the caption is not repeated from ImageInfo.

Adds golang.org/x/image for scaling and WebP/BMP decoding.

* fix(models): report no thinking levels for embedding, rerank and ASR models

Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.

* fix(compaction): never read a cancelled turn's stream as a summary

streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.

Check the context before each receive and again when the stream closes.

* feat(knowledge): put image vectors behind a knowledge-base switch, off by default

Image vector indexing and recall used to turn on whenever the knowledge
base's embedding model declared image input. A model that takes images is
often chosen for text alone, so an upgrade would have added an embedding
call per image, widened vector recall 1.5x and lowered the image threshold
to 0.1 for those knowledge bases without anyone asking.

ImageProcessingConfig gains image_vector_enabled (default false; stored
JSON without the key reads as off, so existing knowledge bases stay off
with no migration). Both sides now need the switch and the model:

- indexImageVector returns before touching the model unless the KB opted
  in (KnowledgeBase.IsImageVectorEnabled, which also needs vector indexing).
- applyImageRecall resolves the embedding model only when some KB in the
  search opted in, and records the opted-in KBs per store group
  (storeGroup.ImageKBIDs). Only those groups widen the pool and lower the
  threshold; image hits of any other KB, including ones indexed before the
  switch was turned off, are dropped.

The KB editor shows the switch in the image processing section, with a
warning when the selected embedding model does not declare image input;
docs describe the switch, its cost and what turning it off does.

* fix(knowledge): leave image vectors behind when cloning into a KB that is off

CloneChunk copied every chunk of the source document and then CopyIndices
for all of them, so copying a document from a knowledge base with image
vectors on into one with the switch off (same model and store) still
created image_vector chunks and their index rows, bypassing the switch
that indexImageVector honours.

Image vector chunks are now dropped from the copy when the target has not
opted in. Every engine's CopyIndices copies only the rows of chunks in the
mapping, so their index rows stay behind too.

* fix(search): refill the vector pool past stale image vectors

Image hits of a knowledge base whose switch is off were dropped only after
the engine had ranked and cut its results at TopK. Image vectors indexed
before the switch was turned off could fill that pool and push out valid
text hits: with MatchCount 5 the pool is 50, and 50 stale image rows
ranked above the text left the search empty.

No engine can filter rows by source type (Postgres and SQLite do not even
take ExcludeChunkIDs), so the exclusion cannot move into the query. When
the document vector pool comes back full and holds stale image rows, that
search alone runs again on a pool twice as large, until it holds TopK rows
that are not stale, the index runs out, or the pool reaches
maxRetrievalPoolSize (at most four extra calls from a pool of 50). The pool
is then cut back to TopK such rows, matching an engine-side exclusion
unless more than maxRetrievalPoolSize-TopK stale rows rank above the text.

* feat(i18n): translate the image vector switch and image input keys into zh-TW

The zh-TW locale arrived from main after this branch added five keys: the
knowledge base's image vector switch (label, description, unsupported-model
warning) and the embedding model's image input flag (label, description).

* fix(knowledge): leave image vectors behind when moving into a KB that is off

A reuse_vectors move relocates a document's chunks and index rows in place:
MoveKnowledgeIndices rebinds the rows by document ID and
MoveChunksByKnowledgeID rebinds the chunks. Moving a document from a
knowledge base with image vectors on into one with the switch off therefore
carried its image_vector chunks and their vectors along, the same bypass the
clone path had.

When the target has not opted in, the image_vector chunks are now deleted
during the move, their index rows first and then the chunks, so a retry
finds the chunks again rather than leaving orphaned rows. A target that
opted in still receives them unchanged.

* fix(retriever): keep image vectors out of keyword-only engines

BatchIndexVectors handed image rows to every engine of the composite,
including one that serves keyword retrieval alone (Elasticsearch v7 next to
a vector store). There the row holds nothing but the image's caption, which
the caption's own chunk already indexes, and retrieval discards every
keyword hit on it.

Image rows now go only to engines that serve vector retrieval. An engine
that serves both keeps one row for both kinds, so its keyword search still
meets them; the search side handles that.

* fix(search): refill the keyword pool past image rows

Every engine that serves both kinds of retrieval stores one row per index
entry and runs its keyword search over that row's content, so an image
vector row, whose content is its caption, is a keyword hit wherever the
caption's own chunk is. filterImageHits drops all of them, but only after
the engine has cut its ranking at TopK, so they crowded out text and caption
hits; a keyword ranking that put the image rows first came back empty.

The keyword pool is now refilled the way the vector pool is refilled past
stale image vectors: when it came back full and held image rows, the
keyword search runs again on a pool twice as large, up to
maxRetrievalPoolSize, and is cut back to TopK hits that are kept.

Leaving the rows out of the keyword index instead would mean writing them
without searchable content (Milvus and Tencent VectorDB derive a BM25
sparse vector from it, and how they take an empty one is unverified) or a
source-type filter in every engine's keyword query (ten engines, and
filterable fields that existing collections lack).

* revert: let image vectors move with the document into a KB that is off

This reverts commit e259977fe915f7d0bc30c4b02bc7af04da1c8dd4.

Deleting the image vector index rows by chunk ID before the move is not
reliable: the Weaviate and Elasticsearch v8 delete wrappers ignore partial
failures, and OpenSearch rejects more than 1000 IDs with ErrBatchTooLarge.
A partial delete followed by the chunk delete leaves index rows whose chunks
are gone, which nothing ever cleans up.

Without it, the image_vector chunks and their index rows move together
with the document. While the target's switch is off they are filtered at
query time (the pool refill keeps them from crowding out text), they are
removed with the document on delete, reparse or knowledge base delete, both
chunks and index rows by knowledge ID, and they become active again if the
target turns the switch on. The clone path keeps dropping them, since there
the rows are copied rather than moved.

* fix(search): keep the image row refill from losing hits it already had

refillPool replaced a pool with the answer to each larger search. A refill
that answered only in part (RetrieveResult.Error, swallowed into a warning
by retainPartialResults, as when some collections of the store did not
answer) could hold fewer text rows than the complete smaller pool, or none:
40 stale image rows above 10 text came back from a partial refill as the
40 image rows alone, and the search lost all 10 text hits. A refill that
failed outright failed the whole store group, discarding a first round that
had succeeded.

A refill is now best effort. A failed or incomplete refill search stops
the refill and leaves the last complete pool in place, cut back to TopK
kept rows as before, so a refill can no longer leave a search worse off
than it was without it. Merging the partial answer into the pool was not
done: its rows would need deduplicating and its ranking is cut at a
different TopK over a different set of collections, while keeping the
previous pool is exact.

* docs(knowledge): say what happens to image vectors in a KB that is off and what the refill costs

The image vector section now says that with the switch off, image vectors
that were indexed before it was turned off, or that came along when the
document was moved in from a KB that has it on, are kept but filtered at
query time, are removed when the document is deleted or reparsed or the
KB is deleted, and are recalled again once the switch is turned on; a copy
into such a KB leaves them behind.

It also states the cost of keeping dropped image rows from crowding text
out of the pools: the vector and keyword pools are searched again with
TopK doubled up to the 500 candidate cap, which adds queries for store
groups that hold such rows, and text can still be missed when more than
about 500 candidates are image rows. A failed or partial refill keeps the
previous complete pool.

* fix(search): log unexpected refill result sets, and describe the refill stop condition precisely
2026-10-08 18:19:53 +08:00
..
.vitepress fix(website-docs): serve everything but the homepage from /docs/ 2026-09-24 15:24:16 +08:00
01-getting-started fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891) 2026-10-08 14:02:42 +08:00
02-architecture feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
03-features feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
04-api feat: index and recall images with multimodal embedding models (#3683) 2026-10-08 18:19:53 +08:00
05-clients chore: bump version to 0.8.2 2026-09-24 14:18:55 +08:00
06-development docs: count 29 built-in model vendors and list Hugging Face TEI and Pinecone (#4048) 2026-10-08 18:03:07 +08:00
07-releases docs: date the v0.8.2 release and describe it in the READMEs 2026-09-24 14:18:55 +08:00
deploy feat(website): add product homepage and unified docs deployment (#3297) 2026-09-16 11:58:37 +08:00
homepage docs: count 29 built-in model vendors and list Hugging Face TEI and Pinecone (#4048) 2026-10-08 18:03:07 +08:00
public docs: sync website-docs, homepage and changelog for v0.8.2 (#3632) 2026-09-24 11:26:34 +08:00
sample-data feat(website-docs): add quickstart sample data and local MCP demo 2026-08-06 19:54:40 +08:00
scripts fix(docs): spawn nested npm through Node on Windows 2026-09-29 17:19:25 +08:00
shared fix(website-docs): serve everything but the homepage from /docs/ 2026-09-24 15:24:16 +08:00
.dockerignore feat(website): add product homepage and unified docs deployment (#3297) 2026-09-16 11:58:37 +08:00
.gitignore feat(website): add product homepage and unified docs deployment (#3297) 2026-09-16 11:58:37 +08:00
.nvmrc feat(website): add product homepage and unified docs deployment (#3297) 2026-09-16 11:58:37 +08:00
Dockerfile docs: sync website-docs, homepage and changelog for v0.8.2 (#3632) 2026-09-24 11:26:34 +08:00
MIGRATION.md docs: 迁移有效文档至 website-docs 并清理旧文档 (#3580) 2026-09-22 23:25:40 +08:00
README.md fix(website-docs): serve everything but the homepage from /docs/ 2026-09-24 15:24:16 +08:00
VERSION chore: bump version to 0.8.2 2026-09-24 14:18:55 +08:00
docker-entrypoint.sh docs: sync website-docs, homepage and changelog for v0.8.2 (#3632) 2026-09-24 11:26:34 +08:00
index.md feat(website): add product homepage and unified docs deployment (#3297) 2026-09-16 11:58:37 +08:00
package-lock.json chore: bump version to 0.8.2 2026-09-24 14:18:55 +08:00
package.json chore: bump version to 0.8.2 2026-09-24 14:18:55 +08:00

README.md

WeKnora 官网与文档

website-docs/ 包含官网、文档、共享样式和全部构建部署脚本,可独立复制和构建,不依赖仓库外层文件。官网和文档共用一个域名、一次构建和一份部署产物。

产品、部署、API 和开发文档统一在这里维护。旧 docs/ 的有效内容已按主题合并,过时和重复正文已删除;迁移取舍及仍保留的工程资源见迁移记录;该记录仅供维护者使用,不发布到站点。

  • /:产品首页。
  • /docs/:直接进入“快速上手”。
  • /docs/…:完整文档,保留分类导航、搜索和侧栏。

站点只占用这两个路径:根目录下只有 index.html、404.html 和 docs/。官网的脚本、样式和图片都放在 /docs/_home/ 下,所以上游网关只需转发 /(精确匹配)和 /docs/ 前缀。build 会拒绝引用这两个路径以外资源的页面。

顶栏高 64px、横向铺满。两边使用 README 原版商标,共用颜色、字体和深浅色偏好;Logo 在当前标签返回官网。官网和文档各自使用适合当前页面的导航。

发布方式:静态文件 + Nginx

1. 准备部署包

使用 Node.js 24,在仓库的 website-docs/ 目录执行。首次构建或依赖锁文件变更后,先安装依赖:

cd website-docs
npm run setup
npm run build
npm run package:site

build 会编译官网、检查文档链接和 Mermaid、编译文档,并校验合并后的资源路径。全部通过后更新 static-site/;package:site 将其打包到 releases/。

2. 上传并解压

只需把压缩包上传到服务器。服务器不需要 Node.js,也不需要 WeKnora 后端或独立文档服务。

每次发布使用一个新的空目录,避免混入旧站点文件。下面的 20260915-1 是示例发布编号,后续发布换一个编号:

sudo mkdir -p /srv/www/weknora/releases/20260915-1
sudo tar -xzf weknora-site-v0.8.2.tar.gz -C /srv/www/weknora/releases/20260915-1

解压后该目录下应直接有 index.html、404.html、docs/,无需再套一层 static-site/。

3. 配置域名根目录

以 deploy/nginx.conf 为模板添加站点配置,修改:

server_name 你的域名;
root /srv/www/weknora/releases/20260915-1;

如果域名已有 HTTPS 配置,保留原来的证书及监听设置,把模板中的 root、index 和 location 规则合入现有站点配置。域名应指向这台服务器。

必须保留文档的 .html 路由解析规则;不要把所有未知路径回退到官网 index.html。当前构建部署在域名根路径,不支持直接放进 /weknora/ 等子目录。

检查配置后重载:

sudo nginx -t
sudo nginx -s reload

4. 确认上线

访问以下路径:

  • /:官网。
  • /docs/:快速上手。
  • /docs/03-features/23-memory:长期记忆文档,直接打开和刷新均正常。
  • /docs/not-found:返回 404。

再确认搜索、深浅色切换和 Logo 返回官网正常。

后续更新重复构建、打包、上传到新目录,再修改 Nginx root 并重载。需要回滚时,将 root 改回上一发布目录。确认新版本稳定后再清理旧发布目录。

可选:Docker 部署

从干净源码即可构建,无需在宿主机安装 Node.js 或提前生成静态文件。Dockerfile 使用两个阶段:Node.js 24 按两份锁文件安装依赖并构建官网与文档,最终 Nginx 镜像只包含静态产物和服务配置。

在仓库根目录执行:

docker build -t weknora-site:0.8.2 website-docs
docker run -d --name weknora-site --restart unless-stopped -p 8080:80 weknora-site:0.8.2

访问 http://服务器地址:8080/。如使用域名和 HTTPS,让现有反向代理转发到该端口即可。镜像内已经包含官网、文档和 Nginx 路由配置。

容器内的 Nginx 默认监听 80 端口,可以用环境变量 WEBSITE_NGINX_PORT 修改,无需重新构建。使用 --network host,或者部署平台要求容器监听指定端口时,这样设置:

docker run -d --name weknora-site --restart unless-stopped -e WEBSITE_NGINX_PORT=8080 -p 8080:8080 weknora-site:0.8.2

只是想换一个对外端口时,改 -p 左侧的宿主机端口就够了,例如 -p 9000:80。

也可以只复制 website-docs/ 目录,在该目录执行 docker build -t weknora-site:0.8.2 .。构建上下文必须是 website-docs/,宿主机的依赖、旧构建产物和部署包由 .dockerignore 排除。

从旧文档镜像迁移时,将容器端口映射或反向代理目标端口从 8081 改为 80(宿主机端口可自行选择,例如 -p 8081:80)。域名根路径 / 现在提供官网,/docs/ 提供文档,反向代理需覆盖整个站点并保留请求路径。容器使用 Nginx 官方镜像的默认入口,无需额外的 docker-entrypoint.sh。

构建后可在安装了 Node.js 24 和 Docker 的机器上执行以下检查,无需安装 npm 依赖。检查会临时启动容器并自动清理,验证两站路由、静态资源、404、压缩和响应头:

cd website-docs
npm run test:docker -- weknora-site:0.8.2

本地开发

使用 Node.js 24 LTS(目录内提供 .nvmrc,使用 nvm 时可执行 nvm use)。首次获取源码后安装依赖(分别按文档和官网的锁文件安装):

cd website-docs
npm run setup
npm run build
npm run preview

预览地址:http://127.0.0.1:3000/。修改源码后重新构建并刷新;可用 npm run preview -- --port 8080 指定其他端口。

npm run check
npm run test:integration

集成检查需要安装 Google Chrome,验证主题同步、两边导航、搜索、手机端控件及直达文档路由。可用 SITE_TEST_URL 指定待验证的站点。

开发时可分别使用 npm run dev:homepage(官网)和 npm run dev:docs(文档)获得热更新;完整站内跳转请使用统一构建后的 npm run preview。

npm run build:docs 只编译文档,npm run build:homepage 只编译官网;正式发布使用 npm run build,输出会同时包含两者。VERSION 是这份站点构建使用的发布版本。

工程目录(相对于 website-docs)

  • homepage/:Next.js 官网源码、品牌素材及独立依赖锁文件。
  • 01-getting-started/ 至 06-development/:文档正文,原有路径不变。
  • .vitepress/:文档站配置与主题。
  • public/、sample-data/:文档截图和示例。
  • shared/:品牌变量、顶栏样式及共用图标。
  • scripts/:统一构建、检查、预览和打包。
  • static-site/:唯一的部署目录,由构建生成。
  • deploy/、Dockerfile:Nginx 和 Docker 部署配置。
  • releases/:生成的部署压缩包。

品牌素材来源见 homepage/BRAND-ASSETS.md。产品视频使用 README 中的 GitHub 原始附件,播放需要访问 GitHub;页面、文档和截图随包部署。