web_fetch stopped at 60s inside the page fetch. The tool wrapper, storePage,
and the HTTP and chromedp fetcher each had their own cap, so a page that
needed about 150s never finished.
WEKNORA_WEB_FETCH_TIMEOUT is the single page budget (default 3m). HTTP and
chromedp share it, and the agent wrapper waits 5 seconds longer so the tool
can return its own result. Other tools and the 15s pipeline fetcher keep
their current timeouts.
Fixes#2298
Co-authored-by: Frank-zhu2000 <332350688+Frank-zhu2000@users.noreply.github.com>
An embedding row with spec.input image and no image_format or image_field
was treated as text-only on every vendor except generic and Jina. Default
that case to the vLLM messages format. An explicit object format without a
field stays refused.
Fixes#4343
* ci(build): add manual docker build workflow
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects
- folder mirror: wiki/drive connectors emit path-qualified FileName so
knowledge lands in KB folders mirroring the Feishu tree; shortcut
subtrees deduped; sanitized directory names; legacy datasources
converge via one-shot resync marker (incremental sync upgrades to
full once, then clears)
- parse mode: replace FEISHU_DOCX_PARSE_MODE env with per-datasource
settings.parse_mode (blocks default, export escape hatch); frontend
selector + 5 locales; zero-migration read-time fallback
- block-level GFM: extend docx block coverage (callout/grid/iframe/
board/mindnote/agenda/synced blocks), rich text elements (links,
inline code, KaTeX equations, mentions), real ordered-list numbering,
code language mapping, merged-cell fill, truncation note
- embedded objects: images (gated off multimodal toggle) and boards
(download_as_image, 403-degraded) stored as knowledge with
weknora-img://<n> markers resolved in the document-processing
pipeline via knowledge FilePath; document attachments whitelisted
(pdf/md/xls(x)/csv/doc(x)/ppt(x)) as standalone knowledge in the
parent folder with bidirectional parent/child metadata and filename
list in the body; videos skipped with placeholder + source link
- golden baseline explicitly re-recorded; docs (.env.example,
docker-compose, docs/*, website-docs) synced
* fix(datasource): address Feishu deep-adaptation smoke-test findings
- folder mapping: drop the space/root-folder name prefix — the selected
sync root maps to the knowledge-base root, subdirectories stay relative
- docx blocks: render view(33) blocks silently as containers (children
only), removing the stray '[飞书块: 视图]' placeholder before file lists
- sheet inline tables: fetch merge ranges via sheets/v3 sheets/query and
fill the top-left value across the merged region (closed-interval
interpretation, flagged for real-environment verification)
- knowledge list: hide embedded image/board rows (metadata
embedded_image=true) from the document list and folder-tree counts;
rows remain for VLM indexing and marker resolution; search untouched
- frontend: hydrate protected images (resource:// etc.) in the document
preview so embedded Feishu images/boards actually render
* fix(datasource): resolve embedded image markers at ingest time
The weknora-img://<n> markers were only resolved during document
processing, so chunks carried real resource:// URLs while the stored
markdown file and source_content kept raw markers — file preview
showed blank images. Resolve them in ingestItem before
CreateKnowledgeFromFile (children are emitted first, so image rows
with FilePath already exist); markers with missing children are kept
for the ProcessDocument fallback to degrade. Stored file,
source_content and chunks now agree. Existing datasources converge on
the next full re-sync (update=delete+recreate).
* feat(datasource): resolve Feishu mention users to real names at ingest
docx mention_user elements only carry an open_id, so they rendered as
a generic '@成员'. Resolve the display name via
GET /contact/v3/users/:id (user_id_type=open_id) while generating the
markdown during FetchDocxWithBlocks — before storage, per user
decision; the frontend performs no lookups. A per-sync closure caches
positive and negative lookups; permission errors and empty names
silently degrade to '@成员' (debug log only, ingest never blocked).
Requires one of the contact read scopes (e.g.
contact:user.base:readonly) on the Feishu app.
* refactor(datasource): polish round — defect fixes and structural cleanup
- attachment placeholder patching now targets the unique
'- weknora-att://<token>' marker line rendered per attachment instead of
matching by display name, so an identical earlier bullet is never
replaced by mistake (regression-tested)
- markdown link targets with spaces or parens are percent-escaped
(iframe / mention_doc / inline links), keeping them parseable
- drive folder-tree walking consolidated into core.WalkDriveTree,
deleting the 174-line copy in drive/paths.go
- parse mode threading made explicit: DocxFetchInput.ParseMode replaces
the private client field (connector entry resolves via
ParseFeishuConfig and passes it through ops)
- maxFeishuDownloadBytes restored to a const; the oversize-degrade test
now declares an over-cap Content-Length and downloadRawBytes rejects
early on the header
* refactor(datasource): align Feishu image handling with the platform image pipeline
Replace the image-as-separate-knowledge-row channel with the platform's
orthodox flow: the connector downloads image/board bytes at fetch time and
embeds them in the parent markdown as base64 data URIs; the document
processing image resolver (image_resolver.ResolveAndStore, the same stage
used for parsed PDFs/Word) stores them on the tenant file service, swaps in
persistent provider:// URLs and produces OCR/caption child chunks when a VLM
is configured.
Removed with the marker pipeline:
- weknora-img:// ingest resolution in ingestItem
- the ProcessDocument Step-1.9 marker hook
- the document-list/folder-tree filter for embedded_image rows
- the feishu-specific runtime parse_mode backfill; legacy data sources are
now stamped by migration 000096 (sqlite 000017) which writes the one-shot
resync_required marker — the generic full-upgrade + clear consumption stays
Attachment (PDF/Word/md) sub-items are unchanged.
* style: gofmt test files
* fix(datasource): resolve Drive sub-selection kind from the root listing
A single-file Drive selection (rootFolderToken:fileToken) was probed with
folder APIs before falling back: folder meta returned 91202 and the folder
list 1061002 on every sync, logging API errors for a working configuration.
listDriveFilesForResource now walks the root subtree first and reads the
selected node's kind off the listing: a file filters straight out of the
already-fetched results (zero error calls), a sub-folder still gets its
dedicated walk, and only a token missing from the listing (partial listing /
permission edge) keeps the legacy probe + fallback.
Also pins sub-folder selection semantics with a dedicated test.
* fix(datasource): address review findings across the Feishu deep-adaptation branch
Four parallel reviewer slices audited the branch; this applies the accepted
findings.
Content fidelity (blocks rendering):
- todo blocks parse style.done — completed items render "- [x]" instead of
always unchecked
- code fences size past the longest backtick run in the content, so a line
of ``` inside code can no longer close the fence early
- ordered-list "auto" numbering only resets on a sibling of the last item;
child blocks rendered flat between items no longer renumber the run
Image pipeline:
- inline base64 images are capped at the document-processing resolver's
10 MB budget; larger images degrade to a placeholder instead of leaving a
permanent raw-base64 blob in stored markdown
- image markers carry a per-document nonce so user-typed look-alike text
can never be patched in place of a real marker
- attachment marker patching anchors on the token's line end, closing the
token-prefix collision
- SupportedImageExt recovers image/svg+xml (board exports), which the
platform resolver already stores natively
Sync correctness:
- clearResyncMarker re-reads the row before writing: a long upgraded sync
no longer risks reverting concurrent user edits with a stale full-row
update
- wiki EditTime keys on max(obj_edit_time, node_edit_time): pure
moves/renames re-file the document on the next incremental sync
- file-typed wiki/drive base names are sanitised; a "/" in a title can no
longer fake an extra folder level
Docs/tests/comments:
- datasources docs rewritten to the v3 image contract (no image rows,
no image_map, no parse_mode backfill claim)
- DataSourceEditorDialog drops settings.parse_mode when backtracking to a
non-Feishu connector type
- stale comments corrected (types.go parse-mode transport, repository
folder-counts, wiki List partial-error branch, test headers); regression
tests added for todo done, fence sizing, ordered numbering, svg sniff
* fix(datasource): repair markdown conversion defects found in live smoke test
Live re-sync of the all-block-type Feishu test document surfaced three
converter defects:
- Code blocks: the closing fence reused the opening fence string, so a
tagged block ended with ```java — CommonMark requires a bare closing
fence, and every renderer left the block unterminated. The closing
fence is now backticks only.
- Feishu heading7-9 emitted 7-9 hash marks; GFM stops at H6, so those
lines rendered as literal '#######' text. They now degrade to bold.
- The docx API percent-encodes link targets (http%3A%2F%2F...); escapeURL
now reverses that encoding before markdown-escaping, so stored links
point at real URLs instead of percent-encoded strings.
Verified against the live document via a throwaway API probe: the
'i|g|k' table cells questioned during review are the document's own
content (the sheets copy says i|j|k; the two docx tables genuinely
contain 'g'), not a converter bug.
* fix(frontend): render GFM task lists in document preview
Task list items (- [x] / - [ ]) rendered as plain bullets: marked emits
<input type="checkbox"> for them, but the preview DOMPurify config's
ALLOWED_TAGS allowlist has no 'input', so the element was stripped whole.
The preview renderer now overrides checkbox to emit a styled
task-checkbox span (class/attr both allowlisted) and the preview styles
draw the box with a check mark, hiding the list bullet via :has().
Same gap exists in the chat markdown surface; left untouched there.
* fix(datasource): crop blank canvas borders from whiteboard exports
The board download_as_image API renders the WHOLE board canvas, not the
content bounding box — editors routinely leave the canvas far larger than
the drawing, so board/mindnote/diagram exports surfaced with large blank
areas below the content. The API has no crop parameter, so the connector
now trims near-uniform blank borders after download (bottom-right pixel
seeds the background color, 14/255 per-channel tolerance absorbs JPEG
noise, 8px margin kept). PNG and JPEG are re-encoded; anything
undecodable, fully blank, or already tight passes through unchanged, and
oversized images (>64MP) skip the scan to bound memory.
* fix(datasource): require corner majority before trimming board exports
The blank-border trim seeded the background color from the bottom-right
pixel alone. A large solid shape reaching that corner would seed the
background with its own fill color, and fully-filled rows could then be
trimmed as blank — cutting real content. The background color is now the
majority of the four corner pixels (>=3 must agree); on a split vote
content reaches the corners and the image passes through untrimmed.
Content visible against the background is untouched either way: a row or
column survives if even one pixel differs from the background by more
than the 14/255 JPEG-noise tolerance.
* fix(datasource): widen board trim margin and lower JPEG quality to 80
The 8px margin read as a tight crop — a 32px white frame now visually
bounds the content region. JPEG re-encode quality drops 90 -> 80 to keep
the re-encoded size closer to the original export.
* feat(datasource): timeline widget Markdown degradation + table column alignment
Two content-fidelity gaps from the live smoke test:
- 文档小组件 (add_ons, block_type 40) carry their full data inline in
record; there is no server-side render to download. The timeline
component (component_type_id blk_6358a421bca0001c...) now degrades to
a Markdown list (- **time · title**:text), honoring contentShow and
omitting empty segments; unknown component types and malformed records
keep the previous placeholder note.
- docx text blocks carry style.align (1 left / 2 center / 3 right,
absent unless the author set it). Native-table rendering now collects
per-column alignment from the first explicit cell and emits GFM colon
separators (:---: / ---:). Embedded sheets/bitable keep the plain
separator — their value APIs expose no alignment.
* feat(datasource): show document titles for 云文档 mention links
The docx renderer emitted mention_doc references as [url](url) because
the payload carries no title. Feishu doc URLs now emit a
[weknora-docname://token](url) placeholder and FetchDocxWithBlocks
resolves the titles in one drive metadata batch call
(POST /drive/v1/metas/batch_query, docx/doc/sheet/bitable/wiki/...),
patching the link text to [文档名](url). Refs whose metadata cannot be
fetched — no permission (970003), deleted/mismatched (970005), API
absent — degrade to the previous URL-as-text form with a warn log.
This is display-only: referenced docs are not pulled as sub-documents
(deferred design lives in the task PRD).
* docs(datasource): document mention rendering enhancement and drive rename limitation
- Add optional mention scopes (contact:user.base:readonly,
drive:drive.metadata:readonly, board:whiteboard:node:read) with
degrade behavior to the Feishu permission tables
- Describe mention_user real-name lookup, mention_doc title backfill
and timeline widget placeholders in the blocks rendering flow
- Document the Drive known limitation: folder renames / file moves do
not bump modified_time, so incremental sync keeps the old KB folder
structure until a full sync; Wiki sources are unaffected
- Clarify the directory-mapping wording in website-docs to distinguish
Wiki (rename-aware) from Drive (rename-blind) sources
* docs(datasource): add blocks vs export tradeoff summary to website-docs
* docs(datasource): document docx block-type coverage for blocks mode
- Add a full block-type support table (block_type 1-52 grouped) to the
datasource dev doc: text/heading/list/task/quote structures, code
fences with language aliases, containers (callout/quote/grid/view),
native tables with merged-cell splitting (top-left value fill) and
column alignment, embedded sheet/bitable converted to in-document
Markdown tables, image/board inline pipeline, attachment whitelist,
iframe links, timeline add-on degradation and the placeholder set
- Summarize the same coverage in the drive user guide, the wiki
quick-start doc and website-docs; fix the stale claim that board
blocks degrade to placeholders (they ride the image pipeline now)
* style: resolve diff-scoped lint findings across feishu packages
- Wrap over-length signatures and calls (lll, max 120)
- Add doc comments to exported Block* payload types (revive)
- Rename unused closure parameters to _ (revive)
- Check w.Write / resp.Body.Close error returns (errcheck)
- Apply De Morgan rewrite in docname token validation (staticcheck)
- gofumpt-format touched files
golangci-lint run --new-from-rev=main ./... now reports 0 issues.
* ci: drop fork-only docker-build workflow
The manual docker-build workflow lives on the fork's main and is not
part of the upstream contribution; it entered the branch through the
fork-main merge.
* fix(migrations): renumber feishu resync migrations to avoid version collision with upstream
Upstream added 000017_dingtalk_stream_only (sqlite) and 000096_dingtalk_stream_only
(versioned) before our merge, colliding with our 000017/000096 feishu resync
migrations. golang-migrate rejects duplicate versions. Renumber to the next free
slots (sqlite 000027, versioned 000108) and update comments + test constant.
* fix(migrations): renumber feishu resync migrations ahead of upstream 000027/000108
Upstream main now also claims sqlite 000027 and versioned 000108
(session_sandbox_config_tenant), colliding with our just-renumbered
resync migrations. Move to 000028 / 000109.
* fix(feishu): address review — deprecated env fallback and upgrade notes
- Keep FEISHU_DOCX_PARSE_MODE as a one-release deprecated fallback when a
data source has no per-source parse_mode (deprecation warning logged);
annotate it in .env.example / docker-compose.yml instead of removing.
- Point stale migration-number comments at the final numbers
(versioned 000110 / sqlite 000030) in datasource_service.go and tests.
- website-docs: add Feishu upgrade notes (env deprecation, default mode
flip to blocks, one-shot full resync with embedding/VLM recompute);
refresh 24-feishu-drive.md parse-mode section to the per-source
setting and new blocks semantics.
* docs(feishu): document necessary and optional connector permissions
Restore the permission matrix lost with the docs/ removal: necessary
scopes (wiki/drive/export/docx) with missing-permission symptoms and
optional enhancement scopes (embedded sheets/bitable, board export,
mention user/doc rendering) with their degrade behavior, in
10-datasource.md's Feishu section; make 24-feishu-drive.md list the
three necessary drive scopes and link to the full table.
* Revert "docs(feishu): document necessary and optional connector permissions"
This reverts commit 3be19cb1b656b3596f2cb6afb2cda8c2a1d12be1.
* docs(feishu): permission matrix by connector, parse mode and necessity
Necessary-scope tables split by connector (feishu/lark wiki vs
feishu_drive/lark_drive) with blocks/export applicability columns and
missing-permission symptoms; optional enhancement scopes with their
degrade behavior; scope names cross-checked against the Feishu open
platform API docs (files/folder list, medias download, export tasks,
metas batch_query). Full matrix lives in 10-datasource.md; the drive
guide mirrors its own table and links back.
* test(feishu): adapt mention_user_test to upstream retryTestServer signature
* feat(feishu): mirror the sync root (wiki space / drive folder) as a top-level KB folder
So a knowledge base linking several feishu sources can tell them apart at a
glance: syncing folder 测试1 yields 测试1/... and syncing wiki space
飞书知识库同步测试 yields 飞书知识库同步测试/.... Root-name resolution
failure falls back to the previous root-less behaviour without failing the
sync; existing sources converge through the one-shot resync marker.
* refactor(feishu): cache wiki space folder prefix per run, refresh stale comments
Review follow-ups for d012e593f: cache the sanitized space prefix per spaceID
for the whole sync run (K selected spaces now cost one ListWikiSpaces paging
pass instead of K), fix the misleading warn wording, and realign the
WalkDriveTree / driveFolderName doc comments with root-mirror semantics.
* docs(feishu): drop stale and duplicated comments from the deep-adaptation
- board.go: trimBoardImage no longer claims a bottom-right seed (the code
votes the background over the four corners)
- shared.go: rewrite the FetchDocxWithBlocks note (images/boards inline via
base64 now; removes a stale-marker splice artefact)
- markdown.go: drop an internal task-number leftover
- datasource_service.go: collapse the ProcessSync/processSyncStreaming notes
duplicated verbatim from resyncRequired's doc comment, keep the call-site
why (marker cleared only on success)
- datasource_resync_test.go: drop the obsolete 'backfill' wording
* feat(i18n): 添加解析模式及其提示信息到繁體中文語言包
* fix(frontend): use radius token for task checkbox; fix simplified chars in zh-TW parse mode
CI guard fixes: the PR-added task-checkbox style pushed the radius-literal
count to 161 (> baseline 160); the new zh-TW keys missed the locale parity
audit (user's dc6fdcff9) — also normalise the simplified 文档 leftovers.
* fix(feishu): preserve sync scope, nested content, and legacy settings
Cover shortcut selection, ancestor path changes, nested containers and table boards with regression tests. Avoid redundant Drive traversal, repair resync migrations for legacy settings, and preserve inherited parse modes. Bound board image dimensions before decoding.
* docs(feishu): clarify shortcut scope and incremental path tracking
* Fix Feishu sync reconciliation and embedded media regressions
* Bound Feishu table rendering by actual cell data
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* feat(assistant): let the assistant manage knowledge bases
Create, update, and delete knowledge bases, tags, and manual Markdown,
and show those confirmations as labeled rows with model names.
* fix(assistant): satisfy lint on knowledge base operations
Check new error returns, wrap lines past 120 characters, and document the exported tag request types.
* fix(assistant): enforce knowledge base settings permissions and preserve patch fields
* feat(assistant): let the assistant manage agents
Expose custom agents to the operation catalog (list, get, create, copy,
update, delete) plus read-only model and MCP service lookups, so the
assistant can set up agents end to end.
- Add PATCH /api/v1/agents/:id. PUT replaces the whole config, so a
one-field change from the assistant would wipe the rest; PATCH only
overlays the fields it carries and reuses the PUT validation path.
- Add x-agent-detail so long fields such as the system prompt are only
projected for single-object reads, not lists.
- Add label_param so a create without a target is labelled by its name.
- The approval card lists agent setting changes and resolves model,
knowledge base and MCP IDs to names.
- Deny sharing, publishing channels, credentials and OAuth routes; model
and MCP endpoints, headers and secrets are never projected.
- Update the assistant prompt, starters, API docs and Go SDK.
* feat(assistant): list skills and configure agent skill scope
- Expose usable skill names for an agent sandbox without exposing skill
administration or connection settings to the model.
- Validate PATCH selections against the workspace sandbox usable set so
invented names and missing sandbox selections cannot be saved.
- Show skill changes in approvals and align the assistant prompt, API
documentation, and Go SDK with focused refusal and projection tests.
* feat(assistant): isolate conversation history
- Keep assistant conversations out of ordinary session lists, search, artifacts and delete-all cleanup.
- Add private, paginated assistant history with session switching, confirmed deletion and stale-session recovery.
- Backfill session kinds safely on Postgres and SQLite, preserve kinds on forks, and align API docs and SDK coverage.
* feat(assistant): refresh pages after successful data changes
- Carry decoded mutation targets through live tool-result events without
exposing refresh metadata to the model or persisted chat history.
- Coalesce page and selector refreshes, update document drawers, and warn
about changed agents while preserving open editor drafts.
- Cover mutation, approval, stream, lifecycle and cache behavior with Go
and Node tests using the existing frontend runner and component fixtures.
* feat(assistant): add agent page context and resource scope limits
- Attach verified agent context and persist personal scope across turns and approvals.
- Enforce resource boundaries before operation and Wiki approvals, including edited arguments.
- Add scope controls, translations, API and SDK support, and regression coverage.
* refactor(assistant): remove the scope limit
The scope picker confused users and duplicated what RBAC and the access
level already enforce. Keep the agent page context: the agent editor
still publishes agent_id and the backend still verifies it.
* feat(assistant): render operation and product-doc results
Show find/run operations and WeKnora docs as structured cards instead of raw JSON.
* feat(assistant): default new agents to smart-reasoning
Quick-answer cannot call MCP, and agent.update could not change the mode. New agents now start in smart-reasoning, and a patch can switch agent_mode.
* fix: resolve assistant management CI failures
* feat(agent): let the WeKnora assistant read the bundled documentation
Embeds the website-docs pages (01-07) into the server and adds
search_weknora_docs / read_weknora_doc for the built-in assistant, so it
answers how WeKnora works from the docs of the running version.
- pages split into sections with the site's VitePress anchors; Mermaid
dropped, screenshots reduced to captions, links rewritten to public URLs
- in-memory BM25 over sections, with Han character pairs and whole
identifiers such as OLLAMA_BASE_URL or v0.8.3
- granted to the built-in assistant by ID, like the operation tools; the
docs are public, so no signed-in user is required
- tests check every embedded page parses and every internal link resolves
* fix(agent): read long doc sections to the end and keep docs on a broken page
- read_weknora_doc takes an offset: a section too long for one read is cut
at a line break and returns next_offset, so leaf sections such as the
model field reference are readable to the end instead of losing the tail.
- Reads follow the registry's output budget (capped at 8000 runes).
- Percent-encoded anchors, as a browser copies them, and http:// doc URLs
resolve.
- productdocs.Load leaves out a page it cannot parse and reports it, so one
broken page no longer withholds the documentation tools.
- App CI runs on changes to the embedded doc pages and the sidebar config,
where the parse and link tests now apply.
* docs(swagger): add shared response envelope types
* docs(swagger): type knowledge base responses
* docs(swagger): add concrete response body types
* docs(swagger): type chunk list responses
* docs(swagger): type knowledge mutation responses
* docs(swagger): type wiki search and autofix responses
* docs(swagger): type FAQ responses
* docs(swagger): type tag responses
* docs(swagger): type message responses and SSE schemas
* docs(swagger): type agent responses
* docs(swagger): type remaining handler responses
* docs(swagger): regenerate OpenAPI spec
* docs(swagger): type knowledge, activity and wiki maintenance endpoints
Covers the endpoints left out of the earlier passes: knowledge base
detail, document list/detail/spans/reparse/cancel/delete, knowledge
base activity and the Wiki stats, issues, lint and rebuild-links
endpoints. The spans handler now writes the documented structs instead
of gin.H, so the schema cannot drift from the body; the JSON is
unchanged. List filters gain their real enums and bounds, including
the finalizing, cancelled and deleting parse states.
* docs(swagger): document JSON-valued fields as JSON, not byte arrays
types.JSON and json.RawMessage are byte slices, so swag rendered every
metadata, details and config field as an array of integers. A swag
overrides file documents them as objects; the four fields that hold
arrays (relation chunk ids, allowed origins, image refs) say so with a
swaggertype tag.
* chore(lint): exempt swag annotation lines from the line-length limit
swag reads each @Param, @Success and @Failure as a single line, so an
enum list or a composed response type cannot be wrapped to fit.
* chore(lint): exempt struct tags carrying swag attributes from the line limit
Struct tags cannot wrap either; a field documented with swaggertype or
extensions keeps its tags on the declaration line.
The agent completion event carries the turn's aggregated token usage
but not the chat model that produced it. A relay or gateway that meters
agent chat cannot price that usage: unit prices, and the discounts for
cache reads, cache writes and uncached input, differ per model, and
changing an agent's model blends the old and the new numbers in one
total.
The completion event's data now carries model_id, taken from the
engine's chat model. The turn's usage is aggregated only from calls
made through that model (each round and the final answer), so it names
the model the usage belongs to. A consumer resolves the model's name
and settings through the model API with its own credentials. The
message row's model_id keeps the requested binding and can differ when
a request override is invalid and the turn falls back to the agent's
model; the event names the model that actually ran. The key is omitted
when no model is known. The quick-answer path is not changed.
The model name is left out on purpose. The completion event reaches
everyone who chats with the agent, including shared-agent users and
embed visitors, and an operator may not want to reveal which model
serves an agent; the provider-facing string can also be a deployment
override that the model API hides from non-admin callers. The message
history already returns the same ID to these users.
mermaid >= 11.17 dynamically imports the UMD package fastdom. Vite's
dependency scanner misses that import during vitepress dev, so the
browser loads raw fastdom.js and throws "does not provide an export
named 'default'". Pre-bundle fastdom only when this install can resolve
it. The lockfile pins mermaid 11.16.0, which does not depend on
fastdom, and an unconditional optimizeDeps.include makes Vite report
"Failed to resolve dependency".
Fixes#2899
Co-authored-by: Frank-zhu2000 <332350688+Frank-zhu2000@users.noreply.github.com>
WeKnora runs behind other applications: they sign their users in and
call WeKnora with a tenant API key and X-External-User-ID, and WeKnora
keeps sessions and MCP OAuth tokens per external user. When an agent
uses an OAuth-protected MCP service, the only way to get that user's
token is the browser authorization code flow, and in this deployment
the user never sees WeKnora, so there is no browser to complete it.
Independently, an OAuth MCP service can only get its client through
dynamic client registration (RFC 7591), so authorization servers
without a registration endpoint cannot be used at all ("server does not
support dynamic client registration"), and servers that allow a grant
only to clients an administrator registered refuse a dynamic one.
JWT bearer grant. The application in front already knows who the user
is. POST /api/v1/mcp-services/:id/oauth/assertion lets it present a
signed assertion of that identity, such as the Identity Assertion JWT
Authorization Grant (ID-JAG) used by MCP's Enterprise-Managed
Authorization extension. WeKnora sends it to the service's
authorization server with the JWT bearer grant (RFC 7523) and stores
the token exactly like the callback does, so status, revoke, runtime
connections and, when the response carries a refresh token, refresh
use it unchanged. The assertion is forwarded unread; validating it is
up to the authorization server. The request reuses the principal from
authorize-url, the SSRF-safe HTTP client and the callback's connection
recycling, and the client the service already has. It does not run
dynamic client registration, which describes a redirect-based client;
without a client it answers 400. The token request outlives the caller,
so a token issued for an assertion that can only be used once is not
lost. A 4xx response other than 408 and 429, or a 2xx response carrying
an OAuth error, answers 400 with the OAuth error code and is logged; a
5xx, 408 or 429 response or a network failure answers 500. Request
logging masks the assertion field.
Pre-registered client. auth_config.client_id names a client registered
with the authorization server in advance, and client_secret is its
secret for a confidential client. When client_id is set it takes the
place of dynamic registration everywhere a client is read: the
authorization URL, the code exchange, the assertion grant and runtime
connections with their token refreshes. The values live on the service
rather than in mcp_oauth_clients, because a refresh rejected with
invalid_client deletes that row so the next authorization registers
again; an administrator's client would silently disappear. client_id is
non-secret: it is set through the main PUT, where an empty value clears
it, and returned in auth_config. client_secret follows api_key and
token: encrypted at rest, never returned, accepted in the create body,
and changed afterwards through the credentials subresource. Changing
client_id rebuilds the service's connections, and a connection built
with a pre-registered client is not treated as holding a replaced
registration when an authorization recycles connections. The MCP
service dialog adds optional Client ID and Client Secret fields to the
OAuth section, in all six locales.
Nothing changes for existing services: with client_id empty every
existing path reads the stored registration and registers dynamically
as before, and the authorization code flow, its state, the callback and
the dynamic registration request are untouched. The authorization
server HTTP client is moved into a function that both paths call.
* fix(deploy): bump paradedb image to v0.25.11-pg17
Batch-importing large knowledge bases on the pinned
paradedb/paradedb:v0.22.6-pg17 image drives PostgreSQL into a
PANIC/crash-recovery loop (corrupted page pointers in shared buffers,
amcheck clean, no OOM), taking every ingest task down with it.
Three independent fixes upstream cover this exact write path, and the
pinned image predates all of them:
- pgvector 0.8.2 fixed a buffer overflow in parallel HNSW index
builds, 0.8.3 fixed possible index corruption during HNSW vacuuming,
and 0.8.4 fixed inserts during HNSW vacuuming; the image ships
pgvector 0.8.1, and the embeddings table maintains three HNSW
halfvec indexes on import.
- pg_search gained WAL-based crash recovery in 0.24.0 (paradedb#4901):
without it, any backend crash corrupts BM25 segment metadata and
turns a single crash into a recovery loop (paradedb#4935).
- PostgreSQL core moved 17.9 -> 17.11; its release notes contain no
fix matching this failure signature, ruling the core out.
Bump the compose image tags to v0.25.11-pg17 (postgres and
langfuse-db-init, prod and dev) and update the docs that pin the old
tag. Fresh installs are covered by the existing schema SQL (verified
against 0.25.11: extensions, BM25 index and HNSW halfvec indexes all
create cleanly). Existing data volumes keep working across the image
swap without any action (keyword and vector queries byte-identical in
testing); ALTER EXTENSION pg_search/vector UPDATE aligns the SQL
catalog afterwards, and migration 000099 is a no-op for 0.25.x by
design. The Helm chart pins a different image line and is left alone.
Fixes#4077
* fix(migrations): upgrade pg_search catalog to 0.25.11 with the image
Swapping in the v0.25.11 image leaves the extension catalog at 0.22.x.
Calling extension functions such as paradedb.version_info() against the
mismatched catalog segfaults the backend and sends the whole server into
crash recovery, so the catalog upgrade is required rather than optional.
- Add migration 000115 to update pg_search 0.22.0-0.25.10 to 0.25.11 and
pgvector 0.8.0-0.8.3 to 0.8.4 when the server ships those versions.
- Skip 000099 on newer images instead of advising UPDATE TO '0.22.6',
which leaves the catalog behind the shared library and crashes.
- Rewrite the upgrade guide as one 0.22.x -> 0.25.11 path with a required
ALTER EXTENSION UPDATE and a note that rollback needs a backup.
- Extend scripts/test_paradedb_upgrade.py to verify 0.22.2 -> 0.25.11 by
default (--to 0.22.6 keeps the old path) and run CONCURRENTLY
migrations outside a transaction like golang-migrate does.
---------
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
These three models were missing from the catalog, so they resolved to the
anthropic defaults: budget thinking (they reject budget_tokens) and
temperature/top_p (they reject non-default sampling values).
seed.json pins their metadata from the models overview and pricing pages:
1M context, 128K output, and base, 5m cache write, cache hit and output
prices. Haiku 5.5 is priced by prompt length, and the catalog has one price
per model, so it carries the prompts-up-to-100k tier. overrides.json gives
all three adaptive thinking with effort (all five levels), with temperature
and top_p off.
Thinking off differs per model. Opus 5.5 can't turn thinking off, so off is
unsupported, like Fable 5.1. Haiku 5.5 accepts "disabled". Sonnet 5.5
rejects "disabled" and turns off up-front thinking with "between_tools". The
new thinking_off_type compat field carries that value. Empty keeps
"disabled", and it is omitempty so the parity hashes of other entries don't
change.
Claude-Session: https://claude.ai/code/session_01BeNHkjF4spsH9d18UiYPiU
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Distinguish vector and keyword result pools, initial and refill phases, and diagnostic decisions from actual retrieval and model calls. Clarify that aggregate counts cover all candidates while detailed samples focus on image vectors, and document the new trace names.
Batch independent Qwen3-VL embedding conversations through an explicit compatibility setting. Preserve singleton requests and serial overrides, validate response ordering, and add wire tests and opt-in live numerical parity checks.
Live verification confirms valid batched responses but does not meet strict numerical parity bounds; document the deployment-side reproducibility limitation.
* feat(mcp): support modern and legacy protocol connections
* fix(mcp): preserve annotated headers on modern tool calls
* fix(mcp): load parameter header schemas before tool calls
* fix(mcp): keep a working legacy session when discovery retry fails
Modern method-level 404s stay connected, while an expired SSE session still disconnects. A transient discover failure retries once, and a failed retry keeps the handshake that already succeeded.
* fix(mcp): wrap protocol negotiation lines for the linter
golangci-lint rejects lines longer than 120 characters.
* fix(mcp): retry protocol discovery without dropping the live session
A second initialize on the same transport clears a stateful legacy session when discover returns 404. Probe on a separate connection, and drop cached parameter headers after another client refreshes the directory.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(models): align configuration defaults and multimodal workflows
* feat(image): enhance image retrieval diagnostics and filtering
- Implemented `retrieveObservedImagePool` to observe engine calls and record image recall decisions without altering results.
- Added detailed diagnostics in `summarizeImagePool` and `summarizeImageCandidateCut` to track image filtering decisions and scores.
- Introduced tests for image pool diagnostics to ensure accurate filtering and decision recording.
- Updated existing functions to utilize new retrieval and summarization methods for improved observability and accuracy in image handling.
* feat(embeddings): enhance image handling with new formats and message support
- Introduced `EmbeddingImageFormat` to manage image input formats, allowing for both object and messages formats.
- Updated `EmbeddingsSettings` to include new fields for `ImageFormat`, `ImagePrompt`, and `TextAsMessages` to support multimodal embedding workflows.
- Enhanced `Client` methods to build request bodies for images in both formats, ensuring compatibility with vLLM's requirements.
- Added tests to validate the new image message handling and ensure correct request formatting for different scenarios.
- Updated documentation to reflect changes in image input handling and configuration options.
* feat(similarity): add embedding similarity mode for enhanced document scoring
- Introduced a new `similarity` option in `ModelDebugOptions` to score documents and an optional image against a query.
- Updated `ModelDebugDrawer.vue` to support a new input mode for similarity, allowing users to select similarity as an input type.
- Enhanced the UI to display similarity results, including ranked candidates based on cosine similarity.
- Added tests to validate the functionality of the new similarity mode in embedding models.
- Updated localization files to include new strings related to similarity mode in multiple languages.
* feat(embeddings): add Qwen3-VL-Embedding support for multimodal input
- Introduced Qwen3-VL-Embedding model with support for both text and image inputs.
- Updated model configuration files to include compatibility settings for the new embedding type.
- Enhanced tests to validate the functionality of Qwen3-VL-Embedding, ensuring correct handling of inputs and outputs.
- Updated documentation to reflect the new model's capabilities and usage instructions.
* test(models): wrap embedding assertions to satisfy lint
* test(opensearch): isolate HTTP transports between parallel tests
* fix(agent): align web_search count limit with the configured maximum
count 的参数描述写死了 "at most 20",而实际生效的上限来自 agent 配置
(web_search_max_results,常见值 7)。模型按描述填 8/20 就会被 Execute 里的
"count must be between 1 and N" 硬拒——那一轮连 provider 都没被调用,而且在
trace 里与真正的故障同为 span error,无法区分。
- count 的静态描述去掉写死的 20;NewWebSearchTool 再按实际生效的 maxResults
动态改写 schema 描述(解析失败原样返回,不影响工具可用性)
- count 超范围由"整轮失败"改为"截断到上限",输出里附一句纠正提示,日志降为 Warn。
同类标量 search_knowledge.limit / search_memory.limit 走的也是截断;且工具本来
就会静默返回少于 count 条(结果被 URL 校验过滤时 break),同一字段不该两套策略
行为变更:count 传 0/负数/超过上限不再返回 Success=false,而是按上限执行。
TestSearchOptionsAndExplicitPageContent 原先把 count 与非法过滤值并列断言
"必须失败且不得调用 provider",该断言已相应调整,新行为由
TestAgentWebSearchClampsCountToConfiguredMaximum 覆盖。
测试:go test ./internal/agent/tools/... -count=1
* fix(agent): clamp only an over-maximum web_search count
count is documented as 1..maximum, so a value below 1 is a malformed
argument rather than a near miss: reject it before the provider runs and
restore the previous message ("count must be between 1 and N"). Only a
count above the configured maximum is clamped, which keeps the change
scoped to the near-miss case it was meant to fix.
The correction now travels in Data as well as Output. web_search enables
sourceOutput, so once there is at least one row the model reads the
rendered retrieval block, which is rebuilt from Data["results"]; the note
written to Output never reached it. annotateWebSearchNotes restates the
clamp as <count_clamped> inside that block, the same way
annotateSearchNotes surfaces <omitted> and <subset>. The zero-result early
return carries the note too, instead of returning before it is built.
Also documents the clamping in website-docs and moves the added comments
back to English, keeping only the ones that describe what the code does.
Pass the app APT mirror to all build stages and expose optional Playwright download hosts for docreader through the build script and Docker Compose. Document the mirror settings and WebKit-specific host precedence.
* feat(models): expose embedding input modalities
Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.
- Catalog: declare input modalities for the Doubao and Tongyi vision
embeddings, the Aliyun multimodal / qwen-vl embedding families and
jina-embeddings-v4, as listed in each vendor's documentation. The
migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
the catalog and writes spec.input only when the row differs from it;
picking another catalog model or switching vendor drops the
declaration. Model cards show the image badge for such models.
* feat(embedding): embed images with multimodal embedding models
Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.
- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
object store is usually unreachable from the vendor) and an
ImageEmbedder implemented by all four protocols, each in the image
shape its reference documents: Ark image_url, DashScope contents[].image,
Gemini inlineData parts, and on the OpenAI shape an object keyed by the
vendor's image_field (Jina: "image"). A plain OpenAI-compatible
endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
max_image_bytes and image_mime_types, filled from vendor docs for the
DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
AsImageEmbedder, so existing Embedder implementations and fakes are
untouched. An embedder takes images only when the catalog (or the
row's spec.input) lists image AND the endpoint can carry one. Images
are checked against the documented limits before any request, then
split at the documented images per request. The concurrency, debug
and Langfuse decorators forward it; logs and traces carry the MIME
type and size, never the bytes.
The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.
* feat(knowledge): index images by their own vector and recall them
When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.
Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
the image (embedding.PrepareImage converts and shrinks it to the
vendor's documented limits) and stores it as a new image_vector chunk:
Content is the caption (OCR text when there is none) for reranking and
the answer context; its index row carries the image's vector with
SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
so a second row would overwrite the caption's text vector, and copy,
move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
through each engine's normal BatchIndex, keeping row order (the
composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
syncChunkIndex only syncs its enabled flag, updateChunkVector skips
it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
backends with a column default were storing them as enabled.
Retrieval
- A text query scores lower against an image than against text, so when
the model embeds images the document vector search is widened by half
(capped at 500) and run at min(VectorThreshold, 0.1); each store
group's hits are then held to their own threshold before score
normalization: text at VectorThreshold, images at min(VectorThreshold,
0.1). Keyword hits on image rows are dropped, their Content being the
already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
context is attached, and the caption is not repeated from ImageInfo.
Adds golang.org/x/image for scaling and WebP/BMP decoding.
* fix(models): report no thinking levels for embedding, rerank and ASR models
Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.
* fix(compaction): never read a cancelled turn's stream as a summary
streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.
Check the context before each receive and again when the stream closes.
* feat(rerank): score images with multimodal rerank models
Rerank models that read images can now be sent an image document and
score it against a text query on the same scale as text, which is what a
recall made by an image's own vector needs: a text reranker only sees the
caption, and rejects an image for what the caption left out.
- api: ImageReranker, implemented by the Cohere and NIM shapes. Image
documents go in requests of their own, so text requests keep their
exact shape. NIM sends image-only passages without text, as base64 data
URLs (its VLM reranking reference requires both). The Cohere shape sends
{"<image_field>": data URI} documents and asks for no echo of them.
- The image settings embeddings already had (image_field,
max_image_batch_size, max_image_bytes, image_mime_types) move into a
shared ImageInput embedded in both settings; resolved embedding settings
serialize exactly as before.
- Catalog: jina-reranker-m0 and nvidia/llama-nemotron-rerank-vl-1b-v2 take
text and image. Jina names "image" documents; its base64 form is not
shown in any reranker example, so the data URI its embeddings reference
documents is used and marked unverified. NVIDIA lists its six formats
and 512 passages per request.
- rerank: an optional ImageReranker reached through AsImageReranker, gated
on the model's catalog input and the endpoint; images are checked against
the documented limits before any request, scores are normalized like
text, and the debug and Langfuse decorators forward it without logging
image bytes.
- Image preparation (format conversion, shrink to fit) moves from the
embedding package to internal/models/imageprep, since rerank needs it
too.
* feat(retrieval): keep a vector hit's own similarity through fusion
Hybrid fusion replaces a hit's Score with a rank-based RRF score. That is
right for ordering, but it erases the evidence reranking needs to judge an
image found by its own vector: how close the vector actually was. Record
the engine's similarity on each vector hit (IndexWithScore.VectorScore)
before normalization and fusion rewrite Score, and carry it onto the
search result (SearchResult.VectorScore, not serialized).
* feat(reranking): judge image hits by their image, not only their caption
An image found by its own vector was reranked on its caption and OCR text,
so a text reranker rejected it for exactly what the caption left out.
- With a reranker that reads images and a LoadImage callback, image_vector
hits are scored by their image (prepared to the model's limits) on the
same scale as text, and pass or fail the same threshold. A hit whose
image cannot be read, prepared or scored falls back to its text.
- With a text reranker, its verdict stands for images that are mostly
text (OCR of 200 characters or more). A pictorial hit it rejects is kept
when its vector similarity (VectorScore) reaches Options.ImageKeepScore:
at most two, strongest vector first, appended after the ranked results
and marked kept_by=image_vector. This also returns an image when no text
passes at all.
- Diagnostics report images_scored and images_kept.
* feat(rerank): read image hits' images in every rerank path
The chat pipeline, the agent search_knowledge tool and the hybrid-search
API now pass the rerank stage a way to read an image hit's image and an
image keep score, so a multimodal reranker scores images by what they show
and a text reranker can no longer drop a pictorial image its caption
undersells.
- KnowledgeBaseService.ReadChunkImage reads the first image of a result's
image_info from the storage of its knowledge base (the knowledge base's
tenant, so shared knowledge bases read from their owner's storage),
capped at 32 MB. Plain http(s) images go through the SSRF-safe
downloader.
- The backend resolution the image multimodal task used moves into a
shared knowledgeImageReader, so ingestion and reranking find an image in
the same place.
- reranking.ImageKeepScoreFor: an image is kept over a text reranker only
when its vector was at least as close as a text hit had to be, and never
below 0.25 (empirical).
* feat(chat): show a vision model the images its answer rests on
An image recalled by its own vector reached the answer as its caption only,
so a vision model could not answer from what the caption left out.
- Knowledge QA: INTO_CHAT_MESSAGE reads, for a model that supports vision,
the images of the contexts that rest on an image vector match (at most
three distinct ones, prepared to what every vision API takes: PNG, JPEG,
WebP or GIF within 5 MB), attaches them after the user's own images, and
names the contexts they belong to.
- Agent: search_knowledge attaches the same images when the agent's model
supports vision (the agent describes tool images to models that cannot
see them, and these already carry captions). Stored history drops them;
they are read again from storage.
- An image hit and its caption hit read the same once merged, and
de-duplication keeps either. The kept copy now inherits the image
(image_vector_match), in the pipeline's exact and partial-overlap
de-duplication and in the agent tool's.
- Results the rerank stage kept outside its top-k (kept_by) no longer fall
to the later top-k cuts: the pipeline's FilterTopK and the agent tool's
MMR and limit let them ride along after the ranked results.
* docs(website): describe how image hits are reranked and shown to vision models
* Fix image evidence preservation and model source mapping
* feat(models): expose embedding input modalities
Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.
- Catalog: declare input modalities for the Doubao and Tongyi vision
embeddings, the Aliyun multimodal / qwen-vl embedding families and
jina-embeddings-v4, as listed in each vendor's documentation. The
migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
the catalog and writes spec.input only when the row differs from it;
picking another catalog model or switching vendor drops the
declaration. Model cards show the image badge for such models.
* feat(embedding): embed images with multimodal embedding models
Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.
- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
object store is usually unreachable from the vendor) and an
ImageEmbedder implemented by all four protocols, each in the image
shape its reference documents: Ark image_url, DashScope contents[].image,
Gemini inlineData parts, and on the OpenAI shape an object keyed by the
vendor's image_field (Jina: "image"). A plain OpenAI-compatible
endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
max_image_bytes and image_mime_types, filled from vendor docs for the
DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
AsImageEmbedder, so existing Embedder implementations and fakes are
untouched. An embedder takes images only when the catalog (or the
row's spec.input) lists image AND the endpoint can carry one. Images
are checked against the documented limits before any request, then
split at the documented images per request. The concurrency, debug
and Langfuse decorators forward it; logs and traces carry the MIME
type and size, never the bytes.
The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.
* feat(knowledge): index images by their own vector and recall them
When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.
Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
the image (embedding.PrepareImage converts and shrinks it to the
vendor's documented limits) and stores it as a new image_vector chunk:
Content is the caption (OCR text when there is none) for reranking and
the answer context; its index row carries the image's vector with
SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
so a second row would overwrite the caption's text vector, and copy,
move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
through each engine's normal BatchIndex, keeping row order (the
composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
syncChunkIndex only syncs its enabled flag, updateChunkVector skips
it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
backends with a column default were storing them as enabled.
Retrieval
- A text query scores lower against an image than against text, so when
the model embeds images the document vector search is widened by half
(capped at 500) and run at min(VectorThreshold, 0.1); each store
group's hits are then held to their own threshold before score
normalization: text at VectorThreshold, images at min(VectorThreshold,
0.1). Keyword hits on image rows are dropped, their Content being the
already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
context is attached, and the caption is not repeated from ImageInfo.
Adds golang.org/x/image for scaling and WebP/BMP decoding.
* fix(models): report no thinking levels for embedding, rerank and ASR models
Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.
* fix(compaction): never read a cancelled turn's stream as a summary
streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.
Check the context before each receive and again when the stream closes.
* feat(knowledge): put image vectors behind a knowledge-base switch, off by default
Image vector indexing and recall used to turn on whenever the knowledge
base's embedding model declared image input. A model that takes images is
often chosen for text alone, so an upgrade would have added an embedding
call per image, widened vector recall 1.5x and lowered the image threshold
to 0.1 for those knowledge bases without anyone asking.
ImageProcessingConfig gains image_vector_enabled (default false; stored
JSON without the key reads as off, so existing knowledge bases stay off
with no migration). Both sides now need the switch and the model:
- indexImageVector returns before touching the model unless the KB opted
in (KnowledgeBase.IsImageVectorEnabled, which also needs vector indexing).
- applyImageRecall resolves the embedding model only when some KB in the
search opted in, and records the opted-in KBs per store group
(storeGroup.ImageKBIDs). Only those groups widen the pool and lower the
threshold; image hits of any other KB, including ones indexed before the
switch was turned off, are dropped.
The KB editor shows the switch in the image processing section, with a
warning when the selected embedding model does not declare image input;
docs describe the switch, its cost and what turning it off does.
* fix(knowledge): leave image vectors behind when cloning into a KB that is off
CloneChunk copied every chunk of the source document and then CopyIndices
for all of them, so copying a document from a knowledge base with image
vectors on into one with the switch off (same model and store) still
created image_vector chunks and their index rows, bypassing the switch
that indexImageVector honours.
Image vector chunks are now dropped from the copy when the target has not
opted in. Every engine's CopyIndices copies only the rows of chunks in the
mapping, so their index rows stay behind too.
* fix(search): refill the vector pool past stale image vectors
Image hits of a knowledge base whose switch is off were dropped only after
the engine had ranked and cut its results at TopK. Image vectors indexed
before the switch was turned off could fill that pool and push out valid
text hits: with MatchCount 5 the pool is 50, and 50 stale image rows
ranked above the text left the search empty.
No engine can filter rows by source type (Postgres and SQLite do not even
take ExcludeChunkIDs), so the exclusion cannot move into the query. When
the document vector pool comes back full and holds stale image rows, that
search alone runs again on a pool twice as large, until it holds TopK rows
that are not stale, the index runs out, or the pool reaches
maxRetrievalPoolSize (at most four extra calls from a pool of 50). The pool
is then cut back to TopK such rows, matching an engine-side exclusion
unless more than maxRetrievalPoolSize-TopK stale rows rank above the text.
* feat(i18n): translate the image vector switch and image input keys into zh-TW
The zh-TW locale arrived from main after this branch added five keys: the
knowledge base's image vector switch (label, description, unsupported-model
warning) and the embedding model's image input flag (label, description).
* fix(knowledge): leave image vectors behind when moving into a KB that is off
A reuse_vectors move relocates a document's chunks and index rows in place:
MoveKnowledgeIndices rebinds the rows by document ID and
MoveChunksByKnowledgeID rebinds the chunks. Moving a document from a
knowledge base with image vectors on into one with the switch off therefore
carried its image_vector chunks and their vectors along, the same bypass the
clone path had.
When the target has not opted in, the image_vector chunks are now deleted
during the move, their index rows first and then the chunks, so a retry
finds the chunks again rather than leaving orphaned rows. A target that
opted in still receives them unchanged.
* fix(retriever): keep image vectors out of keyword-only engines
BatchIndexVectors handed image rows to every engine of the composite,
including one that serves keyword retrieval alone (Elasticsearch v7 next to
a vector store). There the row holds nothing but the image's caption, which
the caption's own chunk already indexes, and retrieval discards every
keyword hit on it.
Image rows now go only to engines that serve vector retrieval. An engine
that serves both keeps one row for both kinds, so its keyword search still
meets them; the search side handles that.
* fix(search): refill the keyword pool past image rows
Every engine that serves both kinds of retrieval stores one row per index
entry and runs its keyword search over that row's content, so an image
vector row, whose content is its caption, is a keyword hit wherever the
caption's own chunk is. filterImageHits drops all of them, but only after
the engine has cut its ranking at TopK, so they crowded out text and caption
hits; a keyword ranking that put the image rows first came back empty.
The keyword pool is now refilled the way the vector pool is refilled past
stale image vectors: when it came back full and held image rows, the
keyword search runs again on a pool twice as large, up to
maxRetrievalPoolSize, and is cut back to TopK hits that are kept.
Leaving the rows out of the keyword index instead would mean writing them
without searchable content (Milvus and Tencent VectorDB derive a BM25
sparse vector from it, and how they take an empty one is unverified) or a
source-type filter in every engine's keyword query (ten engines, and
filterable fields that existing collections lack).
* revert: let image vectors move with the document into a KB that is off
This reverts commit e259977fe915f7d0bc30c4b02bc7af04da1c8dd4.
Deleting the image vector index rows by chunk ID before the move is not
reliable: the Weaviate and Elasticsearch v8 delete wrappers ignore partial
failures, and OpenSearch rejects more than 1000 IDs with ErrBatchTooLarge.
A partial delete followed by the chunk delete leaves index rows whose chunks
are gone, which nothing ever cleans up.
Without it, the image_vector chunks and their index rows move together
with the document. While the target's switch is off they are filtered at
query time (the pool refill keeps them from crowding out text), they are
removed with the document on delete, reparse or knowledge base delete, both
chunks and index rows by knowledge ID, and they become active again if the
target turns the switch on. The clone path keeps dropping them, since there
the rows are copied rather than moved.
* fix(search): keep the image row refill from losing hits it already had
refillPool replaced a pool with the answer to each larger search. A refill
that answered only in part (RetrieveResult.Error, swallowed into a warning
by retainPartialResults, as when some collections of the store did not
answer) could hold fewer text rows than the complete smaller pool, or none:
40 stale image rows above 10 text came back from a partial refill as the
40 image rows alone, and the search lost all 10 text hits. A refill that
failed outright failed the whole store group, discarding a first round that
had succeeded.
A refill is now best effort. A failed or incomplete refill search stops
the refill and leaves the last complete pool in place, cut back to TopK
kept rows as before, so a refill can no longer leave a search worse off
than it was without it. Merging the partial answer into the pool was not
done: its rows would need deduplicating and its ranking is cut at a
different TopK over a different set of collections, while keeping the
previous pool is exact.
* docs(knowledge): say what happens to image vectors in a KB that is off and what the refill costs
The image vector section now says that with the switch off, image vectors
that were indexed before it was turned off, or that came along when the
document was moved in from a KB that has it on, are kept but filtered at
query time, are removed when the document is deleted or reparsed or the
KB is deleted, and are recalled again once the switch is turned on; a copy
into such a KB leaves them behind.
It also states the cost of keeping dropped image rows from crowding text
out of the pools: the vector and keyword pools are searched again with
TopK doubled up to the 500 candidate cap, which adds queries for store
groups that hold such rows, and text can still be missed when more than
about 500 candidates are image rows. A failed or partial refill keeps the
previous complete pool.
* fix(search): log unexpected refill result sets, and describe the refill stop condition precisely
#3956 (Hugging Face TEI) and #3957 (Pinecone) added two rerank-only
vendors, but the READMEs, the models page, the extension-points page and
the homepage still said 27, and the vendor table had no rows for them.
The v0.8.2 release notes keep their historical 27.
The rerank limits row said every limit splits the request automatically.
Only max_documents and max_request_chars do; an over-long query or
document (max_query_chars / max_document_chars, or a single document
that does not fit the request budget beside the query) fails the call.
Taken from the documentation part of #3757; max_documents keeps
0 = unlimited.
Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
* fix(datasource): omit last_sync_cursor from the data source list response
The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.
* feat(datasource): add Seafile connector with fingerprint-based incremental sync
Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.
- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils
* feat(datasource): add Seafile frontend connector and docs
Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.
- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
updates, and Seafile in the README data source lists
---------
Co-authored-by: zaqqmm <q-zhang10@qq.com>
* fix(kb): let update API set vlm_config
UpdateKnowledgeBaseRequest only carried name/description/config, so the
PUT /knowledge-bases/:id endpoint could never change a knowledge base's
multimodal (VLM) config, even though CreateKnowledgeBase accepts a
top-level vlm_config. Thread an optional vlm_config through the handler
and service so the update path mirrors create (nil = no change).
Fixes#1723
* test(kb): pass the vlm_config argument in tests added on main
* fix(kb): accept only model-managed vlm_config fields on update
The update API copied the request's VLMConfig wholesale, so a caller
could set the legacy ModelName/BaseURL/APIKey/InterfaceType fields and
have the VLM client call BaseURL directly, outside model management and
its SSRF checks. Only Enabled/ModelID/DescriptionLanguage/
CustomInstructions are taken now; legacy fields keep their stored value.
A referenced model must exist and be a VLLM model (400 otherwise), and
disabling clears ModelID, matching the initialization config endpoint.
* fix(kb): validate and gate vlm_config on the update API
- Enforce the 4000-character custom_instructions cap
(types.ValidateKnowledgeBasePromptInstructions), as create and
/initialization/config already do.
- vlm_config is a KB setting: require kbSettingsAccess (owner workspace
or admin share) like /initialization/config, so share editors get 403.
- Stored legacy model_name/base_url keep IsEnabled() true on their own,
so disabling returned 200 but left VLM on, and managed -> disabled fell
back to the old endpoint. Clear them when the request enables a managed
model_id, or disables VLM without echoing back the stored
model_name+base_url (GET-then-PUT keeps legacy on). Request legacy
fields are only compared, never written. Enabling without model_id on a
non-legacy KB is now a 400 instead of a silent no-op.
- Document the vlm_config semantics in the KB API reference.
* fix(kb): keep the legacy VLM endpoint only for pure legacy knowledge bases
CreateKnowledgeBase stores vlm_config as sent, so a managed model_id can
sit next to legacy model_name/base_url; resolveVLM then uses the managed
model. The update's keep-legacy paths (disable echoing the stored legacy
fields, enabled=true without model_id) cleared model_id but kept the
legacy fields, so IsEnabled() stayed true and image calls moved to the
stored base_url.
Those paths now apply only when no model_id is stored. On a mixed config
a disable clears everything, and enabled=true without model_id is a 400,
the same as on any managed knowledge base.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(data-analysis): skip style-only worksheets before DuckDB loading
* chore: retrigger CI for XLSX sheet fix
* fix(data-analysis): keep all sheets when the worksheet preflight fails
The style-only sheet preflight is best-effort: if it cannot read the
workbook parts, log a warning and load the sheets DuckDB enumerated
instead of failing the whole workbook. Context cancellation still
aborts the load.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>