WeKnora/internal/modelcontext
lyingbug 65f88f4121
feat: judge image hits by their image in rerank and show them to vision models (#4013)
* feat(models): expose embedding input modalities

Whether an embedding model maps images into the same space as text is
what will decide if a knowledge base can store image vectors. The
catalog already had the field, but most multimodal embedding models
were missing it and embedding rows never reported it.

- Catalog: declare input modalities for the Doubao and Tongyi vision
  embeddings, the Aliyun multimodal / qwen-vl embedding families and
  jina-embeddings-v4, as listed in each vendor's documentation. The
  migration baseline changes for exactly these nine entries.
- Model responses resolve capabilities for embedding rows too, so
  capabilities.input reflects the catalog merged with spec.input.
- Model editor: an "Image input" switch for embedding models. It follows
  the catalog and writes spec.input only when the row differs from it;
  picking another catalog model or switching vendor drops the
  declaration. Model cards show the image badge for such models.

* feat(embedding): embed images with multimodal embedding models

Adds the image side of an embedder, for models that map images into the
same space as their text vectors so a text query can find an image.

- api: EmbedImage (bytes + MIME type, sent inline as base64 because the
  object store is usually unreachable from the vendor) and an
  ImageEmbedder implemented by all four protocols, each in the image
  shape its reference documents: Ark image_url, DashScope contents[].image,
  Gemini inlineData parts, and on the OpenAI shape an object keyed by the
  vendor's image_field (Jina: "image"). A plain OpenAI-compatible
  endpoint names no image field and takes no images.
- Settings: image_field, max_image_batch_size (default one per request),
  max_image_bytes and image_mime_types, filled from vendor docs for the
  DashScope multimodal models, gemini-embedding-2 and Jina.
- embedding: an optional ImageEmbedder interface reached through
  AsImageEmbedder, so existing Embedder implementations and fakes are
  untouched. An embedder takes images only when the catalog (or the
  row's spec.input) lists image AND the endpoint can carry one. Images
  are checked against the documented limits before any request, then
  split at the documented images per request. The concurrency, debug
  and Langfuse decorators forward it; logs and traces carry the MIME
  type and size, never the bytes.

The migration baseline changes only for the entries whose resolved
embedding settings now carry image limits.

* feat(knowledge): index images by their own vector and recall them

When the knowledge base's embedding model embeds images, each image now
gets a vector of its own next to its caption and OCR chunks, so a query
can find an image by what it shows even where the caption left it out.

Ingestion
- After the caption/OCR chunks are indexed, the multimodal task embeds
  the image (embedding.PrepareImage converts and shrinks it to the
  vendor's documented limits) and stores it as a new image_vector chunk:
  Content is the caption (OCR text when there is none) for reranking and
  the answer context; its index row carries the image's vector with
  SourceType ImageSourceType. Scanned PDF pages are skipped, their OCR
  text serves better. A failure here is logged and never fails the task.
- The image vector gets its own chunk rather than a second index row on
  the caption chunk: OpenSearch and Weaviate key documents by ChunkID,
  so a second row would overwrite the caption's text vector, and copy,
  move and enable/disable all follow chunk IDs.
- CompositeRetrieveEngine.BatchIndexVectors writes precomputed vectors
  through each engine's normal BatchIndex, keeping row order (the
  composite's de-duplication does not).
- Paths that rebuild vectors from text leave image_vector alone:
  syncChunkIndex only syncs its enabled flag, updateChunkVector skips
  it, and a caption edit updates its Content only.
- The multimodal task's caption/OCR index rows now set IsEnabled; only
  backends with a column default were storing them as enabled.

Retrieval
- A text query scores lower against an image than against text, so when
  the model embeds images the document vector search is widened by half
  (capped at 500) and run at min(VectorThreshold, 0.1); each store
  group's hits are then held to their own threshold before score
  normalization: text at VectorThreshold, images at min(VectorThreshold,
  0.1). Keyword hits on image rows are dropped, their Content being the
  already-indexed caption.
- image_vector hits are searchable and merge like caption hits: parent
  context is attached, and the caption is not repeated from ImageInfo.

Adds golang.org/x/image for scaling and WebP/BMP decoding.

* fix(models): report no thinking levels for embedding, rerank and ASR models

Embedding rows now carry capabilities, and the end-to-end run showed them
listing off/auto/minimal/low/medium/high: these references resolve no chat
protocol, and an empty level map reads as "every level supported". A model
without a chat protocol does not think, so its capabilities say so.

* fix(compaction): never read a cancelled turn's stream as a summary

streamSummary selects on the turn's context and the provider stream at
once. When the turn is already cancelled both are ready (providers close
the stream once their context ends), and select picks at random, so a
stopped turn could come back as a completed, partial summary with a nil
error. TestSummarizationIsNotRetriedOnceTheTurnIsCancelled caught it
intermittently in CI and reproduces under -race with several -cpu values.

Check the context before each receive and again when the stream closes.

* feat(rerank): score images with multimodal rerank models

Rerank models that read images can now be sent an image document and
score it against a text query on the same scale as text, which is what a
recall made by an image's own vector needs: a text reranker only sees the
caption, and rejects an image for what the caption left out.

- api: ImageReranker, implemented by the Cohere and NIM shapes. Image
  documents go in requests of their own, so text requests keep their
  exact shape. NIM sends image-only passages without text, as base64 data
  URLs (its VLM reranking reference requires both). The Cohere shape sends
  {"<image_field>": data URI} documents and asks for no echo of them.
- The image settings embeddings already had (image_field,
  max_image_batch_size, max_image_bytes, image_mime_types) move into a
  shared ImageInput embedded in both settings; resolved embedding settings
  serialize exactly as before.
- Catalog: jina-reranker-m0 and nvidia/llama-nemotron-rerank-vl-1b-v2 take
  text and image. Jina names "image" documents; its base64 form is not
  shown in any reranker example, so the data URI its embeddings reference
  documents is used and marked unverified. NVIDIA lists its six formats
  and 512 passages per request.
- rerank: an optional ImageReranker reached through AsImageReranker, gated
  on the model's catalog input and the endpoint; images are checked against
  the documented limits before any request, scores are normalized like
  text, and the debug and Langfuse decorators forward it without logging
  image bytes.
- Image preparation (format conversion, shrink to fit) moves from the
  embedding package to internal/models/imageprep, since rerank needs it
  too.

* feat(retrieval): keep a vector hit's own similarity through fusion

Hybrid fusion replaces a hit's Score with a rank-based RRF score. That is
right for ordering, but it erases the evidence reranking needs to judge an
image found by its own vector: how close the vector actually was. Record
the engine's similarity on each vector hit (IndexWithScore.VectorScore)
before normalization and fusion rewrite Score, and carry it onto the
search result (SearchResult.VectorScore, not serialized).

* feat(reranking): judge image hits by their image, not only their caption

An image found by its own vector was reranked on its caption and OCR text,
so a text reranker rejected it for exactly what the caption left out.

- With a reranker that reads images and a LoadImage callback, image_vector
  hits are scored by their image (prepared to the model's limits) on the
  same scale as text, and pass or fail the same threshold. A hit whose
  image cannot be read, prepared or scored falls back to its text.
- With a text reranker, its verdict stands for images that are mostly
  text (OCR of 200 characters or more). A pictorial hit it rejects is kept
  when its vector similarity (VectorScore) reaches Options.ImageKeepScore:
  at most two, strongest vector first, appended after the ranked results
  and marked kept_by=image_vector. This also returns an image when no text
  passes at all.
- Diagnostics report images_scored and images_kept.

* feat(rerank): read image hits' images in every rerank path

The chat pipeline, the agent search_knowledge tool and the hybrid-search
API now pass the rerank stage a way to read an image hit's image and an
image keep score, so a multimodal reranker scores images by what they show
and a text reranker can no longer drop a pictorial image its caption
undersells.

- KnowledgeBaseService.ReadChunkImage reads the first image of a result's
  image_info from the storage of its knowledge base (the knowledge base's
  tenant, so shared knowledge bases read from their owner's storage),
  capped at 32 MB. Plain http(s) images go through the SSRF-safe
  downloader.
- The backend resolution the image multimodal task used moves into a
  shared knowledgeImageReader, so ingestion and reranking find an image in
  the same place.
- reranking.ImageKeepScoreFor: an image is kept over a text reranker only
  when its vector was at least as close as a text hit had to be, and never
  below 0.25 (empirical).

* feat(chat): show a vision model the images its answer rests on

An image recalled by its own vector reached the answer as its caption only,
so a vision model could not answer from what the caption left out.

- Knowledge QA: INTO_CHAT_MESSAGE reads, for a model that supports vision,
  the images of the contexts that rest on an image vector match (at most
  three distinct ones, prepared to what every vision API takes: PNG, JPEG,
  WebP or GIF within 5 MB), attaches them after the user's own images, and
  names the contexts they belong to.
- Agent: search_knowledge attaches the same images when the agent's model
  supports vision (the agent describes tool images to models that cannot
  see them, and these already carry captions). Stored history drops them;
  they are read again from storage.
- An image hit and its caption hit read the same once merged, and
  de-duplication keeps either. The kept copy now inherits the image
  (image_vector_match), in the pipeline's exact and partial-overlap
  de-duplication and in the agent tool's.
- Results the rerank stage kept outside its top-k (kept_by) no longer fall
  to the later top-k cuts: the pipeline's FilterTopK and the agent tool's
  MMR and limit let them ride along after the ranked results.

* docs(website): describe how image hits are reranked and shown to vision models

* Fix image evidence preservation and model source mapping
2026-10-09 13:01:48 +08:00
..
README.md refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
citations.go fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794) 2026-09-27 20:32:26 +08:00
graph_truncation_test.go fix(agent): 图谱查询的三处截断必须可见,模型不再把子图当全集 (#3885) 2026-10-07 23:59:50 +08:00
handle_table.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
handles.go fix(agent): harden search-target scope and model-handle policies 2026-07-27 11:05:43 +08:00
image_sources.go feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
image_sources_test.go feat: judge image hits by their image in rerank and show them to vision models (#4013) 2026-10-09 13:01:48 +08:00
leaks.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
leaks_test.go refactor(modelcontext): consolidate source_refs parsing, fixed data_analysis table, and identifier leak report (#3372) 2026-09-17 22:08:16 +08:00
mcp.go fix(mcp): normalize JSON-encoded argument envelopes 2026-09-09 16:11:52 +08:00
mcp_sources.go fix(agent): prevent MCP answers from citing unrelated KB sources (#3144) 2026-09-10 00:31:43 +08:00
mcp_sources_test.go fix(agent): prevent MCP answers from citing unrelated KB sources (#3144) 2026-09-10 00:31:43 +08:00
mcp_test.go fix(mcp): normalize JSON-encoded argument envelopes 2026-09-09 16:11:52 +08:00
model_output.go fix(chat): 在模型上下文中标注 FILTER_TOP_K 的子集截断,避免把前 N 条当作全集作答 (#3758) 2026-10-09 10:54:37 +08:00
omitted_results_test.go fix(chat): 在模型上下文中标注 FILTER_TOP_K 的子集截断,避免把前 N 条当作全集作答 (#3758) 2026-10-09 10:54:37 +08:00
registry.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
registry_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
resources.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
resources_test.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
sources.go refactor(models): rebuild vendor integration on protocol + catalog + vendor layers (#3470) 2026-09-20 21:28:23 +08:00
sources_test.go fix(agent): prevent MCP answers from citing unrelated KB sources (#3144) 2026-09-10 00:31:43 +08:00
stream.go feat(mcp)!: streamline setup and use short routing handles 2026-09-09 16:11:52 +08:00
tool_policy.go refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381) 2026-09-18 14:12:22 +08:00
tool_policy_test.go refactor(modelcontext): unify source-key gating behind one dispatch table 2026-07-27 11:05:43 +08:00

README.md

Model context boundary

modelcontext.Registry is the only application-facing boundary for values that are shortened before an LLM call and restored afterwards. The whole codec lives in this one package:

File Role
registry.go Registry facade — the only type request lifecycles use
handle_table.go handleTable[M], the generic bidirectional primitive behind every handle space
sources.go source handles (cN/dN/bN/wN) and the tool-argument codec
citations.go protocol prompts, <ref/> → <kb/>/<web/> expansion, citation stream expander
model_output.go compact source-centric renderings of tool results
resources.go durable-resource handles (res://NNNN)
stream.go streamHold suffix-hold primitive and every streaming decoder
tool_policy.go the single policy layer: per-tool key contracts + sourceKeySpaces dispatch
mcp.go MCP bridge routing handles (msN/mtN), definition projection and envelope codec
mcp_sources.go bounded citation sidecar for HTTP(S) links in successful MCP results
handles.go exported HandleTable for invocation-local spaces (iN, ref-N, c000)
leaks.go LeakedIdentifiers report of raw UUIDs that survive EncodeMessages, logged at every model-call site

Identity rules

  • Durable application identities: UUIDs, Wiki slugs, URLs, and resource://... references.
  • Request-local model handles: cN, dN, bN, wN, msN, mtN, and res://NNNN.
  • Tool-private request handles: iN for Wiki issues.
  • Ingest-call handles: cNNN and ref-N, allocated with HandleTable.
  • Temporary handles are decoded before tool execution, persistence, or UI delivery. They are never accepted as durable authorization or routing data.

Request lifecycle

  1. Create one Registry for the complete request or Agent execution.
  2. Register structured source identities as they enter the context. For Agent tools, use EncodeTools to project MCP routing enums and source summaries.
  3. Call EncodeMessages immediately before the model call.
  4. Decode tool calls through DecodeToolCalls before parsing or execution.
  5. Decode complete responses with DecodeResponse, or streaming text with one StreamDecoder per response channel.

Unknown handle-shaped values in declared tool fields are rejected before tool execution. The same per-tool policy is applied when historical assistant tool calls and tool results are replayed, so handles neither disappear nor drift to new numbers across Agent rounds.

The registry owns codec ordering. Resource references are encoded before source IDs so a Wiki slug such as summary/<knowledge-id> cannot be corrupted into summary/d1.

Source addressability is separate from citation eligibility. Use RegisterContextChunk for bound-KB directory entries. Replayed citations, legacy tool history, and tool arguments also register navigation handles only. ModelToolResultForTool and explicitly supplied retrieval results authorize citations for the current execution; rereading a historical source promotes the same handle without renumbering it. Both complete and streaming decoders drop references that have not been backed by current evidence.

Observability

Every model-call site runs LeakedIdentifiers on the encoded messages and logs a warning naming the role or tool whose text still carries a raw UUID. A registered leak means the codec missed a rewrite site; an unregistered leak means a producer emitted an identifier that never entered the registry. Text-level compaction (CompactKnownText) stays in place as defense in depth until those producers are fixed and the report stays quiet.

Langfuse generation observations contain the exact encoded payload sent to and returned by the model. Agent tool spans contain both model_arguments (the model-emitted handles) and resolved_arguments (the durable values actually executed), plus argument_resolution and any unresolved handles. Sensitive tools report only argument keys and resolution counts.

Langfuse is an observer of this boundary; it must not implement another handle mapping.

Built-in tool policies

Canonical ID-bearing arguments for knowledge, graph, data, web, and Wiki tools are decoded centrally through a tool-name plus JSON-field allowlist. The sourceKeySpaces table is the single source of truth for which keys are ID-bearing and which handle space they register into; per-tool sourceIDKeys contracts gate which of those keys each tool may use. database_query.sql has an explicit policy because source handles can be embedded inside quoted SQL values when the model filters the real knowledges/chunks tables; unquoted SQL aliases and arbitrary prose are never rewritten. data_analysis.sql carries no handles: the selected document is always exposed as the fixed table dataset, so only knowledge_id is decoded. Wiki issue IDs use the same lifecycle via an iN handle space.

Dynamic MCP tools are intentionally opaque in both arguments and results. Their schemas and identity semantics are controlled by the MCP server, so the application must not guess that an arbitrary id, knowledge_id, or url field is durable or rewrite it. Durable resource handles inside MCP arguments still use the normal res://NNNN codec. A future MCP ID mapping must be an explicit server/tool annotation and should plug into this registry rather than create a parallel mapper.

Successful MCP execution results additionally receive an external_source_candidates sidecar with up to 50 distinct observed HTTP(S) links mapped to wN. The external payload is not rewritten, discovery schemas are not scanned, and article IDs are never used to guess URLs. A candidate link does not establish that its page was read: the model must select the link whose associated result supports the claim. When no handle is supplied, the citation protocol permits an exact supplied source URL as a Markdown link, never a borrowed knowledge-base citation.

The application-owned MCP bridge is an explicit exception for routing fields: discover_mcp_tools.server_id uses msN, and call_mcp_tool.tool_ref uses mtN. Tool definition enums, service summaries, discovery results and replayed calls share the same request registry. Only directory envelope fields are registered; input_schema, call_mcp_tool.arguments, and external execution results remain opaque. Arguments are restored before authorization, validation, execution, persistence and UI events. Unknown routing handles fail closed.

For model-emitted call_mcp_tool calls, one extra JSON-string encoding of the arguments envelope is accepted only when it contains a complete JSON object (including {}). Normalization runs before resource-handle decoding; original model arguments are retained for tracing. Nested business strings are unchanged, and malformed JSON, arrays, null, and further encoding layers remain invalid. The canonical object still passes the existing routing, schema and approval checks.

Wiki routing

WikiRouteResolver is intentionally separate. It stores server-side slug-to-knowledge-base provenance gathered during Wiki search/read operations, and can only return knowledge bases already present in the Agent's authorized Wiki scope. It is routing state, not a model handle table. Reads scan every legal Wiki KB so cached provenance cannot hide a duplicate slug; mutations require one unique owner. New pages use link provenance, a single Wiki scope, or a single source-document owner and reject ambiguous routing instead of asking the model for a durable KB ID.

Document/tag-constrained Wiki and graph requests are filtered by the same server-owned SearchTargets provenance. Uncited/global Wiki pages and graph results outside that subset fail closed; only an explicit whole-KB target authorizes whole-KB content.