Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
Replace the broad channel-runtime principal check with owner-scoped rules so
API callers can still read their own api_tenant_key and api_external_user
sessions without opening IM sessions matched via legacy empty user_id scope.
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.
中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:
- Startup skips a store whose engine fails to construct and never
retries it, so that instance cannot serve the store again -- not
even after a restart, if the backend is slow to come up.
- A store registered on one instance is absent on every other until
they restart, because registration is not broadcast.
- Registration can fail on the instance handling the create, leaving
the store missing even there.
Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.
Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.
Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.
Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.
Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.
For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.
A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.
Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.
Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
Restore HTTP-only 15s pipeline fetching, reject SPA shell pages when
browser rendering fails, surface per-URL failure fields in the UI, and
keep structured model output when all URLs fail.
Align English question and exclamation marks with period handling so
URL query strings are not split during overlap. Add regression tests for
URL overlap and inline code protection.
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
IM channels (DingTalk/Feishu) showed broken images because resource://
image references resolve to WeKnora capability links that were neither
reachable nor renderable:
- nginx never proxied /r/, so <APP_EXTERNAL_URL>/r/<token> links fell into
the SPA fallback and returned a blank page. Add a /r/ proxy block
mirroring /files, using the ${APP_SCHEME}/${APP_HOST}/${APP_PORT} vars.
- rewriteStorageURLs treated any "dst != src" as success and emitted
non-HTTP results (e.g. an internal storage:// path when APP_EXTERNAL_URL
is unset) that IM clients cannot fetch. Only substitute http(s) URLs;
otherwise leave the reference unchanged and log an actionable WARN.
- Warn once at startup when IM channels are active but APP_EXTERNAL_URL is
unset — the config state that silently breaks resource:// images on the
default MinIO/local deployment.
- Document that IM knowledge-base images need an IM-reachable http URL via
one of two paths: a publicly reachable storage backend, or APP_EXTERNAL_URL
routing resource:// through nginx /r/.
Remote http(s) images written as <img src="..."> reached no resolver.
Markdown images went through ResolveRemoteImages, HTML data-URIs and HTML
relative paths each had their own pass, but nothing claimed the remaining
combination, so those images were dropped without a log line and the
multimodal stage was skipped for documents that rely on HTML for layout.
ResolveRemoteImages now scans HTML img tags with a quoted src as a second
pass and rewrites only the src attribute value, leaving the tag intact
because the surrounding markup is why these documents use HTML at all.
The per-image work (whitelist, SSRF, download, icon filter, upload) moves
into one helper so the two syntaxes cannot drift apart on policy. Each
pass carries its own image budget; sharing one would let HTML images eat
into the number of Markdown images a document already gets resolved.
An attribute value is trimmed, HTML-unescaped and its scheme lowercased
before use, because HTML permits padding, spells a query separator &
and tolerates any scheme case, while every fetcher downstream compares the
scheme byte-for-byte. Markdown targets get none of that normalization, so
no Markdown reference resolves differently than it does today. A
whitelisted host keeps serving the image from where it already lives, and
its span is rewritten with the normalized URL rather than the raw attribute
text: later stages both locate the image by searching the document for its
serving URL and fetch that same string, so the two have to agree.
The src attribute must now be preceded by whitespace, so a hyphenated
attribute name is no longer mistaken for it. Among HTML tags this cuts two
ways: one carrying both a hyphenated *-src and a real src now resolves the
real src instead of the placeholder, while one carrying only a hyphenated
*-src is no longer resolved at all. Recovering the latter needs an
attribute parser rather than a regex, and resolving a lazy-loading
placeholder while the real image goes unvisited is the worse of the two
failures. Both halves are pinned by tests.
Image-derived text also has to survive into the answer, which is why this
touches internal/searchutil. The chat enricher and the image_info filters
matched Markdown links only, so an HTML-only passage looked image-free:
every entry was dropped and the model received bare markup with the
caption and OCR text stripped, even though the analysis had run. Both now
recognise HTML img tags through one shared regex, so a tag the parser
stores can always be matched back. Both syntaxes are located against the
original passage and spliced in a single pass; scanning one regex over the
other's output let a metadata block that itself quotes an image reference
be rescanned and duplicated. The chat variant still enriches only images
present in the passage, so orphan entries stay out.
Known limits:
- An unquoted src and a srcset-only tag are not matched, nor is a tag
whose attribute values hold a bare > or a quoted src= of their own.
- imgHTMLDataURI still matches src without a boundary, so a data-src
holding a base64 placeholder is consumed as if it were the src and the
whole tag replaced. Worth the same fix, but it changes what an existing
lazy-loading document resolves, so it belongs on its own.
- A reference shown as a documentation example is rewritten like any
other. Recognising code spans needs a Markdown parser; a hand-rolled
scanner risks the far worse failure of treating a document's real
screenshots as examples.
- Rewriting a whitelisted src writes the unescaped URL back into an HTML
attribute, so a query parameter whose name spells an entity renders
differently than the author wrote it. Keeping the encoded form would
mean unescaping on every matcher instead.
- Image text reaches the model only when the whole reference falls inside
the retrieved window; an img tag is several times longer than a
Markdown link and straddles a chunk boundary more often.
- A reference repeated in one document is fetched once per occurrence,
which is what gives each chunk its own analysed copy.
- Nothing bounds how long one document's fetches may take in total.
- Parent-context pruning removes only Markdown images.
Co-authored-by: wizardchen <wizardchen@tencent.com>
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.
Co-authored-by: Cursor <cursoragent@cursor.com>
CleanMarkdown regex patterns (script, onclick=, javascript:, etc.)
unconditionally delete matching substrings from chunk content, even when
they appear in code blocks or inline code.
JSON APIs should return raw stored data; XSS protection is the frontend
markdown renderer's responsibility.
SanitizeForDisplay and CleanMarkdown are retained in security.go.
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.
Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
Markdown tables — searchable text, instead of being lost or (post-export)
rendered as unsearchable images. Bitable date/datetime cells render in the
table's timezone (configurable via Settings["timezone"], default GMT+8) rather
than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
pipeline, gated by the KB's multimodal setting: the image's external_id always
stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
records) break on an empty page so a malformed has_more response cannot spin
until the task deadline.
Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
"<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
means a single failed attachment can't wipe its siblings, and dupIsSameNode
guards the sweep so a file_hash-only dedup collision against a different node
cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
(never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
metadata external_id prefix scan.
Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.
Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.
This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
Batch wiki reads used to lose middle pages when the registry applied
head/tail truncation to the combined output. wiki_read_page now budgets
per page, caps neighbour link summaries, and names omitted slugs explicitly.
Also align registry truncation with rune-based limits and raise the default
tool output ceiling to 24k characters.
router.go now holds only engine assembly (RouterParams / NewRouter /
trustedProxies, 300 lines); the Register* functions move to domain files —
routes_auth_tenant.go, routes_knowledge.go, routes_chat.go, routes_agent.go,
routes_infra.go — and the five file-proxy surfaces (/files, KB-scoped files,
presigned, presigned-preview, /r/:token grants) move to files.go with the
frontend SPA server in static.go. Pure moves; every route, guard chain and
API-key policy declaration is unchanged, as verified by the existing route
tests and the startup policy self-check.
files.go additionally consolidates the plumbing the four handlers each
hand-rolled: LOCAL_STORAGE_BASE_DIR resolution (localStorageAbsDir),
backend-ID/provider parsing (parseStorageTarget), file_path query validation
(requireFilePathQuery), resource-catalog tenancy mapping
(resolveCatalogResource), tenant-storage resolution with and without the
global-service fallback, and the shared success response including HEAD
short-circuit (streamStoredFile).
Also reattaches the RegisterMCPServiceRoutes doc comment that had been
stranded on top of RegisterSystemAdminRoutes, drops the swagger blank import
duplicated into every split file, and documents why AllowCredentials is
inert under the wildcard CORS origin.
The five near-identical "c.Set + context.WithValue" blocks (JWT branch,
tenantless branch, both API-key attach helpers, EmbedAuth) now go through
one applyAuthSession helper, so the c.Keys surface and the request-context
surface can no longer drift apart. The 140-line JWT branch inside Auth is
split into authenticateJWTUser / resolveTargetTenant, which also removes a
duplicate GetTenantByID roundtrip on the X-Tenant-ID switch path.
Fixes folded in:
- delete GetTenantIDFromContext: it read ctx.Value("tenantID") while the
value is stored under types.TenantIDContextKey ("TenantID"), so it could
never succeed (dead, broken, uncalled)
- isPlatformTenantOptionalAPI: match "/api/v1/system/admin" exactly or as
a "/"-delimited prefix instead of a bare HasPrefix that would also admit
sibling paths sharing the prefix
- 401 for a presented-but-invalid Bearer token now says "invalid or
expired token" instead of the misleading "missing authentication"
- replace the remaining log.Printf calls in the auth path with the
request-scoped structured logger
New tests lock the both-surfaces invariant of applyAuthSession, the
tenantless "no tenant/role keys" contract, and bearerToken parsing.