Commit Graph

1414 Commits (37837c00a92d976efe2373c74bd3c232bea8a2aa)

Author SHA1 Message Date
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00
wizardchen ab9f705843 fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
wizardchen db9bdc27cd fix(session): scope API runtime admin-console bypass to owned sessions
Replace the broad channel-runtime principal check with owner-scoped rules so
API callers can still read their own api_tenant_key and api_external_user
sessions without opening IM sessions matched via legacy empty user_id scope.
2026-07-30 11:56:05 +08:00
wizardchen 123f2d6349 fix(session): include external users in API session group 2026-07-30 11:56:05 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wizardchen c291c6a59f fix(kb): retry KB delete when engine resolution is deferred
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
2026-07-29 12:25:18 +08:00
ochan.kwon c15c3e972f fix(retriever): rebuild a missing vector store engine on demand
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:

  - Startup skips a store whose engine fails to construct and never
    retries it, so that instance cannot serve the store again -- not
    even after a restart, if the backend is slow to come up.
  - A store registered on one instance is absent on every other until
    they restart, because registration is not broadcast.
  - Registration can fail on the instance handling the create, leaving
    the store missing even there.

Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.

Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.

Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.

Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.

Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.

For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.

A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.

Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.

Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
2026-07-29 12:25:18 +08:00
wizardchen bf68db4653 fix(storage): allow image uploads to use file service fallback 2026-07-29 11:36:24 +08:00
wizardchen 5db186a0fc fix(knowledge): harden file-type scoped duplicate detection
Use case-insensitive file_type matching, scope the filename fallback by type, and document the dedup contract for datasource sweep handling.
2026-07-29 10:36:17 +08:00
koishi514-Z 18b95ebb3a fix(knowledge): distinguish duplicate files by type 2026-07-29 10:36:17 +08:00
wizardchen 2358564bde fix(knowledge): default custom metadata on create 2026-07-28 22:22:59 +08:00
wizardchen b581094c7b fix(web_fetch): address review follow-ups for structured fetch results
Restore HTTP-only 15s pipeline fetching, reject SPA shell pages when
browser rendering fails, surface per-URL failure fields in the UI, and
keep structured model output when all URLs fail.
2026-07-28 22:17:31 +08:00
langcaiye 75a2eabe20 fix(agent): restore safe rendered web fetch fallback 2026-07-28 22:17:31 +08:00
langcaiye a3da5cb57a fix(agent): preserve web evidence when page fetches fail 2026-07-28 22:17:31 +08:00
wizardchen c193b949aa fix(chunker): require space after English ? and ! in overlap boundaries
Align English question and exclamation marks with period handling so
URL query strings are not split during overlap. Add regression tests for
URL overlap and inline code protection.
2026-07-28 22:02:30 +08:00
ttommybot 21861f4b0f D:\xinchen.ju\Code\WeKnora 2026-07-28 22:02:30 +08:00
ttommybot 2dfc696740 fix: preserve semantic boundaries in chunk overlap 2026-07-28 22:02:30 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
koishi514-Z 0e1e5a63ba fix(chat): prevent duplicate terminal answer events 2026-07-28 21:33:33 +08:00
nullkey 4e3b512af6 fix(im): render resource:// images in IM channels (nginx /r/ + config guards)
IM channels (DingTalk/Feishu) showed broken images because resource://
image references resolve to WeKnora capability links that were neither
reachable nor renderable:

- nginx never proxied /r/, so <APP_EXTERNAL_URL>/r/<token> links fell into
  the SPA fallback and returned a blank page. Add a /r/ proxy block
  mirroring /files, using the ${APP_SCHEME}/${APP_HOST}/${APP_PORT} vars.
- rewriteStorageURLs treated any "dst != src" as success and emitted
  non-HTTP results (e.g. an internal storage:// path when APP_EXTERNAL_URL
  is unset) that IM clients cannot fetch. Only substitute http(s) URLs;
  otherwise leave the reference unchanged and log an actionable WARN.
- Warn once at startup when IM channels are active but APP_EXTERNAL_URL is
  unset — the config state that silently breaks resource:// images on the
  default MinIO/local deployment.
- Document that IM knowledge-base images need an IM-reachable http URL via
  one of two paths: a publicly reachable storage backend, or APP_EXTERNAL_URL
  routing resource:// through nginx /r/.
2026-07-28 21:25:12 +08:00
Ochan Kwon c9a342d965
fix(docparser): resolve remote images referenced by HTML img tags (#2355)
Remote http(s) images written as <img src="..."> reached no resolver.
Markdown images went through ResolveRemoteImages, HTML data-URIs and HTML
relative paths each had their own pass, but nothing claimed the remaining
combination, so those images were dropped without a log line and the
multimodal stage was skipped for documents that rely on HTML for layout.

ResolveRemoteImages now scans HTML img tags with a quoted src as a second
pass and rewrites only the src attribute value, leaving the tag intact
because the surrounding markup is why these documents use HTML at all.
The per-image work (whitelist, SSRF, download, icon filter, upload) moves
into one helper so the two syntaxes cannot drift apart on policy. Each
pass carries its own image budget; sharing one would let HTML images eat
into the number of Markdown images a document already gets resolved.

An attribute value is trimmed, HTML-unescaped and its scheme lowercased
before use, because HTML permits padding, spells a query separator &amp;
and tolerates any scheme case, while every fetcher downstream compares the
scheme byte-for-byte. Markdown targets get none of that normalization, so
no Markdown reference resolves differently than it does today. A
whitelisted host keeps serving the image from where it already lives, and
its span is rewritten with the normalized URL rather than the raw attribute
text: later stages both locate the image by searching the document for its
serving URL and fetch that same string, so the two have to agree.

The src attribute must now be preceded by whitespace, so a hyphenated
attribute name is no longer mistaken for it. Among HTML tags this cuts two
ways: one carrying both a hyphenated *-src and a real src now resolves the
real src instead of the placeholder, while one carrying only a hyphenated
*-src is no longer resolved at all. Recovering the latter needs an
attribute parser rather than a regex, and resolving a lazy-loading
placeholder while the real image goes unvisited is the worse of the two
failures. Both halves are pinned by tests.

Image-derived text also has to survive into the answer, which is why this
touches internal/searchutil. The chat enricher and the image_info filters
matched Markdown links only, so an HTML-only passage looked image-free:
every entry was dropped and the model received bare markup with the
caption and OCR text stripped, even though the analysis had run. Both now
recognise HTML img tags through one shared regex, so a tag the parser
stores can always be matched back. Both syntaxes are located against the
original passage and spliced in a single pass; scanning one regex over the
other's output let a metadata block that itself quotes an image reference
be rescanned and duplicated. The chat variant still enriches only images
present in the passage, so orphan entries stay out.

Known limits:

  - An unquoted src and a srcset-only tag are not matched, nor is a tag
    whose attribute values hold a bare > or a quoted src= of their own.
  - imgHTMLDataURI still matches src without a boundary, so a data-src
    holding a base64 placeholder is consumed as if it were the src and the
    whole tag replaced. Worth the same fix, but it changes what an existing
    lazy-loading document resolves, so it belongs on its own.
  - A reference shown as a documentation example is rewritten like any
    other. Recognising code spans needs a Markdown parser; a hand-rolled
    scanner risks the far worse failure of treating a document's real
    screenshots as examples.
  - Rewriting a whitelisted src writes the unescaped URL back into an HTML
    attribute, so a query parameter whose name spells an entity renders
    differently than the author wrote it. Keeping the encoded form would
    mean unescaping on every matcher instead.
  - Image text reaches the model only when the whole reference falls inside
    the retrieved window; an img tag is several times longer than a
    Markdown link and straddles a chunk boundary more often.
  - A reference repeated in one document is fetched once per occurrence,
    which is what gives each chunk its own analysed copy.
  - Nothing bounds how long one document's fetches may take in total.
  - Parent-context pruning removes only Markdown images.

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-07-28 21:08:31 +08:00
xiaowang 82d3e0b271 fix(wiki): prevent finalizing stalls on enqueue failure 2026-07-28 20:54:48 +08:00
zhangzherui eb447365cd fix(knowledge): clear stale errors on reparse 2026-07-28 20:44:34 +08:00
zhangzherui d59151d872 fix(im): recover after leader lock loss 2026-07-28 20:40:58 +08:00
wizardchen 497e8312e3 fix(knowledge): harden summary refresh after review feedback
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 20:23:48 +08:00
wizardchen 0766c5671f fix(retrieval): restore image parent context 2026-07-28 20:23:48 +08:00
wizardchen 48b661134b fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
wizardchen 6a439a4c3b fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
wizardchen 850a073c83 fix(summary): restore tenant context for refresh tasks 2026-07-28 20:23:48 +08:00
wizardchen 1b9858ecb8 fix(knowledge): enqueue stale summary refreshes 2026-07-28 20:23:48 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen 95775ae909 fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
MidoriKurage 002434bbb3 fix: remove SanitizeForDisplay from chunk handler responses
CleanMarkdown regex patterns (script, onclick=, javascript:, etc.)
unconditionally delete matching substrings from chunk content, even when
they appear in code blocks or inline code.

JSON APIs should return raw stored data; XSS protection is the frontend
markdown renderer's responsibility.

SanitizeForDisplay and CleanMarkdown are retained in security.go.
2026-07-28 18:18:56 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
wizardchen 9714eaa9a7 fix(im): handle disabled channels in IM callback and improve connection lifecycle management 2026-07-27 20:39:12 +08:00
wizardchen 53ddad57c0 fix(im): harden connection lifecycle across replicas 2026-07-27 20:39:12 +08:00
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
wizardchen 80a5003cc9 feat(wiki): add page revision history with diff, revert, and manual editing
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
2026-07-27 15:03:24 +08:00
wizardchen 1d64fc6234 fix(knowledge): reconcile wiki state when moving documents across KBs
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
2026-07-27 15:02:02 +08:00
MidoriKurage 3b476c07df fix(pipeline): deterministic merge output ordering
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.

Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
2026-07-27 13:30:21 +08:00
MidoriKurage dfb371154c test(pipeline): add failing test for deterministic merge output ordering
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.

This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
2026-07-27 13:30:21 +08:00
wizardchen 4a5fc4487b fix(agent): keep batched wiki_read_page results within output budget
Batch wiki reads used to lose middle pages when the registry applied
head/tail truncation to the combined output. wiki_read_page now budgets
per page, caps neighbour link summaries, and names omitted slugs explicitly.
Also align registry truncation with rune-based limits and raise the default
tool output ceiling to 24k characters.
2026-07-27 13:22:16 +08:00
koishi 9e66389126 Add files via upload 2026-07-27 11:39:06 +08:00
xiaowang a7dc78810c fix(knowledge): clean bound vector stores in batch delete 2026-07-27 11:32:30 +08:00
wizardchen 234110b540 refactor(router): split the 2390-line router.go by domain and dedupe the file proxies
router.go now holds only engine assembly (RouterParams / NewRouter /
trustedProxies, 300 lines); the Register* functions move to domain files —
routes_auth_tenant.go, routes_knowledge.go, routes_chat.go, routes_agent.go,
routes_infra.go — and the five file-proxy surfaces (/files, KB-scoped files,
presigned, presigned-preview, /r/:token grants) move to files.go with the
frontend SPA server in static.go. Pure moves; every route, guard chain and
API-key policy declaration is unchanged, as verified by the existing route
tests and the startup policy self-check.

files.go additionally consolidates the plumbing the four handlers each
hand-rolled: LOCAL_STORAGE_BASE_DIR resolution (localStorageAbsDir),
backend-ID/provider parsing (parseStorageTarget), file_path query validation
(requireFilePathQuery), resource-catalog tenancy mapping
(resolveCatalogResource), tenant-storage resolution with and without the
global-service fallback, and the shared success response including HEAD
short-circuit (streamStoredFile).

Also reattaches the RegisterMCPServiceRoutes doc comment that had been
stranded on top of RegisterSystemAdminRoutes, drops the swagger blank import
duplicated into every split file, and documents why AllowCredentials is
inert under the wildcard CORS origin.
2026-07-27 11:25:23 +08:00
wizardchen a55b09ea82 refactor(auth): unify session-context attachment and split the Auth middleware
The five near-identical "c.Set + context.WithValue" blocks (JWT branch,
tenantless branch, both API-key attach helpers, EmbedAuth) now go through
one applyAuthSession helper, so the c.Keys surface and the request-context
surface can no longer drift apart. The 140-line JWT branch inside Auth is
split into authenticateJWTUser / resolveTargetTenant, which also removes a
duplicate GetTenantByID roundtrip on the X-Tenant-ID switch path.

Fixes folded in:
- delete GetTenantIDFromContext: it read ctx.Value("tenantID") while the
  value is stored under types.TenantIDContextKey ("TenantID"), so it could
  never succeed (dead, broken, uncalled)
- isPlatformTenantOptionalAPI: match "/api/v1/system/admin" exactly or as
  a "/"-delimited prefix instead of a bare HasPrefix that would also admit
  sibling paths sharing the prefix
- 401 for a presented-but-invalid Bearer token now says "invalid or
  expired token" instead of the misleading "missing authentication"
- replace the remaining log.Printf calls in the auth path with the
  request-scoped structured logger

New tests lock the both-surfaces invariant of applyAuthSession, the
tenantless "no tenant/role keys" contract, and bearerToken parsing.
2026-07-27 11:25:23 +08:00