Commit Graph

814 Commits (37837c00a92d976efe2373c74bd3c232bea8a2aa)

Author SHA1 Message Date
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00
wizardchen ab9f705843 fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
wizardchen db9bdc27cd fix(session): scope API runtime admin-console bypass to owned sessions
Replace the broad channel-runtime principal check with owner-scoped rules so
API callers can still read their own api_tenant_key and api_external_user
sessions without opening IM sessions matched via legacy empty user_id scope.
2026-07-30 11:56:05 +08:00
wizardchen 123f2d6349 fix(session): include external users in API session group 2026-07-30 11:56:05 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wizardchen c291c6a59f fix(kb): retry KB delete when engine resolution is deferred
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
2026-07-29 12:25:18 +08:00
ochan.kwon c15c3e972f fix(retriever): rebuild a missing vector store engine on demand
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:

  - Startup skips a store whose engine fails to construct and never
    retries it, so that instance cannot serve the store again -- not
    even after a restart, if the backend is slow to come up.
  - A store registered on one instance is absent on every other until
    they restart, because registration is not broadcast.
  - Registration can fail on the instance handling the create, leaving
    the store missing even there.

Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.

Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.

Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.

Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.

Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.

For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.

A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.

Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.

Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
2026-07-29 12:25:18 +08:00
wizardchen bf68db4653 fix(storage): allow image uploads to use file service fallback 2026-07-29 11:36:24 +08:00
wizardchen 5db186a0fc fix(knowledge): harden file-type scoped duplicate detection
Use case-insensitive file_type matching, scope the filename fallback by type, and document the dedup contract for datasource sweep handling.
2026-07-29 10:36:17 +08:00
koishi514-Z 18b95ebb3a fix(knowledge): distinguish duplicate files by type 2026-07-29 10:36:17 +08:00
wizardchen 2358564bde fix(knowledge): default custom metadata on create 2026-07-28 22:22:59 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
koishi514-Z 0e1e5a63ba fix(chat): prevent duplicate terminal answer events 2026-07-28 21:33:33 +08:00
xiaowang 82d3e0b271 fix(wiki): prevent finalizing stalls on enqueue failure 2026-07-28 20:54:48 +08:00
zhangzherui eb447365cd fix(knowledge): clear stale errors on reparse 2026-07-28 20:44:34 +08:00
wizardchen 497e8312e3 fix(knowledge): harden summary refresh after review feedback
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 20:23:48 +08:00
wizardchen 0766c5671f fix(retrieval): restore image parent context 2026-07-28 20:23:48 +08:00
wizardchen 48b661134b fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
wizardchen 6a439a4c3b fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
wizardchen 850a073c83 fix(summary): restore tenant context for refresh tasks 2026-07-28 20:23:48 +08:00
wizardchen 1b9858ecb8 fix(knowledge): enqueue stale summary refreshes 2026-07-28 20:23:48 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen 95775ae909 fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
wizardchen 80a5003cc9 feat(wiki): add page revision history with diff, revert, and manual editing
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
2026-07-27 15:03:24 +08:00
wizardchen 1d64fc6234 fix(knowledge): reconcile wiki state when moving documents across KBs
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
2026-07-27 15:02:02 +08:00
MidoriKurage 3b476c07df fix(pipeline): deterministic merge output ordering
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.

Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
2026-07-27 13:30:21 +08:00
MidoriKurage dfb371154c test(pipeline): add failing test for deterministic merge output ordering
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.

This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
2026-07-27 13:30:21 +08:00
xiaowang a7dc78810c fix(knowledge): clean bound vector stores in batch delete 2026-07-27 11:32:30 +08:00
wizardchen 718b7347d0 fix(agent): harden search-target scope and model-handle policies
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
2026-07-27 11:05:43 +08:00
wizardchen 5e413c6478 refactor(agent): introduce modelcontext registry and fix handle-codec bugs
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.

Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
  UnresolvedToolHandles)
2026-07-27 11:05:43 +08:00
wizardchen 5d0326f987 fix(wiki): harden slug integrity across ingest and agent write paths
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:

- Ingest dedup now groups each new item with only its own trigram
  candidates and validates merges against that per-item scope, rejecting
  cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
  translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
  system-owned summary pages, and auto-repairs dead [[slug]] links
  (rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
  storage handles so they round-trip safely through chat.

Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
2026-07-24 15:57:08 +08:00
wizardchen d283fabc99 fix(doris): harden Stream Load HTTP with SSRF-safe client
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
2026-07-24 15:10:44 +08:00
Jack Wang 43839f7830 fix(agent): honor suggested-questions limit parameter
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.

Fixes #2238
2026-07-24 10:46:53 +08:00
wizardchen 6fb858105a feat(llm): add prompt-cache observability and wiki cache-friendly prompts
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
2026-07-23 19:17:30 +08:00
ttommybot a287808597 fix: prevent API keys from assigning owner role 2026-07-23 16:04:42 +08:00
wizardchen 8ae82573ab fix(mcp): coordinate OAuth token refresh and authorization polling
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
2026-07-23 15:35:24 +08:00
wizardchen 986a6f3699 fix(tasks): detach KB deletion queue cleanup from request ctx
Run the synchronous best-effort queue scrub in DeleteKnowledgeBase with
context.WithoutCancel and kbTaskCleanupTimeout so client disconnects
cannot truncate cancellation mid-scan, matching ProcessKBDelete cleanup
semantics.
2026-07-23 12:20:03 +08:00
zhangzherui 37de0afad0 fix(tasks): dequeue work for deleted knowledge bases 2026-07-23 12:17:14 +08:00
wizardchen 8ca9c117bb feat(knowledge): enhance FAQ entry filtering and tagging
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.

This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
2026-07-23 10:50:32 +08:00
wizardchen b6a3b347da feat(knowledge): enhance FAQ export functionality
- Updated the exportFAQEntries function to support exporting FAQ entries in both CSV and JSON formats.
- Modified the frontend to include options for selecting the export format.
- Enhanced localization files to reflect the new export options in English, Russian, and Chinese.
- Implemented new tests to validate the export functionality for both formats.

This commit improves the flexibility of FAQ exports, allowing users to choose their preferred format for better integration with other tools.
2026-07-23 10:50:32 +08:00
wizardchen 135033e6df feat(knowledge): enhance FAQ import result tracking and UI
- Added new fields for tracking added, merged, and partially failed entries during FAQ imports.
- Updated the import result statistics to include detailed counts for added and merged entries.
- Enhanced the UI to display progress hints and improved import result summaries.
- Implemented a new method for saving chunks in a single transaction to improve performance.

This commit improves the visibility and accuracy of FAQ import results, providing better user feedback and tracking capabilities.
2026-07-23 10:50:32 +08:00
wizardchen 5af8eae81d feat(knowledge): enhance activity logging for FAQ imports
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.

This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.
2026-07-23 10:50:32 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
wolfkill f195e0494c fix(wiki): support orphan count on sqlite 2026-07-23 10:44:08 +08:00
Changjin Li 8248752e62 修复共享知识库文档预览路径 2026-07-23 10:43:31 +08:00