Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.
Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
Markdown tables — searchable text, instead of being lost or (post-export)
rendered as unsearchable images. Bitable date/datetime cells render in the
table's timezone (configurable via Settings["timezone"], default GMT+8) rather
than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
pipeline, gated by the KB's multimodal setting: the image's external_id always
stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
records) break on an empty page so a malformed has_more response cannot spin
until the task deadline.
Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
"<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
means a single failed attachment can't wipe its siblings, and dupIsSameNode
guards the sweep so a file_hash-only dedup collision against a different node
cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
(never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
metadata external_id prefix scan.
Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.
Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.
This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.
Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
UnresolvedToolHandles)
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:
- Ingest dedup now groups each new item with only its own trigram
candidates and validates merges against that per-item scope, rejecting
cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
system-owned summary pages, and auto-repairs dead [[slug]] links
(rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
storage handles so they round-trip safely through chat.
Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.
Fixes#2238
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
Run the synchronous best-effort queue scrub in DeleteKnowledgeBase with
context.WithoutCancel and kbTaskCleanupTimeout so client disconnects
cannot truncate cancellation mid-scan, matching ProcessKBDelete cleanup
semantics.
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.
This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
- Updated the exportFAQEntries function to support exporting FAQ entries in both CSV and JSON formats.
- Modified the frontend to include options for selecting the export format.
- Enhanced localization files to reflect the new export options in English, Russian, and Chinese.
- Implemented new tests to validate the export functionality for both formats.
This commit improves the flexibility of FAQ exports, allowing users to choose their preferred format for better integration with other tools.
- Added new fields for tracking added, merged, and partially failed entries during FAQ imports.
- Updated the import result statistics to include detailed counts for added and merged entries.
- Enhanced the UI to display progress hints and improved import result summaries.
- Implemented a new method for saving chunks in a single transaction to improve performance.
This commit improves the visibility and accuracy of FAQ import results, providing better user feedback and tracking capabilities.
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.
This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.
Automatically remove wiki folders that become empty when knowledge is
retracted, using the durable finalize lane so pruning waits until ingest
drains. Repository DeleteFolder now atomically checks emptiness to avoid
races with concurrent page or child-folder writes.
Two backend fixes for the v0.7.0 regression where a deleted document keeps
showing in the list ('delete success' toast) and a follow-up reparse then
reports the file is missing:
1. List filtering: applyKnowledgeListFilter now hides rows in the transient
'deleting' state from the default document list. The async delete pipeline
marks a row 'deleting' before tearing down its resources, so without this
the entry lingers as a normal doc until it is physically gone. Rows whose
delete task exhausts retries are flipped to 'failed' by the dead-letter
callback and stay visible so the failure remains actionable; an explicit
parse_status=deleting filter can still surface in-flight rows.
2. Delete ordering: DeleteKnowledge / DeleteKnowledgeList now delete the DB
row FIRST and defer physical file (+ extracted image) cleanup until after
the row is gone. Previously file removal ran concurrently with the
index/chunk/graph cleanup; when one of those failed the row was kept but
the file could already be deleted, producing a 'file missing but row
present' zombie that could neither be reparsed nor cleanly re-deleted.
Orphaning a file after the row is gone is the tolerable failure mode.
Adds a repository regression test covering the default/explicit deleting
visibility rules.
Store authorization timestamps as timestamptz and normalize Unix expiry
inputs to UTC so tenant/platform API keys expire at the intended instant
even when the app process timezone differs from the PostgreSQL session.
Cloned images were unreachable via GET /knowledge-bases/:id/files:
copyOwnedObject used CopyFile, which writes to the knowledge-scoped upload
layout ({tenant}/{knowledgeID}/...), but the KB proxy only serves objects
under {tenant}/exports/ (ValidateKBScopedStoragePath). Read the source
bytes and re-save via SaveBytes instead, mirroring how images are
persisted during ingestion, so copies land in the servable exports
namespace and remain independent of the source object.
For document knowledge, extracted images live in independent image_ocr/
image_caption child chunks that carry image_info, while the parent text
chunk embeds the  reference with an empty image_info. Rewriting
content per-chunk therefore missed the parent text chunk, leaving it
pointing at soon-to-be-deleted source storage.
Move content URL rewriting to a final pass over all cloned chunks, run
once urlCache holds the complete old->new mapping. FAQ chunks are
self-contained, so they are rewritten inline from the same cache.
When cloning knowledge chunks, image_info was deep-copied to new storage
objects but Content still embedded the source markdown image URLs. After
the source knowledge is deleted, those in-content references break while
image_info points at the new copies. Rewrite Content alongside image_info
so both stay consistent.
- Added AminusB method to the knowledgeRepository, which returns IDs of knowledge in A that have no counterpart in B, comparing by file_hash as a multiset.
- This implementation addresses issues with handling duplicate entries and NULL file_hash values, ensuring accurate synchronization between knowledge bases.
- Updated the KnowledgeRepository interface to reflect the new functionality and improved documentation for clarity.
Introduce platform API keys with capability-based access to tenant management,
system settings, runtime queues, and audit logs, plus admin UI for key lifecycle
and system audit log viewing.
Large Feishu wikis stopped syncing after the first batch (Tencent/WeKnora#2136:
5000+ docs, only ~150 ingested; #1262: a child-list failure aborted the run).
Three compounding causes, fixed in layers:
1. Rate-limit / transient failures had no retry. Feishu's wiki/drive/export
APIs are aggressively rate limited; a thousand-doc sync issues tens of
thousands of calls, and a single 429/5xx burst failed whole swathes of docs.
Add exponential backoff honouring Retry-After to doRequest (JSON) and
downloadRawBytes (files): 429 up to 3x, 5xx once, transport errors back off.
2. The batch path fetched the entire tree into memory before ingesting, so a
sync that hit the 2h task timeout lost all progress and restarted from
scratch, and a transient per-document export failure advanced the cursor and
skipped that doc permanently. Add an optional StreamingConnector interface,
implemented for Feishu: FetchStream interleaves fetch -> ingest -> checkpoint,
bounding memory to one item and persisting a per-node edit-time cursor at
page boundaries. The service dispatches to it via type assertion; connectors
without it fall back to FetchAll / FetchIncremental unchanged (non-breaking).
A resumed/incremental run skips nodes already recorded at their current edit
time (converges), a failed fetch does not advance the cursor (retried next
run instead of dropped), and partial listing failures emit error items and
carry prior edit times forward instead of aborting the sync.
3. Failures and skips were invisible. Unsupported node types (mindnote/slides/…)
were skipped with no item, error or log ("13 docs synced only 3"). FetchAll
now tallies discovered/fetched/failed/skipped per resource with a per-type
skip breakdown and a progress line every 100 nodes. The backend already
persisted per-item errors in SyncLog.result.errors but the frontend type
omitted the field; declare `result` on SyncLog and render up to 50 failed
documents in the sync-log drawer (#1262). i18n for zh-CN/en-US/ko-KR/ru-RU.
result.errors is a jsonb column shipped in every list response, so it is
capped at 100 samples server-side (recordSyncError); the exact failure total
comes from the bounded items_failed counter, which the drawer uses for its
count and "N more" line so a sync that fails thousands of docs stays bounded.
Each sample is a structured, localisable SyncItemError instead of a raw
"title: ...api error: status=500 body={json}" dump that repeated the title
and leaked status/body/log_id. The connector (feishuFailure) classifies the
error into a stable i18n code (+ the feishu code as a param); the service
builds {title, code, params, message}; the drawer localises via vue-i18n to
the viewer's language (zh/en/ko/ru), falling back to message. A custom
UnmarshalJSON keeps old string-form logs readable. The banner is localised
from items_failed too. Raw errors stay in the server logs. Follows the
two-tier norm of Airbyte/Fivetran/Onyx: human, actionable, localised
summaries in the UI; raw detail in logs. Transient errors say "will retry"
(the cursor is retained); auth/permission errors point at the fix instead.
Tests: client_retry_test (backoff), tally_test (accounting), connector_stream /
datasource_stream (streaming dispatch + checkpoint), connector_convergence
(timeout + transient-failure resume converges to full coverage; proven to catch
#2136 via bug injection), datasource_result_cap (error-sample cap + structured
localisable SyncItemError, no raw body), connector_error_reason (error→i18n-code
classification), sync_item_error (legacy-string + structured back-compat) — all
in the normal suite. connector_realapi / connector_seed are an opt-in real-Feishu
harness gated by the feishu_integration build tag AND env vars, never run in CI.
Restrict channel-managed sessions (API keys, IM, embed) to tenant admin
in the Web console for list/read paths, hide channel sidebar buckets from
non-admins, and add CountSessionsBySource for aggregate stats without
exposing session rows to viewers.
Add GetOwnedSession method to enforce strict ownership checks for session mutations, ensuring tenant admins can read but not modify API-key sessions. Update relevant handlers to utilize this method for operations that alter session data, maintaining security and integrity of API-key sessions.
Let tenant admins list and open API-key sessions from the Web console while
keeping per-key isolation for everyone else. Pin the session source filter so
it no longer jumps when switching buckets, and resolve session origin on read
so deep links stay in sync with the active folder after refresh.
Use bounded detached contexts for wiki ingest tail cleanup so pending row
settlement, span closure, finalize enqueueing, and follow-up scheduling can
finish even after the asynq task context is cancelled.
Return settlement errors from pending row trimming and failed-op requeueing,
and only mark claims settled after rows are deleted, released, or
dead-lettered successfully. Requeue documents whose reduce updates were not
applied after context cancellation to avoid silently dropping wiki work.
Add unit coverage for detached cleanup contexts and settlement error handling.
Decouple the references SSE event from the per-agent citation output toggle
so the retrieval-results drawer still works, and align follow-up suggestion
card layout with the chat column.
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
Move citation protocol injection to the system layer so custom prompts no
longer own citation syntax. Agents can disable inline citations while source
aliases still encode and decode tool arguments correctly.
Replace raw chunk/document/KB IDs in LLM context with short cN/dN/bN/wN
aliases, decode tool arguments on the way in, and expand compact <ref/>
citations to the public <kb>/<web> format before user-facing output.
Add functionality to normalize temporary attachment IDs, ensuring deduplication and enforcing a maximum limit of 5 attachments per message. Update related handlers and tests to support this new behavior, enhancing the robustness of attachment management in chat sessions.
Persist resolved temporary-attachment content onto the stored user message
so later turns can replay it in history. Close the retrieval progress spinner
on early pipeline exits, and reshape model-fallback messages with a proper
system role plus trailing user question.