A blob that a commit emptied is still a blob, and ingestion reads an item with
neither content nor URL as nothing to sync: applyFetchedItem counts it as
skipped and the sync reports success. The body indexed from an earlier commit
stays fully searchable, and no later sync revisits it — a full sync still sees
the path in the tree, and an incremental sync only walks the diffs of commits
newer than the recorded head, which the emptying commit already moved past.
Emit the file's path as the content in that case so ingestion replaces the stale
body, matching what the Notion (b9be7d0d), Confluence and IMA (b71b31c8)
connectors already do. The check is on length alone: a blob holding only
whitespace still carries content the repository has.
* fix(feishu): 修复同步请求限流与共享重试等待
* fix(feishu): 完善知识库限流冷却和导出轮询
Bound export polling by poll count so shared pacing and cooldowns retain the
same export ticket without causing false 60-second timeouts. Preserve the
feishu_timeout failure classification when the poll budget is exhausted.
Cap cooldowns, isolate reactive waits by API path, and use a configurable
10-second default when no valid retry hint is available. Refresh expired
tokens before rejoining the request gate to preserve pacing and new cooldowns.
Skip JSON decoding when the legacy rate-limit code is absent, derive the
prefilter from the shared protocol constant, and update configuration docs.
Isolate shared gates in tests and cover cooldowns, cancellation, API grouping,
concurrent token refresh, and timeout classification.
Validation: Feishu core tests pass; diff-scoped Feishu lint reports 0 issues;
gofmt and git diff --check pass. No live Feishu sync was run for this commit.
* fix(feishu): grant request gate slots in arrival order and improve rate-limit logs
- Reserve the next slot before sleeping so concurrent callers are paced in
FIFO order instead of racing after a shared wake-up.
- Restore the error log for non-rate-limited HTTP 400 download failures.
- Include the API gate name in rate-limit warnings.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(datasource/dingtalk): tighten Validate sampling edges
Keep probing later documents in the same workspace, hold listing budget
for sibling folders that are already queued, fail closed when any root
page cannot be listed, and say when the probe cap stops a conclusive walk.
Fixes#4021
Co-authored-by: fight for you <wangrui19971019@users.noreply.github.com>
* fix(datasource/dingtalk): stop Validate walk once probe cap is spent
- Break the workspace walk as soon as maxValidateProbes is spent, so a
later unlistable folder can no longer turn failed probes into acceptance.
- Treat a skipped folder tail as covered only by a probe made after the
skip, not by any earlier probe in the same workspace.
- Keep the document name on probe context errors.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: fight for you <wangrui19971019@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(datasource): omit last_sync_cursor from the data source list response
The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.
* feat(datasource): add Seafile connector with fingerprint-based incremental sync
Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.
- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils
* feat(datasource): add Seafile frontend connector and docs
Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.
- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
updates, and Seafile in the README data source lists
---------
Co-authored-by: zaqqmm <q-zhang10@qq.com>
A partial sync recorded each doc's new content_updated_at before fetching
its detail. When a detail request failed after client retries, the cursor
still held the new version, so every later incremental sync treated the
doc as unchanged and never re-fetched it (mixed success/failure batches
persist the cursor; only all-failed batches are aborted).
Rewind the cursor entry to the previously synced version (existing docs)
or drop it (new docs) on detail failure so recovery re-syncs the doc.
Deletion detection is unaffected: currentDocs is set at listing time.
Fixes#3690
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
* fix(server): signal shutdown raced Fatalf exit before ResourceCleaner ran
The signal goroutine closed the listener before calling server.Shutdown,
so Serve returned the raw accept error ('use of closed network connection')
instead of ErrServerClosed. The invoke error path then hit logger.Fatalf
(os.Exit(1)) in the same millisecond, killing the process before the
shutdown goroutine reached resourceCleaner.Cleanup. Every registered
cleaner — BrowserSkill daemon, sandbox reapers, IMService, AntsPool,
Langfuse — was skipped on every SIGTERM/SIGINT, leaking the shared bsk
daemon process and its temp home on each restart.
Two changes:
- Drop the manual listener.Close(): Shutdown closes all listeners as its
first step, releasing the port just as fast without racing Serve.
- Give resourceCleaner.Cleanup its own fresh timeout context so a slow
connection drain cannot leave Cleanup with an already-expired context,
which made it skip every entry without running any.
* fix(server): keep signal cleanup inside one grace period
Desktop quit still closed the listener before Shutdown and returned
before ResourceCleaner finished, so Fatalf could skip every hook.
Share ShutdownTimeout between drain and cleanup, run BrowserSkill
first, and do not drop later hooks when the deadline expires.
* fix(server): satisfy errcheck and line-length lint
---------
Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:
* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)
The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
* feat(datasource/yuque): preserve TOC folder hierarchy for synced documents
The Yuque connector emitted a bare "<title>.md" file name, and folder_path
is derived from FileName alone (SplitKnowledgeRelativePath ->
NormalizeKnowledgeFolderPath). With no "/" in the name the derived folder is
always empty, so every synced document landed at the knowledge base root no
matter where the book's TOC places it.
Read the book's TOC once per sync, fold it into a per-document path by walking
parent_uuid upward, and prepend the result to FileName. The GitLab connector
already works this way; this supplies the missing input rather than changing
the FileName -> folder_path machinery.
Three optional settings, all defaulting to the previous behaviour so the
emitted FileName stays byte-for-byte identical unless folder_mode is set to
"toc". A TOC failure degrades to flat names rather than failing the sync, and
the toc_only filter is skipped entirely when no TOC was fetched.
* feat(datasource/yuque): make toc_only an admission filter, drop include_book_title
Review follow-ups for the TOC folder hierarchy change.
toc_only used to run before the document was recorded for deletion detection,
so a document that was present in the source but not admitted came back as
IsDeleted — and with sync_deletions on, the sync service then deleted it from
the knowledge base. That is the wrong semantics for a key named after
filtering, and on Yuque it is unusually damaging: the web UI does not show
documents created through the API that were never attached to the TOC, so a
knowledge base is often the only place such a document can be reached at all.
A filtered document is now recorded as seen but not ingested, so enabling the
filter removes nothing. It is deliberately kept out of the cursor so that
admitting it later — by attaching it to the TOC, or by switching the filter
off — ingests it instead of dismissing it as unchanged.
include_book_title is removed. Leaving the book segment out silently merges
same-named top-level groups from different books, and the "which book"
dimension has no other carrier: the automatic tag granularity is one tag per
data source, not one per book. Under folder_mode=toc the book segment is now
unconditional; a source still carrying the key keeps syncing with the value
ignored.
Also supplies the UI the feature was missing — a Yuque-only folder_mode select
in the sync-strategy step, with newly created sources opting into the TOC
layout and existing ones left untouched — and documents both settings in the
Yuque section of the datasource guide.
* feat(datasource/yuque): expose the TOC admission filter in the datasource editor
UI follow-ups from review.
The folder-layout control now uses the pill-button group that the neighbouring
sync-mode and conflict-strategy settings already use, instead of a select — a
two-way choice was the only thing on that step rendered as a dropdown.
toc_only is exposed beside it, disabled unless the TOC layout is selected: the
connector only fetches the table of contents under folder_mode=toc, so the flag
does nothing in the flat layout, and the control says so. Its label names the
documents it admits as the ones visible in the Yuque table of contents, which
is what separates them from documents created through the API that were never
attached to one.
Drops the include_book_title paragraph from the datasource guide. That key was
never in a released build, so describing it as removed documents a history no
reader has; the tolerance test for unknown settings keys stays, without the
retired-key framing.
* docs(datasource): note Yuque folder-mode sync limitations
---------
Co-authored-by: niannian <10008660+RichardChengshaojin@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
* feat(confluence): support page subtree selection
* feat(confluence): add private image asset resolver
* fix(confluence): preserve pages during partial syncs
* fix(confluence): cap total inlined image bytes per page
Up to 30 images of 10 MB each could be inlined as base64, and every
downstream conversion copies the page again, so one extreme page could
peak above 1 GB. Cap the data URIs a page holds, in the download cache
and the rewritten HTML, at 50 MB; images past the budget keep their
original src like any other skipped image.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
* fix(datasource): preserve schedule state on invalid updates
* docs(datasource): document PUT /datasource/:id as a full replace
The update now always persists sync_schedule, so a partial body clears
the schedule. Show a full-object example and state that omitted fields
are reset rather than preserved.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
On a Server/DC instance with the #3596 behaviour every listing ends
early, so skipping deletion reconciliation for such runs meant pages
deleted in Confluence were never removed from the knowledge base.
When a listing is cut short, look up each previously synced page it did
not return with GET /rest/api/content/{id}?expand=space and delete it
only when Confluence confirms it is gone from the space: a 404, a
non-current status, or a different space. Pages that are still there,
that fail to load (including 403), or that exceed the 200-probe budget
per run stay tracked and are checked again next time.
Some Server/Data Center builds keep returning _links.next from
/rest/api/space/{key}/content/page after the last page, with empty
results (#3596). The connector followed it until the 10000-hop cap, so a
sync ran for hours, imported nothing and then failed.
Stop after three consecutive empty pages that still link onward. A
single empty page is not taken as the end, since permission filtering
can empty one window mid-listing. A listing cut short this way is
reported as incomplete: its pages are imported, pages it did not reach
stay in the cursor, and deletion reconciliation is skipped for that run
so an early stop cannot mirror-delete documents.
* feat(datasource): add Confluence sync
* fix(confluence): resume full sync and list Server pages without CQL
CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.
* fix(confluence): satisfy package lint for pre-push against main
* fix(confluence): parse Server page lists and keep version expand
The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.
* fix(confluence): classify errors and fail closed on empty listings
Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
parseYuqueConfig and parseIMAConfig validate DefaultBaseURL through
ValidateConnectorBaseURL, which resolves the host with net.LookupIP.
Their TestMain only whitelists loopback, so the config tests fail on
any machine where www.yuque.com or ima.qq.com resolves to a restricted
address (a local DNS sinkhole, for example), and the result depends on
the network the tests happen to run on.
Whitelist the default host as well, which is what the feishu tests
already do for open.feishu.cn.
* feat(datasource): persist source document creation time and use content edit time for Feishu
Follow-up to 160c806b (Refs #3016), which started persisting the source
last-modified time as metadata.source_updated_at. Two gaps remained:
1. Creation time was never carried. FetchedItem gains CreatedAt and
ingestItem persists it as metadata.source_created_at (RFC3339, UTC)
with the same absent-means-unknown rule as source_updated_at.
2. The Feishu wiki connector filled FetchedItem.UpdatedAt from
NodeEditTime, which moves when a node is renamed or dragged in the
tree without any content change. The connector's own cursor already
prefers ObjEditTime for exactly this reason; use the same rule for the
persisted value (contentEditTime / contentCreateTime, node_* as
fallback), so source_updated_at reflects document edits.
Feishu Drive populates CreatedAt from created_time. GitLab stops
fabricating UpdatedAt = time.Now(): a fetch timestamp is not a document
timestamp, and an absent value is the documented "unknown" (the commit
time is not fetched today; leaving it unset is honest until it is).
Tests: ingestItem covers both keys present/absent; the wiki fetch test
asserts obj_* times win and node_* are the fallback.
* style(gitlab): break up the FetchedItem literal (lll)
---------
Co-authored-by: xiaoyaner0201 <>
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
Notes (media_type=11) were skipped as "unsupported by wiki OpenAPI". The wiki
namespace indeed never exposes a note body, but IMA reads it from a second
namespace: get_media_info reports a notebook_id, which
/openapi/note/v1/get_doc_content turns into the note text. The connector
already parsed that notebook_id into item metadata and then dropped the
record, so this wires up the one call that was missing.
Note bodies arrive as plain text and are ingested as Markdown, since IMA notes
are authored as rich text and the export keeps its heading and list markers.
An unreadable note now takes the same retry path as a failed download instead
of being recorded as synced; empty notes are skipped.
AI sessions (12) and video parses (16) stay skipped, as those genuinely have
no read endpoint, and the connector description now names them specifically
rather than implying notes are unsupported.
Also accept the {retcode, errmsg} spelling of the response envelope alongside
{code, msg}. The reference MCP implementation confirms {code, msg} is correct,
but several published IMA references disagree, and under the wrong spelling
the status code would decode as 0 and turn every API error into a silent
success.
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.
Alongside that:
- Drop media_type from logicalKey. get_knowledge_list omits the field, so
folding it in meant every external_id would shift the day IMA starts
populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.