* fix(datasource): omit last_sync_cursor from the data source list response
The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.
* feat(datasource): add Seafile connector with fingerprint-based incremental sync
Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.
- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils
* feat(datasource): add Seafile frontend connector and docs
Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.
- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
updates, and Seafile in the README data source lists
---------
Co-authored-by: zaqqmm <q-zhang10@qq.com>
A partial sync recorded each doc's new content_updated_at before fetching
its detail. When a detail request failed after client retries, the cursor
still held the new version, so every later incremental sync treated the
doc as unchanged and never re-fetched it (mixed success/failure batches
persist the cursor; only all-failed batches are aborted).
Rewind the cursor entry to the previously synced version (existing docs)
or drop it (new docs) on detail failure so recovery re-syncs the doc.
Deletion detection is unaffected: currentDocs is set at listing time.
Fixes#3690
Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
* fix(server): signal shutdown raced Fatalf exit before ResourceCleaner ran
The signal goroutine closed the listener before calling server.Shutdown,
so Serve returned the raw accept error ('use of closed network connection')
instead of ErrServerClosed. The invoke error path then hit logger.Fatalf
(os.Exit(1)) in the same millisecond, killing the process before the
shutdown goroutine reached resourceCleaner.Cleanup. Every registered
cleaner — BrowserSkill daemon, sandbox reapers, IMService, AntsPool,
Langfuse — was skipped on every SIGTERM/SIGINT, leaking the shared bsk
daemon process and its temp home on each restart.
Two changes:
- Drop the manual listener.Close(): Shutdown closes all listeners as its
first step, releasing the port just as fast without racing Serve.
- Give resourceCleaner.Cleanup its own fresh timeout context so a slow
connection drain cannot leave Cleanup with an already-expired context,
which made it skip every entry without running any.
* fix(server): keep signal cleanup inside one grace period
Desktop quit still closed the listener before Shutdown and returned
before ResourceCleaner finished, so Fatalf could skip every hook.
Share ShutdownTimeout between drain and cleanup, run BrowserSkill
first, and do not drop later hooks when the deadline expires.
* fix(server): satisfy errcheck and line-length lint
---------
Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:
* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)
The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
* feat(datasource/yuque): preserve TOC folder hierarchy for synced documents
The Yuque connector emitted a bare "<title>.md" file name, and folder_path
is derived from FileName alone (SplitKnowledgeRelativePath ->
NormalizeKnowledgeFolderPath). With no "/" in the name the derived folder is
always empty, so every synced document landed at the knowledge base root no
matter where the book's TOC places it.
Read the book's TOC once per sync, fold it into a per-document path by walking
parent_uuid upward, and prepend the result to FileName. The GitLab connector
already works this way; this supplies the missing input rather than changing
the FileName -> folder_path machinery.
Three optional settings, all defaulting to the previous behaviour so the
emitted FileName stays byte-for-byte identical unless folder_mode is set to
"toc". A TOC failure degrades to flat names rather than failing the sync, and
the toc_only filter is skipped entirely when no TOC was fetched.
* feat(datasource/yuque): make toc_only an admission filter, drop include_book_title
Review follow-ups for the TOC folder hierarchy change.
toc_only used to run before the document was recorded for deletion detection,
so a document that was present in the source but not admitted came back as
IsDeleted — and with sync_deletions on, the sync service then deleted it from
the knowledge base. That is the wrong semantics for a key named after
filtering, and on Yuque it is unusually damaging: the web UI does not show
documents created through the API that were never attached to the TOC, so a
knowledge base is often the only place such a document can be reached at all.
A filtered document is now recorded as seen but not ingested, so enabling the
filter removes nothing. It is deliberately kept out of the cursor so that
admitting it later — by attaching it to the TOC, or by switching the filter
off — ingests it instead of dismissing it as unchanged.
include_book_title is removed. Leaving the book segment out silently merges
same-named top-level groups from different books, and the "which book"
dimension has no other carrier: the automatic tag granularity is one tag per
data source, not one per book. Under folder_mode=toc the book segment is now
unconditional; a source still carrying the key keeps syncing with the value
ignored.
Also supplies the UI the feature was missing — a Yuque-only folder_mode select
in the sync-strategy step, with newly created sources opting into the TOC
layout and existing ones left untouched — and documents both settings in the
Yuque section of the datasource guide.
* feat(datasource/yuque): expose the TOC admission filter in the datasource editor
UI follow-ups from review.
The folder-layout control now uses the pill-button group that the neighbouring
sync-mode and conflict-strategy settings already use, instead of a select — a
two-way choice was the only thing on that step rendered as a dropdown.
toc_only is exposed beside it, disabled unless the TOC layout is selected: the
connector only fetches the table of contents under folder_mode=toc, so the flag
does nothing in the flat layout, and the control says so. Its label names the
documents it admits as the ones visible in the Yuque table of contents, which
is what separates them from documents created through the API that were never
attached to one.
Drops the include_book_title paragraph from the datasource guide. That key was
never in a released build, so describing it as removed documents a history no
reader has; the tolerance test for unknown settings keys stays, without the
retired-key framing.
* docs(datasource): note Yuque folder-mode sync limitations
---------
Co-authored-by: niannian <10008660+RichardChengshaojin@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
* feat(confluence): support page subtree selection
* feat(confluence): add private image asset resolver
* fix(confluence): preserve pages during partial syncs
* fix(confluence): cap total inlined image bytes per page
Up to 30 images of 10 MB each could be inlined as base64, and every
downstream conversion copies the page again, so one extreme page could
peak above 1 GB. Cap the data URIs a page holds, in the download cache
and the rewritten HTML, at 50 MB; images past the budget keep their
original src like any other skipped image.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
* fix(datasource): preserve schedule state on invalid updates
* docs(datasource): document PUT /datasource/:id as a full replace
The update now always persists sync_schedule, so a partial body clears
the schedule. Show a full-object example and state that omitted fields
are reset rather than preserved.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
On a Server/DC instance with the #3596 behaviour every listing ends
early, so skipping deletion reconciliation for such runs meant pages
deleted in Confluence were never removed from the knowledge base.
When a listing is cut short, look up each previously synced page it did
not return with GET /rest/api/content/{id}?expand=space and delete it
only when Confluence confirms it is gone from the space: a 404, a
non-current status, or a different space. Pages that are still there,
that fail to load (including 403), or that exceed the 200-probe budget
per run stay tracked and are checked again next time.
Some Server/Data Center builds keep returning _links.next from
/rest/api/space/{key}/content/page after the last page, with empty
results (#3596). The connector followed it until the 10000-hop cap, so a
sync ran for hours, imported nothing and then failed.
Stop after three consecutive empty pages that still link onward. A
single empty page is not taken as the end, since permission filtering
can empty one window mid-listing. A listing cut short this way is
reported as incomplete: its pages are imported, pages it did not reach
stay in the cursor, and deletion reconciliation is skipped for that run
so an early stop cannot mirror-delete documents.
* feat(datasource): add Confluence sync
* fix(confluence): resume full sync and list Server pages without CQL
CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.
* fix(confluence): satisfy package lint for pre-push against main
* fix(confluence): parse Server page lists and keep version expand
The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.
* fix(confluence): classify errors and fail closed on empty listings
Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
parseYuqueConfig and parseIMAConfig validate DefaultBaseURL through
ValidateConnectorBaseURL, which resolves the host with net.LookupIP.
Their TestMain only whitelists loopback, so the config tests fail on
any machine where www.yuque.com or ima.qq.com resolves to a restricted
address (a local DNS sinkhole, for example), and the result depends on
the network the tests happen to run on.
Whitelist the default host as well, which is what the feishu tests
already do for open.feishu.cn.
* feat(datasource): persist source document creation time and use content edit time for Feishu
Follow-up to 160c806b (Refs #3016), which started persisting the source
last-modified time as metadata.source_updated_at. Two gaps remained:
1. Creation time was never carried. FetchedItem gains CreatedAt and
ingestItem persists it as metadata.source_created_at (RFC3339, UTC)
with the same absent-means-unknown rule as source_updated_at.
2. The Feishu wiki connector filled FetchedItem.UpdatedAt from
NodeEditTime, which moves when a node is renamed or dragged in the
tree without any content change. The connector's own cursor already
prefers ObjEditTime for exactly this reason; use the same rule for the
persisted value (contentEditTime / contentCreateTime, node_* as
fallback), so source_updated_at reflects document edits.
Feishu Drive populates CreatedAt from created_time. GitLab stops
fabricating UpdatedAt = time.Now(): a fetch timestamp is not a document
timestamp, and an absent value is the documented "unknown" (the commit
time is not fetched today; leaving it unset is honest until it is).
Tests: ingestItem covers both keys present/absent; the wiki fetch test
asserts obj_* times win and node_* are the fallback.
* style(gitlab): break up the FetchedItem literal (lll)
---------
Co-authored-by: xiaoyaner0201 <>
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
Notes (media_type=11) were skipped as "unsupported by wiki OpenAPI". The wiki
namespace indeed never exposes a note body, but IMA reads it from a second
namespace: get_media_info reports a notebook_id, which
/openapi/note/v1/get_doc_content turns into the note text. The connector
already parsed that notebook_id into item metadata and then dropped the
record, so this wires up the one call that was missing.
Note bodies arrive as plain text and are ingested as Markdown, since IMA notes
are authored as rich text and the export keeps its heading and list markers.
An unreadable note now takes the same retry path as a failed download instead
of being recorded as synced; empty notes are skipped.
AI sessions (12) and video parses (16) stay skipped, as those genuinely have
no read endpoint, and the connector description now names them specifically
rather than implying notes are unsupported.
Also accept the {retcode, errmsg} spelling of the response envelope alongside
{code, msg}. The reference MCP implementation confirms {code, msg} is correct,
but several published IMA references disagree, and under the wrong spelling
the status code would decode as 0 and turn every API error into a silent
success.
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.
Alongside that:
- Drop media_type from logicalKey. get_knowledge_list omits the field, so
folding it in meant every external_id would shift the day IMA starts
populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.
* feat(datasource): add Feishu/Lark Drive (云盘) connector
Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.
Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
/ feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
(FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
the wiki connector's three cursor semantics (resume fast-path,
failure-does-not-advance-cursor, toSyncCursor snapshot), plus
ListResources / ResolveResourceAncestors (top-down shared traversal,
best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata
Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".
Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
Drive root folder_token input + load button before the lazy-load tree,
edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
input group
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): refine Feishu Drive connector UI, channel and root folder
A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:
Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
"飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.
Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
switch); the tree area shows a "load a folder first" placeholder until
the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
actionable i18n hints.
Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
to resolve the root folder's human-readable name; the list API only
returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
suffix) so it matches the saved resource_id form and the direct
children's ParentID, which previously broke selection matching on edit.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): sync a single selected Drive file without folder error
When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".
The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.
A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks extraction design spec
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks implementation plan
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): sync feishu drive docx via blocks API with export fallback
- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
sub-items), falling back to the export API on blocks errors or empty
renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): label feishu drive knowledge as 飞书云盘 in list
The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): polish feishu drive resource-step UI in datasource editor
- Replace the two stacked t-alert banners under the folder_token input:
shareHint becomes an always-visible form-desc line, rootNotSupportedHint
moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
status + tips on the input (replacing the global MessagePlugin) when
empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
fixed and the resource region (placeholder / loading / empty / tree)
fills the remaining drawer height, with the tree scrolling internally.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu drive datasource integration guide
User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine
Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.
Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.
Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): split feishu connector into core/wiki/drive packages
Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.
container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).
Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).
Co-Authored-By: Claude <noreply@anthropic.com>
* test(datasource): migrate feishu tests to core/wiki/drive packages
Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.
All tests pass: core 4s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test
Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).
Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.
All tests pass: core 3.5s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): repair feishu cleanup fallout
Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.
All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(frontend): feishu drive data source UI and i18n
Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: add trellis journal merge=union rule
Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): rename feishu/drive drive_connector.go to connector.go
Aligns naming with feishu/wiki subpackage (connector.go).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): allow data:image base64 URIs in sanitized markdown
DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path
The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export
Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.
- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
they don't break under the new default
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout
- 飞书云盘接入说明: add section on docx parse mode (export default vs
blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
add pros/cons table, fix source-file table to core/wiki/drive layout
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: 修复合并main后丢失新增的国际化问题
* fix(i18n): 修复误删的国际化内容
* docs: 删除误提交的文件
---------
Co-authored-by: Claude <noreply@anthropic.com>
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.
Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
Markdown tables — searchable text, instead of being lost or (post-export)
rendered as unsearchable images. Bitable date/datetime cells render in the
table's timezone (configurable via Settings["timezone"], default GMT+8) rather
than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
pipeline, gated by the KB's multimodal setting: the image's external_id always
stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
records) break on an empty page so a malformed has_more response cannot spin
until the task deadline.
Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
"<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
means a single failed attachment can't wipe its siblings, and dupIsSameNode
guards the sweep so a file_hash-only dedup collision against a different node
cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
(never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
metadata external_id prefix scan.
Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.