Commit Graph

92 Commits (main)

Author SHA1 Message Date
davien 2c0d838439
fix(feishu): stop logging tenant access token fragments (#4399) 2026-10-17 23:25:02 +08:00
linus-liu-web 26ff988823
fix(notion): 递归子块分页补重复游标与跳数上限,兜底不再被空页绕过 (#4411) 2026-10-17 23:17:21 +08:00
linus-liu-web b2e81a9ac4
fix(notion): 附件下载失败中止本轮不再推进游标,附件不再永久缺失 (#4413) 2026-10-17 23:17:16 +08:00
linus-liu-web 4d767d9dc4
fix(yuque): 两处 offset 分页补跳数上限、重复页检测与 meta.total 提前结束 (#4415) 2026-10-17 23:17:11 +08:00
linus-liu-web d92728d88c
fix(datasource/dingtalk): Validate 只校验所选范围,不再扫全租户 (#4421) 2026-10-17 23:17:02 +08:00
linus-liu-web e849eaad5c
feat(datasource/dingtalk): 保留来源目录结构(开关控制,默认关) (#4422) 2026-10-17 23:16:57 +08:00
davien b373374bf3
fix(feishu): keep nested text in native table cells (#4400) 2026-10-17 23:00:01 +08:00
lyingbug 0d7516ea0a
feat(assistant): expand management operations and polish interaction UI (#4427)
* fix(assistant): polish interaction cards and stabilize composer context

Localize operation labels across all six UI languages and expose answer copying in the assistant panel. Refine question and page-link cards with a single open/return action, and keep the current-page context row at a stable height.

* feat(assistant): add common data source, chunk and knowledge base operations

* feat(assistant): support parsing configuration and batch FAQ updates

* feat(assistant): add MCP tool directory query and sync operations

* feat(assistant): manage web search providers and data source sync settings

* feat(assistant): add model configuration copy operation

* feat(assistant): add workspace profile members and system information operations

* feat(assistant): add system administrator queries and runtime diagnostics

* fix(frontend): polish assistant and resource card UI

Refine assistant composer layout, context controls and reply actions, including narrow-window alignment. Unify knowledge base, agent and shared-space cards with consistent icons, hover feedback, favorites, menus and compact localized statistics.

* fix(assistant): preserve concurrent settings and exclude upstream error secrets
2026-10-17 14:35:58 +08:00
Greenplumwine 973d9d051f
feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects (#3497)
* ci(build): add manual docker build workflow

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects

- folder mirror: wiki/drive connectors emit path-qualified FileName so
  knowledge lands in KB folders mirroring the Feishu tree; shortcut
  subtrees deduped; sanitized directory names; legacy datasources
  converge via one-shot resync marker (incremental sync upgrades to
  full once, then clears)
- parse mode: replace FEISHU_DOCX_PARSE_MODE env with per-datasource
  settings.parse_mode (blocks default, export escape hatch); frontend
  selector + 5 locales; zero-migration read-time fallback
- block-level GFM: extend docx block coverage (callout/grid/iframe/
  board/mindnote/agenda/synced blocks), rich text elements (links,
  inline code, KaTeX equations, mentions), real ordered-list numbering,
  code language mapping, merged-cell fill, truncation note
- embedded objects: images (gated off multimodal toggle) and boards
  (download_as_image, 403-degraded) stored as knowledge with
  weknora-img://<n> markers resolved in the document-processing
  pipeline via knowledge FilePath; document attachments whitelisted
  (pdf/md/xls(x)/csv/doc(x)/ppt(x)) as standalone knowledge in the
  parent folder with bidirectional parent/child metadata and filename
  list in the body; videos skipped with placeholder + source link
- golden baseline explicitly re-recorded; docs (.env.example,
  docker-compose, docs/*, website-docs) synced

* fix(datasource): address Feishu deep-adaptation smoke-test findings

- folder mapping: drop the space/root-folder name prefix — the selected
  sync root maps to the knowledge-base root, subdirectories stay relative
- docx blocks: render view(33) blocks silently as containers (children
  only), removing the stray '[飞书块: 视图]' placeholder before file lists
- sheet inline tables: fetch merge ranges via sheets/v3 sheets/query and
  fill the top-left value across the merged region (closed-interval
  interpretation, flagged for real-environment verification)
- knowledge list: hide embedded image/board rows (metadata
  embedded_image=true) from the document list and folder-tree counts;
  rows remain for VLM indexing and marker resolution; search untouched
- frontend: hydrate protected images (resource:// etc.) in the document
  preview so embedded Feishu images/boards actually render

* fix(datasource): resolve embedded image markers at ingest time

The weknora-img://<n> markers were only resolved during document
processing, so chunks carried real resource:// URLs while the stored
markdown file and source_content kept raw markers — file preview
showed blank images. Resolve them in ingestItem before
CreateKnowledgeFromFile (children are emitted first, so image rows
with FilePath already exist); markers with missing children are kept
for the ProcessDocument fallback to degrade. Stored file,
source_content and chunks now agree. Existing datasources converge on
the next full re-sync (update=delete+recreate).

* feat(datasource): resolve Feishu mention users to real names at ingest

docx mention_user elements only carry an open_id, so they rendered as
a generic '@成员'. Resolve the display name via
GET /contact/v3/users/:id (user_id_type=open_id) while generating the
markdown during FetchDocxWithBlocks — before storage, per user
decision; the frontend performs no lookups. A per-sync closure caches
positive and negative lookups; permission errors and empty names
silently degrade to '@成员' (debug log only, ingest never blocked).
Requires one of the contact read scopes (e.g.
contact:user.base:readonly) on the Feishu app.

* refactor(datasource): polish round — defect fixes and structural cleanup

- attachment placeholder patching now targets the unique
  '- weknora-att://<token>' marker line rendered per attachment instead of
  matching by display name, so an identical earlier bullet is never
  replaced by mistake (regression-tested)
- markdown link targets with spaces or parens are percent-escaped
  (iframe / mention_doc / inline links), keeping them parseable
- drive folder-tree walking consolidated into core.WalkDriveTree,
  deleting the 174-line copy in drive/paths.go
- parse mode threading made explicit: DocxFetchInput.ParseMode replaces
  the private client field (connector entry resolves via
  ParseFeishuConfig and passes it through ops)
- maxFeishuDownloadBytes restored to a const; the oversize-degrade test
  now declares an over-cap Content-Length and downloadRawBytes rejects
  early on the header

* refactor(datasource): align Feishu image handling with the platform image pipeline

Replace the image-as-separate-knowledge-row channel with the platform's
orthodox flow: the connector downloads image/board bytes at fetch time and
embeds them in the parent markdown as base64 data URIs; the document
processing image resolver (image_resolver.ResolveAndStore, the same stage
used for parsed PDFs/Word) stores them on the tenant file service, swaps in
persistent provider:// URLs and produces OCR/caption child chunks when a VLM
is configured.

Removed with the marker pipeline:
- weknora-img:// ingest resolution in ingestItem
- the ProcessDocument Step-1.9 marker hook
- the document-list/folder-tree filter for embedded_image rows
- the feishu-specific runtime parse_mode backfill; legacy data sources are
  now stamped by migration 000096 (sqlite 000017) which writes the one-shot
  resync_required marker — the generic full-upgrade + clear consumption stays

Attachment (PDF/Word/md) sub-items are unchanged.

* style: gofmt test files

* fix(datasource): resolve Drive sub-selection kind from the root listing

A single-file Drive selection (rootFolderToken:fileToken) was probed with
folder APIs before falling back: folder meta returned 91202 and the folder
list 1061002 on every sync, logging API errors for a working configuration.

listDriveFilesForResource now walks the root subtree first and reads the
selected node's kind off the listing: a file filters straight out of the
already-fetched results (zero error calls), a sub-folder still gets its
dedicated walk, and only a token missing from the listing (partial listing /
permission edge) keeps the legacy probe + fallback.

Also pins sub-folder selection semantics with a dedicated test.

* fix(datasource): address review findings across the Feishu deep-adaptation branch

Four parallel reviewer slices audited the branch; this applies the accepted
findings.

Content fidelity (blocks rendering):
- todo blocks parse style.done — completed items render "- [x]" instead of
  always unchecked
- code fences size past the longest backtick run in the content, so a line
  of ``` inside code can no longer close the fence early
- ordered-list "auto" numbering only resets on a sibling of the last item;
  child blocks rendered flat between items no longer renumber the run

Image pipeline:
- inline base64 images are capped at the document-processing resolver's
  10 MB budget; larger images degrade to a placeholder instead of leaving a
  permanent raw-base64 blob in stored markdown
- image markers carry a per-document nonce so user-typed look-alike text
  can never be patched in place of a real marker
- attachment marker patching anchors on the token's line end, closing the
  token-prefix collision
- SupportedImageExt recovers image/svg+xml (board exports), which the
  platform resolver already stores natively

Sync correctness:
- clearResyncMarker re-reads the row before writing: a long upgraded sync
  no longer risks reverting concurrent user edits with a stale full-row
  update
- wiki EditTime keys on max(obj_edit_time, node_edit_time): pure
  moves/renames re-file the document on the next incremental sync
- file-typed wiki/drive base names are sanitised; a "/" in a title can no
  longer fake an extra folder level

Docs/tests/comments:
- datasources docs rewritten to the v3 image contract (no image rows,
  no image_map, no parse_mode backfill claim)
- DataSourceEditorDialog drops settings.parse_mode when backtracking to a
  non-Feishu connector type
- stale comments corrected (types.go parse-mode transport, repository
  folder-counts, wiki List partial-error branch, test headers); regression
  tests added for todo done, fence sizing, ordered numbering, svg sniff

* fix(datasource): repair markdown conversion defects found in live smoke test

Live re-sync of the all-block-type Feishu test document surfaced three
converter defects:

- Code blocks: the closing fence reused the opening fence string, so a
  tagged block ended with ```java — CommonMark requires a bare closing
  fence, and every renderer left the block unterminated. The closing
  fence is now backticks only.
- Feishu heading7-9 emitted 7-9 hash marks; GFM stops at H6, so those
  lines rendered as literal '#######' text. They now degrade to bold.
- The docx API percent-encodes link targets (http%3A%2F%2F...); escapeURL
  now reverses that encoding before markdown-escaping, so stored links
  point at real URLs instead of percent-encoded strings.

Verified against the live document via a throwaway API probe: the
'i|g|k' table cells questioned during review are the document's own
content (the sheets copy says i|j|k; the two docx tables genuinely
contain 'g'), not a converter bug.

* fix(frontend): render GFM task lists in document preview

Task list items (- [x] / - [ ]) rendered as plain bullets: marked emits
<input type="checkbox"> for them, but the preview DOMPurify config's
ALLOWED_TAGS allowlist has no 'input', so the element was stripped whole.
The preview renderer now overrides checkbox to emit a styled
task-checkbox span (class/attr both allowlisted) and the preview styles
draw the box with a check mark, hiding the list bullet via :has().

Same gap exists in the chat markdown surface; left untouched there.

* fix(datasource): crop blank canvas borders from whiteboard exports

The board download_as_image API renders the WHOLE board canvas, not the
content bounding box — editors routinely leave the canvas far larger than
the drawing, so board/mindnote/diagram exports surfaced with large blank
areas below the content. The API has no crop parameter, so the connector
now trims near-uniform blank borders after download (bottom-right pixel
seeds the background color, 14/255 per-channel tolerance absorbs JPEG
noise, 8px margin kept). PNG and JPEG are re-encoded; anything
undecodable, fully blank, or already tight passes through unchanged, and
oversized images (>64MP) skip the scan to bound memory.

* fix(datasource): require corner majority before trimming board exports

The blank-border trim seeded the background color from the bottom-right
pixel alone. A large solid shape reaching that corner would seed the
background with its own fill color, and fully-filled rows could then be
trimmed as blank — cutting real content. The background color is now the
majority of the four corner pixels (>=3 must agree); on a split vote
content reaches the corners and the image passes through untrimmed.
Content visible against the background is untouched either way: a row or
column survives if even one pixel differs from the background by more
than the 14/255 JPEG-noise tolerance.

* fix(datasource): widen board trim margin and lower JPEG quality to 80

The 8px margin read as a tight crop — a 32px white frame now visually
bounds the content region. JPEG re-encode quality drops 90 -> 80 to keep
the re-encoded size closer to the original export.

* feat(datasource): timeline widget Markdown degradation + table column alignment

Two content-fidelity gaps from the live smoke test:

- 文档小组件 (add_ons, block_type 40) carry their full data inline in
  record; there is no server-side render to download. The timeline
  component (component_type_id blk_6358a421bca0001c...) now degrades to
  a Markdown list (- **time · title**:text), honoring contentShow and
  omitting empty segments; unknown component types and malformed records
  keep the previous placeholder note.
- docx text blocks carry style.align (1 left / 2 center / 3 right,
  absent unless the author set it). Native-table rendering now collects
  per-column alignment from the first explicit cell and emits GFM colon
  separators (:---: / ---:). Embedded sheets/bitable keep the plain
  separator — their value APIs expose no alignment.

* feat(datasource): show document titles for 云文档 mention links

The docx renderer emitted mention_doc references as [url](url) because
the payload carries no title. Feishu doc URLs now emit a
[weknora-docname://token](url) placeholder and FetchDocxWithBlocks
resolves the titles in one drive metadata batch call
(POST /drive/v1/metas/batch_query, docx/doc/sheet/bitable/wiki/...),
patching the link text to [文档名](url). Refs whose metadata cannot be
fetched — no permission (970003), deleted/mismatched (970005), API
absent — degrade to the previous URL-as-text form with a warn log.

This is display-only: referenced docs are not pulled as sub-documents
(deferred design lives in the task PRD).

* docs(datasource): document mention rendering enhancement and drive rename limitation

- Add optional mention scopes (contact:user.base:readonly,
  drive:drive.metadata:readonly, board:whiteboard:node:read) with
  degrade behavior to the Feishu permission tables
- Describe mention_user real-name lookup, mention_doc title backfill
  and timeline widget placeholders in the blocks rendering flow
- Document the Drive known limitation: folder renames / file moves do
  not bump modified_time, so incremental sync keeps the old KB folder
  structure until a full sync; Wiki sources are unaffected
- Clarify the directory-mapping wording in website-docs to distinguish
  Wiki (rename-aware) from Drive (rename-blind) sources

* docs(datasource): add blocks vs export tradeoff summary to website-docs

* docs(datasource): document docx block-type coverage for blocks mode

- Add a full block-type support table (block_type 1-52 grouped) to the
  datasource dev doc: text/heading/list/task/quote structures, code
  fences with language aliases, containers (callout/quote/grid/view),
  native tables with merged-cell splitting (top-left value fill) and
  column alignment, embedded sheet/bitable converted to in-document
  Markdown tables, image/board inline pipeline, attachment whitelist,
  iframe links, timeline add-on degradation and the placeholder set
- Summarize the same coverage in the drive user guide, the wiki
  quick-start doc and website-docs; fix the stale claim that board
  blocks degrade to placeholders (they ride the image pipeline now)

* style: resolve diff-scoped lint findings across feishu packages

- Wrap over-length signatures and calls (lll, max 120)
- Add doc comments to exported Block* payload types (revive)
- Rename unused closure parameters to _ (revive)
- Check w.Write / resp.Body.Close error returns (errcheck)
- Apply De Morgan rewrite in docname token validation (staticcheck)
- gofumpt-format touched files

golangci-lint run --new-from-rev=main ./... now reports 0 issues.

* ci: drop fork-only docker-build workflow

The manual docker-build workflow lives on the fork's main and is not
part of the upstream contribution; it entered the branch through the
fork-main merge.

* fix(migrations): renumber feishu resync migrations to avoid version collision with upstream

Upstream added 000017_dingtalk_stream_only (sqlite) and 000096_dingtalk_stream_only
(versioned) before our merge, colliding with our 000017/000096 feishu resync
migrations. golang-migrate rejects duplicate versions. Renumber to the next free
slots (sqlite 000027, versioned 000108) and update comments + test constant.

* fix(migrations): renumber feishu resync migrations ahead of upstream 000027/000108

Upstream main now also claims sqlite 000027 and versioned 000108
(session_sandbox_config_tenant), colliding with our just-renumbered
resync migrations. Move to 000028 / 000109.

* fix(feishu): address review — deprecated env fallback and upgrade notes

- Keep FEISHU_DOCX_PARSE_MODE as a one-release deprecated fallback when a
  data source has no per-source parse_mode (deprecation warning logged);
  annotate it in .env.example / docker-compose.yml instead of removing.
- Point stale migration-number comments at the final numbers
  (versioned 000110 / sqlite 000030) in datasource_service.go and tests.
- website-docs: add Feishu upgrade notes (env deprecation, default mode
  flip to blocks, one-shot full resync with embedding/VLM recompute);
  refresh 24-feishu-drive.md parse-mode section to the per-source
  setting and new blocks semantics.

* docs(feishu): document necessary and optional connector permissions

Restore the permission matrix lost with the docs/ removal: necessary
scopes (wiki/drive/export/docx) with missing-permission symptoms and
optional enhancement scopes (embedded sheets/bitable, board export,
mention user/doc rendering) with their degrade behavior, in
10-datasource.md's Feishu section; make 24-feishu-drive.md list the
three necessary drive scopes and link to the full table.

* Revert "docs(feishu): document necessary and optional connector permissions"

This reverts commit 3be19cb1b656b3596f2cb6afb2cda8c2a1d12be1.

* docs(feishu): permission matrix by connector, parse mode and necessity

Necessary-scope tables split by connector (feishu/lark wiki vs
feishu_drive/lark_drive) with blocks/export applicability columns and
missing-permission symptoms; optional enhancement scopes with their
degrade behavior; scope names cross-checked against the Feishu open
platform API docs (files/folder list, medias download, export tasks,
metas batch_query). Full matrix lives in 10-datasource.md; the drive
guide mirrors its own table and links back.

* test(feishu): adapt mention_user_test to upstream retryTestServer signature

* feat(feishu): mirror the sync root (wiki space / drive folder) as a top-level KB folder

So a knowledge base linking several feishu sources can tell them apart at a
glance: syncing folder 测试1 yields 测试1/... and syncing wiki space
飞书知识库同步测试 yields 飞书知识库同步测试/.... Root-name resolution
failure falls back to the previous root-less behaviour without failing the
sync; existing sources converge through the one-shot resync marker.

* refactor(feishu): cache wiki space folder prefix per run, refresh stale comments

Review follow-ups for d012e593f: cache the sanitized space prefix per spaceID
for the whole sync run (K selected spaces now cost one ListWikiSpaces paging
pass instead of K), fix the misleading warn wording, and realign the
WalkDriveTree / driveFolderName doc comments with root-mirror semantics.

* docs(feishu): drop stale and duplicated comments from the deep-adaptation

- board.go: trimBoardImage no longer claims a bottom-right seed (the code
  votes the background over the four corners)
- shared.go: rewrite the FetchDocxWithBlocks note (images/boards inline via
  base64 now; removes a stale-marker splice artefact)
- markdown.go: drop an internal task-number leftover
- datasource_service.go: collapse the ProcessSync/processSyncStreaming notes
  duplicated verbatim from resyncRequired's doc comment, keep the call-site
  why (marker cleared only on success)
- datasource_resync_test.go: drop the obsolete 'backfill' wording

* feat(i18n): 添加解析模式及其提示信息到繁體中文語言包

* fix(frontend): use radius token for task checkbox; fix simplified chars in zh-TW parse mode

CI guard fixes: the PR-added task-checkbox style pushed the radius-literal
count to 161 (> baseline 160); the new zh-TW keys missed the locale parity
audit (user's dc6fdcff9) — also normalise the simplified 文档 leftovers.

* fix(feishu): preserve sync scope, nested content, and legacy settings

Cover shortcut selection, ancestor path changes, nested containers and table boards with regression tests. Avoid redundant Drive traversal, repair resync migrations for legacy settings, and preserve inherited parse modes. Bound board image dimensions before decoding.

* docs(feishu): clarify shortcut scope and incremental path tracking

* Fix Feishu sync reconciliation and embedded media regressions

* Bound Feishu table rendering by actual cell data

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-16 16:49:44 +08:00
ordinary being 7925f774d1
fix(datasource/gitlab): emptied files no longer leave their old body searchable (#4008)
A blob that a commit emptied is still a blob, and ingestion reads an item with
neither content nor URL as nothing to sync: applyFetchedItem counts it as
skipped and the sync reports success. The body indexed from an earlier commit
stays fully searchable, and no later sync revisits it — a full sync still sees
the path in the tree, and an incremental sync only walks the diffs of commits
newer than the recorded head, which the emptying commit already moved past.

Emit the file's path as the content in that case so ingestion replaces the stale
body, matching what the Notion (b9be7d0d), Confluence and IMA (b71b31c8)
connectors already do. The check is on length alone: a blob holding only
whitespace still carries content the repository has.
2026-10-12 13:06:19 +08:00
Lyang77 fe444c851e
fix(feishu): add shared request pacing and improve HTTP 429 / error 99991400 retries in Feishu Wiki synchronization (#4039)
* fix(feishu): 修复同步请求限流与共享重试等待

* fix(feishu): 完善知识库限流冷却和导出轮询

Bound export polling by poll count so shared pacing and cooldowns retain the
same export ticket without causing false 60-second timeouts. Preserve the
feishu_timeout failure classification when the poll budget is exhausted.

Cap cooldowns, isolate reactive waits by API path, and use a configurable
10-second default when no valid retry hint is available. Refresh expired
tokens before rejoining the request gate to preserve pacing and new cooldowns.

Skip JSON decoding when the legacy rate-limit code is absent, derive the
prefilter from the shared protocol constant, and update configuration docs.
Isolate shared gates in tests and cover cooldowns, cancellation, API grouping,
concurrent token refresh, and timeout classification.

Validation: Feishu core tests pass; diff-scoped Feishu lint reports 0 issues;
gofmt and git diff --check pass. No live Feishu sync was run for this commit.

* fix(feishu): grant request gate slots in arrival order and improve rate-limit logs

- Reserve the next slot before sleeping so concurrent callers are paced in
  FIFO order instead of racing after a shared wake-up.
- Restore the error log for non-rate-limited HTTP 400 download failures.
- Include the API gate name in rate-limit warnings.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-12 13:00:33 +08:00
fight for you 57f8a65fec
fix(datasource/dingtalk): tighten Validate sampling edges (#4079)
* fix(datasource/dingtalk): tighten Validate sampling edges

Keep probing later documents in the same workspace, hold listing budget
for sibling folders that are already queued, fail closed when any root
page cannot be listed, and say when the probe cap stops a conclusive walk.

Fixes #4021

Co-authored-by: fight for you <wangrui19971019@users.noreply.github.com>

* fix(datasource/dingtalk): stop Validate walk once probe cap is spent

- Break the workspace walk as soon as maxValidateProbes is spent, so a
  later unlistable folder can no longer turn failed probes into acceptance.
- Treat a skipped folder tail as covered only by a probe made after the
  skip, not by any earlier probe in the same workspace.
- Keep the document name on probe context errors.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: fight for you <wangrui19971019@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-10 20:32:05 +08:00
linus-liu-web 8319d59dd0
feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf) (#3786)
* feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf)

钉钉连接器此前只导入原生 .adoc 文档:知识库里的上传文件既不会出现在选择器里,
也不会被同步,而且没有任何提示。

- 节点判定拆成 isOnlineDocument / isBinaryDocument / binaryContentType,isDocument()
  变成"在线文档 ∪ 二进制文档",上传文件因此可被枚举与摄取;表格(axls)等其它原生
  类型仍不在其中,避免枚举出同步读不了的节点。
- 上传文件通过钉盘下载 API 逐字节传输,交给 WeKnora 自己的解析器:先用
  /v2.0/doc/dentries/{id}/queryDentryId 把节点 UUID 换成数字 spaceId/dentryId
  (存储 API 拒绝非数字 spaceId),再取预签名 URL 下载。
- 大小上限加在传输上而不是列表元数据上:node.size 与存储对象并不一致。
- 预签名 URL 短命且带版本,传输失败会用新签名重试。
- Validate 对上传文件"解析位置但不传 payload",凭证校验不必为每个候选文件下载
  数 MB;adoc 仍走 blocks API。
- 同步按类型分派:adoc 渲染 blocks,上传文件落原始字节并带上具体 MIME 类型与只含
  一个扩展名的文件名;下载失败与读取失败一样是可重试的失败项,不推进游标。

本提交只含 C(上传文件摄取)。不含:校验循环容错(A)、跳过项日志与可见性(B)、
原生表格 axls(D)、多维表/手工引用/前端(E)。

* test(datasource/dingtalk): 上传文件用例改用合成文档名与占位 ID

- 夹具、断言与文件名注释里的文档名统一换成合成名 Employee-Handbook.docx,
  用例语义不变:仍覆盖"名字已带扩展名时不重复追加"与 PK\x03\x04 字节断言。
- 去掉测试注释里"as verified against the live tenant"的表述,改为描述调用顺序。
- dentry 响应示例与用例中的数字 ID 换成明确占位符,不再出现可被误认的真实 ID。

* fix(datasource/dingtalk): 上传文件摄取改为数据源级开关,超限不再重试

评审意见(#3786):上传文件是本次新增的摄取路径,存量数据源升级后无需用户显式启用
就会在下一次同步(含增量)开始下载并向量化所选范围内的 docx/pptx/xlsx/pdf,可能
增加存储与 embedding 成本;同时单次下载里超出体积上限被当成可重试错误,没有
Content-Length 的超大文件会重签名再下满 3 次。

- 新增 `Settings.include_uploaded_files`(默认 false,未设置/非法值一律按 false)。
  读取走与本仓库语雀连接器同名的 settingBool(v, def),并新增 documentSettings /
  parseDocumentSettings 承载开关。
- 判定拆成"有没有读取路径"(isDocument)与"这个数据源收不收"(isIngestible):
  Validate 的探测、ListResources、全量同步与增量同步(共用 scanWorkspace)四处
  全部改用 isIngestible,关闭时行为与 main 一致。
- skipReason 对"能读但没开开关"的上传文件给出可操作的原因:说明
  include_uploaded_files 未启用,而不是伪装成类型不支持;媒体文件与真正没有
  摄取路径的类型保持原有说法。
- readLimited 改为返回哨兵错误 errDocumentTooLarge(同时包装上限字节数),
  fetchSigned 据此返回 retryable=false:体积是对象的属性,新签名只会拿到同样的
  字节。普通读错误仍然 retryable=true,仍会重签名重试。
- 前端数据源编辑弹窗(同步策略步骤)加对应开关,未触碰时不写回 settings;5 个
  语言包补齐文案。
- website-docs 钉钉一节写明支持范围、默认值、64 MiB 单文件上限与成本影响。

测试补 include_uploaded_files 的未设置/显式关闭/显式开启三种状态:校验、资源列表、
全量同步、增量同步四处各自断言,直接选择上传文件在关闭时仍是失败的资源;流式超限
用例断言签名查询与对象下载各只发生一次,另加"普通读错误仍重试"的用例。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:56:04 +08:00
诉说晚秋 b14bcbcd54
fix(notion): 保留正文抓取失败时的同步游标 (#3693) 2026-10-09 10:43:59 +08:00
zaqqmm 19542f2a1d
feat(datasource): add Seafile data source connector (#3616)
* fix(datasource): omit last_sync_cursor from the data source list response

The settings page polls GET /datasource every few seconds during a sync,
and connector cursors can grow to megabytes. No list consumer reads the
cursor, so NewDataSourceResponses now blanks it; detail, create and
update responses still return it through NewDataSourceResponse.

* feat(datasource): add Seafile connector with fingerprint-based incremental sync

Sync folders and files from one Seafile library per data source. The
connector walks the selected scope one directory level at a time,
downloads files through the link + fileserver flow without credentials,
and reconciles deletions against a per-file oid/mtime/size fingerprint
cursor that is checkpointed every 50 mutations.

- resource ids are "<repo_id>:<path>"; ancestors collapse, one library
- per-file failures become coded error items and keep the path pending
- directory failures and Retry-After > 60s abort without deletions
- forced full runs keep the old cursor as a deletion baseline
- API token never reaches cursors, metadata, logs or error text
- share the supported import extension set through internal/utils

* feat(datasource): add Seafile frontend connector and docs

Wire the Seafile data source into the editor dialog, channel labels
and the five locales, and document how to connect a Seafile 10.0.1
library.

- connector definition with base_url + API token fields and icon
- picker guard: one library per data source, at least one selection
- "library" resource type and icon in the lazy tree
- channel label in the knowledge base filter, document views and chat
- i18n for labels, guide steps and the eight seafile_* sync error codes
- website-docs integration guide, data source and extension point doc
  updates, and Seafile in the README data source lists

---------

Co-authored-by: zaqqmm <q-zhang10@qq.com>
2026-10-08 16:18:09 +08:00
linus-liu-web bcbd65a459
fix(datasource/dingtalk): Validate 不再因为某个工作区读不到就整库判失败 (#3782)
* fix(datasource/dingtalk): 多工作区租户里一个工作区读不到就整库判失败

Validate 原来在第一个读不到的工作区或文档上直接 return err。租户的
listWorkspaces 会返回组织内的全部工作区,而 operator 通常只被授权其中
一部分,于是"知识库里混了多个工作区、其中一个没权限"会被判成"凭据
错误",整库都建不起来,哪怕另一个工作区完全可读。

改为在循环里累计 lastErr / workspaceFail / sawDocument:某个工作区列不出
节点、或某个文档读不到,都只记录原因并 continue,不再中断;循环结束后
只有"一个能读的都没有"才报错。一个文档都没暴露的租户仍然放行——没有
可读之物就无法证伪凭据,只有"存在却读不到"才算问题。

文档探测抽成 verifyDocument,后续接入上传文件、表格时在这里加分支。

验证:
- 容器内 go build ./internal/...(仅第三方 sqlite-vec cgo 因环境缺
  sqlite3.h 失败)+ go test ./internal/datasource/... 全部通过
- 反向验证:把 Validate 还原成"第一个失败即返回"后,
  TestValidateSkipsWorkspacesTheOperatorCannotRead 与
  TestValidateSurvivesWorkspaceListingFailure 立即失败

* test(datasource/dingtalk): 钉住 Validate 收紧路径并修正 verifyDocument 注释

Validate 改成继续遍历所有工作区后,多出一条比改动前更严的失败路径:
"工作区 1 只有文件夹(可读)+ 工作区 2 有一个本 operator 读不到的文档"
这种配置,改动前在第一个工作区末尾就 return nil,现在会报错。这条路径是
有意的——存在却读不到的文档意味着数据源同步不出任何东西,不是无关工作区
的权限噪音——但之前没有任何测试。

新增 TestValidateRejectsTenantWhoseOnlyDocumentIsUnreadable 固定该行为与
意图;并给既有的"全部文档都读不到"用例补上"错误必须带工作区名/文档名"的
断言(原先只 Contains("no permission"))。

顺带把 verifyDocument 的注释从"上传文件与表格目前不可摄取、永远到不了
这个函数"改成描述不变量:只有 isDocument 接受的原生类型才会进入该函数,
一个类型获得摄取路径时同时成为 document、获得同步读路径并在该函数获得
探测,上传文件与表格随各自的摄取路径一起接入。原措辞在表格/上传 PR 合并
后即失效。

验证:
- go test ./internal/datasource/connector/dingtalk/... -count=1 通过
- 反向验证:把 Validate 换回改动前的实现(首个工作区无文档即 return nil)
  后,新用例失败并报 "Validate must fail";恢复后通过

* fix(datasource/dingtalk): Validate 每个工作区只探 1 个文档,并给探测总数加上限

应用缺少文档读权限时每次探测都会失败,原先会把每个工作区根目录下的每个文档
都串行探一遍,大租户里一次校验就是几百次调用。改为探测失败后直接跳到下一个
工作区,并且最多探测 maxValidateProbes(5) 个文档后返回最后一个错误。

* fix(datasource/dingtalk): Validate 进入文件夹找文档,根目录只有文件夹的工作区不再被误判

根目录只有文件夹、文件夹里有可读文档的工作区,原先会被另一个读不到的工作区拖累,
整体判失败(main 上这种情况是通过的)。Validate 现在按 scanWorkspace 的走法对
每个工作区做广度优先查找,取第一个文档探测;子文件夹列举共用 20 次额度,额度用尽
仍有文件夹没看完时视为无法证伪,直接通过。

* fix(datasource/dingtalk): Validate 子文件夹列举失败不再当空目录,并按页限额、防环

- 非根文件夹列举失败时把本次校验记为无法确定(放行),不再当成空目录导致
  另一个工作区的不可读文档把整库判死;取消/超时错误直接返回,不再被当成
  凭据有效,也不会被之前记下的 inconclusive 吞掉。
- BFS 加 visited 集合,子节点 ID 等于祖先或工作区根时不再无限列举。
- 新增 listNodesPage 单页列举,Validate 的列举额度按实际 HTTP 分页计数:
  节点列举请求最多 len(workspaces) + maxValidateListings 次(另加客户端重试)。

* fix(datasource/dingtalk): Validate 先翻完同层分页再进子文件夹,重复 nextToken 时停下

子文件夹排在父目录下一页之前时,根目录第一页全是文件夹、文档在第二页的
工作区会把列举额度花在子文件夹上,第二页的文档永远探不到。现在每个文件夹
先翻完自己的分页再下钻,和 listNodes 给 scanWorkspace 的顺序一致;同一
文件夹重复返回 nextToken 时停下并按无法确定处理。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: 🏠 <linus@192.168.10.6>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 15:20:27 +08:00
诉说晚秋 d84cdabac4
fix(rss): refresh titles and source links when article text is unchanged (#3824)
* fix(rss): 修复正文未变时标题和来源链接不同步

* fix(datasource/rss): 旧 h: 游标只升级不重建,原文抓取失败时保留旧游标

升级后第一次增量原先会把 feed 里每一条仍在的旧条目删掉重建(Wiki、手工标签、
知识 ID 都会丢),原文接口临时失败时还会把库里的全文降级成 feed 摘要。

现在对旧 h: 游标的条目:重新解析出的正文哈希与 h: 相同时,只把游标升级为 i:
指纹、不输出;原文抓取失败时沿用 h: 游标、不输出,留到下次重试;只有正文真的
变了才输出。

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:56 +08:00
yetuge a75e8e9ac0
fix(datasource): 语雀/Notion/IMA/飞书连接器 API 响应体补体积上限 (#3985)
* fix(datasource): 语雀/Notion/IMA/飞书 API 响应体补体积上限

#3887 给 GitLab 客户端补了 readCapped 上限;语雀、Notion、IMA、飞书四个
连接器的 API 读取仍是裸 io.ReadAll,破损或恶意端点可让同步进程缓冲无界
响应体。各连接器比照 GitLab 同步同一上限(16 MiB)与显式超限报错,limit
做成字段以便测试下调。飞书 429/5xx 错误路径的 best-effort 读取刻意不动。

* fix(datasource): 响应体上限为零值时回退到默认 16 MiB

测试里直接用字面量构造的 Client 不经过 NewClient,jsonLimit 为 0,
readCapped 会拒绝任何响应,导致 feishu/core 11 个现有测试失败。
readCapped 遇到非正数 limit 时改用 maxJSONResponseBytes。

* fix(datasource): 响应体超限不再重试;补齐飞书 token 与下载错误分支的读取上限

- yuque/ima/feishu/notion 的 readCapped 改用哨兵错误 errResponseTooLarge,
  文案不变;yuque doRequest、ima callAPIAt、feishu DoRequest 遇到超限直接返回,
  不再按退避重试 3 次(同一请求必然再次超限,只会白耗限流额度)。
- 飞书 GetTenantAccessToken 改为 readCapped + json.Unmarshal,和其他 API 响应同一上限。
- 飞书 downloadRawBytes 的 429/5xx/非 200 错误预览只读 4 KiB,
  download failed 错误不再携带完整响应体。

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:29 +08:00
linus-liu-web dabbdaf156
fix(notion): 分页循环补重复游标防护与跳数上限 (#3890)
paginatePages 与 GetBlockChildrenFlat 都只以 has_more / 空 next_cursor 为终止
条件:既不记录用过的游标,也没有页数上限。Notion(或中间网关)只要回
has_more=true 且 next_cursor 不前进——包括"空 results + has_more=true"这种厂商
形状——循环就永远退不出来,同一页被反复抓取:SearchPages(资源发现)与
QueryDatabaseAll(数据库记录)都汇入 paginatePages,客户端自带 3 req/s 限流,
所以表现为以 3 rps 稳定空转到任务 deadline,同步任务永不进终态;若伴随非空
results,allPages 还会无界增长。

同文件的 getBlockChildrenRecursive 已有 maxBlocksPerPage 兜底,Confluence
(seen + maxPaginationHops)、DingTalk(seenTokens + maxPages)也有同族守卫,
这两条循环漏了。

改动:
- 抽一条 advancePaginationCursor:每页取回 next_cursor 后调用,重复游标立即报
  错,页数达到 maxPaginationHops(10000,对齐 Confluence)时 Warn 并报错。触顶
  必须留日志:从外部看这种失败和"还在正常同步"一模一样。
- paginatePages 与 GetBlockChildrenFlat 一并接入(同文件同型缺陷,发现阶段扫子
  页面走的正是后者)。

新增用例:重复游标(第 2 次请求即报错,含"空页 + has_more=true"形态)、跳数上限
(正好 10000 次请求后报错且日志含 "pagination exceeded 10000 pages")、真实多页
仍正常合并。测试把 3 rps 限流换成不限速,否则回归场景要跑 55 分钟。去掉本次改动
后三条用例全部失败(重复游标场景实测跑到 10006 次请求才被假服务器的预算拦下)。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 01:29:20 +08:00
linus-liu-web 5077f35800
fix(notion): 超过 block 上限截断时补 Warn,避免内容与子页面静默缺失 (#3771)
* fix(notion): 超过 block 上限截断时补 Warn,避免内容与子页面静默缺失

单个页面/块超过 maxBlocksPerPage(1000) 时客户端会静默停止分页:第 1000
个 block 之后的正文不会进入 markdown,其后的 child_page/child_database
也不会被 fetchPage 递归同步。又因为游标记录的是 last_edited_time,页面
未变更时同步会直接跳过该页,截断因此长期存在且无任何提示。飞书同类上限
(feishu/core/blocks.go 的 maxDocumentBlocks)已有 Warn,这里对齐其措辞
补齐可观测性。

- cap 触发且 API 仍有下一页时打一条 Warn,含块/页面 ID 与上限数字,说明
  剩余 block 不会被同步;是否真为截断由纯函数 blocksTruncated 判定,恰好
  1000 个 block 的页面不会误报;
- 不改 cap 数值、page_size、递归逻辑与游标语义。

测试 truncation_test.go:伪造返回 1001 个 block 的 children 接口,断言
Warn 内容且第 11 次请求未发出;另覆盖恰好 1000 的边界与 blocksTruncated
真值表。

* fix(notion): has_more 但 cursor 为空时不再静默丢内容

blocksTruncated 的第三个条件 nextCursor != "" 在 Notion 契约下与 has_more
冗余:has_more=false 时 next_cursor 必为 null。于是"has_more=true 但
next_cursor 为空"这种异常响应仍会静默截断——未命中 cap 时循环本就因空游标
break,命中 cap 时又因该条件跳过 Warn,两种情况都不留痕迹。

- 谓词收敛为 currentCount >= maxBlocksPerPage && hasMore,不再要求游标非空:
  cap 命中且 has_more=true 即视为截断;
- has_more=true 而 next_cursor 为空时单独打一条 Warn,带上块 ID 与已取块数,
  说明无法继续翻页、剩余内容不会被同步(无论是否命中 cap 都打);
- 不改 cap 数值、break 条件、page_size、递归深度与游标语义。

测试:真值表去掉游标列,并删掉把"cap 命中 + 空游标 = 不告警"固化成期望的
用例;新增 TestBlockCapWarnsWhenHasMoreHasNoCursor(cap 命中 + has_more=true
+ 空游标须同时出现 cap Warn 与游标 Warn,且不发出第 11 次请求)与
TestHasMoreWithoutCursorWarnsBelowTheCap(cap 以下同样要告警);原有正常翻页
用例补断言:格式正确的分页不得出现游标异常 Warn。反向验证:谓词还原为原样
后两个新用例均失败。

* test(notion): rename block-cap log helper to avoid clashing with #3845

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 01:29:12 +08:00
linus-liu-web a6c6c35ad2
fix(ima): 三条游标分页循环补重复游标防护与跳数上限 (#3889)
ListResources 的两条循环(get_addable_knowledge_base_list、search_knowledge_base
兜底)与 listAllKBFiles 的 get_knowledge_list 循环都只以 is_end / 空 next_cursor
为终止条件:既不记录用过的游标,也没有跳数上限。厂商只要回 is_end=false 且
next_cursor 不前进,循环就会一直请求同一页,bases / out 随迭代重复增长,直到任务
deadline —— 资源选择器接口不返回,同步任务永远不进终态。

同族的 DingTalk(seenTokens + maxPages)、Confluence(seen +
maxPaginationHops)、飞书(seenPageTokens)都已有守卫,本包漏了。

改动:
- 抽一条 nextPageCursor:每页取回 next_cursor 后调用,重复游标立即返回错误(不再
  重试),页数达到 maxPaginationHops(10000,对齐 Confluence)时 Warn 并报错。
  触顶必须留日志:从外部看这种失败和"还在正常同步"一模一样。
- 三条循环按同一写法接入;listAllKBFiles 的守卫错误补上 get_knowledge_list 前缀。
- 假 IMA 服务器支持覆盖某个 action 的分页字段(固定游标 / 每次换新游标 / 有限页数 /
  每次换 payload / 请求预算),并给"无防护时会空转"的场景设请求上限,使回归用例失
  败而不是挂死。

新增用例:三条循环的重复游标(都在第 2 次请求即报错)、跳数上限(正好 10000 次
请求后报错且日志含 "pagination exceeded 10000 pages")、真实多页仍正常合并。
去掉本次改动后这四条用例全部失败(重复游标场景实测跑到 10006 次请求才被假服务器
的预算拦下)。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:44 +08:00
linus-liu-web a2aeb97a01
fix(gitlab): 分页循环补重复页防护与跳数上限 (#3841)
projects() 与 tree() 只依据服务端返回的 X-Next-Page 判断是否继续,
该值重复或不前进时循环不会结束:同一页被反复请求、结果切片持续
增长,只能等上层 context 到期。

两条循环改用 trackPage:页码已请求过即返回错误,不静默当作读完;
页数达到 maxPaginationHops 时记警告并返回错误。正常翻页路径不变。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:37 +08:00
linus-liu-web b5c74b2dd6
fix(feishu): 多维表格字段分页补重复 page_token 防护与条数上限 (#3840)
内嵌多维表格的 list-fields 分页循环只判断 has_more 与 token 是否为空:
厂商在 has_more=true 时返回已读过的 page_token,循环就会反复请求同一页并把
同一批字段重复追加进 cols,直到任务 deadline(实测 2s 内 5 万余次请求仍未
返回)。该循环也是同文件里唯一既无去重、又无尺寸上限的分页循环。

- 按同库 wiki 空间/节点与云盘分页已有写法加 seenFieldPageTokens 集合,重复的
  非空 token 立即返回 error,不再静默当作读完;
- 补 maxBitableFields = 5000 的条数上限,命中时 Warn 并截断。

测试:重复 token 在第二次出现时报错并停止请求;厂商每页都返回新 token 时在
第 5000 个字段处停止并打 Warn;正常 100 条/页 分页不受影响。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:31 +08:00
LuckVd 0d3a053eb6
fix(yuque): don't acknowledge failed detail fetches in the incremental cursor (#3702)
A partial sync recorded each doc's new content_updated_at before fetching
its detail. When a detail request failed after client retries, the cursor
still held the new version, so every later incremental sync treated the
doc as unchanged and never re-fetched it (mixed success/failure batches
persist the cursor; only all-failed batches are aborted).

Rewind the cursor entry to the previously synced version (existing docs)
or drop it (new docs) on detail failure so recovery re-syncs the doc.
Deletion detection is unaffected: currentDocs is set at listing time.

Fixes #3690

Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
2026-10-08 01:21:23 +08:00
linus-liu-web 402f69832e
fix(notion): 查询撞厂商上限被截断时不再静默,截断轮次不下发删除 (#3845)
* fix(notion): 查询撞厂商行数上限不再静默截断,截断轮次跳过删除判定

Notion 单次 data source 查询最多返回 10,000 行。撞到上限时厂商把 has_more 置为
false,并在响应里带 request_status.type == "incomplete"
(incomplete_reason = "query_result_limit_reached"),而 connector 只认
has_more、不解析 request_status,于是"被截断"与"读完"在代码里完全同形:

- QueryDatabaseAll 返回 err == nil、10,000 条记录,日志照常打
  "database <id> (<title>): 10000 records",同步看起来完全成功;
- 增量同步里,本轮查询未返回、又没被 /v1/search 带出来的行,会被当成源端已删除
  下发 IsDeleted。开启删除同步时,已在库的文档连同向量被真实删除,而同步日志仍
  显示成功。

改动:

- types.go:paginatedResponse 增加 request_status 解析(notionRequestStatus /
  isIncomplete)。
- client.go:paginatePages 在每一页反序列化后判定 incomplete(厂商文档要求逐页
  检查,标记可能出现在最后一页之前),命中即返回可识别的
  errQueryResultTruncated,连同该页已取回的行,调用方既能拿到部分数据,也不会
  再把缺失行当作"源端已删除"的证据;QueryDatabaseAll 遇到截断不再走
  "当作 database_id 再解析一次"的回退,避免重复同一次受限查询并把截断错误埋在
  查找失败后面。
- connector.go:截断信号经 queryDatabaseRecords 传到
  fetchDatabaseIncremental / fetchDatabase / fetchPage,本轮只要出现过截断,删除
  判定整体跳过(与 #3682"发现请求失败即中止、不删"同一取向);计数与日志改为
  带截断标记的 Warn,让"同步成功但文档数偏少"可解释。

新增回归测试 internal/datasource/connector/notion/query_truncation_test.go:截断
(含中途页截断)必须返回哨兵错误且不重试、端到端不产生 IsDeleted、子数据库路径
同样传出截断;同时锁定正常 has_more=false 无 request_status 的结果集行为不变
(正常读完仍照旧判定删除)。反向验证:去掉逐页判定后 4 条断言失败的用例失败、
3 条保护性用例通过,随后逐字节还原。

按 created_time 开新窗口的分页改写(厂商大数据源食谱)改动面较大,留作后续改动:
本次修复只保证截断可见、不误删;超过上限的记录仍不会被这批同步收进知识库。

* fix(notion): 数据源查询按 created_time 开窗口,超过 10,000 行的记录也能取全

Notion 单次 data source 查询最多返回 10,000 行:撞限时厂商把 has_more 置为 false,
并带 request_status.type == "incomplete"(incomplete_reason ==
"query_result_limit_reached")。本 PR 的前一版让截断可见、不再误删,但被截断的记录
仍然进不了知识库——一次查询的上限就是一次查询的上限,只跟着 next_cursor 走永远读
不到第 10,001 行。

按厂商"大数据源"食谱改为时间窗口分页(用 created_time,不用会随编辑移动边界的
last_edited_time):

- types.go:notionPage 增加 created_time,作为窗口边界。
- client.go:数据源查询改走 queryDataSourceAll / queryDataSourceWindow。每个窗口都
  带 sorts: [{timestamp: created_time, direction: ascending}],窗口内继续按
  next_cursor 翻页;只要该窗口任意一页带 incomplete 标记(厂商明确要求逐页检查,
  标记可能出现在最后一页之前),就用 filter.created_time.on_or_after = 本窗口最后
  一行的时间戳开下一个窗口,累计结果按行 ID 去重(相邻窗口共享边界行,边界行只会
  入库一次)。窗口推进必须严格前进:没有任何行带 created_time,或窗口最后一行的时间
  戳没有超过窗口起点(同一 created_time 的行数超过单次查询上限)时停止续窗、打 Warn
  并返回 errQueryResultTruncated,绝不原地重查。每次续窗都打一条 Warn,截断仍然留痕。
- connector.go:queryDatabaseRecords 的截断日志改为"撞上限且窗口无法推进";"截断轮次
  跳过删除判定"的行为保持不变(现在的截断只可能来自无法推进的窗口)。

回归测试:新增 query_windowing_test.go,用按请求里的窗口过滤条件返回数据的假服务端
覆盖跨 4 个窗口取全 5 行并按行 ID 去重、同一窗口内的重复行去重、incomplete 标记落在
非末页时仍然开新窗口、未撞上限的正常路径行为不变(request_status 为 complete 与完全
缺失两种形态)。query_truncation_test.go 改写为:窗口无法推进(同一 created_time 超
限、行不带 created_time、空窗口仍 incomplete)时停止并告警、端到端不产生 IsDeleted;
另加窗口取全后仍照常判定源端真实删除的用例,防止"一有窗口就不再删"的过度修复。

反向验证:把窗口续读改回"撞上限即中止"后,窗口取全相关用例全部失败(QueryDatabaseAll
报 "vendor limit reached ... 2 records fetched",端到端漏报 r4 的删除),"正常路径行为
不变"与"截断不误删"的用例继续通过,随后逐字节还原。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-07 23:54:08 +08:00
linus-liu-web 1dbe9d0c1a
fix(gitlab): 仓库文件下载补体积上限,超限报错而不是无界读入 (#3887)
* fix(gitlab): 仓库文件下载没有体积上限,超限改为显式报错

getRaw 直接用 io.ReadAll 缓冲整个 blob,get/getPage 的 JSON 读取同样无界。
GitLab 的 tree/compare 接口不返回 blob 大小,下载处是唯一能拦的地方,所以
同步一个音频或大 PDF 就会把一个完整 blob 读进进程内存。

对齐兄弟 connector(钉钉 16MiB JSON 上限、飞书 512MiB 下载上限):

- maxJSONResponseBytes(16 MiB)用于 get/getPage;
- maxRawFileBytes(512 MiB,与飞书 maxFeishuDownloadBytes 对齐)用于 getRaw;
- base64 回退走 getCapped,其请求体是 base64 后的 blob(约 4/3)加信封。

超限返回错误而不是截断:截断的正文会被当成完整文档入库。非 2xx 状态先于
读取判定,避免超大错误页把真实状态码盖掉。

* fix(gitlab): 将 compare 响应上限与元数据分开

compare 正文包含完整提交和 patch,沿用 16MiB 元数据上限会中断增量同步且游标无法前进。超限错误改为带上未编码的仓库路径。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-30 14:33:15 +08:00
linus-liu-web b71b31c897 fix(ima): 笔记正文被清空后旧正文仍留在知识库
fetchNote 在正文为空时返回 fetchSkipped,walk 只把该 key 记进增量游标而
不产出任何 item。于是重新读到一篇已被清空的笔记时,入库层收不到「内容变
空」的信号:旧正文既不会被替换,游标还把它记为已同步。

Notion(b9be7d0d)与 Confluence 早就是这个写法:正文为空时仍然产出只有
标题的 markdown,让入库去替换旧正文。这里对齐同一写法,空正文改成
"# <title>\n" 照常走 fetchOK。
2026-09-30 13:43:55 +08:00
yeliheng-010 15d4845521 fix(rss): 修复全文回退后增量同步不重试 2026-09-29 00:53:27 +08:00
yeliheng-010 b9be7d0d0d fix(notion): 修复页面清空后旧正文残留 2026-09-29 00:52:07 +08:00
linus-liu-web a282d1091e
fix(feishu): 云盘分页补上 #3645 给 wiki 加的重复 page token 防护 (#3770)
ListDriveFilesAllPages 只以 next_page_token 是否为空作为终止条件,既不看
has_more,也没有重复 token 检测;同一文件里同一厂商的 ListWikiSpaces /
ListWikiNodes 已在 #3645 修过,云盘这两条防护都漏了。

厂商返回 has_more=false 但残留非空 token,或返回重复的 page_token 时,循环会
拿同一个 token 反复请求同一页:定时同步任务会在 asynq 的 2h 超时前一直打 API
(scheduler 侧 MaxRetry 5),资源选择器
GET /api/v1/datasources/:id/resources 永不返回,同时 all 每轮 append 不断增长。

按 #3645 的既有写法原样修复:listDriveFiles 透出 has_more,循环改为
if !hasMore || next == "" 才退出,并用 seenPageTokens 记录已用过的 token,重复即
返回 "feishu drive file pagination repeated page token" 错误。正常多页语义不变
(仍返回完整列表),只是不再空转。

测试:TestClientDriveListFilesPagination(drive 包)用 httptest 覆盖重复 token、
has_more=false 带残留 token、正常两页三种情况;注释掉守卫后该用例在 2s 内发出
4 万+ 次请求并以 context deadline exceeded 失败。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-29 00:51:19 +08:00
yeliheng-010 72e105f46a fix(gitlab): 按提交快照比较增量同步变更 2026-09-29 00:50:05 +08:00
linus-liu-web 7d1d71e50d
fix(datasource/dingtalk): 块级渲染不再静默丢内容(列表按线上真实形状渲染 + 未建模块类型可见) (#3822)
* fix(datasource/dingtalk): 块级渲染不再静默丢内容(列表按契约渲染 + 未建模块类型可见)

渲染器有三处会无声地丢正文:

- 列表块只在 inline 子节点渲染结果为空时才递归子块,于是 children 里既有
  inline 又有块级元素时,块级子节点的文本永远不渲染 —— 而且连 unknown 标记
  都不设,完全无痕;块级列表项还会被 strings.Fields 折叠进同一个 bullet 行;
- 未建模的块类型(code / attachment / slot / sheet / unknown …)payload 解析
  不出来,正文整块丢失,只剩类型名写进文档 metadata,全仓没有一行日志;
- 未建模的 inline 元素类型(slot 等,官方文档里文本放在 children)在默认分支
  只写 text 字段,children 里的正文直接消失。

改动:

- 列表按官方契约渲染:一个列表块 = 一个列表项,缩进用 list.level(0-based),
  同一列表的多项是共享 listId 的兄弟块(扁平模型),不假设结构嵌套;契约外的
  块级子节点仍然逐个渲染,作为防御,不再二选一、不再塌行;
- 未建模的 inline 元素在自身没有文本时渲染 children,避免整段消失;
- 未建模块类型在 Fetch 路径上显式暴露(这一层同时拿得到 ctx 与节点身份):
  collectUnknownBlockTypes 产出可测值,同时喂给 logger.Warnf 与
  common.PipelineWarn;文档 metadata 保持不变。

code / attachment / sheet / slot 的正文渲染不在本次范围:它们的块级 JSON 形状
仍无可靠证据(官方与公开 CLI 参考的键名互相冲突),先靠告警把真实 payload
找出来再按形状补,免得用猜测的字段把"可见的丢失"变回"静默的丢失"。

* fix(datasource/dingtalk): 按实测形状补 code / attachment / unknown 三类块的渲染

只读探针在真实 adoc 文档上取到的形状(不再依赖公开参考里互相冲突的键名):

- code: {"syntax":"bash","text":"...","title":"..."},title 有时缺;语言键是
  syntax 而不是 language;
- attachment: {"name":..,"size":..,"type":..,"viewType":..,"resourceId":..},
  resourceId 有时缺,payload 里没有任何下载 URL;
- unknown: {"rawType":"..."},只有真实类型名,没有正文;
- table.cells 的读取路径与实测一致,未改动。

改动:

- code 渲染成围栏代码块:语言取 syntax(含反引号或换行这类无法作为 info string
  的值时省略),围栏长度取 max(3, 正文最长反引号串+1),避免正文自带的围栏把代码块
  提前截断;title 非空时在围栏前输出一行加粗标题;text 为空不输出空围栏,仍按未建模
  类型记录并告警;
- attachment 渲染成一行纯文本 Attachment: <name> (<size>):名称做 Markdown 转义,
  size 按 1024 进制保留一位小数(B/KB/MB/GB),size<=0 时省略括号;payload 没有可
  解析的下载地址,因此不拿 resourceId 拼任何链接;名称为空不输出,仍按未建模类型
  记录并告警;
- unknown:default 分支在 rawType 非空时把标记写成 "unknown:"+rawType,使
  unknown_block_types metadata 与 warnUnknownBlockTypes 的日志/pipeline 事件直接
  带真实类型名,而不是无信息量的 "unknown"(字段名、签名与调用位置均未动),
  children 仍逐层递归。

既有普通块(段落/标题/引用/表格/列表)的输出逐字节不变,回归用例
TestRenderDocumentKeepsOrdinaryBlocksStable 未修改即通过。原先拿 code/attachment
当“未建模”夹具的两个用例改用空载荷的同类块与带 rawType 的 unknown 块,覆盖新的
“渲染不出内容仍要告警”语义。

* fix(datasource/dingtalk): 列表按线上真实形状渲染(块自带 text)

线上 Blocks API 返回的是一个列表项一个块,正文在块自带的 text 字段里,
没有 children、没有 list.level、也没有 listId。列表是唯一一类没有回退到
「块自带 text」的正文块 —— paragraph / heading / blockquote 都有
renderInlines(children) 取不到就取自身 text 的回退,列表直接进
renderListBlock,而那个函数只遍历 children,于是列表正文被整条丢弃;
又因为 blockType 是已知类型,switch 命中了它,连 unknown_block_types
都不会记一笔。

同一形状问题还影响 unknown:sdt 块(文档目录)的正文就在 unknown.text 里,
此前只写标记不渲染正文。

夹具取自线上 payload(markdown_live_shape_test.go),不按转述的契约自造。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-29 00:48:20 +08:00
linus-liu-web 08f7e702cf fix(datasource/dingtalk): 跳过不再静默,展开叶子不再报错
同步过去只保留能读的内容:没有 ingest 路径的节点被直接丢掉,日志里
一个字都没有,一个 19 个文件的库只同步出 14 个也无从得知。现在

- scanWorkspace 把「看到了但读不了」的节点一并以 skipped 返回,
  sync 逐个打印身份与原因(视频/媒体是刻意不下载,原生类型是还没实现),
  每个 scope 再补一行 total/synced/skipped/failed 汇总,partial sync
  再也装不成一次干净的运行;
- 可重试的失败仍按失败计(不计入 skipped),并显式记一条 warn,
  与确定性跳过区分开;
- 资源选择器展开一个其实没有子节点的条目(document)时返回空列表而不是
  ErrResourceNotFound:叶子本来就没有子节点,而前端把 error 渲染成
  展开失败的气泡,会让一次同步完全正常的操作看起来出错。前端
  DataSourceEditorDialog 收到空列表后会把该行 has_children 置 false 并
  保留,因此第二次展开不会再发一次空请求(该逻辑已在上游 main 中)。

跳过只在日志里报告:FetchedItem 没有 skip 字段,也不打算为它新增。
2026-09-29 00:47:02 +08:00
Fei.Chen ec0ca46334
fix(server): run ResourceCleaner on signal shutdown instead of racing Fatalf exit (#3804)
* fix(server): signal shutdown raced Fatalf exit before ResourceCleaner ran

The signal goroutine closed the listener before calling server.Shutdown,
so Serve returned the raw accept error ('use of closed network connection')
instead of ErrServerClosed. The invoke error path then hit logger.Fatalf
(os.Exit(1)) in the same millisecond, killing the process before the
shutdown goroutine reached resourceCleaner.Cleanup. Every registered
cleaner — BrowserSkill daemon, sandbox reapers, IMService, AntsPool,
Langfuse — was skipped on every SIGTERM/SIGINT, leaking the shared bsk
daemon process and its temp home on each restart.

Two changes:
- Drop the manual listener.Close(): Shutdown closes all listeners as its
  first step, releasing the port just as fast without racing Serve.
- Give resourceCleaner.Cleanup its own fresh timeout context so a slow
  connection drain cannot leave Cleanup with an already-expired context,
  which made it skip every entry without running any.

* fix(server): keep signal cleanup inside one grace period

Desktop quit still closed the listener before Shutdown and returned
before ResourceCleaner finished, so Fatalf could skip every hook.
Share ShutdownTimeout between drain and cleanup, run BrowserSkill
first, and do not drop later hooks when the deadline expires.

* fix(server): satisfy errcheck and line-length lint

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-28 23:37:36 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug 4328ab26a9
feat(plugin): code plugins — extension protocol, Go SDK, plugin host, web search and data source adapters (#3720)
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
2026-09-26 14:45:15 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
niannian 4364e61afa
feat(datasource/yuque): preserve TOC folder hierarchy for synced documents (#3476)
* feat(datasource/yuque): preserve TOC folder hierarchy for synced documents

The Yuque connector emitted a bare "<title>.md" file name, and folder_path
is derived from FileName alone (SplitKnowledgeRelativePath ->
NormalizeKnowledgeFolderPath). With no "/" in the name the derived folder is
always empty, so every synced document landed at the knowledge base root no
matter where the book's TOC places it.

Read the book's TOC once per sync, fold it into a per-document path by walking
parent_uuid upward, and prepend the result to FileName. The GitLab connector
already works this way; this supplies the missing input rather than changing
the FileName -> folder_path machinery.

Three optional settings, all defaulting to the previous behaviour so the
emitted FileName stays byte-for-byte identical unless folder_mode is set to
"toc". A TOC failure degrades to flat names rather than failing the sync, and
the toc_only filter is skipped entirely when no TOC was fetched.

* feat(datasource/yuque): make toc_only an admission filter, drop include_book_title

Review follow-ups for the TOC folder hierarchy change.

toc_only used to run before the document was recorded for deletion detection,
so a document that was present in the source but not admitted came back as
IsDeleted — and with sync_deletions on, the sync service then deleted it from
the knowledge base. That is the wrong semantics for a key named after
filtering, and on Yuque it is unusually damaging: the web UI does not show
documents created through the API that were never attached to the TOC, so a
knowledge base is often the only place such a document can be reached at all.
A filtered document is now recorded as seen but not ingested, so enabling the
filter removes nothing. It is deliberately kept out of the cursor so that
admitting it later — by attaching it to the TOC, or by switching the filter
off — ingests it instead of dismissing it as unchanged.

include_book_title is removed. Leaving the book segment out silently merges
same-named top-level groups from different books, and the "which book"
dimension has no other carrier: the automatic tag granularity is one tag per
data source, not one per book. Under folder_mode=toc the book segment is now
unconditional; a source still carrying the key keeps syncing with the value
ignored.

Also supplies the UI the feature was missing — a Yuque-only folder_mode select
in the sync-strategy step, with newly created sources opting into the TOC
layout and existing ones left untouched — and documents both settings in the
Yuque section of the datasource guide.

* feat(datasource/yuque): expose the TOC admission filter in the datasource editor

UI follow-ups from review.

The folder-layout control now uses the pill-button group that the neighbouring
sync-mode and conflict-strategy settings already use, instead of a select — a
two-way choice was the only thing on that step rendered as a dropdown.

toc_only is exposed beside it, disabled unless the TOC layout is selected: the
connector only fetches the table of contents under folder_mode=toc, so the flag
does nothing in the flat layout, and the control says so. Its label names the
documents it admits as the ones visible in the Yuque table of contents, which
is what separates them from documents created through the API that were never
attached to one.

Drops the include_book_title paragraph from the datasource guide. That key was
never in a released build, so describing it as removed documents a history no
reader has; the tolerance test for unknown settings keys stays, without the
retired-key framing.

* docs(datasource): note Yuque folder-mode sync limitations

---------

Co-authored-by: niannian <10008660+RichardChengshaojin@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:33:48 +08:00
jacentsao e7c27ccc9e
feat(confluence): add directory sync and private image assets (#3319)
* feat(confluence): support page subtree selection

* feat(confluence): add private image asset resolver

* fix(confluence): preserve pages during partial syncs

* fix(confluence): cap total inlined image bytes per page

Up to 30 images of 10 MB each could be inlined as base64, and every
downstream conversion copies the page again, so one extreme page could
peak above 1 GB. Cap the data URIs a page holds, in the download cache
and the rewritten HTML, at 50 MB; images past the budget keep their
original src like any other skipped image.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:23:48 +08:00
Perfecto 00e6366fd2
fix(datasource): validate cron schedules and persist manual-only mode (#3653)
* fix(datasource): preserve schedule state on invalid updates

* docs(datasource): document PUT /datasource/:id as a full replace

The update now always persists sync_schedule, so a partial body clears
the schedule. Show a full-object example and state that omitted fields
are reset rather than preserved.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:23:18 +08:00
xxxxx 5614c04d88
fix(feishu): stop Wiki pagination on repeated page tokens (#3645) 2026-09-25 18:47:47 +08:00
诉说晚秋 605d70fc52
fix(notion): 在资源发现失败时中止增量同步 (#3682) 2026-09-25 18:47:38 +08:00
yeliheng-010 c029b50ef9 fix(rss): 拒绝超限响应以避免正文静默截断 2026-09-24 20:44:56 +08:00
wizardchen 503f1abfb4 fix(confluence): confirm unlisted pages before deleting after a cut-short listing
On a Server/DC instance with the #3596 behaviour every listing ends
early, so skipping deletion reconciliation for such runs meant pages
deleted in Confluence were never removed from the knowledge base.

When a listing is cut short, look up each previously synced page it did
not return with GET /rest/api/content/{id}?expand=space and delete it
only when Confluence confirms it is gone from the space: a 404, a
non-current status, or a different space. Pages that are still there,
that fail to load (including 403), or that exceed the 200-probe budget
per run stay tracked and are checked again next time.
2026-09-24 14:18:55 +08:00
wizardchen eb888d0d3d fix(confluence): stop Server listings that link past the last page
Some Server/Data Center builds keep returning _links.next from
/rest/api/space/{key}/content/page after the last page, with empty
results (#3596). The connector followed it until the 10000-hop cap, so a
sync ran for hours, imported nothing and then failed.

Stop after three consecutive empty pages that still link onward. A
single empty page is not taken as the end, since permission filtering
can empty one window mid-listing. A listing cut short this way is
reported as incomplete: its pages are imported, pages it did not reach
stay in the cursor, and deletion reconciliation is skipped for that run
so an early stop cannot mirror-delete documents.
2026-09-24 14:18:55 +08:00
yeliheng-010 608027179f fix(gitlab): 补齐项目资源列表分页 2026-09-24 07:52:32 +08:00
hahahahahayesyeseys 11edd16819 fix(datasource): fail closed on GitLab compare errors 2026-09-24 07:49:07 +08:00
yushangcan fca1cd06f8 fix(ima): traverse subfolders using list response media IDs 2026-09-19 22:42:57 +08:00
wizardchen 820a14dafa refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
jacentsao 01f0700b35
feat(datasource): add Confluence sync (#3287)
* feat(datasource): add Confluence sync

* fix(confluence): resume full sync and list Server pages without CQL

CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.

* fix(confluence): satisfy package lint for pre-push against main

* fix(confluence): parse Server page lists and keep version expand

The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.

* fix(confluence): classify errors and fail closed on empty listings

Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-16 14:38:16 +08:00