Commit Graph

74 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
linus-liu-web dabbdaf156
fix(notion): 分页循环补重复游标防护与跳数上限 (#3890)
paginatePages 与 GetBlockChildrenFlat 都只以 has_more / 空 next_cursor 为终止
条件:既不记录用过的游标,也没有页数上限。Notion(或中间网关)只要回
has_more=true 且 next_cursor 不前进——包括"空 results + has_more=true"这种厂商
形状——循环就永远退不出来,同一页被反复抓取:SearchPages(资源发现)与
QueryDatabaseAll(数据库记录)都汇入 paginatePages,客户端自带 3 req/s 限流,
所以表现为以 3 rps 稳定空转到任务 deadline,同步任务永不进终态;若伴随非空
results,allPages 还会无界增长。

同文件的 getBlockChildrenRecursive 已有 maxBlocksPerPage 兜底,Confluence
(seen + maxPaginationHops)、DingTalk(seenTokens + maxPages)也有同族守卫,
这两条循环漏了。

改动:
- 抽一条 advancePaginationCursor:每页取回 next_cursor 后调用,重复游标立即报
  错,页数达到 maxPaginationHops(10000,对齐 Confluence)时 Warn 并报错。触顶
  必须留日志:从外部看这种失败和"还在正常同步"一模一样。
- paginatePages 与 GetBlockChildrenFlat 一并接入(同文件同型缺陷,发现阶段扫子
  页面走的正是后者)。

新增用例:重复游标(第 2 次请求即报错,含"空页 + has_more=true"形态)、跳数上限
(正好 10000 次请求后报错且日志含 "pagination exceeded 10000 pages")、真实多页
仍正常合并。测试把 3 rps 限流换成不限速,否则回归场景要跑 55 分钟。去掉本次改动
后三条用例全部失败(重复游标场景实测跑到 10006 次请求才被假服务器的预算拦下)。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 01:29:20 +08:00
linus-liu-web 5077f35800
fix(notion): 超过 block 上限截断时补 Warn,避免内容与子页面静默缺失 (#3771)
* fix(notion): 超过 block 上限截断时补 Warn,避免内容与子页面静默缺失

单个页面/块超过 maxBlocksPerPage(1000) 时客户端会静默停止分页:第 1000
个 block 之后的正文不会进入 markdown,其后的 child_page/child_database
也不会被 fetchPage 递归同步。又因为游标记录的是 last_edited_time,页面
未变更时同步会直接跳过该页,截断因此长期存在且无任何提示。飞书同类上限
(feishu/core/blocks.go 的 maxDocumentBlocks)已有 Warn,这里对齐其措辞
补齐可观测性。

- cap 触发且 API 仍有下一页时打一条 Warn,含块/页面 ID 与上限数字,说明
  剩余 block 不会被同步;是否真为截断由纯函数 blocksTruncated 判定,恰好
  1000 个 block 的页面不会误报;
- 不改 cap 数值、page_size、递归逻辑与游标语义。

测试 truncation_test.go:伪造返回 1001 个 block 的 children 接口,断言
Warn 内容且第 11 次请求未发出;另覆盖恰好 1000 的边界与 blocksTruncated
真值表。

* fix(notion): has_more 但 cursor 为空时不再静默丢内容

blocksTruncated 的第三个条件 nextCursor != "" 在 Notion 契约下与 has_more
冗余:has_more=false 时 next_cursor 必为 null。于是"has_more=true 但
next_cursor 为空"这种异常响应仍会静默截断——未命中 cap 时循环本就因空游标
break,命中 cap 时又因该条件跳过 Warn,两种情况都不留痕迹。

- 谓词收敛为 currentCount >= maxBlocksPerPage && hasMore,不再要求游标非空:
  cap 命中且 has_more=true 即视为截断;
- has_more=true 而 next_cursor 为空时单独打一条 Warn,带上块 ID 与已取块数,
  说明无法继续翻页、剩余内容不会被同步(无论是否命中 cap 都打);
- 不改 cap 数值、break 条件、page_size、递归深度与游标语义。

测试:真值表去掉游标列,并删掉把"cap 命中 + 空游标 = 不告警"固化成期望的
用例;新增 TestBlockCapWarnsWhenHasMoreHasNoCursor(cap 命中 + has_more=true
+ 空游标须同时出现 cap Warn 与游标 Warn,且不发出第 11 次请求)与
TestHasMoreWithoutCursorWarnsBelowTheCap(cap 以下同样要告警);原有正常翻页
用例补断言:格式正确的分页不得出现游标异常 Warn。反向验证:谓词还原为原样
后两个新用例均失败。

* test(notion): rename block-cap log helper to avoid clashing with #3845

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 01:29:12 +08:00
linus-liu-web a6c6c35ad2
fix(ima): 三条游标分页循环补重复游标防护与跳数上限 (#3889)
ListResources 的两条循环(get_addable_knowledge_base_list、search_knowledge_base
兜底)与 listAllKBFiles 的 get_knowledge_list 循环都只以 is_end / 空 next_cursor
为终止条件:既不记录用过的游标,也没有跳数上限。厂商只要回 is_end=false 且
next_cursor 不前进,循环就会一直请求同一页,bases / out 随迭代重复增长,直到任务
deadline —— 资源选择器接口不返回,同步任务永远不进终态。

同族的 DingTalk(seenTokens + maxPages)、Confluence(seen +
maxPaginationHops)、飞书(seenPageTokens)都已有守卫,本包漏了。

改动:
- 抽一条 nextPageCursor:每页取回 next_cursor 后调用,重复游标立即返回错误(不再
  重试),页数达到 maxPaginationHops(10000,对齐 Confluence)时 Warn 并报错。
  触顶必须留日志:从外部看这种失败和"还在正常同步"一模一样。
- 三条循环按同一写法接入;listAllKBFiles 的守卫错误补上 get_knowledge_list 前缀。
- 假 IMA 服务器支持覆盖某个 action 的分页字段(固定游标 / 每次换新游标 / 有限页数 /
  每次换 payload / 请求预算),并给"无防护时会空转"的场景设请求上限,使回归用例失
  败而不是挂死。

新增用例:三条循环的重复游标(都在第 2 次请求即报错)、跳数上限(正好 10000 次
请求后报错且日志含 "pagination exceeded 10000 pages")、真实多页仍正常合并。
去掉本次改动后这四条用例全部失败(重复游标场景实测跑到 10006 次请求才被假服务器
的预算拦下)。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:44 +08:00
linus-liu-web a2aeb97a01
fix(gitlab): 分页循环补重复页防护与跳数上限 (#3841)
projects() 与 tree() 只依据服务端返回的 X-Next-Page 判断是否继续,
该值重复或不前进时循环不会结束:同一页被反复请求、结果切片持续
增长,只能等上层 context 到期。

两条循环改用 trackPage:页码已请求过即返回错误,不静默当作读完;
页数达到 maxPaginationHops 时记警告并返回错误。正常翻页路径不变。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:37 +08:00
linus-liu-web b5c74b2dd6
fix(feishu): 多维表格字段分页补重复 page_token 防护与条数上限 (#3840)
内嵌多维表格的 list-fields 分页循环只判断 has_more 与 token 是否为空:
厂商在 has_more=true 时返回已读过的 page_token,循环就会反复请求同一页并把
同一批字段重复追加进 cols,直到任务 deadline(实测 2s 内 5 万余次请求仍未
返回)。该循环也是同文件里唯一既无去重、又无尺寸上限的分页循环。

- 按同库 wiki 空间/节点与云盘分页已有写法加 seenFieldPageTokens 集合,重复的
  非空 token 立即返回 error,不再静默当作读完;
- 补 maxBitableFields = 5000 的条数上限,命中时 Warn 并截断。

测试:重复 token 在第二次出现时报错并停止请求;厂商每页都返回新 token 时在
第 5000 个字段处停止并打 Warn;正常 100 条/页 分页不受影响。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:31 +08:00
LuckVd 0d3a053eb6
fix(yuque): don't acknowledge failed detail fetches in the incremental cursor (#3702)
A partial sync recorded each doc's new content_updated_at before fetching
its detail. When a detail request failed after client retries, the cursor
still held the new version, so every later incremental sync treated the
doc as unchanged and never re-fetched it (mixed success/failure batches
persist the cursor; only all-failed batches are aborted).

Rewind the cursor entry to the previously synced version (existing docs)
or drop it (new docs) on detail failure so recovery re-syncs the doc.
Deletion detection is unaffected: currentDocs is set at listing time.

Fixes #3690

Co-authored-by: LuckVd <LuckVd@users.noreply.github.com>
2026-10-08 01:21:23 +08:00
linus-liu-web 402f69832e
fix(notion): 查询撞厂商上限被截断时不再静默,截断轮次不下发删除 (#3845)
* fix(notion): 查询撞厂商行数上限不再静默截断,截断轮次跳过删除判定

Notion 单次 data source 查询最多返回 10,000 行。撞到上限时厂商把 has_more 置为
false,并在响应里带 request_status.type == "incomplete"
(incomplete_reason = "query_result_limit_reached"),而 connector 只认
has_more、不解析 request_status,于是"被截断"与"读完"在代码里完全同形:

- QueryDatabaseAll 返回 err == nil、10,000 条记录,日志照常打
  "database <id> (<title>): 10000 records",同步看起来完全成功;
- 增量同步里,本轮查询未返回、又没被 /v1/search 带出来的行,会被当成源端已删除
  下发 IsDeleted。开启删除同步时,已在库的文档连同向量被真实删除,而同步日志仍
  显示成功。

改动:

- types.go:paginatedResponse 增加 request_status 解析(notionRequestStatus /
  isIncomplete)。
- client.go:paginatePages 在每一页反序列化后判定 incomplete(厂商文档要求逐页
  检查,标记可能出现在最后一页之前),命中即返回可识别的
  errQueryResultTruncated,连同该页已取回的行,调用方既能拿到部分数据,也不会
  再把缺失行当作"源端已删除"的证据;QueryDatabaseAll 遇到截断不再走
  "当作 database_id 再解析一次"的回退,避免重复同一次受限查询并把截断错误埋在
  查找失败后面。
- connector.go:截断信号经 queryDatabaseRecords 传到
  fetchDatabaseIncremental / fetchDatabase / fetchPage,本轮只要出现过截断,删除
  判定整体跳过(与 #3682"发现请求失败即中止、不删"同一取向);计数与日志改为
  带截断标记的 Warn,让"同步成功但文档数偏少"可解释。

新增回归测试 internal/datasource/connector/notion/query_truncation_test.go:截断
(含中途页截断)必须返回哨兵错误且不重试、端到端不产生 IsDeleted、子数据库路径
同样传出截断;同时锁定正常 has_more=false 无 request_status 的结果集行为不变
(正常读完仍照旧判定删除)。反向验证:去掉逐页判定后 4 条断言失败的用例失败、
3 条保护性用例通过,随后逐字节还原。

按 created_time 开新窗口的分页改写(厂商大数据源食谱)改动面较大,留作后续改动:
本次修复只保证截断可见、不误删;超过上限的记录仍不会被这批同步收进知识库。

* fix(notion): 数据源查询按 created_time 开窗口,超过 10,000 行的记录也能取全

Notion 单次 data source 查询最多返回 10,000 行:撞限时厂商把 has_more 置为 false,
并带 request_status.type == "incomplete"(incomplete_reason ==
"query_result_limit_reached")。本 PR 的前一版让截断可见、不再误删,但被截断的记录
仍然进不了知识库——一次查询的上限就是一次查询的上限,只跟着 next_cursor 走永远读
不到第 10,001 行。

按厂商"大数据源"食谱改为时间窗口分页(用 created_time,不用会随编辑移动边界的
last_edited_time):

- types.go:notionPage 增加 created_time,作为窗口边界。
- client.go:数据源查询改走 queryDataSourceAll / queryDataSourceWindow。每个窗口都
  带 sorts: [{timestamp: created_time, direction: ascending}],窗口内继续按
  next_cursor 翻页;只要该窗口任意一页带 incomplete 标记(厂商明确要求逐页检查,
  标记可能出现在最后一页之前),就用 filter.created_time.on_or_after = 本窗口最后
  一行的时间戳开下一个窗口,累计结果按行 ID 去重(相邻窗口共享边界行,边界行只会
  入库一次)。窗口推进必须严格前进:没有任何行带 created_time,或窗口最后一行的时间
  戳没有超过窗口起点(同一 created_time 的行数超过单次查询上限)时停止续窗、打 Warn
  并返回 errQueryResultTruncated,绝不原地重查。每次续窗都打一条 Warn,截断仍然留痕。
- connector.go:queryDatabaseRecords 的截断日志改为"撞上限且窗口无法推进";"截断轮次
  跳过删除判定"的行为保持不变(现在的截断只可能来自无法推进的窗口)。

回归测试:新增 query_windowing_test.go,用按请求里的窗口过滤条件返回数据的假服务端
覆盖跨 4 个窗口取全 5 行并按行 ID 去重、同一窗口内的重复行去重、incomplete 标记落在
非末页时仍然开新窗口、未撞上限的正常路径行为不变(request_status 为 complete 与完全
缺失两种形态)。query_truncation_test.go 改写为:窗口无法推进(同一 created_time 超
限、行不带 created_time、空窗口仍 incomplete)时停止并告警、端到端不产生 IsDeleted;
另加窗口取全后仍照常判定源端真实删除的用例,防止"一有窗口就不再删"的过度修复。

反向验证:把窗口续读改回"撞上限即中止"后,窗口取全相关用例全部失败(QueryDatabaseAll
报 "vendor limit reached ... 2 records fetched",端到端漏报 r4 的删除),"正常路径行为
不变"与"截断不误删"的用例继续通过,随后逐字节还原。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-07 23:54:08 +08:00
linus-liu-web 1dbe9d0c1a
fix(gitlab): 仓库文件下载补体积上限,超限报错而不是无界读入 (#3887)
* fix(gitlab): 仓库文件下载没有体积上限,超限改为显式报错

getRaw 直接用 io.ReadAll 缓冲整个 blob,get/getPage 的 JSON 读取同样无界。
GitLab 的 tree/compare 接口不返回 blob 大小,下载处是唯一能拦的地方,所以
同步一个音频或大 PDF 就会把一个完整 blob 读进进程内存。

对齐兄弟 connector(钉钉 16MiB JSON 上限、飞书 512MiB 下载上限):

- maxJSONResponseBytes(16 MiB)用于 get/getPage;
- maxRawFileBytes(512 MiB,与飞书 maxFeishuDownloadBytes 对齐)用于 getRaw;
- base64 回退走 getCapped,其请求体是 base64 后的 blob(约 4/3)加信封。

超限返回错误而不是截断:截断的正文会被当成完整文档入库。非 2xx 状态先于
读取判定,避免超大错误页把真实状态码盖掉。

* fix(gitlab): 将 compare 响应上限与元数据分开

compare 正文包含完整提交和 patch,沿用 16MiB 元数据上限会中断增量同步且游标无法前进。超限错误改为带上未编码的仓库路径。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-30 14:33:15 +08:00
linus-liu-web b71b31c897 fix(ima): 笔记正文被清空后旧正文仍留在知识库
fetchNote 在正文为空时返回 fetchSkipped,walk 只把该 key 记进增量游标而
不产出任何 item。于是重新读到一篇已被清空的笔记时,入库层收不到「内容变
空」的信号:旧正文既不会被替换,游标还把它记为已同步。

Notion(b9be7d0d)与 Confluence 早就是这个写法:正文为空时仍然产出只有
标题的 markdown,让入库去替换旧正文。这里对齐同一写法,空正文改成
"# <title>\n" 照常走 fetchOK。
2026-09-30 13:43:55 +08:00
yeliheng-010 15d4845521 fix(rss): 修复全文回退后增量同步不重试 2026-09-29 00:53:27 +08:00
yeliheng-010 b9be7d0d0d fix(notion): 修复页面清空后旧正文残留 2026-09-29 00:52:07 +08:00
linus-liu-web a282d1091e
fix(feishu): 云盘分页补上 #3645 给 wiki 加的重复 page token 防护 (#3770)
ListDriveFilesAllPages 只以 next_page_token 是否为空作为终止条件,既不看
has_more,也没有重复 token 检测;同一文件里同一厂商的 ListWikiSpaces /
ListWikiNodes 已在 #3645 修过,云盘这两条防护都漏了。

厂商返回 has_more=false 但残留非空 token,或返回重复的 page_token 时,循环会
拿同一个 token 反复请求同一页:定时同步任务会在 asynq 的 2h 超时前一直打 API
(scheduler 侧 MaxRetry 5),资源选择器
GET /api/v1/datasources/:id/resources 永不返回,同时 all 每轮 append 不断增长。

按 #3645 的既有写法原样修复:listDriveFiles 透出 has_more,循环改为
if !hasMore || next == "" 才退出,并用 seenPageTokens 记录已用过的 token,重复即
返回 "feishu drive file pagination repeated page token" 错误。正常多页语义不变
(仍返回完整列表),只是不再空转。

测试:TestClientDriveListFilesPagination(drive 包)用 httptest 覆盖重复 token、
has_more=false 带残留 token、正常两页三种情况;注释掉守卫后该用例在 2s 内发出
4 万+ 次请求并以 context deadline exceeded 失败。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-29 00:51:19 +08:00
yeliheng-010 72e105f46a fix(gitlab): 按提交快照比较增量同步变更 2026-09-29 00:50:05 +08:00
linus-liu-web 7d1d71e50d
fix(datasource/dingtalk): 块级渲染不再静默丢内容(列表按线上真实形状渲染 + 未建模块类型可见) (#3822)
* fix(datasource/dingtalk): 块级渲染不再静默丢内容(列表按契约渲染 + 未建模块类型可见)

渲染器有三处会无声地丢正文:

- 列表块只在 inline 子节点渲染结果为空时才递归子块,于是 children 里既有
  inline 又有块级元素时,块级子节点的文本永远不渲染 —— 而且连 unknown 标记
  都不设,完全无痕;块级列表项还会被 strings.Fields 折叠进同一个 bullet 行;
- 未建模的块类型(code / attachment / slot / sheet / unknown …)payload 解析
  不出来,正文整块丢失,只剩类型名写进文档 metadata,全仓没有一行日志;
- 未建模的 inline 元素类型(slot 等,官方文档里文本放在 children)在默认分支
  只写 text 字段,children 里的正文直接消失。

改动:

- 列表按官方契约渲染:一个列表块 = 一个列表项,缩进用 list.level(0-based),
  同一列表的多项是共享 listId 的兄弟块(扁平模型),不假设结构嵌套;契约外的
  块级子节点仍然逐个渲染,作为防御,不再二选一、不再塌行;
- 未建模的 inline 元素在自身没有文本时渲染 children,避免整段消失;
- 未建模块类型在 Fetch 路径上显式暴露(这一层同时拿得到 ctx 与节点身份):
  collectUnknownBlockTypes 产出可测值,同时喂给 logger.Warnf 与
  common.PipelineWarn;文档 metadata 保持不变。

code / attachment / sheet / slot 的正文渲染不在本次范围:它们的块级 JSON 形状
仍无可靠证据(官方与公开 CLI 参考的键名互相冲突),先靠告警把真实 payload
找出来再按形状补,免得用猜测的字段把"可见的丢失"变回"静默的丢失"。

* fix(datasource/dingtalk): 按实测形状补 code / attachment / unknown 三类块的渲染

只读探针在真实 adoc 文档上取到的形状(不再依赖公开参考里互相冲突的键名):

- code: {"syntax":"bash","text":"...","title":"..."},title 有时缺;语言键是
  syntax 而不是 language;
- attachment: {"name":..,"size":..,"type":..,"viewType":..,"resourceId":..},
  resourceId 有时缺,payload 里没有任何下载 URL;
- unknown: {"rawType":"..."},只有真实类型名,没有正文;
- table.cells 的读取路径与实测一致,未改动。

改动:

- code 渲染成围栏代码块:语言取 syntax(含反引号或换行这类无法作为 info string
  的值时省略),围栏长度取 max(3, 正文最长反引号串+1),避免正文自带的围栏把代码块
  提前截断;title 非空时在围栏前输出一行加粗标题;text 为空不输出空围栏,仍按未建模
  类型记录并告警;
- attachment 渲染成一行纯文本 Attachment: <name> (<size>):名称做 Markdown 转义,
  size 按 1024 进制保留一位小数(B/KB/MB/GB),size<=0 时省略括号;payload 没有可
  解析的下载地址,因此不拿 resourceId 拼任何链接;名称为空不输出,仍按未建模类型
  记录并告警;
- unknown:default 分支在 rawType 非空时把标记写成 "unknown:"+rawType,使
  unknown_block_types metadata 与 warnUnknownBlockTypes 的日志/pipeline 事件直接
  带真实类型名,而不是无信息量的 "unknown"(字段名、签名与调用位置均未动),
  children 仍逐层递归。

既有普通块(段落/标题/引用/表格/列表)的输出逐字节不变,回归用例
TestRenderDocumentKeepsOrdinaryBlocksStable 未修改即通过。原先拿 code/attachment
当“未建模”夹具的两个用例改用空载荷的同类块与带 rawType 的 unknown 块,覆盖新的
“渲染不出内容仍要告警”语义。

* fix(datasource/dingtalk): 列表按线上真实形状渲染(块自带 text)

线上 Blocks API 返回的是一个列表项一个块,正文在块自带的 text 字段里,
没有 children、没有 list.level、也没有 listId。列表是唯一一类没有回退到
「块自带 text」的正文块 —— paragraph / heading / blockquote 都有
renderInlines(children) 取不到就取自身 text 的回退,列表直接进
renderListBlock,而那个函数只遍历 children,于是列表正文被整条丢弃;
又因为 blockType 是已知类型,switch 命中了它,连 unknown_block_types
都不会记一笔。

同一形状问题还影响 unknown:sdt 块(文档目录)的正文就在 unknown.text 里,
此前只写标记不渲染正文。

夹具取自线上 payload(markdown_live_shape_test.go),不按转述的契约自造。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-09-29 00:48:20 +08:00
linus-liu-web 08f7e702cf fix(datasource/dingtalk): 跳过不再静默,展开叶子不再报错
同步过去只保留能读的内容:没有 ingest 路径的节点被直接丢掉,日志里
一个字都没有,一个 19 个文件的库只同步出 14 个也无从得知。现在

- scanWorkspace 把「看到了但读不了」的节点一并以 skipped 返回,
  sync 逐个打印身份与原因(视频/媒体是刻意不下载,原生类型是还没实现),
  每个 scope 再补一行 total/synced/skipped/failed 汇总,partial sync
  再也装不成一次干净的运行;
- 可重试的失败仍按失败计(不计入 skipped),并显式记一条 warn,
  与确定性跳过区分开;
- 资源选择器展开一个其实没有子节点的条目(document)时返回空列表而不是
  ErrResourceNotFound:叶子本来就没有子节点,而前端把 error 渲染成
  展开失败的气泡,会让一次同步完全正常的操作看起来出错。前端
  DataSourceEditorDialog 收到空列表后会把该行 has_children 置 false 并
  保留,因此第二次展开不会再发一次空请求(该逻辑已在上游 main 中)。

跳过只在日志里报告:FetchedItem 没有 skip 字段,也不打算为它新增。
2026-09-29 00:47:02 +08:00
Fei.Chen ec0ca46334
fix(server): run ResourceCleaner on signal shutdown instead of racing Fatalf exit (#3804)
* fix(server): signal shutdown raced Fatalf exit before ResourceCleaner ran

The signal goroutine closed the listener before calling server.Shutdown,
so Serve returned the raw accept error ('use of closed network connection')
instead of ErrServerClosed. The invoke error path then hit logger.Fatalf
(os.Exit(1)) in the same millisecond, killing the process before the
shutdown goroutine reached resourceCleaner.Cleanup. Every registered
cleaner — BrowserSkill daemon, sandbox reapers, IMService, AntsPool,
Langfuse — was skipped on every SIGTERM/SIGINT, leaking the shared bsk
daemon process and its temp home on each restart.

Two changes:
- Drop the manual listener.Close(): Shutdown closes all listeners as its
  first step, releasing the port just as fast without racing Serve.
- Give resourceCleaner.Cleanup its own fresh timeout context so a slow
  connection drain cannot leave Cleanup with an already-expired context,
  which made it skip every entry without running any.

* fix(server): keep signal cleanup inside one grace period

Desktop quit still closed the listener before Shutdown and returned
before ResourceCleaner finished, so Fatalf could skip every hook.
Share ShutdownTimeout between drain and cleanup, run BrowserSkill
first, and do not drop later hooks when the deadline expires.

* fix(server): satisfy errcheck and line-length lint

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-28 23:37:36 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug 4328ab26a9
feat(plugin): code plugins — extension protocol, Go SDK, plugin host, web search and data source adapters (#3720)
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
2026-09-26 14:45:15 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
niannian 4364e61afa
feat(datasource/yuque): preserve TOC folder hierarchy for synced documents (#3476)
* feat(datasource/yuque): preserve TOC folder hierarchy for synced documents

The Yuque connector emitted a bare "<title>.md" file name, and folder_path
is derived from FileName alone (SplitKnowledgeRelativePath ->
NormalizeKnowledgeFolderPath). With no "/" in the name the derived folder is
always empty, so every synced document landed at the knowledge base root no
matter where the book's TOC places it.

Read the book's TOC once per sync, fold it into a per-document path by walking
parent_uuid upward, and prepend the result to FileName. The GitLab connector
already works this way; this supplies the missing input rather than changing
the FileName -> folder_path machinery.

Three optional settings, all defaulting to the previous behaviour so the
emitted FileName stays byte-for-byte identical unless folder_mode is set to
"toc". A TOC failure degrades to flat names rather than failing the sync, and
the toc_only filter is skipped entirely when no TOC was fetched.

* feat(datasource/yuque): make toc_only an admission filter, drop include_book_title

Review follow-ups for the TOC folder hierarchy change.

toc_only used to run before the document was recorded for deletion detection,
so a document that was present in the source but not admitted came back as
IsDeleted — and with sync_deletions on, the sync service then deleted it from
the knowledge base. That is the wrong semantics for a key named after
filtering, and on Yuque it is unusually damaging: the web UI does not show
documents created through the API that were never attached to the TOC, so a
knowledge base is often the only place such a document can be reached at all.
A filtered document is now recorded as seen but not ingested, so enabling the
filter removes nothing. It is deliberately kept out of the cursor so that
admitting it later — by attaching it to the TOC, or by switching the filter
off — ingests it instead of dismissing it as unchanged.

include_book_title is removed. Leaving the book segment out silently merges
same-named top-level groups from different books, and the "which book"
dimension has no other carrier: the automatic tag granularity is one tag per
data source, not one per book. Under folder_mode=toc the book segment is now
unconditional; a source still carrying the key keeps syncing with the value
ignored.

Also supplies the UI the feature was missing — a Yuque-only folder_mode select
in the sync-strategy step, with newly created sources opting into the TOC
layout and existing ones left untouched — and documents both settings in the
Yuque section of the datasource guide.

* feat(datasource/yuque): expose the TOC admission filter in the datasource editor

UI follow-ups from review.

The folder-layout control now uses the pill-button group that the neighbouring
sync-mode and conflict-strategy settings already use, instead of a select — a
two-way choice was the only thing on that step rendered as a dropdown.

toc_only is exposed beside it, disabled unless the TOC layout is selected: the
connector only fetches the table of contents under folder_mode=toc, so the flag
does nothing in the flat layout, and the control says so. Its label names the
documents it admits as the ones visible in the Yuque table of contents, which
is what separates them from documents created through the API that were never
attached to one.

Drops the include_book_title paragraph from the datasource guide. That key was
never in a released build, so describing it as removed documents a history no
reader has; the tolerance test for unknown settings keys stays, without the
retired-key framing.

* docs(datasource): note Yuque folder-mode sync limitations

---------

Co-authored-by: niannian <10008660+RichardChengshaojin@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:33:48 +08:00
jacentsao e7c27ccc9e
feat(confluence): add directory sync and private image assets (#3319)
* feat(confluence): support page subtree selection

* feat(confluence): add private image asset resolver

* fix(confluence): preserve pages during partial syncs

* fix(confluence): cap total inlined image bytes per page

Up to 30 images of 10 MB each could be inlined as base64, and every
downstream conversion copies the page again, so one extreme page could
peak above 1 GB. Cap the data URIs a page holds, in the download cache
and the rewritten HTML, at 50 MB; images past the budget keep their
original src like any other skipped image.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:23:48 +08:00
Perfecto 00e6366fd2
fix(datasource): validate cron schedules and persist manual-only mode (#3653)
* fix(datasource): preserve schedule state on invalid updates

* docs(datasource): document PUT /datasource/:id as a full replace

The update now always persists sync_schedule, so a partial body clears
the schedule. Show a full-object example and state that omitted fields
are reset rather than preserved.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:23:18 +08:00
xxxxx 5614c04d88
fix(feishu): stop Wiki pagination on repeated page tokens (#3645) 2026-09-25 18:47:47 +08:00
诉说晚秋 605d70fc52
fix(notion): 在资源发现失败时中止增量同步 (#3682) 2026-09-25 18:47:38 +08:00
yeliheng-010 c029b50ef9 fix(rss): 拒绝超限响应以避免正文静默截断 2026-09-24 20:44:56 +08:00
wizardchen 503f1abfb4 fix(confluence): confirm unlisted pages before deleting after a cut-short listing
On a Server/DC instance with the #3596 behaviour every listing ends
early, so skipping deletion reconciliation for such runs meant pages
deleted in Confluence were never removed from the knowledge base.

When a listing is cut short, look up each previously synced page it did
not return with GET /rest/api/content/{id}?expand=space and delete it
only when Confluence confirms it is gone from the space: a 404, a
non-current status, or a different space. Pages that are still there,
that fail to load (including 403), or that exceed the 200-probe budget
per run stay tracked and are checked again next time.
2026-09-24 14:18:55 +08:00
wizardchen eb888d0d3d fix(confluence): stop Server listings that link past the last page
Some Server/Data Center builds keep returning _links.next from
/rest/api/space/{key}/content/page after the last page, with empty
results (#3596). The connector followed it until the 10000-hop cap, so a
sync ran for hours, imported nothing and then failed.

Stop after three consecutive empty pages that still link onward. A
single empty page is not taken as the end, since permission filtering
can empty one window mid-listing. A listing cut short this way is
reported as incomplete: its pages are imported, pages it did not reach
stay in the cursor, and deletion reconciliation is skipped for that run
so an early stop cannot mirror-delete documents.
2026-09-24 14:18:55 +08:00
yeliheng-010 608027179f fix(gitlab): 补齐项目资源列表分页 2026-09-24 07:52:32 +08:00
hahahahahayesyeseys 11edd16819 fix(datasource): fail closed on GitLab compare errors 2026-09-24 07:49:07 +08:00
yushangcan fca1cd06f8 fix(ima): traverse subfolders using list response media IDs 2026-09-19 22:42:57 +08:00
wizardchen 820a14dafa refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
jacentsao 01f0700b35
feat(datasource): add Confluence sync (#3287)
* feat(datasource): add Confluence sync

* fix(confluence): resume full sync and list Server pages without CQL

CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.

* fix(confluence): satisfy package lint for pre-push against main

* fix(confluence): parse Server page lists and keep version expand

The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.

* fix(confluence): classify errors and fail closed on empty listings

Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-16 14:38:16 +08:00
po-et 90e3977714 test(datasource): keep yuque and ima config tests off live DNS
parseYuqueConfig and parseIMAConfig validate DefaultBaseURL through
ValidateConnectorBaseURL, which resolves the host with net.LookupIP.
Their TestMain only whitelists loopback, so the config tests fail on
any machine where www.yuque.com or ima.qq.com resolves to a restricted
address (a local DNS sinkhole, for example), and the result depends on
the network the tests happen to run on.

Whitelist the default host as well, which is what the feishu tests
already do for open.feishu.cn.
2026-09-15 15:03:45 +08:00
DEMO 5d3114fc9e
feat(datasource): add DingTalk Docs integration with safe sync and localization (#2313)
* feat(datasource): add DingTalk Docs integration

* fix(datasource): harden DingTalk sync and complete translations

* fix(datasource): close DingTalk review gaps in sync and rendering

Prefer millisecond node timestamps, keep full-sync deletion reconciliation, and stop dropping official block payloads so incremental updates and Markdown import stay complete.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-14 19:53:29 +08:00
wizardchen 1f82fbc9b5 fix(datasource): preserve distinct GitLab files during deduplication 2026-09-14 19:32:41 +08:00
小妍儿 ✨ 061df8efdc
feat(datasource): persist source created time; Feishu uses content edit time for source_updated_at (#3083)
* feat(datasource): persist source document creation time and use content edit time for Feishu

Follow-up to 160c806b (Refs #3016), which started persisting the source
last-modified time as metadata.source_updated_at. Two gaps remained:

1. Creation time was never carried. FetchedItem gains CreatedAt and
   ingestItem persists it as metadata.source_created_at (RFC3339, UTC)
   with the same absent-means-unknown rule as source_updated_at.

2. The Feishu wiki connector filled FetchedItem.UpdatedAt from
   NodeEditTime, which moves when a node is renamed or dragged in the
   tree without any content change. The connector's own cursor already
   prefers ObjEditTime for exactly this reason; use the same rule for the
   persisted value (contentEditTime / contentCreateTime, node_* as
   fallback), so source_updated_at reflects document edits.

Feishu Drive populates CreatedAt from created_time. GitLab stops
fabricating UpdatedAt = time.Now(): a fetch timestamp is not a document
timestamp, and an absent value is the documented "unknown" (the commit
time is not fetched today; leaving it unset is honest until it is).

Tests: ingestItem covers both keys present/absent; the wiki fetch test
asserts obj_* times win and node_* are the fallback.

* style(gitlab): break up the FetchedItem literal (lll)

---------

Co-authored-by: xiaoyaner0201 <>
2026-09-07 19:39:01 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
lyingbug 0cc3c33530
fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题 (#2673)
* fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题

避免 project_id 二次 URL 编码,并确保每个数据源使用独立 client;
保存时校验 projects 配置,补齐增量同步过滤与相关单测。

* style(datasource/gitlab): gofmt connector sources
2026-08-12 16:22:02 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00
wizardchen 8d5540e26d feat(datasource/ima): sync IMA notes via the note OpenAPI
Notes (media_type=11) were skipped as "unsupported by wiki OpenAPI". The wiki
namespace indeed never exposes a note body, but IMA reads it from a second
namespace: get_media_info reports a notebook_id, which
/openapi/note/v1/get_doc_content turns into the note text. The connector
already parsed that notebook_id into item metadata and then dropped the
record, so this wires up the one call that was missing.

Note bodies arrive as plain text and are ingested as Markdown, since IMA notes
are authored as rich text and the export keeps its heading and list markers.
An unreadable note now takes the same retry path as a failed download instead
of being recorded as synced; empty notes are skipped.

AI sessions (12) and video parses (16) stay skipped, as those genuinely have
no read endpoint, and the connector description now names them specifically
rather than implying notes are unsupported.

Also accept the {retcode, errmsg} spelling of the response envelope alongside
{code, msg}. The reference MCP implementation confirms {code, msg} is correct,
but several published IMA references disagree, and under the wrong spelling
the status code would decode as 0 and turn every API error into a silent
success.
2026-08-12 14:56:25 +08:00
wizardchen 3d645d2578 fix(datasource/ima): retry transient sync failures instead of losing documents
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.

Alongside that:

- Drop media_type from logicalKey. get_knowledge_list omits the field, so
  folding it in meant every external_id would shift the day IMA starts
  populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
  bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
  image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
  per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.
2026-08-12 14:56:25 +08:00
joeyczheng 664d193ca7 feat(datasource): add Tencent IMA datasource
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-08-12 14:56:25 +08:00
Greenplumwine 439af21f9b
feat(feishu-driver): 新增飞书云盘数据源 (#2466)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-05 19:39:49 +08:00
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
nullkey c695125dce fix(feishu): resilient large-wiki sync — retry, streaming resume, failure visibility
Large Feishu wikis stopped syncing after the first batch (Tencent/WeKnora#2136:
5000+ docs, only ~150 ingested; #1262: a child-list failure aborted the run).
Three compounding causes, fixed in layers:

1. Rate-limit / transient failures had no retry. Feishu's wiki/drive/export
   APIs are aggressively rate limited; a thousand-doc sync issues tens of
   thousands of calls, and a single 429/5xx burst failed whole swathes of docs.
   Add exponential backoff honouring Retry-After to doRequest (JSON) and
   downloadRawBytes (files): 429 up to 3x, 5xx once, transport errors back off.

2. The batch path fetched the entire tree into memory before ingesting, so a
   sync that hit the 2h task timeout lost all progress and restarted from
   scratch, and a transient per-document export failure advanced the cursor and
   skipped that doc permanently. Add an optional StreamingConnector interface,
   implemented for Feishu: FetchStream interleaves fetch -> ingest -> checkpoint,
   bounding memory to one item and persisting a per-node edit-time cursor at
   page boundaries. The service dispatches to it via type assertion; connectors
   without it fall back to FetchAll / FetchIncremental unchanged (non-breaking).
   A resumed/incremental run skips nodes already recorded at their current edit
   time (converges), a failed fetch does not advance the cursor (retried next
   run instead of dropped), and partial listing failures emit error items and
   carry prior edit times forward instead of aborting the sync.

3. Failures and skips were invisible. Unsupported node types (mindnote/slides/…)
   were skipped with no item, error or log ("13 docs synced only 3"). FetchAll
   now tallies discovered/fetched/failed/skipped per resource with a per-type
   skip breakdown and a progress line every 100 nodes. The backend already
   persisted per-item errors in SyncLog.result.errors but the frontend type
   omitted the field; declare `result` on SyncLog and render up to 50 failed
   documents in the sync-log drawer (#1262). i18n for zh-CN/en-US/ko-KR/ru-RU.
   result.errors is a jsonb column shipped in every list response, so it is
   capped at 100 samples server-side (recordSyncError); the exact failure total
   comes from the bounded items_failed counter, which the drawer uses for its
   count and "N more" line so a sync that fails thousands of docs stays bounded.
   Each sample is a structured, localisable SyncItemError instead of a raw
   "title: ...api error: status=500 body={json}" dump that repeated the title
   and leaked status/body/log_id. The connector (feishuFailure) classifies the
   error into a stable i18n code (+ the feishu code as a param); the service
   builds {title, code, params, message}; the drawer localises via vue-i18n to
   the viewer's language (zh/en/ko/ru), falling back to message. A custom
   UnmarshalJSON keeps old string-form logs readable. The banner is localised
   from items_failed too. Raw errors stay in the server logs. Follows the
   two-tier norm of Airbyte/Fivetran/Onyx: human, actionable, localised
   summaries in the UI; raw detail in logs. Transient errors say "will retry"
   (the cursor is retained); auth/permission errors point at the fix instead.

Tests: client_retry_test (backoff), tally_test (accounting), connector_stream /
datasource_stream (streaming dispatch + checkpoint), connector_convergence
(timeout + transient-failure resume converges to full coverage; proven to catch
#2136 via bug injection), datasource_result_cap (error-sample cap + structured
localisable SyncItemError, no raw body), connector_error_reason (error→i18n-code
classification), sync_item_error (legacy-string + structured back-compat) — all
in the normal suite. connector_realapi / connector_seed are an opt-in real-Feishu
harness gated by the feishu_integration build tag AND env vars, never run in CI.
2026-07-21 16:14:22 +08:00
qilifan 9afb8b7b68 feat(im,datasource): support Lark, Feishu's international edition
Lark and Feishu are the same product deployed on two isolated clouds
(open.larksuite.com and open.feishu.cn). WeKnora touches both from two
places — the IM bot and the Wiki data-source connector — and this brings
Lark to parity across both. Neither API surface differs, so rather than
duplicate either implementation, each is parametrized by a Region and
registered twice.

IM (internal/im/feishu):
  Every API call routes through a.api(), which prefixes the region's
  host, and the WebSocket client passes larkws.WithDomain — the SDK
  defaults to open.feishu.cn, so Lark long connections would otherwise
  fail to authenticate.

Data source (internal/datasource/connector/feishu):
  Lark was half-supported: the client honoured a base_url override and a
  LarkBaseURL constant existed, but the connector registered only as
  "feishu" and the UI never exposed base_url, so Lark wikis could not be
  configured at all. Lark is now its own connector type. base_url is kept
  as an explicit override so data sources that pointed a "feishu"
  connector at open.larksuite.com keep working.

Bugs found and fixed along the way:

- Wiki resource links were hardcoded to https://feishu.cn/wiki/{token} in
  four places, so a Lark wiki linked into the wrong cloud entirely.
- Bot identity is now platform-prefixed (feishu:cli_x vs lark:cli_x).
  BotIdentity carries a unique index, so without this a Lark channel
  would be rejected as a duplicate of an unrelated Feishu bot sharing an
  app_id.
- The card image cache is now keyed by app_id. image_keys are issued per
  app on one cloud; a Feishu key handed to a Lark app fails the card
  update with code=200570.
- The datasource doc claimed Lark "adapts automatically" — it could not,
  since the UI had no way to reach base_url.

Permissions are where the clouds genuinely diverge. Identical APIs do not
mean identical permission catalogues: aily:file:read, aily:file:write and
corehr:file:download exist on Feishu but not on Lark, so pasting Feishu's
scope JSON into Lark fails to import. That JSON is also over-broad (it
covers products WeKnora never calls) and simultaneously incomplete (it
omits the drive:*/docx:* scopes the connector needs). Lark therefore gets
its own list, derived from the 17 endpoints the code actually calls and
checked identifier-by-identifier against Lark's own API reference, plus a
full Feishu→Lark mapping table marking each scope verified, unverified,
or absent. Both features use tenant_access_token only — no user_access_
token exists anywhere in the repo — so the user dimension stays empty.

Known limitation: Region.ThinkingText makes Lark's initial card
placeholder English, but once streaming starts the think-block headers
come from the shared MarkdownThinkStyle and are Chinese on every platform
(Lark, Slack, Telegram, Mattermost alike). Fixing that means threading a
locale through FormatIMDisplayContent, which has no platform context — a
cross-platform change, not a Lark one. Documented as such.

The channel-creation 400 message is now derived from validIMPlatforms so
the list cannot drift as platforms are added.

Tests: region invariants and per-region URL construction on both sides,
the Feishu/Lark bot-identity split, platform-to-channel mapping, base_url
override precedence, an end-to-end SendReply against a fake Open Platform
(token fetch, reply, thread-unsupported fallback, hard-error surfacing),
and a ListResources drive asserting wiki links never leak the other
cloud's host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:05:54 +08:00
wizardchen d496d0fc73 feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
2026-07-10 23:51:29 +08:00
wizardchen cd65665894 fix(security): close remaining GHSA gaps from security triage
Block scheme-prefixed loopback storage checks, validate Notion attachment
URLs, sanitize wiki markdown HTML, and require MCP_SERVER_AUTH_TOKEN for
network MCP transports.
2026-07-02 15:03:53 +08:00