Commit Graph

10 Commits (main)

Author SHA1 Message Date
ordinary being 7925f774d1
fix(datasource/gitlab): emptied files no longer leave their old body searchable (#4008)
A blob that a commit emptied is still a blob, and ingestion reads an item with
neither content nor URL as nothing to sync: applyFetchedItem counts it as
skipped and the sync reports success. The body indexed from an earlier commit
stays fully searchable, and no later sync revisits it — a full sync still sees
the path in the tree, and an incremental sync only walks the diffs of commits
newer than the recorded head, which the emptying commit already moved past.

Emit the file's path as the content in that case so ingestion replaces the stale
body, matching what the Notion (b9be7d0d), Confluence and IMA (b71b31c8)
connectors already do. The check is on length alone: a blob holding only
whitespace still carries content the repository has.
2026-10-12 13:06:19 +08:00
linus-liu-web a2aeb97a01
fix(gitlab): 分页循环补重复页防护与跳数上限 (#3841)
projects() 与 tree() 只依据服务端返回的 X-Next-Page 判断是否继续,
该值重复或不前进时循环不会结束:同一页被反复请求、结果切片持续
增长,只能等上层 context 到期。

两条循环改用 trackPage:页码已请求过即返回错误,不静默当作读完;
页数达到 maxPaginationHops 时记警告并返回错误。正常翻页路径不变。

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:37 +08:00
linus-liu-web 1dbe9d0c1a
fix(gitlab): 仓库文件下载补体积上限,超限报错而不是无界读入 (#3887)
* fix(gitlab): 仓库文件下载没有体积上限,超限改为显式报错

getRaw 直接用 io.ReadAll 缓冲整个 blob,get/getPage 的 JSON 读取同样无界。
GitLab 的 tree/compare 接口不返回 blob 大小,下载处是唯一能拦的地方,所以
同步一个音频或大 PDF 就会把一个完整 blob 读进进程内存。

对齐兄弟 connector(钉钉 16MiB JSON 上限、飞书 512MiB 下载上限):

- maxJSONResponseBytes(16 MiB)用于 get/getPage;
- maxRawFileBytes(512 MiB,与飞书 maxFeishuDownloadBytes 对齐)用于 getRaw;
- base64 回退走 getCapped,其请求体是 base64 后的 blob(约 4/3)加信封。

超限返回错误而不是截断:截断的正文会被当成完整文档入库。非 2xx 状态先于
读取判定,避免超大错误页把真实状态码盖掉。

* fix(gitlab): 将 compare 响应上限与元数据分开

compare 正文包含完整提交和 patch,沿用 16MiB 元数据上限会中断增量同步且游标无法前进。超限错误改为带上未编码的仓库路径。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-30 14:33:15 +08:00
yeliheng-010 72e105f46a fix(gitlab): 按提交快照比较增量同步变更 2026-09-29 00:50:05 +08:00
yeliheng-010 608027179f fix(gitlab): 补齐项目资源列表分页 2026-09-24 07:52:32 +08:00
hahahahahayesyeseys 11edd16819 fix(datasource): fail closed on GitLab compare errors 2026-09-24 07:49:07 +08:00
wizardchen 1f82fbc9b5 fix(datasource): preserve distinct GitLab files during deduplication 2026-09-14 19:32:41 +08:00
小妍儿 ✨ 061df8efdc
feat(datasource): persist source created time; Feishu uses content edit time for source_updated_at (#3083)
* feat(datasource): persist source document creation time and use content edit time for Feishu

Follow-up to 160c806b (Refs #3016), which started persisting the source
last-modified time as metadata.source_updated_at. Two gaps remained:

1. Creation time was never carried. FetchedItem gains CreatedAt and
   ingestItem persists it as metadata.source_created_at (RFC3339, UTC)
   with the same absent-means-unknown rule as source_updated_at.

2. The Feishu wiki connector filled FetchedItem.UpdatedAt from
   NodeEditTime, which moves when a node is renamed or dragged in the
   tree without any content change. The connector's own cursor already
   prefers ObjEditTime for exactly this reason; use the same rule for the
   persisted value (contentEditTime / contentCreateTime, node_* as
   fallback), so source_updated_at reflects document edits.

Feishu Drive populates CreatedAt from created_time. GitLab stops
fabricating UpdatedAt = time.Now(): a fetch timestamp is not a document
timestamp, and an absent value is the documented "unknown" (the commit
time is not fetched today; leaving it unset is honest until it is).

Tests: ingestItem covers both keys present/absent; the wiki fetch test
asserts obj_* times win and node_* are the fallback.

* style(gitlab): break up the FetchedItem literal (lll)

---------

Co-authored-by: xiaoyaner0201 <>
2026-09-07 19:39:01 +08:00
lyingbug 0cc3c33530
fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题 (#2673)
* fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题

避免 project_id 二次 URL 编码,并确保每个数据源使用独立 client;
保存时校验 projects 配置,补齐增量同步过滤与相关单测。

* style(datasource/gitlab): gofmt connector sources
2026-08-12 16:22:02 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00