WeKnora/internal/datasource/connector
wizardchen 3d645d2578 fix(datasource/ima): retry transient sync failures instead of losing documents
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.

Alongside that:

- Drop media_type from logicalKey. get_knowledge_list omits the field, so
  folding it in meant every external_id would shift the day IMA starts
  populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
  bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
  image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
  per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.
2026-08-12 14:56:25 +08:00
..
feishu feat(feishu-driver): 新增飞书云盘数据源 (#2466) 2026-08-05 19:39:49 +08:00
ima fix(datasource/ima): retry transient sync failures instead of losing documents 2026-08-12 14:56:25 +08:00
notion ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
rss fix(datasource): optimize RSS incremental sync and surface partial failures 2026-06-23 17:30:38 +08:00
yuque fix(security): harden web_fetch and datasource connector SSRF paths 2026-07-02 12:27:48 +08:00