The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.
Alongside that:
- Drop media_type from logicalKey. get_knowledge_list omits the field, so
folding it in meant every external_id would shift the day IMA starts
populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.