The sync cursor was built from the raw IMA listing before any content was fetched, so an item whose get_media_info or download failed was still recorded as "seen with this media_id". Every later incremental sync then skipped it as unchanged and the document was silently missing from the knowledge base for good. Track the listed keys and the successfully synced keys separately: the cursor now only remembers what was actually resolved, while delete detection keeps using the full listing so a failed download is never mistaken for a document that vanished. Deterministic skips (unsupported media types, items with no URL) are still recorded so they are not re-probed every cycle. Alongside that: - Drop media_type from logicalKey. get_knowledge_list omits the field, so folding it in meant every external_id would shift the day IMA starts populating it, which reads downstream as "every document deleted". - Download URL-backed media that carry auth headers rather than handing the bare URL to the ingest layer, which cannot replay those headers. - Derive image extensions from the response Content-Type; IMA reports every image as media_type=9, so a JPEG was being written as .png. - Stop logging API response bodies, which carry pre-signed storage URLs and per-request auth headers. - Validate media URLs against the SSRF policy before the first connection. - Stop writing the redundant kb_media map into the persisted cursor. - Say in the connector description that notes and AI sessions are skipped. |
||
|---|---|---|
| .. | ||
| packages | ||
| public | ||
| src | ||
| .dockerignore | ||
| .gitignore | ||
| Dockerfile | ||
| docker-entrypoint.sh | ||
| embed.html | ||
| env.d.ts | ||
| index.html | ||
| nginx.conf | ||
| package-lock.json | ||
| package.json | ||
| package.json.md5 | ||
| pnpm-workspace.yaml | ||
| tsconfig.app.json | ||
| tsconfig.json | ||
| tsconfig.node.json | ||
| vite.config.ts | ||