WeKnora/internal/infrastructure
ordinary being c44f939211
fix(web-search): bound provider response bodies instead of buffering them whole (#4370)
Every provider read its response with an unbounded io.ReadAll, or handed
resp.Body straight to json.NewDecoder, so a misconfigured or hostile endpoint
decided how much memory a single search allocated. Two providers made it worse
by reading before they looked at the status code and then interpolating the whole
body into both a log line and the error the caller sees - so a failing endpoint
could push an unbounded blob into the error string as well as into memory.

Reuse the cap the datasource connectors already apply to their API bodies
(readCapped + maxJSONResponseBytes + errResponseTooLarge, added for feishu, ima,
notion and yuque in #3985) rather than inventing a second dialect here. Apply it
to the six providers that read a response body, and bound failed-response
previews the way searxng.go already bounded its own and the connectors bound
theirs.

The two providers that decoded straight from resp.Body now buffer through
readCapped first, since a decoder cannot be capped without buffering.
2026-10-16 17:16:39 +08:00
..
chunker fix(chunker): respect Markdown fences during heading detection 2026-10-08 21:06:59 +08:00
docparser feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects (#3497) 2026-10-16 16:49:44 +08:00
web_fetch feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393) 2026-09-23 12:02:06 +08:00
web_search fix(web-search): bound provider response bodies instead of buffering them whole (#4370) 2026-10-16 17:16:39 +08:00