WeKnora/internal/container
Captain a5bdf82a02
fix(storage): let object storage transfers outlive 30 seconds (#3368)
* fix(storage): let object storage transfers outlive 30 seconds

The S3, COS, KS3, OBS and OSS file services build their SDK client from
DefaultSSRFSafeHTTPClientConfig, whose 30-second Timeout is Go's
whole-request timeout, body included. So every request had to finish
within 30 seconds whatever it carried: a 40 MB PDF needs about
11 Mbit/s of uplink to make it as one COS Object.Put (OSS uploads in
10 MB parts, so there the cap was per part), and the same file coming
back from a distant region was cut the same way, while the local backend
took any size (#3306). MinIO and TOS, which only hand the SDK a
transport, never had the cap.

Give the object storage clients a config without the whole-request
timeout, and bound connection setup on their transport instead (TLS
handshake, response headers, idle connections), so a peer that accepts
the connection and then goes silent still fails within about a minute;
the header wait starts after the request body is written, so it does
not cap an upload. A transfer is then bounded by its request context.
The bucket probe every constructor runs, which had relied on the client
timeout, gets a 30-second context of its own, and the OSS and KS3
connectivity checks get the 10-second deadline the other backends
already set. KS3 called its SDK without a context at all; it now passes
the caller's, so the same bounds apply there.

* fix(storage): bound object transfers and split bucket probes

Dropping the 30s client timeout left asynq/API transfers unbounded, and Head+Create shared one setup context so a slow miss aborted create. Add a 30-minute fallback deadline, give each probe its own timeout, and only create a KS3 bucket on a real 404.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-19 21:30:55 +08:00
..
audit_sink.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
audit_sink_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
container.go fix(storage): let object storage transfers outlive 30 seconds (#3368) 2026-09-19 21:30:55 +08:00
datasource_registry_test.go feat(datasource): add DingTalk Docs integration with safe sync and localization (#2313) 2026-09-14 19:53:29 +08:00
engine_factory.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
engine_factory_opensearch_test.go fix(retriever): rebuild a missing vector store engine on demand 2026-07-29 12:25:18 +08:00
engine_factory_test.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
recover_pending_wiki_tasks.go fix(tasks): dequeue work for deleted knowledge bases 2026-07-23 12:17:14 +08:00
reset_pending_tasks.go fix(knowledge): recover tasks safely after server restart 2026-07-14 17:55:26 +08:00
reset_pending_tasks_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
retrieve_registry_wiring_test.go fix(retriever): rebuild a missing vector store engine on demand 2026-07-29 12:25:18 +08:00
sandbox.go feat(session): fork conversations from a historical user message (#3304) 2026-09-17 14:20:18 +08:00
sandbox_test.go feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断 2026-08-11 15:42:02 +08:00
ssrf_test.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
storagebackend_migration_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00