WeKnora/internal
Captain a5bdf82a02
fix(storage): let object storage transfers outlive 30 seconds (#3368)
* fix(storage): let object storage transfers outlive 30 seconds

The S3, COS, KS3, OBS and OSS file services build their SDK client from
DefaultSSRFSafeHTTPClientConfig, whose 30-second Timeout is Go's
whole-request timeout, body included. So every request had to finish
within 30 seconds whatever it carried: a 40 MB PDF needs about
11 Mbit/s of uplink to make it as one COS Object.Put (OSS uploads in
10 MB parts, so there the cap was per part), and the same file coming
back from a distant region was cut the same way, while the local backend
took any size (#3306). MinIO and TOS, which only hand the SDK a
transport, never had the cap.

Give the object storage clients a config without the whole-request
timeout, and bound connection setup on their transport instead (TLS
handshake, response headers, idle connections), so a peer that accepts
the connection and then goes silent still fails within about a minute;
the header wait starts after the request body is written, so it does
not cap an upload. A transfer is then bounded by its request context.
The bucket probe every constructor runs, which had relied on the client
timeout, gets a 30-second context of its own, and the OSS and KS3
connectivity checks get the 10-second deadline the other backends
already set. KS3 called its SDK without a context at all; it now passes
the caller's, so the same bounds apply there.

* fix(storage): bound object transfers and split bucket probes

Dropping the 30s client timeout left asynq/API transfers unbounded, and Head+Create shared one setup context so a slow miss aborted create. Add a 30-minute fallback deadline, give each probe its own timeout, and only create a KS3 bucket on a real 404.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-19 21:30:55 +08:00
..
agent fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
application fix(storage): let object storage transfers outlive 30 seconds (#3368) 2026-09-19 21:30:55 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
browserskill fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
common fix(markdown): sanitize invalid UTF-8 and NUL bytes 2026-08-31 12:24:35 +08:00
config feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
container fix(storage): let object storage transfers outlive 30 seconds (#3368) 2026-09-19 21:30:55 +08:00
database feat(agent): size agent history by the window and calibrate token estimates (#3413) 2026-09-18 22:51:31 +08:00
datasource refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
embedpolicy fix(embed): enforce host origin allowlists on framed pages 2026-09-14 18:03:57 +08:00
errors feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
event feat(sandbox): stream live shell and install command output 2026-09-12 10:16:39 +08:00
filetransport refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
handler fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
im fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
infrastructure fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
ipclass fix: align client sessions and integration request handling (#3305) 2026-09-16 16:20:57 +08:00
logger fix(sandbox): record product-level Langfuse spans and detach idle sweeps 2026-08-31 19:42:32 +08:00
mcp fix: align client sessions and integration request handling (#3305) 2026-09-16 16:20:57 +08:00
mcpserver refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381) 2026-09-18 14:12:22 +08:00
middleware fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
modelcontext fix(agent): make read_document queries keyword-based and bound its pages 2026-09-18 15:09:25 +08:00
models fix(embedding): preserve batching and concurrency limits for WeKnoraCloud 2026-09-19 20:26:46 +08:00
ratelimit feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
router fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
runtime fix(redis): harden TLS tests and surface TLS config at startup 2026-07-06 23:04:52 +08:00
sandbox perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
searchutil fix(chat): keep OCR/caption text of scanned-page image hits in merge (#3064) 2026-09-07 11:38:47 +08:00
storageallowlist fix(storage): align default provider fallback with STORAGE_ALLOW_LIST 2026-07-15 11:05:00 +08:00
storageurl refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
stream fix(chat): preserve steering continuity and composer focus 2026-09-09 20:15:23 +08:00
textconv fix: remove GPL converter and bundle third-party license sources 2026-09-08 15:19:10 +08:00
tracing/langfuse fix(milvus): preserve keyword retrieval scores 2026-09-17 11:59:40 +08:00
types fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
utils refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00