WeKnora/internal/infrastructure/docparser
ordinary being 1fe804bd5c
fix(docparser): bound MinerU result packages and their expansion (#4396)
A MinerU result package arrives over the network from a third-party parse API and
was buffered whole before anything about it was validated. The Cloud converter
downloaded it with an unbounded io.ReadAll, even though the V1 client already caps
the same artifact at mineruV1MaxZipBytes - the two implementations of one feature
disagreed about the limit.

Decompression was unbounded on every path. readZipEntry and readZipEntryBytes
called io.ReadAll on a decompressed archive entry, and extractMarkdownZip keeps
every extracted image in memory until the caller hands the slice on, so a package
of many small entries expanded without ever tripping a per-entry check. The
declared UncompressedSize64 is attacker or vendor controlled and may understate
the expansion, so the header is checked and the read is capped as well.

Cap the download at the V1 ceiling, each entry at maxZipEntryBytes, and the
accumulated images at maxZipImagesTotalBytes. The image budget is a parameter on
extractMarkdownZipBounded so a test can exercise it without materialising the
production figure, mirroring the jsonLimit field added in #3985.
2026-10-16 19:37:27 +08:00
..
anydoc fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938) 2026-10-08 01:21:57 +08:00
anydoc_reader.go feat(docparser): upgrade AnyDoc to 0.2.4 and parse assets once 2026-09-30 15:25:01 +08:00
anydoc_reader_linked_test.go feat(docparser): upgrade AnyDoc to 0.2.4 and parse assets once 2026-09-30 15:25:01 +08:00
anydoc_reader_test.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
anydoc_vector_image_test.go fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938) 2026-10-08 01:21:57 +08:00
builtin_converter.go fix(docparser): preserve standalone image uploads from icon filter 2026-04-29 11:58:02 +08:00
cloud_poll_timeout_test.go fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891) 2026-10-08 14:02:42 +08:00
doc.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
engine_registry.go revert: take the plugin system back out of main 2026-09-26 14:58:18 +08:00
engine_registry_reader_test.go fix(parser): default ppt/pptx to markitdown when no engine rules (#2882) 2026-08-31 14:58:15 +08:00
engine_registry_test.go fix(parser): keep PDFs on builtin instead of defaulting to anydoc 2026-09-21 16:28:02 +08:00
engines.go feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU 2026-09-24 20:40:52 +08:00
grpc_parser.go fix(milvus): validate connections and preserve gRPC resolver defaults (#3654) 2026-10-08 00:48:45 +08:00
grpc_parser_resolver_test.go fix(milvus): validate connections and preserve gRPC resolver defaults (#3654) 2026-10-08 00:48:45 +08:00
helpers.go fix(docparser): make the MinerU parse timeouts configurable 2026-09-25 18:38:25 +08:00
html_table_normalizer.go fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
html_table_normalizer_test.go fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
http_parser.go feat(chat): open cited documents at the cited passage (#3700) 2026-09-25 13:26:16 +08:00
image_resolver.go feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects (#3497) 2026-10-16 16:49:44 +08:00
image_resolver_relative_html_test.go fix(docparser): address review feedback on PR #1404 2026-05-21 11:42:56 +08:00
image_resolver_test.go feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects (#3497) 2026-10-16 16:49:44 +08:00
json_converter.go fix(knowledge): reject invalid JSON uploads before async parse (#3360) 2026-09-25 22:23:27 +08:00
json_converter_test.go fix(knowledge): reject invalid JSON uploads before async parse (#3360) 2026-09-25 22:23:27 +08:00
markdown_image_scanner.go fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092) 2026-09-08 19:23:13 +08:00
markdown_image_scanner_test.go fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092) 2026-09-08 19:23:13 +08:00
mineru_cloud_converter.go fix(docparser): bound MinerU result packages and their expansion (#4396) 2026-10-16 19:37:27 +08:00
mineru_converter.go fix(docparser): make the MinerU parse timeouts configurable 2026-09-25 18:38:25 +08:00
mineru_converter_ssrf_test.go feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU 2026-09-24 20:40:52 +08:00
mineru_converter_test.go feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU 2026-09-24 20:40:52 +08:00
mineru_layout.go fix(docparser): bound MinerU result packages and their expansion (#4396) 2026-10-16 19:37:27 +08:00
mineru_layout_precision_test.go fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794) 2026-09-27 20:32:26 +08:00
mineru_timeout_test.go fix(docparser): make the MinerU parse timeouts configurable 2026-09-25 18:38:25 +08:00
mineru_v1_client.go feat(chat): open cited documents at the cited passage (#3700) 2026-09-25 13:26:16 +08:00
mineru_v1_client_test.go feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU 2026-09-24 20:40:52 +08:00
paddleocr_vl_cloud_converter.go fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891) 2026-10-08 14:02:42 +08:00
paddleocr_vl_converter.go fix(docparser): make the MinerU parse timeouts configurable 2026-09-25 18:38:25 +08:00
paddleocr_vl_converter_test.go fix(docparser): make PaddleOCR-VL request timeout configurable (#3537) 2026-09-23 11:58:59 +08:00
paddleocr_vl_layout.go feat(chat): open cited documents at the cited passage (#3700) 2026-09-25 13:26:16 +08:00
resolve_remote_html_images_test.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
resolve_remote_images_test.go fix: deep-copy stored files and images when cloning a knowledge base 2026-06-03 14:45:59 +08:00
source_blocks.go feat(chat): open cited documents at the cited passage (#3700) 2026-09-25 13:26:16 +08:00
source_blocks_test.go feat(chat): open cited documents at the cited passage (#3700) 2026-09-25 13:26:16 +08:00
weknoracloud_http_reader.go fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891) 2026-10-08 14:02:42 +08:00
zip_limits.go fix(docparser): bound MinerU result packages and their expansion (#4396) 2026-10-16 19:37:27 +08:00
zip_limits_test.go fix(docparser): bound MinerU result packages and their expansion (#4396) 2026-10-16 19:37:27 +08:00