WeKnora/internal/infrastructure/docparser
GUO Zhaogong 8d7298fb5d
fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092)
* fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders

KnowledgePostProcessService.Handle gathered "text-like" chunks through
chunkService.ListChunksByKnowledgeID, which is text-only by design, so the
image_ocr / image_caption chunks written by multimodal parsing were never
returned. For a scanned PDF the text chunks contain nothing but markdown
image links, so graph extraction ran on empty input: most calls failed
with "unexpected end of JSON input" and the rest echoed the few-shot
example from the extract config into the graph.

- Fetch the exact chunk types the post-process enriches via
  chunkRepo.ListChunksByKnowledgeIDAndTypes (text, image_ocr,
  image_caption); the service now receives interfaces.ChunkRepository
  through dig like knowledgeService already does
- Build a dedicated graphChunks list for graph extraction: skip
  image_caption chunks (they duplicate the OCR text) and chunks that
  contain only image links (chunkHasExtractableText)
- graphChunkCount, the enqueue loop and the log line use graphChunks;
  summary, wiki, auto-tag and question generation keep using textChunks
- Tests: fake ChunkRepository in the wiki-enqueue test, table test for
  chunkHasExtractableText

* fix(graph): skip figure OCR and harden empty-chunk detection

Reuse the CommonMark image scanner plus extractRealText so titles with
parentheses and <image> wrappers are not treated as prose. Only enqueue
OCR for graph extraction when the parent text chunk has no extractable
text, and skip link-only chunks for question generation as well.

* style(graph): wrap long post-process extract log line

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-08 19:23:13 +08:00
..
anydoc fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎 2026-08-16 13:31:51 +08:00
anydoc_reader.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
anydoc_reader_test.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
builtin_converter.go fix(docparser): preserve standalone image uploads from icon filter 2026-04-29 11:58:02 +08:00
doc.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
engine_registry.go feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档 2026-08-16 11:37:27 +08:00
engine_registry_reader_test.go fix(parser): default ppt/pptx to markitdown when no engine rules (#2882) 2026-08-31 14:58:15 +08:00
engine_registry_test.go fix(parser): prefer anydoc for every type it converts when linked 2026-08-31 14:58:15 +08:00
engines.go fix(parser): prefer anydoc for every type it converts when linked 2026-08-31 14:58:15 +08:00
grpc_parser.go refactor(parser): reorganize Markdown parser and enhance gRPC document reading 2026-06-03 12:29:13 +08:00
helpers.go feat(weknoracloud): implement credential management and status checks 2026-04-14 17:36:01 +08:00
html_table_normalizer.go fix(docparser): normalize PaddleOCR-VL HTML tables (#1725) 2026-06-22 17:31:26 +08:00
html_table_normalizer_test.go fix(docparser): normalize PaddleOCR-VL HTML tables (#1725) 2026-06-22 17:31:26 +08:00
http_parser.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
image_resolver.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
image_resolver_relative_html_test.go fix(docparser): address review feedback on PR #1404 2026-05-21 11:42:56 +08:00
image_resolver_test.go fix: preserve MHTML image resolution and Markdown structure (#1743) 2026-06-22 15:53:54 +08:00
json_converter.go feat(docparser): enhance sortedKeys function for numeric string sorting 2026-03-25 22:08:29 +08:00
json_converter_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
markdown_image_scanner.go fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092) 2026-09-08 19:23:13 +08:00
markdown_image_scanner_test.go fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092) 2026-09-08 19:23:13 +08:00
mineru_cloud_converter.go refactor(parser): reorganize Markdown parser and enhance gRPC document reading 2026-06-03 12:29:13 +08:00
mineru_converter.go fix(docparser): resolve MinerU results by upload filename stem 2026-08-13 18:51:27 +08:00
mineru_converter_ssrf_test.go fix(security): redact IM list credentials and harden MinerU SSRF 2026-07-02 17:58:20 +08:00
mineru_converter_test.go style(docparser): gofmt mineru converter tests 2026-08-13 18:51:27 +08:00
paddleocr_vl_cloud_converter.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
paddleocr_vl_converter.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00
resolve_remote_html_images_test.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
resolve_remote_images_test.go fix: deep-copy stored files and images when cloning a knowledge base 2026-06-03 14:45:59 +08:00
weknoracloud_http_reader.go fix(security): harden outbound URL SSRF protection (#2626) 2026-08-10 13:24:14 +08:00