Commit Graph

66 Commits (fc2c12e8ec08efc9b948438fb720db1ff88ca9ca)

Author SHA1 Message Date
LuckVd 81142dfd17
fix(docreader): keep DOCX table content in parsed text (#2898)
* fix(docreader): keep DOCX table content in parsed text

Docx.__call__ returns tables separately from the text lines, and
parse_into_text used to drop them, silently losing all table content
in documents parsed with the builtin engine. Render the tables as GFM
Markdown (header row + delimiter + data rows) and append them to the
parsed text.

Fixes #2881

* fix(docreader): render DOCX tables from grid cells in document order

Skip the HTML round-trip that treated equal or empty adjacent cells as
colspan and dropped columns from GFM. Emit one column per python-docx
grid slot and interleave tables with paragraphs in body order.

---------

Co-authored-by: root <root@localhost.localdomain>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:56:17 +08:00
ResistanceTo 77245f0f4b
fix(docreader): handle Header object when stripping mhtml header (#2902)
* fix(docreader): handle Header object when stripping mhtml header

* fix(docreader): decode 8-bit MHTML headers without crashing (#2901)

compat32 wraps non-ASCII 8-bit headers as Header; str() turns them into
U+FFFD and HTML Content-Location still crashed on .lower()/urljoin.

Refs: https://github.com/Tencent/WeKnora/issues/2901

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:14:58 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
wizardchen 91186b81b7 fix(docreader): fail URL scrape instead of indexing error text (#2883)
Empty scrape/parse results were returned as "Error parsing web page: …"
and then chunked into the knowledge index, hiding the failure. Raise
WebParseError so DocReader returns error and the knowledge is marked failed.

Refs: https://github.com/Tencent/WeKnora/issues/2883
2026-08-31 14:12:11 +08:00
sufan721 3842068d22
fix: stop markdown table regex catastrophic backtracking (#2770)
* fix: stop markdown table regex catastrophic backtracking

The line_pattern in MarkdownTableUtil used overlapping character classes
([\t ]* and [^|\r\n]*) so a `|`-prefixed line missing its trailing `|`
forced the engine to explore an exponential number of backtracking paths,
stalling the single-threaded docreader gRPC event loop (100% CPU, health
check timeout).

Replace whole-text regex substitution with an O(n) line-based pass that
only treats rows beginning and ending with `|` as tables, and preserves
the previous align_pattern semantics via _is_alignment_row.

Fixes #2768

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: keep lone pipe rows and normalize CRLF in table formatter

A single "|" was rewritten into an empty table row and then dropped.
Unify CR/LF before the line scan so exported Markdown tables format
without mixed endings.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-24 18:08:07 +08:00
wizardchen f20427df13 fix(docreader): keep pdfium bitmap alive while copying SMask pixels
to_pil() shares the BGRA buffer; dropping the PdfBitmap before convert
use-after-frees. Copy pixels first, keep opaque RGB/L modes, and log when
render=True falls back to the raw (mask-blind) decode.
2026-08-24 14:41:54 +08:00
xie 0fe0b39461 fix(docreader): apply PDF image SMask so embedded figures are not all-black
get_bitmap() decodes only the base image plane and ignores any /SMask
(soft transparency mask). Figures exported from plotting tools often
store their visible content in the mask while the base RGB plane is
black, so extraction produced all-black JPEGs for such images.

Decode embedded figures via get_bitmap(render=True), which renders
through pdfium's imaging pipeline and carries the mask as an alpha
channel, then composite over white (JPEG has no alpha; PDF pages render
on white). Falls back to the raw decode when rendering is unavailable.

On a production PDF with 7 SMask figures, dark-pixel ratio drops from
~100% to 0.7%-4.5%. Opaque images are unchanged.
2026-08-24 14:41:54 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
jiahao6635 cc690bbe7f fix(docreader): avoid chart serialization during xlsx preprocessing 2026-08-12 13:26:19 +08:00
LionLe ffd1b043d2
fix(docreader): preserve vertically merged table cells (#2657)
* fix(docreader): preserve vertically merged table cells

* test(docreader): cover docx preprocessing boundaries

* test(docreader): isolate separate vertical merge groups
2026-08-12 10:59:49 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
Xi(Darcy) Chen 090612da54
fix: pptx/ppt attachment parsing fails — missing parser engine + markitdown extras (#2621) 2026-08-09 16:23:36 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
wizardchen 083e9738f6 fix(docreader): detect legacy doc payloads 2026-07-11 12:21:56 +08:00
wizardchen b6d3420a5d fix: harden document pipeline concurrency and diagnostics
- docreader: serialize all pdfium access behind a global lock. pypdfium2
  is process-global and not thread-safe, so concurrent PDF uploads could
  corrupt its shared state and deadlock the whole process (requests stuck
  in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
  plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
  multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
  mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
2026-07-09 13:26:40 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
zhouxun 4db727ac69 fix(excel): filter out image function strings from parsed content
WPS embeds images in cells using =DISPIMG("ID",mode) which may appear as plain text prefixed with "_xlfn." when processed by openpyxl. These function strings pollute the RAG-indexed document content, causing incorrect AI recall.

Add a regex filter (_is_image_function) that detects _xlfn.DISPIMG(...) and _xlfn.IMAGE(...) patterns and excludes them during row processing in ExcelParser.parse_into_text().

Refs: https://github.com/Tencent/WeKnora/issues/1779
2026-06-24 14:35:37 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 959eba2136 fix(doc_parser): enhance DOC to DOCX conversion reliability
- Implemented a retry mechanism for DOC to DOCX conversion to handle concurrent `soffice` invocations, ensuring each attempt uses a dedicated user profile directory.
- Added logging for each conversion attempt, including success and failure messages, to improve visibility into the conversion process.
- Adjusted the handling of temporary directories for both conversion output and user profiles, enhancing robustness against conversion failures.
2026-06-01 20:50:02 +08:00
wizardchen 13301ca026 feat(parser): enhance web parser with improved image extension handling and XPath prioritization
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
2026-05-25 19:15:17 +08:00
wizardchen d65e647f95 chore: bump Go to 1.26 and slim docreader dependencies
- Bump base image in docker/Dockerfile.app from golang:1.24 to golang:1.26
  to match `go 1.26` declared in go.mod (fixes CI build failure on
  `go mod download`).
- Drop unused docreader components and their dependencies:
  - Remove `docreader/ocr/` package (paddle/vlm/dummy backends are
    unreferenced by the main flow; OCR/VLM is handled by the Go App).
  - Remove `docreader/parser/storage.py` (dead code; image persistence
    happens in the Go App via inline ImageRef bytes).
  - Remove `docreader/scripts/download_deps.py` (PaddleOCR pre-download).
  - Drop deps: paddleocr, paddlepaddle, openai, ollama, minio,
    cos-python-sdk-v5, oss2, asyncio, pypdf2, markdown, mistletoe,
    goose3, markdownify, pdfplumber, antiword, urllib3.
- Re-lock uv.lock: 145 -> 79 packages.
- Update docreader/README.md to reflect that OCR/VLM/storage are no
  longer configured at the docreader level.
2026-05-09 13:32:40 +08:00
wolfkill 450a5bd2dd fix(docreader): throttle heavy parser concurrency 2026-05-07 17:36:09 +08:00
wizardchen d5f6c7ba21 fix(docreader): remove default 100-page limit for DOCX parsing
The default DOCREADER_DOCX_MAX_PAGES=100 silently truncates large
documents, causing users to see at most ~1000 chunks regardless of
document length. Change the default to 0 (no limit) so all pages are
processed. Operators who need a cap can still set the env var.

Fixes #719
2026-04-28 21:50:15 +08:00
wizardchen 3e61b91efd fix(storage): remove ListBucket permission from MinIO bucket policy 2026-04-16 18:13:21 +08:00
wizardchen 1938094dcc feat(parser): add PDFScannedParser for handling scanned PDFs
- Introduced PDFScannedParser as a fallback parser for scanned PDFs that converts pages into images for OCR processing.
- Updated PDFParser to include PDFScannedParser in the parsing chain, enhancing the document parsing capabilities for scanned content.
- Improved logging for better error tracking during PDF parsing operations.
2026-04-16 18:13:20 +08:00
bingxiang.cheng d11e043142 refactor(storage): 优化OSS存储初始化与URL格式,提升安全性和兼容性
- 添加判断oss2是否安装,未安装时打印错误日志
- 修改OSS桶不存在时不自动创建,避免误创建公共读桶
- OSS下载URL改为虚拟主机风格,提升兼容性
- OSS初始化错误、缺失配置时添加详细日志提示
- 其他存储类代码重排统一风格,提升可维护性
2026-04-13 15:34:28 +08:00
bingxiang.cheng adae28beb2 feat(oss): add OssStorage class to docreader
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
Windfarer 54da98fc24 feat: add docx max pages env config 2026-04-02 10:31:52 +08:00
wizardchen 8e1cfaccb7 refactor(prompt_templates): improve question generation guidelines and context handling
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
2026-04-01 21:59:22 +08:00
wizardchen c4f5db7e88 feat(metadata): enhance document processing to include metadata extraction and handling
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
2026-04-01 15:49:47 +08:00
Windfarer bc0c2d23d5 add comment 2026-04-01 13:37:37 +08:00
Windfarer 6058f1cbe0 add monkey patch for docx parse error 2026-04-01 13:37:37 +08:00
wizardchen 1c9503b063 feat(docparser): enhance image resolution capabilities in markdown and HTML
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
2026-03-25 22:08:29 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
chenrui7 aba7a47b20 fix(parser): resolve chunk index mismatch in logs 2026-01-29 19:32:35 +08:00
ice 6a3c29d455 fix: correctly extract host from completion_url by handling both v1 and non-v1 endpoints 2026-01-28 14:06:43 +08:00
cc 5c7f05189e fix(parser): separate StdMinerUParser and MinerUCloudParser implementation 2026-01-23 10:28:38 +08:00
begoniezhao 88fd42cbc3 refactor: Restructure OCR module and centralize config 2026-01-16 16:05:31 +08:00
cc 74bf76eb3f fix(parser): provide explicit file_extension for markitdown to resolve #544 2026-01-16 11:50:42 +08:00
cc 3f2792dff4 fix(parser): provide explicit file_extension for markitdown to resolve #544 2026-01-16 11:50:42 +08:00
wizardchen 48a0f9b508 Merge remote-tracking branch 'github_public/main' 2026-01-15 14:39:35 +08:00