Commit Graph

30 Commits (ebbcf96fabb651c091c3ee2272d802b80c100c4e)

Author SHA1 Message Date
YixuanLieu 06ae6bcf04
fix(docreader): kill timed-out converter process groups (#2990)
* fix(docreader): kill timed-out converter process groups

(cherry picked from commit 403ee6dd90ad6836f4f073f4f06839884133a7e4)

* fix(docreader): bound timeout reap so converter cleanup cannot hang

Unbounded communicate() after killpg waited for pipe EOF and could
block a DocReader worker if a descendant left the process group.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-23 12:02:53 +08:00
Captain 171097f06e
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS

A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).

SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.

Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.

Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.

Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.

docreader enforces the same switch in its own validator, and compose
passes the variable to both services.

Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.

The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.

Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.

The switch defaults to off, so nothing changes for deployments that do
not set it.

* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs

docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.

The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 12:02:06 +08:00
VinhChu ae185dd1a9 fix(docreader): stop a non-ASCII base64 payload from failing the whole parse
encode_image documents errors="ignore" as "return empty bytes on decoding
errors", but it only catches binascii.Error. base64.b64decode raises that for
bad padding or a bad alphabet; for a str that is not pure ASCII it fails in
_bytes_from_decode_data first and raises a plain ValueError, which escapes.

The one caller that passes errors="ignore" is the markdown base64-image
extractor, whose payload group is `([^\)]+)` — everything up to the closing
paren, not the base64 alphabet. A CJK image title therefore lands inside the
payload:

    ![x](data:image/png;base64,iVBORw0KGgo= "图片")
                                            ^^^^ captured as base64

MarkdownImageBase64.parse_into_text does not catch, PipelineParser does not
catch, and BaseParser.parse does not catch, so the ValueError reaches the gRPC
handler and the whole document comes back as
ReadResponse(error="string argument should contain only ASCII characters").
One image title fails the entire upload, and the message names nothing the
user can act on.

The same title written in ASCII decodes fine, because b64decode with
validate=False discards the quotes and spaces. The failure only shows up for
titles in the languages this project is mostly used in.

Catching ValueError covers both cases — binascii.Error subclasses it — so the
undecodable image is skipped and logged, which is what the extractor's own
`if not image_byte` branch already expects to happen.

Not addressed here: the pattern swallowing the title is a separate bug, and
narrowing that group changes which markdown the extractor matches. Filed as
its own change rather than folded in.
2026-09-22 19:37:21 +08:00
林SO f307d3dc94
fix: remove temp file when TempFileContext write fails inside __enter__ (#3495)
* fix: remove temp file when TempFileContext write fails inside __enter__

TempFileContext.__enter__ creates a NamedTemporaryFile(delete=False) and then
calls write()/flush() with no error handling. If either raises (ENOSPC, quota,
any I/O error), the exception escapes __enter__, and Python's with-statement
never calls __exit__ for a context manager whose __enter__ failed. __exit__ is
the only place the file is closed and removed, so the file leaks permanently;
delete=False means GC will not remove it either.

doc_parser.py uses this to spill every uploaded .doc's full bytes to disk, so
one disk-pressure event leaves the whole document behind and makes the
pressure worse.

Wrap write/flush in try/except, close and remove the file, then re-raise.
Success path and the exception-inside-body path are unchanged.

Adds docreader/tests/test_tempfile_context.py covering the failure-injection
case (fails on the old code), the normal roundtrip, and body exceptions.

* fix: unlink temp file even if close() re-raises after ENOSPC

close() flushes again, so the ENOSPC path can skip os.remove and still leak.
Share one cleanup helper that always unlinks and preserves the original error.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-21 14:21:07 +08:00
lyingbug d7ccd5ba51
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling

* fix: preserve trusted Doris redirects and prepare browser tests

* fix: align proxy address checks and integration test coverage
2026-09-16 16:20:57 +08:00
LuckVd 81142dfd17
fix(docreader): keep DOCX table content in parsed text (#2898)
* fix(docreader): keep DOCX table content in parsed text

Docx.__call__ returns tables separately from the text lines, and
parse_into_text used to drop them, silently losing all table content
in documents parsed with the builtin engine. Render the tables as GFM
Markdown (header row + delimiter + data rows) and append them to the
parsed text.

Fixes #2881

* fix(docreader): render DOCX tables from grid cells in document order

Skip the HTML round-trip that treated equal or empty adjacent cells as
colspan and dropped columns from GFM. Emit one column per python-docx
grid slot and interleave tables with paragraphs in body order.

---------

Co-authored-by: root <root@localhost.localdomain>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:56:17 +08:00
ResistanceTo 77245f0f4b
fix(docreader): handle Header object when stripping mhtml header (#2902)
* fix(docreader): handle Header object when stripping mhtml header

* fix(docreader): decode 8-bit MHTML headers without crashing (#2901)

compat32 wraps non-ASCII 8-bit headers as Header; str() turns them into
U+FFFD and HTML Content-Location still crashed on .lower()/urljoin.

Refs: https://github.com/Tencent/WeKnora/issues/2901

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:14:58 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
wizardchen 91186b81b7 fix(docreader): fail URL scrape instead of indexing error text (#2883)
Empty scrape/parse results were returned as "Error parsing web page: …"
and then chunked into the knowledge index, hiding the failure. Raise
WebParseError so DocReader returns error and the knowledge is marked failed.

Refs: https://github.com/Tencent/WeKnora/issues/2883
2026-08-31 14:12:11 +08:00
sufan721 3842068d22
fix: stop markdown table regex catastrophic backtracking (#2770)
* fix: stop markdown table regex catastrophic backtracking

The line_pattern in MarkdownTableUtil used overlapping character classes
([\t ]* and [^|\r\n]*) so a `|`-prefixed line missing its trailing `|`
forced the engine to explore an exponential number of backtracking paths,
stalling the single-threaded docreader gRPC event loop (100% CPU, health
check timeout).

Replace whole-text regex substitution with an O(n) line-based pass that
only treats rows beginning and ending with `|` as tables, and preserves
the previous align_pattern semantics via _is_alignment_row.

Fixes #2768

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: keep lone pipe rows and normalize CRLF in table formatter

A single "|" was rewritten into an empty table row and then dropped.
Unify CR/LF before the line scan so exported Markdown tables format
without mixed endings.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-24 18:08:07 +08:00
wizardchen f20427df13 fix(docreader): keep pdfium bitmap alive while copying SMask pixels
to_pil() shares the BGRA buffer; dropping the PdfBitmap before convert
use-after-frees. Copy pixels first, keep opaque RGB/L modes, and log when
render=True falls back to the raw (mask-blind) decode.
2026-08-24 14:41:54 +08:00
xie 0fe0b39461 fix(docreader): apply PDF image SMask so embedded figures are not all-black
get_bitmap() decodes only the base image plane and ignores any /SMask
(soft transparency mask). Figures exported from plotting tools often
store their visible content in the mask while the base RGB plane is
black, so extraction produced all-black JPEGs for such images.

Decode embedded figures via get_bitmap(render=True), which renders
through pdfium's imaging pipeline and carries the mask as an alpha
channel, then composite over white (JPEG has no alpha; PDF pages render
on white). Falls back to the raw decode when rendering is unavailable.

On a production PDF with 7 SMask figures, dark-pixel ratio drops from
~100% to 0.7%-4.5%. Opaque images are unchanged.
2026-08-24 14:41:54 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
jiahao6635 cc690bbe7f fix(docreader): avoid chart serialization during xlsx preprocessing 2026-08-12 13:26:19 +08:00
LionLe ffd1b043d2
fix(docreader): preserve vertically merged table cells (#2657)
* fix(docreader): preserve vertically merged table cells

* test(docreader): cover docx preprocessing boundaries

* test(docreader): isolate separate vertical merge groups
2026-08-12 10:59:49 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
wizardchen 083e9738f6 fix(docreader): detect legacy doc payloads 2026-07-11 12:21:56 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
zhouxun 4db727ac69 fix(excel): filter out image function strings from parsed content
WPS embeds images in cells using =DISPIMG("ID",mode) which may appear as plain text prefixed with "_xlfn." when processed by openpyxl. These function strings pollute the RAG-indexed document content, causing incorrect AI recall.

Add a regex filter (_is_image_function) that detects _xlfn.DISPIMG(...) and _xlfn.IMAGE(...) patterns and excludes them during row processing in ExcelParser.parse_into_text().

Refs: https://github.com/Tencent/WeKnora/issues/1779
2026-06-24 14:35:37 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
JojaCola312 4c5379df9c test(docreader): align pdf JPEG quality default assertion 2026-06-17 17:18:57 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wolfkill 450a5bd2dd fix(docreader): throttle heavy parser concurrency 2026-05-07 17:36:09 +08:00