Commit Graph

111 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
Lukas e7aaf8ae02
fix(docreader): keep the space held by a whitespace-only inline element (#3978)
markdownify renders an emphasis, code or link element whose text is only
whitespace as "", and the whitespace goes with it. HTML and MHTML
uploads therefore lost word boundaries: `further<strong> </strong>
reference` became `furtherreference`, and `<b>First</b><b> </b><b>Last</b>`
became `**First****Last**`. Editors produce that markup whenever a single
space between two words carries different formatting.

Before conversion, unwrap such elements so their whitespace stays as plain
text. Only elements with no child elements are touched, innermost first,
so a linked image keeps its link and nested wrappers come off completely.
2026-10-08 01:31:48 +08:00
Lukas c9723b1198
fix(docreader): read EPUB chapters in spine order (#3977)
EPUBParser took the chapter order from book.get_table_of_contents(),
which ebooklib's EpubBook does not have. The AttributeError was caught,
so every book fell through to get_items(), which follows the manifest.
The manifest may list files in any order; the spine is the reading order.
Standard Ebooks' Pride and Prejudice, for example, lists its title page
and imprint after chapter 61 in the manifest.

Read the documents in spine order, followed by any the spine leaves out
in manifest order, so the set of documents read is unchanged.
2026-10-08 01:21:50 +08:00
AI即世界 afc6cbd502
fix(milvus): validate connections and preserve gRPC resolver defaults (#3654)
* fix(docreader): preserve gRPC resolver defaults

* fix(milvus): validate connection with SDK handshake

* fix(docparser): preserve gRPC resolver defaults

* fix(milvus): satisfy connection check lint

---------

Co-authored-by: tunan <tunan@ahyk.com>
2026-10-08 00:48:45 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
Captain 1dd3fe867e
fix(docreader): drop data validation ranges openpyxl cannot parse instead of failing the sheet (#3599) (#3608)
* fix(docreader): drop data validation ranges openpyxl cannot parse instead of failing the sheet (#3599)

A Feishu sheet/bitable exported as .xlsx could fail to ingest with
"TypeError: expected <class 'openpyxl.worksheet.cell_range.MultiCellRange'>".
openpyxl reads a data validation's `sqref`, and a scenario list's, through a
converter that turns the real range-parse ValueError into that message and
aborts the whole workbook — in load_workbook (used by fill_merged_cells_xlsx)
and in pandas alike.

Measured on openpyxl 3.1.5, the version docreader locks: a whole column
(`C:C`), a whole row (`1:1`), a comma-separated list, `#REF!` and an
out-of-bounds range each raise it. Conditional formatting goes through the
same converter, but openpyxl already catches it there and drops the rule
with a warning, so it is left alone.

strip_unreadable_ranges_xlsx sits next to repair_xlsx_bytes and follows the
same contract: it returns rewritten bytes only when some range would fail
openpyxl's own MultiCellRange constructor — the exact step that raises — and
removes just those elements, logging how many. Every other workbook passes
through byte for byte. Neither element changes a cell's value, so dropping
one costs text extraction nothing, while keeping it cost the document.

* fix(docreader): strip unreadable ranges on the MarkItDown xlsx path too

MarkItDown reads spreadsheets through pandas/openpyxl, so a whole-column
data validation still failed that engine with FileConversionException
after the builtin engine was fixed (#3599). Run the same
strip_unreadable_ranges_xlsx pre-step before handing .xlsx to MarkItDown.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:29:27 +08:00
Fei.Chen a7ac21ce76
fix(docreader): sanitize non-conforming styles.xml fills before openpyxl loads (#3637) (#3638)
* fix(docreader): sanitize non-conforming styles.xml fills before openpyxl loads

openpyxl raises TypeError('expected ... Fill') on <fill> elements with no
patternFill/gradientFill child — cloud-doc exporters, LabView and Acumatica
ERP emit these, upstream treats the workbooks as non-conformant, and both
parse engines died on files Excel opens fine (#3637).

Extend the zip-level repair module with sanitize_xlsx_styles: scan
xl/styles.xml, replace each malformed fill in place with a no-op
patternFill (index-preserving, so cellXf fillId references stay valid),
rewrite the zip only when something needed fixing. The builtin chain
retries the merge-cell prefill once on sanitized bytes; the markitdown
chain sanitizes xlsx/xlsm before MarkItDown sees them. Fixes #3637.

* fix(docreader): log sanitized fill count and never raise from the sanitizer

Fold review feedback into the styles sanitizer: a structured log line on
every repair (count included, doubling as the recurrence meter for #3637,
mirroring the sharedStrings repair log) and a never-raise wrapper so an
internal sanitizer failure passes the original bytes through instead of
trading the caller's error for its own.

* fix(docreader): read the workbook stylesheet by its exact part name

The styles sanitizer picked its part with a suffix match over a set of
zip member names, so a workbook that also carries
xl/richData/richStyles.xml (Excel 365 in-cell images / data types) got
the wrong part on some PYTHONHASHSEED values and skipped the repair.
Read xl/styles.xml, the one stylesheet openpyxl loads.

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:23:57 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
YixuanLieu 06ae6bcf04
fix(docreader): kill timed-out converter process groups (#2990)
* fix(docreader): kill timed-out converter process groups

(cherry picked from commit 403ee6dd90ad6836f4f073f4f06839884133a7e4)

* fix(docreader): bound timeout reap so converter cleanup cannot hang

Unbounded communicate() after killpg waited for pipe EOF and could
block a DocReader worker if a descendant left the process group.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-23 12:02:53 +08:00
Captain 171097f06e
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS

A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).

SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.

Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.

Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.

Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.

docreader enforces the same switch in its own validator, and compose
passes the variable to both services.

Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.

The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.

Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.

The switch defaults to off, so nothing changes for deployments that do
not set it.

* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs

docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.

The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 12:02:06 +08:00
VinhChu ae185dd1a9 fix(docreader): stop a non-ASCII base64 payload from failing the whole parse
encode_image documents errors="ignore" as "return empty bytes on decoding
errors", but it only catches binascii.Error. base64.b64decode raises that for
bad padding or a bad alphabet; for a str that is not pure ASCII it fails in
_bytes_from_decode_data first and raises a plain ValueError, which escapes.

The one caller that passes errors="ignore" is the markdown base64-image
extractor, whose payload group is `([^\)]+)` — everything up to the closing
paren, not the base64 alphabet. A CJK image title therefore lands inside the
payload:

    ![x](data:image/png;base64,iVBORw0KGgo= "图片")
                                            ^^^^ captured as base64

MarkdownImageBase64.parse_into_text does not catch, PipelineParser does not
catch, and BaseParser.parse does not catch, so the ValueError reaches the gRPC
handler and the whole document comes back as
ReadResponse(error="string argument should contain only ASCII characters").
One image title fails the entire upload, and the message names nothing the
user can act on.

The same title written in ASCII decodes fine, because b64decode with
validate=False discards the quotes and spaces. The failure only shows up for
titles in the languages this project is mostly used in.

Catching ValueError covers both cases — binascii.Error subclasses it — so the
undecodable image is skipped and logged, which is what the extractor's own
`if not image_byte` branch already expects to happen.

Not addressed here: the pattern swallowing the title is a separate bug, and
narrowing that group changes which markdown the extractor matches. Filed as
its own change rather than folded in.
2026-09-22 19:37:21 +08:00
林SO f307d3dc94
fix: remove temp file when TempFileContext write fails inside __enter__ (#3495)
* fix: remove temp file when TempFileContext write fails inside __enter__

TempFileContext.__enter__ creates a NamedTemporaryFile(delete=False) and then
calls write()/flush() with no error handling. If either raises (ENOSPC, quota,
any I/O error), the exception escapes __enter__, and Python's with-statement
never calls __exit__ for a context manager whose __enter__ failed. __exit__ is
the only place the file is closed and removed, so the file leaks permanently;
delete=False means GC will not remove it either.

doc_parser.py uses this to spill every uploaded .doc's full bytes to disk, so
one disk-pressure event leaves the whole document behind and makes the
pressure worse.

Wrap write/flush in try/except, close and remove the file, then re-raise.
Success path and the exception-inside-body path are unchanged.

Adds docreader/tests/test_tempfile_context.py covering the failure-injection
case (fails on the old code), the normal roundtrip, and body exceptions.

* fix: unlink temp file even if close() re-raises after ENOSPC

close() flushes again, so the ENOSPC path can skip os.remove and still leak.
Share one cleanup helper that always unlinks and preserves the original error.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-21 14:21:07 +08:00
lyingbug d7ccd5ba51
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling

* fix: preserve trusted Doris redirects and prepare browser tests

* fix: align proxy address checks and integration test coverage
2026-09-16 16:20:57 +08:00
wizardchen 2bc1261b87 chore: remove vendored archives and unused Python helpers 2026-09-16 15:49:36 +08:00
wizardchen ff818b681f fix: share IP classification between SSRF and sandbox URL guards
Keep one classifier for address kinds so user-fetch SSRF and sandbox
control-plane checks cannot disagree, and refuse IPv6 encodings that
embed link-local or other never-routable targets.
2026-09-01 15:35:14 +08:00
LuckVd 81142dfd17
fix(docreader): keep DOCX table content in parsed text (#2898)
* fix(docreader): keep DOCX table content in parsed text

Docx.__call__ returns tables separately from the text lines, and
parse_into_text used to drop them, silently losing all table content
in documents parsed with the builtin engine. Render the tables as GFM
Markdown (header row + delimiter + data rows) and append them to the
parsed text.

Fixes #2881

* fix(docreader): render DOCX tables from grid cells in document order

Skip the HTML round-trip that treated equal or empty adjacent cells as
colspan and dropped columns from GFM. Emit one column per python-docx
grid slot and interleave tables with paragraphs in body order.

---------

Co-authored-by: root <root@localhost.localdomain>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:56:17 +08:00
ResistanceTo 77245f0f4b
fix(docreader): handle Header object when stripping mhtml header (#2902)
* fix(docreader): handle Header object when stripping mhtml header

* fix(docreader): decode 8-bit MHTML headers without crashing (#2901)

compat32 wraps non-ASCII 8-bit headers as Header; str() turns them into
U+FFFD and HTML Content-Location still crashed on .lower()/urljoin.

Refs: https://github.com/Tencent/WeKnora/issues/2901

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:14:58 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
wizardchen 91186b81b7 fix(docreader): fail URL scrape instead of indexing error text (#2883)
Empty scrape/parse results were returned as "Error parsing web page: …"
and then chunked into the knowledge index, hiding the failure. Raise
WebParseError so DocReader returns error and the knowledge is marked failed.

Refs: https://github.com/Tencent/WeKnora/issues/2883
2026-08-31 14:12:11 +08:00
sufan721 3842068d22
fix: stop markdown table regex catastrophic backtracking (#2770)
* fix: stop markdown table regex catastrophic backtracking

The line_pattern in MarkdownTableUtil used overlapping character classes
([\t ]* and [^|\r\n]*) so a `|`-prefixed line missing its trailing `|`
forced the engine to explore an exponential number of backtracking paths,
stalling the single-threaded docreader gRPC event loop (100% CPU, health
check timeout).

Replace whole-text regex substitution with an O(n) line-based pass that
only treats rows beginning and ending with `|` as tables, and preserves
the previous align_pattern semantics via _is_alignment_row.

Fixes #2768

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: keep lone pipe rows and normalize CRLF in table formatter

A single "|" was rewritten into an empty table row and then dropped.
Unify CR/LF before the line scan so exported Markdown tables format
without mixed endings.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-24 18:08:07 +08:00
wizardchen f20427df13 fix(docreader): keep pdfium bitmap alive while copying SMask pixels
to_pil() shares the BGRA buffer; dropping the PdfBitmap before convert
use-after-frees. Copy pixels first, keep opaque RGB/L modes, and log when
render=True falls back to the raw (mask-blind) decode.
2026-08-24 14:41:54 +08:00
xie 0fe0b39461 fix(docreader): apply PDF image SMask so embedded figures are not all-black
get_bitmap() decodes only the base image plane and ignores any /SMask
(soft transparency mask). Figures exported from plotting tools often
store their visible content in the mask while the base RGB plane is
black, so extraction produced all-black JPEGs for such images.

Decode embedded figures via get_bitmap(render=True), which renders
through pdfium's imaging pipeline and carries the mask as an alpha
channel, then composite over white (JPEG has no alpha; PDF pages render
on white). Falls back to the raw decode when rendering is unavailable.

On a production PDF with 7 SMask figures, dark-pixel ratio drops from
~100% to 0.7%-4.5%. Opaque images are unchanged.
2026-08-24 14:41:54 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
jiahao6635 cc690bbe7f fix(docreader): avoid chart serialization during xlsx preprocessing 2026-08-12 13:26:19 +08:00
LionLe ffd1b043d2
fix(docreader): preserve vertically merged table cells (#2657)
* fix(docreader): preserve vertically merged table cells

* test(docreader): cover docx preprocessing boundaries

* test(docreader): isolate separate vertical merge groups
2026-08-12 10:59:49 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
Xi(Darcy) Chen 090612da54
fix: pptx/ppt attachment parsing fails — missing parser engine + markitdown extras (#2621) 2026-08-09 16:23:36 +08:00
wizardchen 4efc987abb fix(ci): make DocReader client tests runnable in PR checks
Start the DocReader gRPC server in the DocReader workflow, skip client
integration tests when the server is unavailable, exclude docreader
packages from the App workflow test scope, and trigger DocReader checks
on go.mod/go.sum changes.
2026-07-30 14:13:15 +08:00
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
wizardchen 74cdd7cdd0 fix(compose): let PDF render parallelism follow CPU-aware code default
Stop hardcoding DOCREADER_PDF_RENDER_PARALLELISM=4 in compose so low-core
containers use min(4,cpu). Clarify WEKNORA_LANGUAGE precedence and align
docreader JPEG quality docs with the 85 default.
2026-07-21 16:02:31 +08:00
wizardchen 083e9738f6 fix(docreader): detect legacy doc payloads 2026-07-11 12:21:56 +08:00
wizardchen ac8c5c5540 chore(deps): resolve open Dependabot security alerts across Go, npm, and pip.
Bump vulnerable runtime and transitive dependencies in docreader, frontend, and the Go module graph to address current GitHub security advisories.
2026-07-11 12:15:39 +08:00
wizardchen b6d3420a5d fix: harden document pipeline concurrency and diagnostics
- docreader: serialize all pdfium access behind a global lock. pypdfium2
  is process-global and not thread-safe, so concurrent PDF uploads could
  corrupt its shared state and deadlock the whole process (requests stuck
  in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
  plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
  multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
  mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
2026-07-09 13:26:40 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
zhouxun 4db727ac69 fix(excel): filter out image function strings from parsed content
WPS embeds images in cells using =DISPIMG("ID",mode) which may appear as plain text prefixed with "_xlfn." when processed by openpyxl. These function strings pollute the RAG-indexed document content, causing incorrect AI recall.

Add a regex filter (_is_image_function) that detects _xlfn.DISPIMG(...) and _xlfn.IMAGE(...) patterns and excludes them during row processing in ExcelParser.parse_into_text().

Refs: https://github.com/Tencent/WeKnora/issues/1779
2026-06-24 14:35:37 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
JojaCola312 4c5379df9c test(docreader): align pdf JPEG quality default assertion 2026-06-17 17:18:57 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 959eba2136 fix(doc_parser): enhance DOC to DOCX conversion reliability
- Implemented a retry mechanism for DOC to DOCX conversion to handle concurrent `soffice` invocations, ensuring each attempt uses a dedicated user profile directory.
- Added logging for each conversion attempt, including success and failure messages, to improve visibility into the conversion process.
- Adjusted the handling of temporary directories for both conversion output and user profiles, enhancing robustness against conversion failures.
2026-06-01 20:50:02 +08:00
wizardchen 13301ca026 feat(parser): enhance web parser with improved image extension handling and XPath prioritization
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
2026-05-25 19:15:17 +08:00
taoyang f30e9ccd0f fix(docreader): bump grpcio floor to 1.78.0 to match generated proto
docreader fails to start on a fresh `uv sync` of main:

    $ uv sync --frozen
    $ uv run python -m docreader.main
    Traceback (most recent call last):
      ...
      File ".../docreader/proto/docreader_pb2_grpc.py", line 19, in <module>
        raise RuntimeError(...)
    RuntimeError: The grpc package installed is at version 1.76.0, but the
    generated code in docreader_pb2_grpc.py depends on grpcio>=1.78.0.
    Please upgrade your grpc module to grpcio>=1.78.0 or downgrade your
    generated code using grpcio-tools<=1.76.0.

`docreader_pb2_grpc.py:8` was regenerated with grpcio-tools 1.78+ and
hardcodes `GRPC_GENERATED_VERSION = '1.78.0'`, then at import time runs
`first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)` and
raises if the installed runtime is older. But `pyproject.toml` still
pins:

    "grpcio>=1.76.0",
    "grpcio-health-checking>=1.76.0",
    "grpcio-tools>=1.76.0",

…and `uv.lock` resolves to grpcio 1.76.0. So every fresh clone +
`uv sync` ends up with a docreader that crashes on import, and a
WeKnora backend that thinks docreader is "not connected" — silently
falling back to the Go-side static engine registry only.

Downstream user-visible symptom: `/api/v1/system/parser-engines`
returns just the Go-side engines. Python-only engines (e.g.
markitdown, plus any custom Python parsers) never appear in the KB
create/edit form. Document parsing falls back to mineru/builtin even
when the user intended to use docreader-backed engines.

Reproduce on main (before this commit):

    cd docreader
    uv sync --frozen
    uv run python -m docreader.main
    # → RuntimeError at import, server never starts

After this commit:

    Resolved 79 packages
    Updated grpcio v1.76.0 -> v1.80.0
    Updated grpcio-health-checking v1.76.0 -> v1.80.0
    Updated grpcio-tools v1.76.0 -> v1.80.0
    ...
    $ uv run python -c "from docreader.proto import docreader_pb2_grpc; print('ok')"
    ok

Fix is the minimum-floor bump on the three grpcio packages, plus the
lockfile update uv emits as a result (all 1.76.0 → 1.80.0, newest
stable matching the proto's floor). No proto regen, no code change —
just aligning the dep declaration with what the generated client
already requires.

The next regen of docreader_pb2_grpc.py (e.g. with grpcio-tools 1.80+)
would raise the floor again; pinning at the proto's declared floor is
the conservative choice that keeps this in sync.
2026-05-17 15:17:09 +08:00
wizardchen a3411899cf fix(docreader/auth): harden gRPC TLS/Token rollout from #1359
Follow-up to #1359. Addresses a set of correctness and security gaps in
the initial docreader auth implementation.

- docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN
  into the WeKnora-app service. Without this the Go client never saw the
  knobs, so enabling token auth on the server broke every RPC.
- client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a
  bearer token cannot be sent over an insecure channel once TLS is on.
- server: load_tls_credentials now raises TLSConfigError on misconfig
  (cert/key missing, file unreadable, mTLS without CA); main.py exits 1
  instead of silently downgrading to insecure.
- server: replace endswith("/Check"|"/Watch") health bypass with exact
  match against /grpc.health.v1.Health/{Check,Watch}.
- server: compare tokens with hmac.compare_digest, warn on tokens < 16B.
- server: AuthInterceptor now returns an abort handler matching the
  original RPC kind (unary/stream) and uses context.abort, so streaming
  RPCs surface UNAUTHENTICATED instead of INTERNAL.
- internal/infrastructure/docparser/grpc_parser.go: drop the duplicated
  TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv +
  BuildDialOptions. Single source of truth for client-side auth.
- Add GRPC_TLS_SERVER_NAME (client SNI override) and
  GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document
  the differing CA semantics between client and server in .env*.example.
- Reject half-configured client mTLS (cert XOR key) loudly.
- Fix missing trailing newline in .env.lite.example.

Verified locally: go build ./... and go vet ./... clean; auth.py
fail-fast / token paths smoke-tested.
2026-05-16 21:45:56 +08:00