Commit Graph

74 Commits (e6ecb9adcc8ddbb500957f77490ca5cb4c2fa8d4)

Author SHA1 Message Date
Lukas e7aaf8ae02
fix(docreader): keep the space held by a whitespace-only inline element (#3978)
markdownify renders an emphasis, code or link element whose text is only
whitespace as "", and the whitespace goes with it. HTML and MHTML
uploads therefore lost word boundaries: `further<strong> </strong>
reference` became `furtherreference`, and `<b>First</b><b> </b><b>Last</b>`
became `**First****Last**`. Editors produce that markup whenever a single
space between two words carries different formatting.

Before conversion, unwrap such elements so their whitespace stays as plain
text. Only elements with no child elements are touched, innermost first,
so a linked image keeps its link and nested wrappers come off completely.
2026-10-08 01:31:48 +08:00
Lukas c9723b1198
fix(docreader): read EPUB chapters in spine order (#3977)
EPUBParser took the chapter order from book.get_table_of_contents(),
which ebooklib's EpubBook does not have. The AttributeError was caught,
so every book fell through to get_items(), which follows the manifest.
The manifest may list files in any order; the spine is the reading order.
Standard Ebooks' Pride and Prejudice, for example, lists its title page
and imprint after chapter 61 in the manifest.

Read the documents in spine order, followed by any the spine leaves out
in manifest order, so the set of documents read is unchanged.
2026-10-08 01:21:50 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
Captain 1dd3fe867e
fix(docreader): drop data validation ranges openpyxl cannot parse instead of failing the sheet (#3599) (#3608)
* fix(docreader): drop data validation ranges openpyxl cannot parse instead of failing the sheet (#3599)

A Feishu sheet/bitable exported as .xlsx could fail to ingest with
"TypeError: expected <class 'openpyxl.worksheet.cell_range.MultiCellRange'>".
openpyxl reads a data validation's `sqref`, and a scenario list's, through a
converter that turns the real range-parse ValueError into that message and
aborts the whole workbook — in load_workbook (used by fill_merged_cells_xlsx)
and in pandas alike.

Measured on openpyxl 3.1.5, the version docreader locks: a whole column
(`C:C`), a whole row (`1:1`), a comma-separated list, `#REF!` and an
out-of-bounds range each raise it. Conditional formatting goes through the
same converter, but openpyxl already catches it there and drops the rule
with a warning, so it is left alone.

strip_unreadable_ranges_xlsx sits next to repair_xlsx_bytes and follows the
same contract: it returns rewritten bytes only when some range would fail
openpyxl's own MultiCellRange constructor — the exact step that raises — and
removes just those elements, logging how many. Every other workbook passes
through byte for byte. Neither element changes a cell's value, so dropping
one costs text extraction nothing, while keeping it cost the document.

* fix(docreader): strip unreadable ranges on the MarkItDown xlsx path too

MarkItDown reads spreadsheets through pandas/openpyxl, so a whole-column
data validation still failed that engine with FileConversionException
after the builtin engine was fixed (#3599). Run the same
strip_unreadable_ranges_xlsx pre-step before handing .xlsx to MarkItDown.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:29:27 +08:00
Fei.Chen a7ac21ce76
fix(docreader): sanitize non-conforming styles.xml fills before openpyxl loads (#3637) (#3638)
* fix(docreader): sanitize non-conforming styles.xml fills before openpyxl loads

openpyxl raises TypeError('expected ... Fill') on <fill> elements with no
patternFill/gradientFill child — cloud-doc exporters, LabView and Acumatica
ERP emit these, upstream treats the workbooks as non-conformant, and both
parse engines died on files Excel opens fine (#3637).

Extend the zip-level repair module with sanitize_xlsx_styles: scan
xl/styles.xml, replace each malformed fill in place with a no-op
patternFill (index-preserving, so cellXf fillId references stay valid),
rewrite the zip only when something needed fixing. The builtin chain
retries the merge-cell prefill once on sanitized bytes; the markitdown
chain sanitizes xlsx/xlsm before MarkItDown sees them. Fixes #3637.

* fix(docreader): log sanitized fill count and never raise from the sanitizer

Fold review feedback into the styles sanitizer: a structured log line on
every repair (count included, doubling as the recurrence meter for #3637,
mirroring the sharedStrings repair log) and a never-raise wrapper so an
internal sanitizer failure passes the original bytes through instead of
trading the caller's error for its own.

* fix(docreader): read the workbook stylesheet by its exact part name

The styles sanitizer picked its part with a suffix match over a set of
zip member names, so a workbook that also carries
xl/richData/richStyles.xml (Excel 365 in-cell images / data types) got
the wrong part on some PYTHONHASHSEED values and skipped the repair.
Read xl/styles.xml, the one stylesheet openpyxl loads.

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-25 22:23:57 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
YixuanLieu 06ae6bcf04
fix(docreader): kill timed-out converter process groups (#2990)
* fix(docreader): kill timed-out converter process groups

(cherry picked from commit 403ee6dd90ad6836f4f073f4f06839884133a7e4)

* fix(docreader): bound timeout reap so converter cleanup cannot hang

Unbounded communicate() after killpg waited for pipe EOF and could
block a DocReader worker if a descendant left the process group.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-23 12:02:53 +08:00
lyingbug d7ccd5ba51
fix: align client sessions and integration request handling (#3305)
* fix: align client sessions and integration request handling

* fix: preserve trusted Doris redirects and prepare browser tests

* fix: align proxy address checks and integration test coverage
2026-09-16 16:20:57 +08:00
LuckVd 81142dfd17
fix(docreader): keep DOCX table content in parsed text (#2898)
* fix(docreader): keep DOCX table content in parsed text

Docx.__call__ returns tables separately from the text lines, and
parse_into_text used to drop them, silently losing all table content
in documents parsed with the builtin engine. Render the tables as GFM
Markdown (header row + delimiter + data rows) and append them to the
parsed text.

Fixes #2881

* fix(docreader): render DOCX tables from grid cells in document order

Skip the HTML round-trip that treated equal or empty adjacent cells as
colspan and dropped columns from GFM. Emit one column per python-docx
grid slot and interleave tables with paragraphs in body order.

---------

Co-authored-by: root <root@localhost.localdomain>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:56:17 +08:00
ResistanceTo 77245f0f4b
fix(docreader): handle Header object when stripping mhtml header (#2902)
* fix(docreader): handle Header object when stripping mhtml header

* fix(docreader): decode 8-bit MHTML headers without crashing (#2901)

compat32 wraps non-ASCII 8-bit headers as Header; str() turns them into
U+FFFD and HTML Content-Location still crashed on .lower()/urljoin.

Refs: https://github.com/Tencent/WeKnora/issues/2901

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:14:58 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
wizardchen 91186b81b7 fix(docreader): fail URL scrape instead of indexing error text (#2883)
Empty scrape/parse results were returned as "Error parsing web page: …"
and then chunked into the knowledge index, hiding the failure. Raise
WebParseError so DocReader returns error and the knowledge is marked failed.

Refs: https://github.com/Tencent/WeKnora/issues/2883
2026-08-31 14:12:11 +08:00
sufan721 3842068d22
fix: stop markdown table regex catastrophic backtracking (#2770)
* fix: stop markdown table regex catastrophic backtracking

The line_pattern in MarkdownTableUtil used overlapping character classes
([\t ]* and [^|\r\n]*) so a `|`-prefixed line missing its trailing `|`
forced the engine to explore an exponential number of backtracking paths,
stalling the single-threaded docreader gRPC event loop (100% CPU, health
check timeout).

Replace whole-text regex substitution with an O(n) line-based pass that
only treats rows beginning and ending with `|` as tables, and preserves
the previous align_pattern semantics via _is_alignment_row.

Fixes #2768

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: keep lone pipe rows and normalize CRLF in table formatter

A single "|" was rewritten into an empty table row and then dropped.
Unify CR/LF before the line scan so exported Markdown tables format
without mixed endings.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-24 18:08:07 +08:00
wizardchen f20427df13 fix(docreader): keep pdfium bitmap alive while copying SMask pixels
to_pil() shares the BGRA buffer; dropping the PdfBitmap before convert
use-after-frees. Copy pixels first, keep opaque RGB/L modes, and log when
render=True falls back to the raw (mask-blind) decode.
2026-08-24 14:41:54 +08:00
xie 0fe0b39461 fix(docreader): apply PDF image SMask so embedded figures are not all-black
get_bitmap() decodes only the base image plane and ignores any /SMask
(soft transparency mask). Figures exported from plotting tools often
store their visible content in the mask while the base RGB plane is
black, so extraction produced all-black JPEGs for such images.

Decode embedded figures via get_bitmap(render=True), which renders
through pdfium's imaging pipeline and carries the mask as an alpha
channel, then composite over white (JPEG has no alpha; PDF pages render
on white). Falls back to the raw decode when rendering is unavailable.

On a production PDF with 7 SMask figures, dark-pixel ratio drops from
~100% to 0.7%-4.5%. Opaque images are unchanged.
2026-08-24 14:41:54 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
jiahao6635 cc690bbe7f fix(docreader): avoid chart serialization during xlsx preprocessing 2026-08-12 13:26:19 +08:00
LionLe ffd1b043d2
fix(docreader): preserve vertically merged table cells (#2657)
* fix(docreader): preserve vertically merged table cells

* test(docreader): cover docx preprocessing boundaries

* test(docreader): isolate separate vertical merge groups
2026-08-12 10:59:49 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
Xi(Darcy) Chen 090612da54
fix: pptx/ppt attachment parsing fails — missing parser engine + markitdown extras (#2621) 2026-08-09 16:23:36 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 36206ae62f fix(docreader): align XLS header override behavior with XLSX
Normalize Excel file types when resolving parser rules and applying the
xlsx_first_row_as_header override for both xlsx and xls. Skip the override
for non-builtin engines, unify legacy XLS parsing with the XLSX default path,
and extend tests for case handling and explicit false overrides.
2026-07-28 22:00:36 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
wizardchen 083e9738f6 fix(docreader): detect legacy doc payloads 2026-07-11 12:21:56 +08:00
wizardchen b6d3420a5d fix: harden document pipeline concurrency and diagnostics
- docreader: serialize all pdfium access behind a global lock. pypdfium2
  is process-global and not thread-safe, so concurrent PDF uploads could
  corrupt its shared state and deadlock the whole process (requests stuck
  in "Parsing document" forever). Non-PDF parsers still run concurrently.
- logger: strip ANSI color escapes from file logs so on-disk logs stay
  plain text while stdout keeps terminal colors.
- storage: log baseDir/resolvedPath on file-open failure and when the
  multimodal service resolves a file service, so a LOCAL_STORAGE_BASE_DIR
  mismatch between writer and reader is immediately visible.
- docker: set TZ (default Asia/Shanghai) for the docreader container.
2026-07-09 13:26:40 +08:00
wizardchen 1d322c525a fix(security): block SSRF via redirects in knowledge URL import
Use the SSRF-safe HTTP client for file_url downloads and add redirect-aware
guards to the docreader Playwright path, closing GHSA-44fp, GHSA-rv6v, and
GHSA-c7m6.
2026-07-02 12:27:48 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
zhouxun 4db727ac69 fix(excel): filter out image function strings from parsed content
WPS embeds images in cells using =DISPIMG("ID",mode) which may appear as plain text prefixed with "_xlfn." when processed by openpyxl. These function strings pollute the RAG-indexed document content, causing incorrect AI recall.

Add a regex filter (_is_image_function) that detects _xlfn.DISPIMG(...) and _xlfn.IMAGE(...) patterns and excludes them during row processing in ExcelParser.parse_into_text().

Refs: https://github.com/Tencent/WeKnora/issues/1779
2026-06-24 14:35:37 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
FFFFFFpy 408364c6c9 fix(docreader): restore epub and mhtml parser registration 2026-06-17 00:30:51 +08:00
wizardchen b58232e65b feat(docreader): add quick availability probe for opendataloader parser
Add a `quick` flag to `opendataloader_available` so registry listing can use
a single short-timeout health check instead of the patient retry/backoff loop.
Parsing keeps the original tolerant probe to handle a hybrid service that is
still starting up.
2026-06-16 16:40:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 959eba2136 fix(doc_parser): enhance DOC to DOCX conversion reliability
- Implemented a retry mechanism for DOC to DOCX conversion to handle concurrent `soffice` invocations, ensuring each attempt uses a dedicated user profile directory.
- Added logging for each conversion attempt, including success and failure messages, to improve visibility into the conversion process.
- Adjusted the handling of temporary directories for both conversion output and user profiles, enhancing robustness against conversion failures.
2026-06-01 20:50:02 +08:00
wizardchen 13301ca026 feat(parser): enhance web parser with improved image extension handling and XPath prioritization
- Added a regex pattern for image file extensions to the utils module for better image detection.
- Updated the BODY_XPATH in the xpaths module to prioritize matching specific content structures in web pages.
- These changes aim to improve the accuracy and efficiency of content extraction from web pages using the StdWebParser class.
2026-05-25 19:15:17 +08:00
wizardchen d65e647f95 chore: bump Go to 1.26 and slim docreader dependencies
- Bump base image in docker/Dockerfile.app from golang:1.24 to golang:1.26
  to match `go 1.26` declared in go.mod (fixes CI build failure on
  `go mod download`).
- Drop unused docreader components and their dependencies:
  - Remove `docreader/ocr/` package (paddle/vlm/dummy backends are
    unreferenced by the main flow; OCR/VLM is handled by the Go App).
  - Remove `docreader/parser/storage.py` (dead code; image persistence
    happens in the Go App via inline ImageRef bytes).
  - Remove `docreader/scripts/download_deps.py` (PaddleOCR pre-download).
  - Drop deps: paddleocr, paddlepaddle, openai, ollama, minio,
    cos-python-sdk-v5, oss2, asyncio, pypdf2, markdown, mistletoe,
    goose3, markdownify, pdfplumber, antiword, urllib3.
- Re-lock uv.lock: 145 -> 79 packages.
- Update docreader/README.md to reflect that OCR/VLM/storage are no
  longer configured at the docreader level.
2026-05-09 13:32:40 +08:00
wolfkill 450a5bd2dd fix(docreader): throttle heavy parser concurrency 2026-05-07 17:36:09 +08:00
wizardchen d5f6c7ba21 fix(docreader): remove default 100-page limit for DOCX parsing
The default DOCREADER_DOCX_MAX_PAGES=100 silently truncates large
documents, causing users to see at most ~1000 chunks regardless of
document length. Change the default to 0 (no limit) so all pages are
processed. Operators who need a cap can still set the env var.

Fixes #719
2026-04-28 21:50:15 +08:00
wizardchen 3e61b91efd fix(storage): remove ListBucket permission from MinIO bucket policy 2026-04-16 18:13:21 +08:00
wizardchen 1938094dcc feat(parser): add PDFScannedParser for handling scanned PDFs
- Introduced PDFScannedParser as a fallback parser for scanned PDFs that converts pages into images for OCR processing.
- Updated PDFParser to include PDFScannedParser in the parsing chain, enhancing the document parsing capabilities for scanned content.
- Improved logging for better error tracking during PDF parsing operations.
2026-04-16 18:13:20 +08:00
bingxiang.cheng d11e043142 refactor(storage): 优化OSS存储初始化与URL格式,提升安全性和兼容性
- 添加判断oss2是否安装,未安装时打印错误日志
- 修改OSS桶不存在时不自动创建,避免误创建公共读桶
- OSS下载URL改为虚拟主机风格,提升兼容性
- OSS初始化错误、缺失配置时添加详细日志提示
- 其他存储类代码重排统一风格,提升可维护性
2026-04-13 15:34:28 +08:00
bingxiang.cheng adae28beb2 feat(oss): add OssStorage class to docreader
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-13 15:34:28 +08:00
Windfarer 54da98fc24 feat: add docx max pages env config 2026-04-02 10:31:52 +08:00
wizardchen 8e1cfaccb7 refactor(prompt_templates): improve question generation guidelines and context handling
- Updated the question generation template to clarify the role of surrounding context and main content.
- Enhanced quality rules for generated questions to better align with user search intent.
- Revised output format and added explicit instructions on what not to generate.
- Improved logging and output in the web parser for better visibility of parsed content and metadata.
2026-04-01 21:59:22 +08:00
wizardchen c4f5db7e88 feat(metadata): enhance document processing to include metadata extraction and handling
- Updated DocReaderServicer to pass metadata in responses.
- Modified PipelineParser to accumulate and merge metadata from all parsers.
- Enhanced StdWebParser to extract and log the title from web page content.
- Implemented logic in knowledge service to update knowledge title based on extracted metadata.
2026-04-01 15:49:47 +08:00
Windfarer bc0c2d23d5 add comment 2026-04-01 13:37:37 +08:00
Windfarer 6058f1cbe0 add monkey patch for docx parse error 2026-04-01 13:37:37 +08:00
wizardchen 1c9503b063 feat(docparser): enhance image resolution capabilities in markdown and HTML
- Updated regex patterns in MarkdownImageUtil to support alt text containing brackets and handle MIME types with hyphens.
- Implemented new functions in ImageResolver for resolving HTML <img> tags with data URIs and bare base64 content, improving image handling in markdown.
- Added comprehensive tests for various image scenarios, ensuring robust handling of data URIs and base64 images.
2026-03-25 22:08:29 +08:00