Commit Graph

6 Commits (578ca1b5193b66b7b08309dfefa7d89db20f8d2a)

Author SHA1 Message Date
Lukas 578ca1b519
fix(docreader): keep a pipe inside an HTML table cell in that cell (#4433)
HTML and MHTML uploads turn tables into GFM through markdownify, which
writes a pipe in a cell's text as is. That pipe ends the cell, so
<td>a|b</td> became two cells and every later cell in the row moved one
column right. Pipes in table cell text are now escaped before the
conversion, as `\|`.

The escape then has to survive MarkdownTableUtil, which normalizes the
converted tables and split rows on every pipe: `| a \| b | c |` came out
as a three-column row with the escape turned into `\ |`. It now splits
on unescaped pipes only; a pipe after an odd run of backslashes is cell
text, as in the Go chunker since #3979.
2026-10-17 22:27:49 +08:00
Lukas e7aaf8ae02
fix(docreader): keep the space held by a whitespace-only inline element (#3978)
markdownify renders an emphasis, code or link element whose text is only
whitespace as "", and the whitespace goes with it. HTML and MHTML
uploads therefore lost word boundaries: `further<strong> </strong>
reference` became `furtherreference`, and `<b>First</b><b> </b><b>Last</b>`
became `**First****Last**`. Editors produce that markup whenever a single
space between two words carries different formatting.

Before conversion, unwrap such elements so their whitespace stays as plain
text. Only elements with no child elements are touched, innermost first,
so a linked image keeps its link and nested wrappers come off completely.
2026-10-08 01:31:48 +08:00
ResistanceTo 77245f0f4b
fix(docreader): handle Header object when stripping mhtml header (#2902)
* fix(docreader): handle Header object when stripping mhtml header

* fix(docreader): decode 8-bit MHTML headers without crashing (#2901)

compat32 wraps non-ASCII 8-bit headers as Header; str() turns them into
U+FFFD and HTML Content-Location still crashed on .lower()/urljoin.

Refs: https://github.com/Tencent/WeKnora/issues/2901

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-31 17:14:58 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00