WeKnora/internal
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
..
agent feat(agent): persist compaction checkpoints across turns (#3407) 2026-09-18 20:38:56 +08:00
application fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
browserskill fix(browser): align BrowserSkill 0.3.0 and repair task lifecycle 2026-09-17 11:54:12 +08:00
common fix(markdown): sanitize invalid UTF-8 and NUL bytes 2026-08-31 12:24:35 +08:00
config feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
container feat(mcp): add built-in MCP server with per-workspace endpoints (#3371) 2026-09-17 22:40:05 +08:00
database feat(agent): persist compaction checkpoints across turns (#3407) 2026-09-18 20:38:56 +08:00
datasource refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
embedpolicy fix(embed): enforce host origin allowlists on framed pages 2026-09-14 18:03:57 +08:00
errors feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
event feat(sandbox): stream live shell and install command output 2026-09-12 10:16:39 +08:00
filetransport refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
handler fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
im refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381) 2026-09-18 14:12:22 +08:00
infrastructure fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
ipclass fix: align client sessions and integration request handling (#3305) 2026-09-16 16:20:57 +08:00
logger fix(sandbox): record product-level Langfuse spans and detach idle sweeps 2026-08-31 19:42:32 +08:00
mcp fix: align client sessions and integration request handling (#3305) 2026-09-16 16:20:57 +08:00
mcpserver refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents (#3381) 2026-09-18 14:12:22 +08:00
middleware feat(audit): 知识库活动记录对 API Key 操作额外显示 Key 名称 (#3405) 2026-09-18 19:39:34 +08:00
modelcontext fix(agent): make read_document queries keyword-based and bound its pages 2026-09-18 15:09:25 +08:00
models feat(agent): persist compaction checkpoints across turns (#3407) 2026-09-18 20:38:56 +08:00
ratelimit feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
router feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
runtime fix(redis): harden TLS tests and surface TLS config at startup 2026-07-06 23:04:52 +08:00
sandbox perf(sandbox): reduce redundant remote calls 2026-09-17 17:47:24 +08:00
searchutil fix(chat): keep OCR/caption text of scanned-page image hits in merge (#3064) 2026-09-07 11:38:47 +08:00
storageallowlist fix(storage): align default provider fallback with STORAGE_ALLOW_LIST 2026-07-15 11:05:00 +08:00
storageurl refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
stream fix(chat): preserve steering continuity and composer focus 2026-09-09 20:15:23 +08:00
textconv fix: remove GPL converter and bundle third-party license sources 2026-09-08 15:19:10 +08:00
tracing/langfuse fix(milvus): preserve keyword retrieval scores 2026-09-17 11:59:40 +08:00
types feat(agent): persist compaction checkpoints across turns (#3407) 2026-09-18 20:38:56 +08:00
utils refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00