WeKnora/internal/infrastructure/chunker
Lukas 98feef38de
fix(chunker): count an escaped pipe as cell text when matching table widths (#3979)
The header tracker compares a row's column count with the active table
header's to tell a new table from more rows of the same one.
tableRowColumnCount split rows on every `|`, so a cell holding an escaped
pipe (`\|`, which GFM writers such as anydoc emit for a pipe in a cell)
counted as one more column. The tracker took that row for the start of a
new table and dropped the header, so the chunks after it no longer began
with the column names.

Split rows on unescaped pipes only. A pipe after an odd run of
backslashes is text inside the cell, as in GFM.
2026-10-08 01:21:16 +08:00
..
header_tracker.go fix(chunker): count an escaped pipe as cell text when matching table widths (#3979) 2026-10-08 01:21:16 +08:00
heading_hierarchy.go feat(chunker): heading hierarchy tracker 2026-05-06 17:17:07 +08:00
heading_hierarchy_test.go feat(chunker): heading hierarchy tracker 2026-05-06 17:17:07 +08:00
heading_splitter.go fix(chunker): carry deep sub-heading context into large-section chunks 2026-06-16 16:56:57 +08:00
heading_splitter_test.go fix(chunker): carry deep sub-heading context into large-section chunks 2026-06-16 16:56:57 +08:00
heuristic_splitter.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
heuristic_splitter_test.go fix(chunker): heuristic splitter drops boundaries inside protected spans 2026-05-06 21:16:48 +08:00
parent_child_offsets_test.go fix(chunker): preserve source offsets for table child chunks (#3969) 2026-10-08 01:21:08 +08:00
patterns.go fix(chunker): recognise CN chapter titles and multi-level numeric headings 2026-05-06 22:14:16 +08:00
patterns_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
profiler.go refactor(chunker): drop redundant TierRecursive 2026-05-06 20:23:05 +08:00
profiler_test.go refactor(chunker): drop redundant TierRecursive 2026-05-06 20:23:05 +08:00
protected_spans_test.go fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404) 2026-09-18 21:10:17 +08:00
splitter.go fix(chunker): preserve source offsets for table child chunks (#3969) 2026-10-08 01:21:08 +08:00
splitter_test.go fix(chunker): count an escaped pipe as cell text when matching table widths (#3979) 2026-10-08 01:21:16 +08:00
strategy.go fix(chunker): preserve source offsets for table child chunks (#3969) 2026-10-08 01:21:08 +08:00
strategy_diagnostics_test.go fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
strategy_test.go fix(chunking): preserve parent-child text child embeddings (#2680) 2026-08-13 11:20:03 +08:00
strategy_token_test.go feat(chunker): recursive priority splitting + token-aware budget 2026-05-06 17:17:07 +08:00
tokens.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
tokens_test.go feat(chunker): foundation for adaptive 3-tier chunking 2026-05-06 17:17:07 +08:00
validator.go feat(chunker): foundation for adaptive 3-tier chunking 2026-05-06 17:17:07 +08:00