WeKnora/internal/infrastructure/chunker
wizardchen d7478094a2 fix(chunker): recognise CN chapter titles and multi-level numeric headings
The heuristic splitter relied on two regexes that were too strict for
real-world Chinese technical documents:

- ChineseChapterPattern required 第 / numeral / unit to be adjacent, so
  the very common "第 1 章 引言" form never matched.
- NumberedSectionPattern required a trailing dot after the numeral, so
  multi-level numbering such as "1.1 文档目的" or "2.2.1 用户与权限"
  was missed.

As a result, documents like the CHAPTER_SAMPLE shipped with the chunking
debug drawer collected zero heuristic markers, and ProfileDocument fell
all the way through to the character-level Legacy tier, producing chunks
that ignored the document's explicit chapter structure.

- Loosen ChineseChapterPattern to tolerate spaces around 第 / 数字 / 单位.
- Allow the multi-level branch of NumberedSectionPattern to drop its
  trailing dot; keep the single-level / roman branch dot-required to
  avoid false positives on version strings.
- Extend patterns_test.go with positive and negative cases covering the
  CHAPTER_SAMPLE wording plus deep-nesting and lone-numeral regressions.
2026-05-06 22:14:16 +08:00
..
header_tracker.go feat(chunker): implement header tracking for Markdown tables during chunking 2026-04-01 13:38:40 +08:00
heading_hierarchy.go feat(chunker): heading hierarchy tracker 2026-05-06 17:17:07 +08:00
heading_hierarchy_test.go feat(chunker): heading hierarchy tracker 2026-05-06 17:17:07 +08:00
heading_splitter.go perf(chunker): thread DocProfile through splitter dispatch 2026-05-06 21:16:48 +08:00
heading_splitter_test.go perf(chunker): thread DocProfile through splitter dispatch 2026-05-06 21:16:48 +08:00
heuristic_splitter.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
heuristic_splitter_test.go fix(chunker): heuristic splitter drops boundaries inside protected spans 2026-05-06 21:16:48 +08:00
patterns.go fix(chunker): recognise CN chapter titles and multi-level numeric headings 2026-05-06 22:14:16 +08:00
patterns_test.go fix(chunker): recognise CN chapter titles and multi-level numeric headings 2026-05-06 22:14:16 +08:00
profiler.go refactor(chunker): drop redundant TierRecursive 2026-05-06 20:23:05 +08:00
profiler_test.go refactor(chunker): drop redundant TierRecursive 2026-05-06 20:23:05 +08:00
splitter.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
splitter_test.go fix(chunker): QA audit — position invariant, recursive split, perf 2026-05-06 17:17:07 +08:00
strategy.go feat(chunker): SplitParentChild children honour configured strategy 2026-05-06 21:16:48 +08:00
strategy_diagnostics_test.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
strategy_test.go feat(chunker): SplitParentChild children honour configured strategy 2026-05-06 21:16:48 +08:00
strategy_token_test.go feat(chunker): recursive priority splitting + token-aware budget 2026-05-06 17:17:07 +08:00
tokens.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
tokens_test.go feat(chunker): foundation for adaptive 3-tier chunking 2026-05-06 17:17:07 +08:00
validator.go feat(chunker): foundation for adaptive 3-tier chunking 2026-05-06 17:17:07 +08:00