WeKnora/internal/searchutil
ChenRussell 2a60a7f30a fix(chunk-merge): stop dropping content when contiguous chunks repeat text
AppendWithOverlap deduplicates chunker-prepended table headers by
  matching a suffix of acc against a head window of next. When chunks are
  strictly contiguous (positionOverlap == 0) there is no overlap to
  deduplicate, but the loop still ran. If a sentence at the tail of acc
  reappeared as real content inside next's head window (same sentence
  written multiple times in the document), the head of next was mistaken
  for a prepended header and deleted — silently losing up to hundreds of
  bytes from the merged result.

  This affects three callers:
    - chat_pipeline.mergeOverlappingChunks: RAG search results returned
      to the LLM with truncated content
    - wiki_ingest.reconstructContent: distillation text fed to the LLM
      is truncated, so wiki search returns 0 hits for present content
    - frontend doc-content.vue port: the "merged" document view drops
      the same segment

  Fix: skip text matching when positionOverlap <= 0 and concatenate
  verbatim. The chunker's prepended-header case only occurs inside the
  overlap region (positionOverlap > 0), which still runs the loop.

  Add a regression test covering a compliance-doc scenario where a
  logging-requirement sentence recurs in a later Compliance section.
2026-08-13 19:58:57 +08:00
..
chunkmerge.go fix(chunk-merge): stop dropping content when contiguous chunks repeat text 2026-08-13 19:58:57 +08:00
chunkmerge_test.go fix(chunk-merge): stop dropping content when contiguous chunks repeat text 2026-08-13 19:58:57 +08:00
conversion.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
imageinfo.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
imageinfo_html_test.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
imageinfo_match.go fix(docparser): resolve remote images referenced by HTML img tags (#2355) 2026-07-28 21:08:31 +08:00
imageinfo_match_test.go fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
normalize.go refactor: Introduce searchutil package for web search result conversion and keyword score normalization 2025-12-01 18:27:12 +08:00
textutil.go refactor(chat_pipeline): enhance merging logic and add partial overlap removal 2026-03-25 22:08:29 +08:00