AppendWithOverlap deduplicates chunker-prepended table headers by
matching a suffix of acc against a head window of next. When chunks are
strictly contiguous (positionOverlap == 0) there is no overlap to
deduplicate, but the loop still ran. If a sentence at the tail of acc
reappeared as real content inside next's head window (same sentence
written multiple times in the document), the head of next was mistaken
for a prepended header and deleted — silently losing up to hundreds of
bytes from the merged result.
This affects three callers:
- chat_pipeline.mergeOverlappingChunks: RAG search results returned
to the LLM with truncated content
- wiki_ingest.reconstructContent: distillation text fed to the LLM
is truncated, so wiki search returns 0 hits for present content
- frontend doc-content.vue port: the "merged" document view drops
the same segment
Fix: skip text matching when positionOverlap <= 0 and concatenate
verbatim. The chunker's prepended-header case only occurs inside the
overlap region (positionOverlap > 0), which still runs the loop.
Add a regression test covering a compliance-doc scenario where a
logging-requirement sentence recurs in a later Compliance section.
|
||
|---|---|---|
| .. | ||
| packages | ||
| public | ||
| src | ||
| .dockerignore | ||
| .gitignore | ||
| Dockerfile | ||
| docker-entrypoint.sh | ||
| embed.html | ||
| env.d.ts | ||
| index.html | ||
| nginx.conf | ||
| package-lock.json | ||
| package.json | ||
| package.json.md5 | ||
| pnpm-workspace.yaml | ||
| tsconfig.app.json | ||
| tsconfig.json | ||
| tsconfig.node.json | ||
| vite.config.ts | ||