WeKnora/internal
wizardchen f92302cc64 fix(knowledge): stitch chunks by text overlap, not by position
Reconstructing full document content from chunks relied on position math
(offset = content length - (EndAt - lastEndAt)) that assumes
len([]rune(Content)) == EndAt - StartAt. Two upstream behaviors break that
invariant and caused the merged "全文" view to drop or duplicate text:

- The parent-child chunker prepends a synthetic table header to split
  tables; that header is zero-width in position space (start == end), so
  Content is longer than EndAt - StartAt.
- Chunk Content can retain HTML entities (", > ...), inflating the
  rune count relative to the source span.

Replace position-based trimming with text-overlap matching: find where the
accumulated text's suffix reappears at the head of the next chunk and join
there. Positions are only used to size the search window. This naturally
skips the prepended header and is immune to entity length drift.

Extract the logic into searchutil.AppendWithOverlap / MergeTextChunks and
reuse it across the three backend stitching sites (wiki_ingest
reconstructContent, graph mergeChunkContents, chat_pipeline
mergeOverlappingChunks). Mirror the same algorithm in the frontend
doc-content merge. Also fix the markdown hr/heading styles (thin solid
divider, primary heading color).
2026-05-30 17:53:47 +08:00
..
agent fix(agent): harden tool parameter parsing against LLM type mismatches (#1505) 2026-05-28 07:50:26 +08:00
application fix(knowledge): stitch chunks by text overlap, not by position 2026-05-30 17:53:47 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
common feat: Implement deadlock retry mechanism for chunk creation 2026-04-22 21:17:21 +08:00
config test: add document process task options tests 2026-05-28 20:27:53 +08:00
container feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
database feat(system-info): surface DB migration errors with troubleshooting links 2026-05-14 16:34:50 +08:00
datasource fix(datasource): support Yuque team token in connector 2026-05-26 20:46:22 +08:00
errors feat(knowledge): track per-stage parsing progress with /stages API 2026-05-28 15:14:45 +08:00
event feat(agent): human-in-the-loop approval for MCP tool calls (#1173) 2026-05-10 22:57:12 +08:00
handler feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
im feat: enhance IM file service tests and refactor storage URL handling 2026-05-29 13:47:32 +08:00
infrastructure feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3) 2026-05-26 20:54:58 +08:00
logger refactor(logger): support LOG_FORMAT template and harden level coloring 2026-05-22 20:31:54 +08:00
mcp feat(mcp): implement reconnection logic for MCP tool calls and tool listing 2026-03-31 11:57:15 +08:00
middleware fix(knowledge): prevent documents from getting stuck in "processing" 2026-05-28 15:14:45 +08:00
models docs(chat): clarify cached-token semantics for explicit-cache providers 2026-05-25 16:47:14 +08:00
router feat: implement tenant validation for file access and enhance related tests 2026-05-29 13:47:32 +08:00
runtime chore(runtime): silence gin per-route logs and emit env config banner at startup 2026-05-17 15:27:52 +08:00
sandbox 增加sandbox对windows编译支持,现在默认是linux的实现,windows直接编译报错 2026-05-25 16:57:56 +08:00
searchutil fix(knowledge): stitch chunks by text overlap, not by position 2026-05-30 17:53:47 +08:00
stream feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
tracing feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
types fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
utils feat: implement tenant validation for file access and enhance related tests 2026-05-29 13:47:32 +08:00