WeKnora/internal/application/service
wizardchen f92302cc64 fix(knowledge): stitch chunks by text overlap, not by position
Reconstructing full document content from chunks relied on position math
(offset = content length - (EndAt - lastEndAt)) that assumes
len([]rune(Content)) == EndAt - StartAt. Two upstream behaviors break that
invariant and caused the merged "全文" view to drop or duplicate text:

- The parent-child chunker prepends a synthetic table header to split
  tables; that header is zero-width in position space (start == end), so
  Content is longer than EndAt - StartAt.
- Chunk Content can retain HTML entities (", > ...), inflating the
  rune count relative to the source span.

Replace position-based trimming with text-overlap matching: find where the
accumulated text's suffix reappears at the head of the next chunk and join
there. Positions are only used to size the search window. This naturally
skips the prepended header and is immune to entity length drift.

Extract the logic into searchutil.AppendWithOverlap / MergeTextChunks and
reuse it across the three backend stitching sites (wiki_ingest
reconstructContent, graph mergeChunkContents, chat_pipeline
mergeOverlappingChunks). Mirror the same algorithm in the frontend
doc-content merge. Also fix the markdown hr/heading styles (thin solid
divider, primary heading color).
2026-05-30 17:53:47 +08:00
..
chat_pipeline fix(knowledge): stitch chunks by text overlap, not by position 2026-05-30 17:53:47 +08:00
file feat: enhance IM file service tests and refactor storage URL handling 2026-05-29 13:47:32 +08:00
memory feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
metric fix: resolve failing test and build errors 2026-04-22 21:17:29 +08:00
retriever refactor(keywords_vector_hybrid_indexer): enhance image payload sanitization logic 2026-05-29 16:56:53 +08:00
agent_history.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_history_test.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_service.go fix(agent): exclude processing docs from prompt 2026-05-18 16:49:14 +08:00
agent_service_test.go fix(agent): exclude processing docs from prompt 2026-05-18 16:49:14 +08:00
agent_share.go fix(rbac): allow source-tenant Admin+ to manage their cross-tenant shares 2026-05-18 17:28:58 +08:00
audit_log.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_retention.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_retention_test.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_test.go feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303) 2026-05-18 17:28:58 +08:00
chunk.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
custom_agent.go feat(custom_agent): enhance custom agent service with KB sharing support 2026-05-18 17:28:58 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource_service.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
evaluation.go feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
extract.go fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
graph.go fix(knowledge): stitch chunks by text overlap, not by position 2026-05-30 17:53:47 +08:00
image_multimodal.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
kbshare.go fix(rbac): allow source-tenant Admin+ to manage their cross-tenant shares 2026-05-18 17:28:58 +08:00
knowledge.go fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
knowledge_clone_move.go bugfix 2026-05-28 20:22:55 +08:00
knowledge_create.go fix: avoid stale knowledge records on upload failure 2026-05-29 16:49:07 +08:00
knowledge_create_test.go fix: avoid stale knowledge records on upload failure 2026-05-29 16:49:07 +08:00
knowledge_delete.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_faq.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledge_faq_import.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
knowledge_housekeeping.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_housekeeping_test.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_image_cleanup_test.go fix(knowledge): delete extracted images from storage when knowledge is removed 2026-04-02 12:04:04 +08:00
knowledge_manual_test.go fix: sanitize tab chars and avoid double .md extension in manual download filename 2026-03-16 18:18:23 +08:00
knowledge_owner_chain_test.go feat(rbac): per-KB ownership for knowledge / chunks / wiki pages (#1303) 2026-05-18 17:28:58 +08:00
knowledge_post_process.go test(knowledge): add comprehensive tests for finalizing subtask behavior 2026-05-30 01:00:15 +08:00
knowledge_process.go fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
knowledge_process_status_test.go fix(knowledge): unstick summary and isolate manual re-index trace attempts 2026-05-29 17:52:01 +08:00
knowledge_shared_access_test.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledge_span_tracker.go fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
knowledge_span_tracker_test.go feat(knowledge): instrument postprocess subspans + polish timeline UI 2026-05-28 15:14:45 +08:00
knowledge_storage_config_test.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
knowledge_summary_test.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
knowledge_task_options.go test: add document process task options tests 2026-05-28 20:27:53 +08:00
knowledge_task_options_test.go test: add document process task options tests 2026-05-28 20:27:53 +08:00
knowledge_util.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledgebase.go feat: implement tenant default storage provider handling in Knowledge Base creation 2026-05-29 13:47:32 +08:00
knowledgebase_pr3_test.go feat: implement tenant default storage provider handling in Knowledge Base creation 2026-05-29 13:47:32 +08:00
knowledgebase_search.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fanout.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fanout_test.go feat(retriever): add OpenSearch driver skeleton + interface stubs (PR 2a of 3) 2026-05-26 20:54:58 +08:00
knowledgebase_search_faq.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fusion.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_results.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
knowledgebase_search_shared.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledgebase_search_storegroup.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
mcp_service.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
mcp_service_test.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
mcp_tool_approval_service.go feat(agent): human-in-the-loop approval for MCP tool calls (#1173) 2026-05-10 22:57:12 +08:00
message.go fix(session): scope session access by user 2026-05-14 15:56:43 +08:00
metric_hook.go refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
model.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
ocr_sanitizer.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
ocr_sanitizer_test.go feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
organization.go feat(organization): transition membership model to tenant-level and enhance invite functionality 2026-05-18 17:28:58 +08:00
session.go feat(session): persist last request state for UI restoration 2026-05-18 22:28:00 +08:00
session_agent_qa.go refactor(api): drop deprecated tenant ConversationConfig field and KV write path 2026-05-18 17:28:58 +08:00
session_knowledge_qa.go fix: respect multi-turn-disabled flag in KnowledgeQA pipeline 2026-05-26 10:45:42 +08:00
session_knowledge_qa_test.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session_qa_helpers.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
session_user_scope_test.go fix(session): scope session access by user 2026-05-14 15:56:43 +08:00
skill_service.go feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
slug_fuzzy.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
slug_fuzzy_test.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
tenant.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_invitation_test.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
tenant_member_test.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
user.go feat(system-admin): implement revocation of system admin privileges with safeguards 2026-05-26 21:13:56 +08:00
user_jwt_test.go fix(auth): honour JWT tenant_id claim in middleware 2026-05-18 17:28:58 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_healthcheck.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_test.go feat(tests): enhance vector store tests with fake Elasticsearch server and add user pinning methods to knowledge base repository 2026-05-18 17:57:46 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
web_search_state.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_ingest.go fix(knowledge): stitch chunks by text overlap, not by position 2026-05-30 17:53:47 +08:00
wiki_ingest_batch.go fix(knowledge): wrap locale code with LanguageLocaleName in wiki ingest prompts 2026-05-30 16:56:32 +08:00
wiki_ingest_cite.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_ingest_cite_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_retry_test.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_ingest_test.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_linkify.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_linkify_test.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_lint.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go perf(wiki): subgraph API and interactive exploration for large graphs 2026-05-08 13:32:54 +08:00