WeKnora/internal/application/service
Claude bc44c31cec fix(chunker): QA audit — position invariant, recursive split, perf
Resolves the issues surfaced by a self-review of the adaptive chunking
work. Tests covering each regression were added first to prove the
fix.

Critical
- Heading splitter no longer prepends breadcrumb to Content. The
  breadcrumb now lives on Chunk.ContextHeader; EmbeddingContent()
  combines header + content for the embedder. This restores the
  End-Start == len(Content) invariant relied on by knowledge.go's
  document-reconstruction path (summary generation, UI highlight) and
  also eliminates the duplicate-heading regression where the section
  heading appeared twice in a chunk's body.
- splitBySeparators is now genuinely recursive: when one separator
  leaves a piece > ChunkSize, the next-priority separator is applied
  inside that piece. Mirrors the Python reference. Without this, a
  document with one paragraph break followed by a long run of
  newline-separated lines could emit a single ~1950-rune chunk for
  ChunkSize=300.

Medium
- SplitParentChild forces children onto the recursive tier, skipping
  the per-parent profile pass (was N extra O(N) scans).
- Heuristic splitter snaps overlap start to the nearest semantic
  boundary or newline instead of slicing mid-line / mid-word.
- Strategy.Split returns the legacy tier's output directly when the
  whole chain rejects rather than running SplitText a second time.

Low
- ensureDefaults caps ChunkOverlap at ChunkSize/2 so pathological
  configs (TokenLimit clamping ChunkSize tiny while Overlap stays
  high) cannot produce ~97% overlap chunks.
- DefaultChunkOverlap docstring spells out the migration story for
  KBs that stored ChunkOverlap=0.

ContextHeader plumbing
- chunker.Chunk and types.ParsedChunk gain ContextHeader plus an
  EmbeddingContent() helper.
- types.Chunk gets a transient (gorm:"-") ContextHeader field so the
  knowledge service can build embedding inputs as
  titlePrefix + chunk.EmbeddingContent() without a DB schema change.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
..
chat_pipeline fix(memory): dedupe episode write on duplicate streaming Done 2026-05-04 23:49:29 +08:00
file feat(ks3): add KS3 object storage support 2026-05-05 00:22:27 +08:00
llmcontext feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
memory feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
metric fix: resolve failing test and build errors 2026-04-22 21:17:29 +08:00
retriever feat(retriever): exponential backoff + safety truncation for embeddings 2026-05-06 17:17:07 +08:00
agent_service.go feat: correct data analysis errors using fileService. 2026-04-27 13:18:08 +08:00
agent_share.go fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
chunk.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
custom_agent.go feat: Enhance suggested questions functionality with wiki integration 2026-04-22 21:18:20 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource_service.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
evaluation.go feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
extract.go feat: correct data analysis errors using fileService. 2026-04-27 13:18:08 +08:00
graph.go feat(graph): enhance graph extraction prompt rendering and language context handling 2026-03-25 22:08:29 +08:00
image_multimodal.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
kbshare.go feat(api): integrate per-organization resource counts into organization listing 2026-02-09 11:40:25 +08:00
knowledge.go fix(chunker): QA audit — position invariant, recursive split, perf 2026-05-06 17:17:07 +08:00
knowledge_image_cleanup_test.go fix(knowledge): delete extracted images from storage when knowledge is removed 2026-04-02 12:04:04 +08:00
knowledge_manual_test.go fix: sanitize tab chars and avoid double .md extension in manual download filename 2026-03-16 18:18:23 +08:00
knowledge_post_process.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
knowledge_shared_access_test.go fix: include shared knowledge in knowledge search 2026-05-05 01:04:49 +08:00
knowledge_storage_config_test.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
knowledge_summary_test.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
knowledgebase.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
knowledgebase_search.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_faq.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
knowledgebase_search_fusion.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_results.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
knowledgebase_search_shared.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
mcp_service.go feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
message.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
metric_hook.go refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
model.go feat(models): support custom HTTP headers across all remote model calls 2026-04-23 23:36:40 +08:00
ocr_sanitizer.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
ocr_sanitizer_test.go feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
organization.go feat(i18n): update localization for knowledge bases and organization features 2026-02-09 11:40:25 +08:00
session.go fix(sessions): review fixes for keyword search / pinning / IM titles 2026-04-30 16:23:00 +08:00
session_agent_qa.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
session_knowledge_qa.go fix(chat): include history in model fallback 2026-05-04 23:24:21 +08:00
session_knowledge_qa_test.go fix(chat): include history in model fallback 2026-05-04 23:24:21 +08:00
session_qa_helpers.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
skill_service.go feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
tag.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
tenant.go fix(tenant): return plaintext API key after create/reset 2026-04-29 19:46:21 +08:00
user.go simplify oidc callback result payload 2026-04-01 20:47:18 +08:00
vectorstore.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
vectorstore_healthcheck.go feat: add VectorStore service layer with types and connection testing 2026-04-13 22:57:17 +08:00
vectorstore_test.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_state.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_ingest.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
wiki_ingest_batch.go chore(comment): update slug source to knowledge ID 2026-05-06 11:23:32 +08:00
wiki_ingest_cite.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_ingest_cite_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_test.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_linkify.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_linkify_test.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_lint.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page_test.go refactor: Update wiki link syntax and rendering in WikiBrowser component 2026-04-22 21:17:26 +08:00