WeKnora/internal
Claude 49adf0d73c docs(chunking): align overlap default + sharpen UI/source/repo docs
Three documentation passes around the adaptive chunking work:

UI
- Frontend ChunkOverlap default consolidated to 80 (was 100), matching
  chunker.DefaultChunkOverlap on the backend. Both DEFAULT_CHUNKING_PRESET
  and initFormData updated. The KB-load fallback also uses 80 when a
  loaded KB has no chunk_overlap stored.
- All four locales (en-US, zh-CN, ko-KR, ru-RU) get rewritten chunking
  setting descriptions: each now states the validated range, the default,
  and the situations where you'd deviate (FAQ vs narrative, embedder
  token limits, language-specific corpora).

Source code
- splitter.go: DefaultChunkSize / DefaultChunkOverlap constants get a
  longer block-comment explaining the per-language token math and the
  use-case sweet spots, plus the migration note on what the old
  inconsistent defaults were.
- KBChunkingSettings.vue: new comment block above ChunkingConfig
  documents the slider min/max for each setting, why those bounds
  exist, and the recommended TokenLimit values per embedding model.

Repo docs
- New docs/CHUNKING.md: end-to-end guide covering why chunking matters,
  the adaptive 3-tier architecture, per-setting reference with ranges
  and sweet spots, parent-child explanation, the token-limit table per
  embedder (OpenAI / Voyage / Cohere / BGE / MiniLM / Jina), 7 use-case
  presets, the debug panel workflow, the API surface, and known
  trade-offs (recursive strategy hidden from UI, no auto-reindex on
  strategy switch, OCR limitations).
- CHANGELOG.md gets a new [Unreleased] section consolidating all the
  adaptive-chunking work shipped on this branch: 5 features, 8
  improvements, 6 fixes, 1 docs entry. The entry references
  docs/CHUNKING.md for deeper explanation.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
..
agent fix(ingest): stop hallucinating summaries from filenames on empty PDFs 2026-05-06 11:23:32 +08:00
application fix(chunker): QA audit — position invariant, recursive split, perf 2026-05-06 17:17:07 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
common feat: Implement deadlock retry mechanism for chunk creation 2026-04-22 21:17:21 +08:00
config feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
container fix(im): rewrite private storage URLs to HTTP in IM channel replies 2026-04-30 11:40:30 +08:00
database fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
datasource feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
errors feat: Add agent configuration and cleanup scripts for database migrations 2025-11-05 23:18:44 +08:00
event fix: improve tenant access token logging and update test cases 2026-04-22 21:17:30 +08:00
handler fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
im fix(im): clamp reconnect backoff to avoid int64 overflow → busy loop 2026-05-05 00:36:18 +08:00
infrastructure docs(chunking): align overlap default + sharpen UI/source/repo docs 2026-05-06 17:17:07 +08:00
logger feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
mcp feat(mcp): implement reconnection logic for MCP tool calls and tool listing 2026-03-31 11:57:15 +08:00
middleware fix(im): rewrite private storage URLs to HTTP in IM channel replies 2026-04-30 11:40:30 +08:00
models fix(embedding): OpenAIEmbedder.doRequestWithRetry shadows err · returns (nil, nil) on connection failure · caller nil-derefs and SIGSEGVs 2026-05-04 23:18:42 +08:00
router feat(handler): add chunker preview endpoint for debugging 2026-05-06 17:17:07 +08:00
runtime init commit 2025-08-05 15:08:07 +08:00
sandbox feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
searchutil feat(prompt): enhance question rewriting and intent classification in prompt templates 2026-04-16 23:15:02 +08:00
stream feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
tracing feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
types test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
utils fix(crypto): fail loudly when encrypted DB fields cannot be decrypted 2026-04-30 15:17:56 +08:00