WeKnora/internal/handler
Claude 8388a50c23 fix(chunker): post-review fixes for preview endpoint robustness
Resolves issues from the review of be326aa..119f5e4. Each fix has a
regression test attached and overclaimed findings (parserEngineRules
defensive copy, runTier dead-code branch, init-vars architecture)
were intentionally not touched after re-evaluation.

Goroutine leak mitigation
- previewMaxChars dropped from 256k to 64k runes. The splitter does
  not accept a context.Context, so when previewTimeout fires the
  worker keeps running. Bounding input size keeps worst-case CPU
  per request well under a second on commodity hardware. Sized so
  10 concurrent timeouts don't pile up faster than they finish.
- Frontend MAX_CHARS lowered to match.
- Comment in handler explains the trade-off and points at the
  follow-up: real cancellation needs the splitter to take a ctx.

Performance
- ApproxTokenCountFromRuneLen variant lets the preview handler
  reuse a single rune-count per chunk for stats + size + token
  estimation. Eliminates the previous triple []rune allocation per
  chunk in the response loop.
- computeChunkSizeStats now takes []int (pre-computed rune lens)
  instead of []chunker.Chunk; sumSq computed in float64 to avoid
  the int*int overflow at l > ~46k.

Correctness / UX
- Preview panel sends strategy / token_limit / languages
  unconditionally, mirroring the buildSubmitData convention so the
  preview faithfully reflects what would happen on save.
- Empty-text returns a friendly 400 ("paste a sample…") instead of
  gin's cryptic 'Field validation failed on the required tag'.

Tests
- TestSplit_DelegatesToSplitWithDiagnostics renamed to
  TestSplit_AndDiagnostics_AgreeOnChunks (the post-audit refactor
  made the original name a misnomer; the test still asserts the
  right invariant under the new name).
- New TestSplitWithDiagnostics_ProfileSetForAuto and
  TestSplitWithDiagnostics_ProfileNilForExplicit lock in the
  profile-reuse contract that the preview endpoint depends on.
- New chunker_debug_test.go covers computeChunkSizeStats edge
  cases (empty / single / varying / no-variance underflow) plus
  PreviewChunking httptest scenarios (auto path, legacy strategy,
  empty-text rejection, oversize rejection, chunk truncation with
  full-set stats).

Doc cleanup
- runTier comment updated; the "stubbed in this scaffold" line was
  obsolete since the heading and heuristic splitters shipped.

https://claude.ai/code/session_01XADhx6mtu2ZYW3DE9Lun6k
2026-05-06 17:17:07 +08:00
..
session fix(sessions): review fixes for keyword search / pinning / IM titles 2026-04-30 16:23:00 +08:00
auth.go feat: implement auto-setup for Lite edition, enhancing user experience by automating user and tenant creation on first launch 2026-04-11 18:51:16 +08:00
chunk.go feat(api): enhance organization shared knowledge base and agent functionalities 2026-02-09 11:40:25 +08:00
chunker_debug.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
chunker_debug_test.go fix(chunker): post-review fixes for preview endpoint robustness 2026-05-06 17:17:07 +08:00
custom_agent.go fix(agent): improve tenant context handling and error reporting 2026-04-24 22:40:57 +08:00
datasource.go feat(datasource): enforce tenant isolation and knowledge base ownership 2026-03-30 20:46:07 +08:00
evaluation.go feat: 新增API Key认证支持,优化Swagger文档安全配置 2025-12-22 15:01:09 +08:00
faq.go feat(api): enhance organization shared knowledge base and agent functionalities 2026-02-09 11:40:25 +08:00
im.go feat(im): tenant-wide IM channels overview under the user menu 2026-04-30 15:29:14 +08:00
initialization.go fix(chunker): QA audit — pointer DTOs, profile reuse, stats correctness 2026-05-06 17:17:07 +08:00
knowledge.go refactor(knowledge-base): address review feedback 2026-04-29 12:01:38 +08:00
knowledgebase.go Merge branch 'main' into storage-allow-list 2026-04-29 19:35:13 +08:00
mcp_service.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
message.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
model.go feat(asr): add ASR model connection check and update related components 2026-04-02 21:27:27 +08:00
organization.go fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
skill_handler.go feat: update sandbox service configuration in Docker Compose files 2026-02-04 21:34:07 +08:00
storage_allowlist.go feat(ks3): add KS3 object storage support 2026-05-05 00:22:27 +08:00
system.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
tag.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
tenant.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
vectorstore.go feat: add VectorStore CRUD API endpoints and documentation 2026-04-14 17:34:57 +08:00
web_search.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
web_search_provider.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
wechat_qrcode.go feat(im): add WeChat integration with QR code login and long-polling support 2026-04-13 14:17:57 +08:00
weknoracloud.go feat(weknoracloud): implement credential management and status checks 2026-04-14 17:36:01 +08:00
wiki_page.go feat: Introduce `wiki_flag_issue` tool for reporting wiki page issues 2026-04-22 21:17:38 +08:00