WeKnora/internal/types
wizardchen 44d6175559 feat: add knowledge parse cancellation with finalizing post-process state
Lets users stop an in-flight document parse to free up LLM / worker
resources without losing the chunks and index already written. The
core insight is that the previous parse_status=completed flipped as
soon as primary chunks landed, while the most expensive subtasks
(graph extract = N LLM calls per chunk, plus summary, question
generation) were still running in the background — so "completed"
wasn't actually terminal from a resource standpoint.

State machine

  pending -> processing -> finalizing -> completed
                              |
                              +-> cancelled (any of the three
                                            in-flight states)
                              +-> failed
                              +-> deleting

`finalizing` is the new post-process fan-out window. parse_status
only promotes to `completed` once pending_subtasks_count (a new
column tracking summary + question + per-chunk graph extract)
drains to zero via atomic FinalizeSubtask. Wiki ingest is
intentionally excluded from the counter — it's a KB-scoped
debounced batch and would otherwise pin parse_status in
`finalizing` for the wiki batch window.

Backend

- New ParseStatusFinalizing + pending_subtasks_count column with
  migration 000056.
- knowledgeRepository.SetFinalizing transitions processing -> finalizing
  conditionally so a racing cancel cannot be clobbered.
- knowledgeRepository.FinalizeSubtask atomically decrements the
  counter and self-promotes the row to completed when it hits zero.
- KnowledgePostProcess restructured to compute expected subtask
  count up front, flip to finalizing (or completed when no
  enrichment is enabled), and only then fan out subtasks. Subtask
  handlers (summary, question, graph extract) defer-decrement on
  terminal exit using the existing isFinalAsynqAttempt convention.
- New POST /api/v1/knowledge/{id}/cancel-parse handler accepting
  pending / processing / finalizing. Marks the row cancelled,
  zeroes the counter, best-effort dequeues asynq tasks via a new
  TaskInspector abstraction (asynq-mode walks pending/scheduled/
  retry queues; Lite-mode noop), and scrubs wiki ingest pending op.
- SpanTracker.AbortAttempt flat-sweeps every still-running span
  for the attempt via a new repo.CancelAllOpenSpans helper so the
  trace viewer's striped bars all flip to cancelled, even leaf
  generations whose parent stage already EndSpan'd (multimodal
  fan-out pattern). knowledge_post_process closes its postSpan
  via SkipSpan on the cancel/deleting entry guard so a worker
  that opens a span AFTER the cancel sweep doesn't leak it.
- Housekeeping and resetPendingTasks sweep finalizing rows
  identically to processing so a crash/restart can't strand them.
- DeleteKnowledge/DeleteKnowledgeList proactively dequeue
  downstream tasks via the same TaskInspector path.
- ChunkExtractService gets a cancel entry guard so the most
  expensive enrichment (graph extract) bails immediately when the
  parent knowledge is aborted.

Frontend

- New cancelKnowledgeParse API client + "Stop parsing" entry in
  both list view and card view more menus, gated on
  pending/processing/finalizing.
- Polling predicate refactored to a shared isParseInFlight helper
  that recognises `finalizing` (previously the doc list silently
  stopped polling once parse_status flipped from processing).
- Knowledge processing timeline: isPolling includes finalizing,
  new isHardTerminal short-circuits LIVE for cancelled/failed/
  completed so stranded child spans cannot pin LIVE on.
- DocumentListView.computeStatus distinguishes finalizing
  ("增强中") from completed and shows the previous "生成摘要中"
  copy when summary_status is still pending under finalizing.
  Added cancelled badge as well.
- i18n: statusFinalizing / statusCancelled / cancelParse* keys
  across zh-CN, en-US, ko-KR, ru-RU.

Docs / SDK

- docs/api/knowledge.md: documents the new finalizing state,
  cancel-parse semantics, and which statuses accept cancel.
- client (Go SDK): CancelKnowledgeParse with docstring listing
  the cancellable statuses.
2026-05-28 20:16:02 +08:00
..
interfaces feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
agent.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
audit_log.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
audit_log_test.go test(system): cover system.* audit constants + emitAdminAudit contract 2026-05-26 21:13:56 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
builtin_models_config.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
builtin_models_config_test.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
chat.go docs(chat): clarify cached-token semantics for explicit-cache providers 2026-05-25 16:47:14 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
chunk.go refactor(chunk): remove VideoInfo field from Chunk struct 2026-05-25 19:15:39 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
const.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers_test.go fix(rbac): address PR 2 review feedback 2026-05-18 17:28:58 +08:00
custom_agent.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
datasource_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embedding.go perf: 优化FAQ索引更新和删除性能 2026-01-14 17:30:04 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(types): enhance FAQ metadata normalization and sanitization 2026-03-19 18:20:09 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_map.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_span.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
knowledgebase.go feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
knowledgebase_test.go feat(knowledge-base): validate vector store bindings on create, copy, and delete 2026-05-18 15:58:46 +08:00
mcp.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
memory.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
message.go feat(chat_pipeline): enhance query handling with image and attachment metadata 2026-04-14 18:21:12 +08:00
model.go fix: add model display name 2026-05-28 19:22:17 +08:00
organization.go feat(organization): transition membership model to tenant-level and enhance invite functionality 2026-05-18 17:28:58 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
qa_request.go feat: attachment_support 2026-04-10 12:49:20 +08:00
retrieval_config.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
retriever.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
search.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
secret.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session.go feat(session): persist last request state for UI restoration 2026-05-18 22:28:00 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
task.go feat(knowledge): instrument postprocess subspans + polish timeline UI 2026-05-28 15:14:45 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
tenant.go feat: add vLLM server URL configuration for MinerU 2026-05-20 16:33:12 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go feat(rbac): tenant member management + invite_only gate (PR 3 of 3, #1303) 2026-05-18 17:28:58 +08:00
tenant_test.go feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
tracing.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
user.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
vectorstore_test.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
web_search_provider.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00