WeKnora/internal/types
wizardchen 84bd95275d fix(knowledge): make finalizing subtask counter reliable + async question fan-out
The finalizing subtask counter (pending_subtasks_count) could get stuck at a
non-zero value, leaving knowledge permanently in "finalizing" and never
promoted to "completed". Root cause and related hardening:

- UpdateKnowledge does a full-row Save; pending_subtasks_count was NOT in the
  omit list, so any concurrent enrichment subtask that loaded the row, did slow
  work (LLM call), then saved an unrelated field wrote back the STALE counter,
  clobbering decrements other subtasks performed in between. Add
  PendingSubtasksCount to omitFieldsOnUpdate so only the atomic helpers
  (SetFinalizing / FinalizeSubtask / explicit column writes) ever touch it.
- FinalizeSubtask no longer gates the finalizing->completed promote on a
  separate re-read of the counter. Every caller unconditionally runs the
  guarded promote UPDATE (WHERE pending_subtasks_count = 0), the single
  authoritative atomic check, so a racy/stale read can't strand the row.
- Decrements run on a context detached from the caller's cancellation
  (context.WithoutCancel + timeout) so graceful shutdown / preemption can't
  silently skip a decrement.
- Reconcile seeded vs actually-enqueued subtask slots (summary/question/graph)
  and release any shortfall so un-enqueued planned slots (e.g. graph with
  NEO4J off) don't strand the row.
- Reparse paths reset pending_subtasks_count via an explicit column write since
  full-row Save now omits it.

Question generation is also moved off the synchronous single-task path onto a
batched async fan-out (windows of text chunks), each batch independently
queued / retried / traced, grouped under a postprocess.question span.
2026-05-30 01:00:15 +08:00
..
interfaces feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
agent.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
audit_log.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
audit_log_test.go test(system): cover system.* audit constants + emitAdminAudit contract 2026-05-26 21:13:56 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
builtin_models_config.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
builtin_models_config_test.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
chat.go docs(chat): clarify cached-token semantics for explicit-cache providers 2026-05-25 16:47:14 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
chunk.go refactor(chunk): remove VideoInfo field from Chunk struct 2026-05-25 19:15:39 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
const.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers_test.go fix(rbac): address PR 2 review feedback 2026-05-18 17:28:58 +08:00
custom_agent.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
datasource_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embedding.go perf: 优化FAQ索引更新和删除性能 2026-01-14 17:30:04 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(types): enhance FAQ metadata normalization and sanitization 2026-03-19 18:20:09 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_map.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_span.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
knowledgebase.go feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
knowledgebase_test.go feat(knowledge-base): validate vector store bindings on create, copy, and delete 2026-05-18 15:58:46 +08:00
mcp.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
memory.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
message.go feat(chat_pipeline): enhance query handling with image and attachment metadata 2026-04-14 18:21:12 +08:00
model.go fix: add model display name 2026-05-28 19:22:17 +08:00
organization.go feat(organization): transition membership model to tenant-level and enhance invite functionality 2026-05-18 17:28:58 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
qa_request.go feat: attachment_support 2026-04-10 12:49:20 +08:00
retrieval_config.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
retriever.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
search.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
secret.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session.go feat(session): persist last request state for UI restoration 2026-05-18 22:28:00 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
task.go fix(knowledge): make finalizing subtask counter reliable + async question fan-out 2026-05-30 01:00:15 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
tenant.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go feat(rbac): tenant member management + invite_only gate (PR 3 of 3, #1303) 2026-05-18 17:28:58 +08:00
tenant_test.go feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
tracing.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
user.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
web_search_provider.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00