WeKnora/migrations/versioned
wizardchen d0144f3586 perf(wiki): move ingest log to event table and index to on-demand API
Fixes O(n²) write amplification during wiki ingest on large KBs. Previously
every ingest/retract op re-wrote the single `slug='log'` row end-to-end and
every batch re-wrote the entire `slug='index'` directory markdown. On a 40k-
doc KB the log row grew to tens of MB and the index row to several MB, so
each batch triggered giant TOAST updates that dominated ingest wall time.

Log: new `wiki_log_entries` event table (`id DESC` indexed per KB) replaces
the single TEXT row. Batch ingest now collects entries and flushes them
once per batch via `AppendBatch`. Each entry stores `pages_affected` as
JSONB `[{slug,title}]` so the UI can render real titles; custom Scan falls
back to legacy `[]string` so older rows still deserialize.

Index: `wiki_pages[slug=index].content` keeps only the LLM-generated intro
(a few KB). The directory is now served by a structured paginated API
(`GetIndexView`) that reads `slug/title/summary` per type with cursor
pagination, so the agent and the frontend only pull the slice they need.
`RebuildIndexPage` degrades to a no-op; agent `wiki_read_page('index')`
synthesizes a small top-K overview and points callers at `wiki_search`.

Ingest resilience: LLM calls wrap with 3-attempt exponential backoff on
transient errors (5xx/408/429, transport resets/timeouts). Summary/extract
failures now bubble up so the batch's failed-op requeue path runs instead
of silently dropping the doc.

Frontend: sidebar Index/Log entries switch to dedicated views. Index view
streams intro → Summary → Entity → Concept → Synthesis → Comparison via
IntersectionObserver (with a nextTick re-check so small KBs still load
every section). Log view uses cursor pagination; pages_affected renders
titles with slug tooltip.

- new migration: migrations/versioned/000040_wiki_log_entries.{up,down}.sql
- tests: log repo pagination + legacy Scan, ListByTypeLight windowing,
  renderIndexOverviewForAgent output, isTransientLLMError classifier

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 21:48:52 +08:00
..
000000_init.down.sql feat(database): update database initialization to conditionally handle embeddings 2025-12-10 17:13:30 +08:00
000000_init.up.sql fix: migration from old version 2025-12-17 22:04:18 +08:00
000001_agent.down.sql feat(database): update database initialization to conditionally handle embeddings 2025-12-10 17:13:30 +08:00
000001_agent.up.sql feat(database): update database initialization to conditionally handle embeddings 2025-12-10 17:13:30 +08:00
000002_embeddings.down.sql feat(database): update database initialization to conditionally handle embeddings 2025-12-10 17:13:30 +08:00
000002_embeddings.up.sql feat(database): update database initialization to conditionally handle embeddings 2025-12-10 17:13:30 +08:00
000003_chunk_flags.down.sql feat: 新增FAQ推荐标签和批量操作支持,优化标签删除逻辑 2025-12-12 19:57:08 +08:00
000003_chunk_flags.up.sql feat: 新增FAQ推荐标签和批量操作支持,优化标签删除逻辑 2025-12-12 19:57:08 +08:00
000004_drop_vlm_model_id.down.sql fix: 强制移除knowledge_bases表中的历史vlm_model_id字段 2025-12-22 13:11:31 +08:00
000004_drop_vlm_model_id.up.sql fix: 强制移除knowledge_bases表中的历史vlm_model_id字段 2025-12-22 13:11:31 +08:00
000005_mentioned_items.down.sql feat: 支持在消息流中展示@的知识库和文件 2025-12-22 13:11:31 +08:00
000005_mentioned_items.up.sql feat: 支持在消息流中展示@的知识库和文件 2025-12-22 13:11:31 +08:00
000006_custom_agents.down.sql refactor: 调整智能体配置,增强智能体选择与状态管理 2025-12-26 21:31:02 +08:00
000006_custom_agents.up.sql refactor: 调整智能体配置,增强智能体选择与状态管理 2025-12-26 21:31:02 +08:00
000007_embeddings_tag_id.down.sql feat: 支持FAQ两级优先级标签过滤 2025-12-30 19:44:53 +08:00
000007_embeddings_tag_id.up.sql docs: sync changes to v0.2.8 changelog 2025-12-31 19:54:03 +08:00
000008_migrate_untagged_faq.down.sql feat: 支持删除文档类型标签 2026-01-15 19:38:16 +08:00
000008_migrate_untagged_faq.up.sql fix: guard pg_search update with skip_embedding 2026-03-19 22:03:09 +08:00
000009_add_last_faq_import_result.down.sql feat: 支持删除文档类型标签 2026-01-15 19:38:16 +08:00
000009_add_last_faq_import_result.up.sql feat: 支持删除文档类型标签 2026-01-15 19:38:16 +08:00
000010_add_seq_id.down.sql feat: 为 FAQ 和标签引入 seq_id 支持 2026-01-14 17:13:44 +08:00
000010_add_seq_id.up.sql feat: 为 FAQ 和标签引入 seq_id 支持 2026-01-14 17:13:44 +08:00
000011_pg_search_update.down.sql feat(migrations): add pg_search extension update migration files 2026-01-30 19:30:35 +08:00
000011_pg_search_update.up.sql fix: guard pg_search update with skip_embedding 2026-03-19 22:03:09 +08:00
000012_organizations.down.sql feat(agent): enhance agent and skill management features 2026-02-09 11:40:25 +08:00
000012_organizations.up.sql feat(agent): enhance agent and skill management features 2026-02-09 11:40:25 +08:00
000013_engine_configs.down.sql feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
000013_engine_configs.up.sql refactor: remove mineru_api_base_url from ParserEngineConfig and related components 2026-03-03 13:24:20 +08:00
000014_storage_provider_config.down.sql feat: introduce storage provider configuration and enhance backward compatibility 2026-03-02 21:21:49 +08:00
000014_storage_provider_config.up.sql feat: introduce storage provider configuration and enhance backward compatibility 2026-03-02 21:21:49 +08:00
000015_add_is_fallback.down.sql feat: add fallback response handling and UI indicators 2026-03-05 18:14:14 +08:00
000015_add_is_fallback.up.sql feat: add fallback response handling and UI indicators 2026-03-05 18:14:14 +08:00
000016_add_kb_pinned.down.sql feat: add pinning functionality for knowledge bases 2026-03-05 18:14:14 +08:00
000016_add_kb_pinned.up.sql feat: add pinning functionality for knowledge bases 2026-03-05 18:14:14 +08:00
000017_mcp_builtin.down.sql feat(security): add AES-256-GCM encryption for API keys at rest 2026-03-09 10:35:07 +08:00
000017_mcp_builtin.up.sql feat(security): add AES-256-GCM encryption for API keys at rest 2026-03-09 10:35:07 +08:00
000018_extend_tenant_api_key.down.sql feat(security): add AES-256-GCM encryption for API keys at rest 2026-03-09 10:35:07 +08:00
000018_extend_tenant_api_key.up.sql feat(security): add AES-256-GCM encryption for API keys at rest 2026-03-09 10:35:07 +08:00
000019_add_agent_duration_ms.down.sql chore: update AWS SDK dependencies and add database migrations 2026-03-09 11:46:56 +08:00
000019_add_agent_duration_ms.up.sql chore: update AWS SDK dependencies and add database migrations 2026-03-09 11:46:56 +08:00
000020_add_message_knowledge_id.down.sql chore: update AWS SDK dependencies and add database migrations 2026-03-09 11:46:56 +08:00
000020_add_message_knowledge_id.up.sql chore: update AWS SDK dependencies and add database migrations 2026-03-09 11:46:56 +08:00
000021_im_channel.down.sql feat: implement IM channel sessions and update message handling 2026-03-16 02:40:42 +08:00
000021_im_channel.up.sql feat: implement IM channel sessions and update message handling 2026-03-16 02:40:42 +08:00
000022_message_images.down.sql feat: implement IM channel sessions and update message handling 2026-03-16 02:40:42 +08:00
000022_message_images.up.sql feat: implement IM channel sessions and update message handling 2026-03-16 02:40:42 +08:00
000023_im_channel_kb_id.down.sql feat(im): add file support and knowledge base integration 2026-03-17 22:23:24 +08:00
000023_im_channel_kb_id.up.sql feat(im): add file support and knowledge base integration 2026-03-17 22:23:24 +08:00
000024_im_channel_bot_identity.down.sql feat(im): add file support and knowledge base integration 2026-03-17 22:23:24 +08:00
000024_im_channel_bot_identity.up.sql feat(im): add file support and knowledge base integration 2026-03-17 22:23:24 +08:00
000025_message_channel.down.sql feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
000025_message_channel.up.sql feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
000026_chunks_query_indexes.down.sql feat(agent): enhance chunk search functionality to include total chunk counts 2026-03-25 22:08:29 +08:00
000026_chunks_query_indexes.up.sql feat(agent): enhance chunk search functionality to include total chunk counts 2026-03-25 22:08:29 +08:00
000027_message_rendered_content.down.sql feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
000027_message_rendered_content.up.sql feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
000028_im_thread_session.down.sql [NELO-1102] feat(im): add thread-based session mode for IM channels 2026-03-30 10:43:14 +08:00
000028_im_thread_session.up.sql [NELO-1102] feat(im): add thread-based session mode for IM channels 2026-03-30 10:43:14 +08:00
000029_datasource_tables.down.sql feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
000029_datasource_tables.up.sql fix(migrations): correct migration numbers and remove broken trigger 2026-04-03 11:31:21 +08:00
000030_web_search_providers.down.sql feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
000030_web_search_providers.up.sql fix(migrations): correct migration numbers and remove broken trigger 2026-04-03 11:31:21 +08:00
000031_add_asr_config.down.sql feat(asr): add ASR model connection check and update related components 2026-04-02 21:27:27 +08:00
000031_add_asr_config.up.sql fix(migrations): correct migration numbers and remove broken trigger 2026-04-03 11:31:21 +08:00
000032_vector_stores.down.sql feat: add VectorStore entity, repository, and migrations 2026-04-09 01:39:51 +08:00
000032_vector_stores.up.sql feat: add VectorStore entity, repository, and migrations 2026-04-09 01:39:51 +08:00
000033_add_video_info_to_chunks.down.sql fix: 修正视频相关迁移文件版本号冲突,从 000032 改为 000033 2026-04-10 22:01:42 +08:00
000033_add_video_info_to_chunks.up.sql fix: 修正视频相关迁移文件版本号冲突,从 000032 改为 000033 2026-04-10 22:01:42 +08:00
000034_add_attachments.down.sql fix: rename sql file 2026-04-14 14:54:21 +08:00
000034_add_attachments.up.sql fix: rename sql file 2026-04-14 14:54:21 +08:00
000035_add_credentials.down.sql refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
000035_add_credentials.up.sql refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
000036_kb_vector_store_id.down.sql feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
000036_kb_vector_store_id.up.sql feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
000037_wiki_and_indexing.down.sql feat: Remove outdated wiki and indexing migration files 2026-04-22 21:24:08 +08:00
000037_wiki_and_indexing.up.sql feat: Remove outdated wiki and indexing migration files 2026-04-22 21:24:08 +08:00
000038_im_session_agent_id.down.sql fix(im): isolate IM sessions per agent and recover from deleted sessions 2026-04-28 20:53:06 +08:00
000038_im_session_agent_id.up.sql fix(im): isolate IM sessions per agent and recover from deleted sessions 2026-04-28 20:53:06 +08:00
000039_session_user_id_and_pin.down.sql feat(sessions): add keyword search, user-scoped pinning, and IM source visibility 2026-04-30 16:23:00 +08:00
000039_session_user_id_and_pin.up.sql feat(sessions): add keyword search, user-scoped pinning, and IM source visibility 2026-04-30 16:23:00 +08:00
000040_wiki_log_entries.down.sql perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
000040_wiki_log_entries.up.sql perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00