WeKnora/internal/types
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
..
interfaces feat(feishu): sync docx via blocks API with per-type downdrill (#2087) 2026-07-27 16:04:56 +08:00
agent.go feat(agent): add per-agent citation output toggle 2026-07-17 10:19:43 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
audit_log.go feat(runtime): add bulk purge for archived queue tasks and chat_attachment i18n 2026-07-23 16:52:35 +08:00
audit_log_test.go feat(runtime): add bulk purge for archived queue tasks and chat_attachment i18n 2026-07-23 16:52:35 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
builtin_models_config.go feat(models): let system admins edit builtin model config 2026-07-14 15:56:02 +08:00
builtin_models_config_test.go feat(models): let system admins edit builtin model config 2026-07-14 15:56:02 +08:00
chat.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go refactor: remove Neo4j-based conversation memory feature 2026-07-18 23:41:48 +08:00
chunk.go refactor(chunk): remove VideoInfo field from Chunk struct 2026-05-25 19:15:39 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
config_redaction.go fix(parser): remove workspace chat parser policy UI 2026-07-17 13:42:02 +08:00
config_redaction_test.go fix(parser): remove workspace chat parser policy UI 2026-07-17 13:42:02 +08:00
const.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
context_helpers.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
context_helpers_test.go feat(llm): add prompt-cache observability and wiki cache-friendly prompts 2026-07-23 19:17:30 +08:00
custom_agent.go feat(agent): add per-agent citation output toggle 2026-07-17 10:19:43 +08:00
custom_agent_test.go feat(agent): add per-agent citation output toggle 2026-07-17 10:19:43 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(feishu): sync docx via blocks API with per-type downdrill (#2087) 2026-07-27 16:04:56 +08:00
datasource_test.go feat(feishu): sync docx via blocks API with per-type downdrill (#2087) 2026-07-27 16:04:56 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embed_channel.go refactor: remove Neo4j-based conversation memory feature 2026-07-18 23:41:48 +08:00
embed_channel_test.go feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
embedding.go feat(agent): enhance suggested questions API with tag scopes 2026-07-17 10:19:43 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go fix: support custom jieba dictionary directory 2026-06-08 20:10:37 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
faq_sync.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
faq_sync_test.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_map.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
knowledge_process.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
knowledge_process_test.go feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
knowledge_span.go fix(spans): expand knowledge span name column and truncate long wiki names 2026-07-13 10:35:44 +08:00
knowledgebase.go feat(resource): add stable resource registry with LLM-context alias compaction 2026-07-15 17:02:43 +08:00
knowledgebase_test.go fix: deep-copy stored files and images when cloning a knowledge base 2026-06-03 14:45:59 +08:00
mcp.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
mcp_oauth.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
message.go fix(types): use correct GORM tag separator in Message.KnowledgeReferences 2026-07-21 12:06:48 +08:00
message_attachment_test.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
message_suggestion.go feat(agent): add question suggestions with after-answer follow-ups 2026-07-14 11:17:25 +08:00
model.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
organization.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
parser_engine_config_test.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
principal.go feat(api-keys): add platform-scoped API keys for control-plane automation 2026-07-21 17:21:35 +08:00
principal_test.go test(api-keys): add unit tests for tenant API key validation 2026-07-07 12:00:13 +08:00
prompt_instructions.go Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00
prompt_instructions_test.go Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00
qa_request.go refactor: remove Neo4j-based conversation memory feature 2026-07-18 23:41:48 +08:00
question_suggestion_test.go feat(agent): add question suggestions with after-answer follow-ups 2026-07-14 11:17:25 +08:00
resource.go feat(resource): add stable resource registry with LLM-context alias compaction 2026-07-15 17:02:43 +08:00
retrieval_config.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
retriever.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
runtime_task.go feat(system): add cursor pagination for runtime queue tasks 2026-07-14 16:53:07 +08:00
search.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
search_scope_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
secret.go fix(security): harden secret redaction follow-ups for PR #1884 2026-07-02 14:41:42 +08:00
secret_test.go fix(security): harden secret redaction follow-ups for PR #1884 2026-07-02 14:41:42 +08:00
session.go feat(session): gate IM/embed/API channel sessions behind admin 2026-07-20 20:31:54 +08:00
session_access_test.go feat(session): gate IM/embed/API channel sessions behind admin 2026-07-20 20:31:54 +08:00
storagebackend.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
storagebackend_test.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
sync_item_error_test.go fix(feishu): resilient large-wiki sync — retry, streaming resume, failure visibility 2026-07-21 16:14:22 +08:00
system_setting.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
tag.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
task.go fix(tasks): dequeue work for deleted knowledge bases 2026-07-23 12:17:14 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_initiator_test.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_queue_test.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
temporary_document.go feat(attachments): implement normalization and limits for temporary attachment IDs 2026-07-16 17:41:27 +08:00
tenant.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
tenant_api_key.go feat(api-keys): add platform-scoped API keys for control-plane automation 2026-07-21 17:21:35 +08:00
tenant_api_key_scope_test.go Add API key member and space capabilities 2026-07-07 12:00:13 +08:00
tenant_api_key_test.go Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
tenant_test.go feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
tracing.go feat(tracing): migrate langfuse client to OTLP/OTel wire protocol 2026-07-17 16:30:45 +08:00
user.go refactor: remove Neo4j-based conversation memory feature 2026-07-18 23:41:48 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
vectorstore_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_test.go feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
web_search_provider.go feat(web-search): add Zhipu AI web search provider support 2026-07-20 12:15:35 +08:00
web_search_provider_test.go feat(web-search): add Zhipu AI web search provider support 2026-07-20 12:15:35 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_log_entry.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
wiki_page.go feat(wiki): add page revision history with diff, revert, and manual editing 2026-07-27 15:03:24 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00