WeKnora/docs
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
..
api feat(api-keys): add platform-scoped API keys for control-plane automation 2026-07-21 17:21:35 +08:00
cloud-image docs(cloud-image): add cloud-agnostic image packaging scripts 2026-05-11 12:25:19 +08:00
dev fix: validate vector store connection addresses against SSRF policy 2026-06-04 15:26:56 +08:00
images chore(release): prepare v0.7.1 2026-07-24 16:53:25 +08:00
wiki docs: clarify workspace owner cardinality 2026-07-23 16:13:35 +08:00
zh refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
BUILTIN_MCP_SERVICES.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
BUILTIN_MODELS.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
CHUNKING.md fix(chunking): post-review polish — error surfacing + doc accuracy 2026-05-06 17:17:07 +08:00
IM集成开发文档.md feat(im,datasource): support Lark, Feishu's international edition 2026-07-17 11:05:54 +08:00
KnowledgeGraph.md docs: 更新多语言文档,新增知识图谱与MCP配置指南及目录结构 2025-11-19 16:56:49 +08:00
LITE.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
Langfuse集成.md feat(llm): add prompt-cache observability and wiki cache-friendly prompts 2026-07-23 19:17:30 +08:00
MCP功能使用说明.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
OIDC认证调用流程.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
QA.md chore(release): prepare v0.7.1 2026-07-24 16:53:25 +08:00
RBAC说明.md docs: clarify workspace owner cardinality 2026-07-23 16:13:35 +08:00
ROADMAP.md feat(roadmap): update progress on document parsing and IM integration 2026-03-31 20:22:34 +08:00
agent-skills.md feat: adjust sandbox setting of agentConfig 2026-02-04 20:08:49 +08:00
docs.go chore(release): prepare v0.7.1 2026-07-24 16:53:25 +08:00
embed-secure-mode.md feat(embed): add widget, visitor chat, and management UI (#1657) 2026-06-12 23:10:39 +08:00
embed-subdomain.md feat(embed): add widget, visitor chat, and management UI (#1657) 2026-06-12 23:10:39 +08:00
migration-troubleshooting.md feat(system-info): surface DB migration errors with troubleshooting links 2026-05-14 16:34:50 +08:00
swagger.json chore(release): prepare v0.7.1 2026-07-24 16:53:25 +08:00
swagger.yaml chore(release): prepare v0.7.1 2026-07-24 16:53:25 +08:00
worker-pool-governance.md feat(worker): replace aggregate asynq concurrency with per-pool governance 2026-07-13 15:51:06 +08:00
使用其他向量数据库.md feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
共享空间说明.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
开发指南.md fix: update dev script checks and Air docs 2026-07-01 11:52:42 +08:00
开启知识图谱功能.md feat: Update VLM configuration in knowledge base and API documentation 2025-11-20 20:08:24 +08:00
快速开发模式说明.md docs: update Air installation command from cosmtrek/air to air-verse/air 2026-02-04 19:27:44 +08:00
数据源导入开发文档.md feat(feishu): sync docx via blocks API with per-type downdrill (#2087) 2026-07-27 16:04:56 +08:00
日志配置.md refactor(logger): support LOG_FORMAT template and harden level coloring 2026-05-22 20:31:54 +08:00
添加新的网络搜索引擎.md feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00