WeKnora/internal/datasource
linus-liu-web 8319d59dd0
feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf) (#3786)
* feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf)

钉钉连接器此前只导入原生 .adoc 文档:知识库里的上传文件既不会出现在选择器里,
也不会被同步,而且没有任何提示。

- 节点判定拆成 isOnlineDocument / isBinaryDocument / binaryContentType,isDocument()
  变成"在线文档 ∪ 二进制文档",上传文件因此可被枚举与摄取;表格(axls)等其它原生
  类型仍不在其中,避免枚举出同步读不了的节点。
- 上传文件通过钉盘下载 API 逐字节传输,交给 WeKnora 自己的解析器:先用
  /v2.0/doc/dentries/{id}/queryDentryId 把节点 UUID 换成数字 spaceId/dentryId
  (存储 API 拒绝非数字 spaceId),再取预签名 URL 下载。
- 大小上限加在传输上而不是列表元数据上:node.size 与存储对象并不一致。
- 预签名 URL 短命且带版本,传输失败会用新签名重试。
- Validate 对上传文件"解析位置但不传 payload",凭证校验不必为每个候选文件下载
  数 MB;adoc 仍走 blocks API。
- 同步按类型分派:adoc 渲染 blocks,上传文件落原始字节并带上具体 MIME 类型与只含
  一个扩展名的文件名;下载失败与读取失败一样是可重试的失败项,不推进游标。

本提交只含 C(上传文件摄取)。不含:校验循环容错(A)、跳过项日志与可见性(B)、
原生表格 axls(D)、多维表/手工引用/前端(E)。

* test(datasource/dingtalk): 上传文件用例改用合成文档名与占位 ID

- 夹具、断言与文件名注释里的文档名统一换成合成名 Employee-Handbook.docx,
  用例语义不变:仍覆盖"名字已带扩展名时不重复追加"与 PK\x03\x04 字节断言。
- 去掉测试注释里"as verified against the live tenant"的表述,改为描述调用顺序。
- dentry 响应示例与用例中的数字 ID 换成明确占位符,不再出现可被误认的真实 ID。

* fix(datasource/dingtalk): 上传文件摄取改为数据源级开关,超限不再重试

评审意见(#3786):上传文件是本次新增的摄取路径,存量数据源升级后无需用户显式启用
就会在下一次同步(含增量)开始下载并向量化所选范围内的 docx/pptx/xlsx/pdf,可能
增加存储与 embedding 成本;同时单次下载里超出体积上限被当成可重试错误,没有
Content-Length 的超大文件会重签名再下满 3 次。

- 新增 `Settings.include_uploaded_files`(默认 false,未设置/非法值一律按 false)。
  读取走与本仓库语雀连接器同名的 settingBool(v, def),并新增 documentSettings /
  parseDocumentSettings 承载开关。
- 判定拆成"有没有读取路径"(isDocument)与"这个数据源收不收"(isIngestible):
  Validate 的探测、ListResources、全量同步与增量同步(共用 scanWorkspace)四处
  全部改用 isIngestible,关闭时行为与 main 一致。
- skipReason 对"能读但没开开关"的上传文件给出可操作的原因:说明
  include_uploaded_files 未启用,而不是伪装成类型不支持;媒体文件与真正没有
  摄取路径的类型保持原有说法。
- readLimited 改为返回哨兵错误 errDocumentTooLarge(同时包装上限字节数),
  fetchSigned 据此返回 retryable=false:体积是对象的属性,新签名只会拿到同样的
  字节。普通读错误仍然 retryable=true,仍会重签名重试。
- 前端数据源编辑弹窗(同步策略步骤)加对应开关,未触碰时不写回 settings;5 个
  语言包补齐文案。
- website-docs 钉钉一节写明支持范围、默认值、64 MiB 单文件上限与成本影响。

测试补 include_uploaded_files 的未设置/显式关闭/显式开启三种状态:校验、资源列表、
全量同步、增量同步四处各自断言,直接选择上传文件在关闭时仍是失败的资源;流式超限
用例断言签名查询与对象下载各只发生一次,另加"普通读错误仍重试"的用例。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:56:04 +08:00
..
connector feat(datasource/dingtalk): 摄取钉钉"上传文件"(docx/pptx/xlsx/pdf) (#3786) 2026-10-09 10:56:04 +08:00
CONNECTOR_IMPLEMENTATION_GUIDE.md revert: take the plugin system back out of main 2026-09-26 14:58:18 +08:00
README.md feat(datasource): implement data source management functionality 2026-03-30 20:46:07 +08:00
connector.go feat(datasource): add Seafile data source connector (#3616) 2026-10-08 16:18:09 +08:00
connector_test.go fix(datasource): align Feishu connector capabilities 2026-06-04 15:33:03 +08:00
errors.go fix(datasource): optimize RSS incremental sync and surface partial failures 2026-06-23 17:30:38 +08:00
filename.go refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
filename_test.go refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
httpclient.go fix(security): harden web_fetch and datasource connector SSRF paths 2026-07-02 12:27:48 +08:00
httpclient_test.go fix(security): harden web_fetch and datasource connector SSRF paths 2026-07-02 12:27:48 +08:00
scheduler.go fix(server): run ResourceCleaner on signal shutdown instead of racing Fatalf exit (#3804) 2026-09-28 23:37:36 +08:00
scheduler_test.go fix(datasource): validate cron schedules and persist manual-only mode (#3653) 2026-09-25 22:23:18 +08:00

README.md

WeKnora Data Source Sync Framework

Overview

The data source sync framework enables WeKnora to automatically import and synchronize content from external platforms (Feishu, Notion, Confluence, etc.) into knowledge bases. This is the foundational layer upon which all specific connectors are built.

Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                     External Data Sources                  │
│     (Feishu, Notion, Confluence, GitHub, Google Drive...) │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│                  Connector Registry & Adapters              │
│    Each platform (feishu/, notion/, confluence/, etc)     │
│           implements the Connector interface               │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│            DataSourceService (Business Logic)               │
│  ├─ CreateDataSource / UpdateDataSource                    │
│  ├─ ManualSync / ListAvailableResources                    │
│  ├─ ValidateConnection / PauseDataSource                   │
│  └─ ProcessSync (asynq task handler)                       │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│        HTTP Handler & API Routes (/api/v1/datasource)      │
│      REST endpoints for UI/programmatic access             │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│              Database (MySQL / PostgreSQL)                  │
│  ├─ data_sources (configuration & state)                   │
│  └─ sync_logs (history & metadata)                         │
└─────────────────────────────────────────────────────────────┘

Task Flow

Manual Trigger or Scheduled Job (cron)
        ↓
  Scheduler enqueues asynq Task
        ↓
  ProcessSync (task handler) executes
        ↓
  Connector.Fetch* methods pull data
        ↓
  Diff engine identifies changes (new/update/delete)
        ↓
  KnowledgeService creates/updates Knowledge entries
        ↓
  Documents parsed → chunks → vectors → indexed

File Structure

internal/datasource/
├── connector.go          # Connector interface & registry
├── errors.go            # Error definitions
└── README.md            # This file

internal/types/
├── datasource.go        # Data models (DataSource, SyncLog, etc)
└── interfaces/datasource.go  # Service interfaces

internal/application/
├── repository/datasource_repo.go  # Database access layer
└── service/datasource_service.go  # Business logic

internal/handler/
└── datasource.go        # HTTP request handlers

internal/router/
├── router.go           # Route registration
├── task.go             # Asynq task registration
└── sync_task.go        # Lite mode task registration

migrations/versioned/
└── 000028_datasource_tables.up/down.sql  # Database schema

Data Models

DataSource

Represents a configured external data source:

  • ID: Unique identifier
  • Type: Connector type (feishu, notion, confluence, etc)
  • Config: Encrypted credentials and configuration (JSON)
  • SyncSchedule: Cron expression (e.g., "0 */6 * * *")
  • SyncMode: "incremental" or "full"
  • Status: active | paused | error
  • ConflictStrategy: overwrite | skip
  • SyncDeletions: Whether to sync deletions from source
  • LastSyncAt: Timestamp of last successful sync
  • LastSyncCursor: State for incremental sync
  • LastSyncResult: Summary of last sync

SyncLog

Tracks execution of each sync operation:

  • Status: running | success | partial | failed | canceled
  • StartedAt/FinishedAt: Execution timestamps
  • Items*: Counters (total, created, updated, deleted, skipped, failed)
  • ErrorMessage: Error details if failed
  • Result: Detailed sync result (JSON)

Connector Interface

All external data sources implement this interface:

type Connector interface {
    Type() string
    Validate(ctx, config) error
    ListResources(ctx, config) ([]Resource, error)
    FetchAll(ctx, config, resourceIDs) ([]FetchedItem, error)
    FetchIncremental(ctx, config, cursor) ([]FetchedItem, *SyncCursor, error)
}

API Endpoints

Data Source Management

POST   /api/v1/datasource              # Create
GET    /api/v1/datasource              # List (by kb_id)
GET    /api/v1/datasource/:id          # Get
PUT    /api/v1/datasource/:id          # Update
DELETE /api/v1/datasource/:id          # Delete

Operations

POST   /api/v1/datasource/:id/validate      # Test connection
GET    /api/v1/datasource/:id/resources     # List available resources
POST   /api/v1/datasource/:id/sync          # Trigger manual sync
POST   /api/v1/datasource/:id/pause         # Pause
POST   /api/v1/datasource/:id/resume        # Resume

Logs

GET    /api/v1/datasource/:id/logs          # List sync logs
GET    /api/v1/datasource/logs/:log_id      # Get specific log

Metadata

GET    /api/v1/datasource/types             # Available connectors

Implementation Guide

Adding a New Connector

  1. Create the connector package in internal/datasource/connector/<type>/

    // connector/<type>/client.go - API client
    // connector/<type>/connector.go - Implements Connector interface
    // connector/<type>/types.go - Type-specific models
    
  2. Implement the Connector interface

    type YourConnector struct {
        // fields
    }
    
    func (c *YourConnector) Type() string {
        return types.ConnectorTypeYourType
    }
    
    func (c *YourConnector) Validate(ctx, config) error {
        // Test connectivity and credentials
    }
    
    func (c *YourConnector) ListResources(ctx, config) ([]types.Resource, error) {
        // List available resources (documents, spaces, etc)
    }
    
    func (c *YourConnector) FetchAll(ctx, config, resourceIDs) ([]types.FetchedItem, error) {
        // Full sync - fetch all items from specified resources
    }
    
    func (c *YourConnector) FetchIncremental(ctx, config, cursor) ([]types.FetchedItem, *types.SyncCursor, error) {
        // Incremental sync - fetch only changed items since cursor
    }
    
  3. Register the connector in the container (in container.go)

    container.Provide(func() datasource.Connector {
        return yourconnector.NewConnector()
    })
    
  4. Add metadata in connector.go

    types.ConnectorTypeYourType: {
        Type: types.ConnectorTypeYourType,
        Name: "Your Platform",
        AuthType: "oauth2",
        // ...
    }
    

Database Schema

data_sources table

  • Stores data source configurations
  • Indexes: tenant_id, knowledge_base_id, type, status
  • Encrypted config field (AES-256-GCM)

sync_logs table

  • Stores sync operation history
  • Indexes: data_source_id, tenant_id, status, started_at
  • Foreign key to data_sources with cascade delete

Key Design Decisions

  1. Adapter Pattern: Each connector is a separate implementation, easy to add new ones
  2. Async Task Queue: Syncs are non-blocking, using asynq (or sync mode in Lite)
  3. Incremental Sync: Supports cursor-based incremental syncs to minimize API calls
  4. Encryption: API keys/tokens stored encrypted with AES-256-GCM
  5. Multi-tenant: All operations are tenant-isolated
  6. Channel Tracking: Knowledge.Channel field tracks data source type for origin tracking

Future Enhancements

  1. Webhook Support: Real-time push from platforms that support webhooks
  2. Conflict Resolution: Advanced merge/conflict strategies for overlapping content
  3. Rate Limiting: Per-connector rate limiting and backoff strategies
  4. Scheduling: Full cron scheduler with time zone support
  5. Monitoring: Metrics, alerting, and sync health dashboards
  6. Filtering: User-defined filters for selective syncing (by title, date, tags, etc)
  7. Transformation: Content transformation pipelines (e.g., extract tables, summarize)
  8. Deduplication: Smart deduplication across multiple sources

Testing

The framework is designed to be testable:

  • Mock connectors can be created for testing
  • No external dependencies required for core logic
  • Database operations isolated and mockable

Error Handling

All operations return detailed error information:

  • Connection failures are captured and status updated
  • Partial failures are logged (some items succeed, some fail)
  • Error messages stored in DataSource for user troubleshooting