WeKnora/internal/datasource
linus-liu-web 402f69832e
fix(notion): 查询撞厂商上限被截断时不再静默,截断轮次不下发删除 (#3845)
* fix(notion): 查询撞厂商行数上限不再静默截断,截断轮次跳过删除判定

Notion 单次 data source 查询最多返回 10,000 行。撞到上限时厂商把 has_more 置为
false,并在响应里带 request_status.type == "incomplete"
(incomplete_reason = "query_result_limit_reached"),而 connector 只认
has_more、不解析 request_status,于是"被截断"与"读完"在代码里完全同形:

- QueryDatabaseAll 返回 err == nil、10,000 条记录,日志照常打
  "database <id> (<title>): 10000 records",同步看起来完全成功;
- 增量同步里,本轮查询未返回、又没被 /v1/search 带出来的行,会被当成源端已删除
  下发 IsDeleted。开启删除同步时,已在库的文档连同向量被真实删除,而同步日志仍
  显示成功。

改动:

- types.go:paginatedResponse 增加 request_status 解析(notionRequestStatus /
  isIncomplete)。
- client.go:paginatePages 在每一页反序列化后判定 incomplete(厂商文档要求逐页
  检查,标记可能出现在最后一页之前),命中即返回可识别的
  errQueryResultTruncated,连同该页已取回的行,调用方既能拿到部分数据,也不会
  再把缺失行当作"源端已删除"的证据;QueryDatabaseAll 遇到截断不再走
  "当作 database_id 再解析一次"的回退,避免重复同一次受限查询并把截断错误埋在
  查找失败后面。
- connector.go:截断信号经 queryDatabaseRecords 传到
  fetchDatabaseIncremental / fetchDatabase / fetchPage,本轮只要出现过截断,删除
  判定整体跳过(与 #3682"发现请求失败即中止、不删"同一取向);计数与日志改为
  带截断标记的 Warn,让"同步成功但文档数偏少"可解释。

新增回归测试 internal/datasource/connector/notion/query_truncation_test.go:截断
(含中途页截断)必须返回哨兵错误且不重试、端到端不产生 IsDeleted、子数据库路径
同样传出截断;同时锁定正常 has_more=false 无 request_status 的结果集行为不变
(正常读完仍照旧判定删除)。反向验证:去掉逐页判定后 4 条断言失败的用例失败、
3 条保护性用例通过,随后逐字节还原。

按 created_time 开新窗口的分页改写(厂商大数据源食谱)改动面较大,留作后续改动:
本次修复只保证截断可见、不误删;超过上限的记录仍不会被这批同步收进知识库。

* fix(notion): 数据源查询按 created_time 开窗口,超过 10,000 行的记录也能取全

Notion 单次 data source 查询最多返回 10,000 行:撞限时厂商把 has_more 置为 false,
并带 request_status.type == "incomplete"(incomplete_reason ==
"query_result_limit_reached")。本 PR 的前一版让截断可见、不再误删,但被截断的记录
仍然进不了知识库——一次查询的上限就是一次查询的上限,只跟着 next_cursor 走永远读
不到第 10,001 行。

按厂商"大数据源"食谱改为时间窗口分页(用 created_time,不用会随编辑移动边界的
last_edited_time):

- types.go:notionPage 增加 created_time,作为窗口边界。
- client.go:数据源查询改走 queryDataSourceAll / queryDataSourceWindow。每个窗口都
  带 sorts: [{timestamp: created_time, direction: ascending}],窗口内继续按
  next_cursor 翻页;只要该窗口任意一页带 incomplete 标记(厂商明确要求逐页检查,
  标记可能出现在最后一页之前),就用 filter.created_time.on_or_after = 本窗口最后
  一行的时间戳开下一个窗口,累计结果按行 ID 去重(相邻窗口共享边界行,边界行只会
  入库一次)。窗口推进必须严格前进:没有任何行带 created_time,或窗口最后一行的时间
  戳没有超过窗口起点(同一 created_time 的行数超过单次查询上限)时停止续窗、打 Warn
  并返回 errQueryResultTruncated,绝不原地重查。每次续窗都打一条 Warn,截断仍然留痕。
- connector.go:queryDatabaseRecords 的截断日志改为"撞上限且窗口无法推进";"截断轮次
  跳过删除判定"的行为保持不变(现在的截断只可能来自无法推进的窗口)。

回归测试:新增 query_windowing_test.go,用按请求里的窗口过滤条件返回数据的假服务端
覆盖跨 4 个窗口取全 5 行并按行 ID 去重、同一窗口内的重复行去重、incomplete 标记落在
非末页时仍然开新窗口、未撞上限的正常路径行为不变(request_status 为 complete 与完全
缺失两种形态)。query_truncation_test.go 改写为:窗口无法推进(同一 created_time 超
限、行不带 created_time、空窗口仍 incomplete)时停止并告警、端到端不产生 IsDeleted;
另加窗口取全后仍照常判定源端真实删除的用例,防止"一有窗口就不再删"的过度修复。

反向验证:把窗口续读改回"撞上限即中止"后,窗口取全相关用例全部失败(QueryDatabaseAll
报 "vendor limit reached ... 2 records fetched",端到端漏报 r4 的删除),"正常路径行为
不变"与"截断不误删"的用例继续通过,随后逐字节还原。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
2026-10-07 23:54:08 +08:00
..
connector fix(notion): 查询撞厂商上限被截断时不再静默,截断轮次不下发删除 (#3845) 2026-10-07 23:54:08 +08:00
CONNECTOR_IMPLEMENTATION_GUIDE.md revert: take the plugin system back out of main 2026-09-26 14:58:18 +08:00
README.md feat(datasource): implement data source management functionality 2026-03-30 20:46:07 +08:00
connector.go revert: take the plugin system back out of main 2026-09-26 14:58:18 +08:00
connector_test.go fix(datasource): align Feishu connector capabilities 2026-06-04 15:33:03 +08:00
errors.go fix(datasource): optimize RSS incremental sync and surface partial failures 2026-06-23 17:30:38 +08:00
filename.go refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
filename_test.go refactor(datasource): share connector filename sanitization 2026-09-17 16:18:52 +08:00
httpclient.go fix(security): harden web_fetch and datasource connector SSRF paths 2026-07-02 12:27:48 +08:00
httpclient_test.go fix(security): harden web_fetch and datasource connector SSRF paths 2026-07-02 12:27:48 +08:00
scheduler.go fix(server): run ResourceCleaner on signal shutdown instead of racing Fatalf exit (#3804) 2026-09-28 23:37:36 +08:00
scheduler_test.go fix(datasource): validate cron schedules and persist manual-only mode (#3653) 2026-09-25 22:23:18 +08:00

README.md

WeKnora Data Source Sync Framework

Overview

The data source sync framework enables WeKnora to automatically import and synchronize content from external platforms (Feishu, Notion, Confluence, etc.) into knowledge bases. This is the foundational layer upon which all specific connectors are built.

Architecture

Core Components

┌─────────────────────────────────────────────────────────────┐
│                     External Data Sources                  │
│     (Feishu, Notion, Confluence, GitHub, Google Drive...) │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│                  Connector Registry & Adapters              │
│    Each platform (feishu/, notion/, confluence/, etc)     │
│           implements the Connector interface               │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│            DataSourceService (Business Logic)               │
│  ├─ CreateDataSource / UpdateDataSource                    │
│  ├─ ManualSync / ListAvailableResources                    │
│  ├─ ValidateConnection / PauseDataSource                   │
│  └─ ProcessSync (asynq task handler)                       │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│        HTTP Handler & API Routes (/api/v1/datasource)      │
│      REST endpoints for UI/programmatic access             │
└─────────────┬───────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────────────────────┐
│              Database (MySQL / PostgreSQL)                  │
│  ├─ data_sources (configuration & state)                   │
│  └─ sync_logs (history & metadata)                         │
└─────────────────────────────────────────────────────────────┘

Task Flow

Manual Trigger or Scheduled Job (cron)
        ↓
  Scheduler enqueues asynq Task
        ↓
  ProcessSync (task handler) executes
        ↓
  Connector.Fetch* methods pull data
        ↓
  Diff engine identifies changes (new/update/delete)
        ↓
  KnowledgeService creates/updates Knowledge entries
        ↓
  Documents parsed → chunks → vectors → indexed

File Structure

internal/datasource/
├── connector.go          # Connector interface & registry
├── errors.go            # Error definitions
└── README.md            # This file

internal/types/
├── datasource.go        # Data models (DataSource, SyncLog, etc)
└── interfaces/datasource.go  # Service interfaces

internal/application/
├── repository/datasource_repo.go  # Database access layer
└── service/datasource_service.go  # Business logic

internal/handler/
└── datasource.go        # HTTP request handlers

internal/router/
├── router.go           # Route registration
├── task.go             # Asynq task registration
└── sync_task.go        # Lite mode task registration

migrations/versioned/
└── 000028_datasource_tables.up/down.sql  # Database schema

Data Models

DataSource

Represents a configured external data source:

  • ID: Unique identifier
  • Type: Connector type (feishu, notion, confluence, etc)
  • Config: Encrypted credentials and configuration (JSON)
  • SyncSchedule: Cron expression (e.g., "0 */6 * * *")
  • SyncMode: "incremental" or "full"
  • Status: active | paused | error
  • ConflictStrategy: overwrite | skip
  • SyncDeletions: Whether to sync deletions from source
  • LastSyncAt: Timestamp of last successful sync
  • LastSyncCursor: State for incremental sync
  • LastSyncResult: Summary of last sync

SyncLog

Tracks execution of each sync operation:

  • Status: running | success | partial | failed | canceled
  • StartedAt/FinishedAt: Execution timestamps
  • Items*: Counters (total, created, updated, deleted, skipped, failed)
  • ErrorMessage: Error details if failed
  • Result: Detailed sync result (JSON)

Connector Interface

All external data sources implement this interface:

type Connector interface {
    Type() string
    Validate(ctx, config) error
    ListResources(ctx, config) ([]Resource, error)
    FetchAll(ctx, config, resourceIDs) ([]FetchedItem, error)
    FetchIncremental(ctx, config, cursor) ([]FetchedItem, *SyncCursor, error)
}

API Endpoints

Data Source Management

POST   /api/v1/datasource              # Create
GET    /api/v1/datasource              # List (by kb_id)
GET    /api/v1/datasource/:id          # Get
PUT    /api/v1/datasource/:id          # Update
DELETE /api/v1/datasource/:id          # Delete

Operations

POST   /api/v1/datasource/:id/validate      # Test connection
GET    /api/v1/datasource/:id/resources     # List available resources
POST   /api/v1/datasource/:id/sync          # Trigger manual sync
POST   /api/v1/datasource/:id/pause         # Pause
POST   /api/v1/datasource/:id/resume        # Resume

Logs

GET    /api/v1/datasource/:id/logs          # List sync logs
GET    /api/v1/datasource/logs/:log_id      # Get specific log

Metadata

GET    /api/v1/datasource/types             # Available connectors

Implementation Guide

Adding a New Connector

  1. Create the connector package in internal/datasource/connector/<type>/

    // connector/<type>/client.go - API client
    // connector/<type>/connector.go - Implements Connector interface
    // connector/<type>/types.go - Type-specific models
    
  2. Implement the Connector interface

    type YourConnector struct {
        // fields
    }
    
    func (c *YourConnector) Type() string {
        return types.ConnectorTypeYourType
    }
    
    func (c *YourConnector) Validate(ctx, config) error {
        // Test connectivity and credentials
    }
    
    func (c *YourConnector) ListResources(ctx, config) ([]types.Resource, error) {
        // List available resources (documents, spaces, etc)
    }
    
    func (c *YourConnector) FetchAll(ctx, config, resourceIDs) ([]types.FetchedItem, error) {
        // Full sync - fetch all items from specified resources
    }
    
    func (c *YourConnector) FetchIncremental(ctx, config, cursor) ([]types.FetchedItem, *types.SyncCursor, error) {
        // Incremental sync - fetch only changed items since cursor
    }
    
  3. Register the connector in the container (in container.go)

    container.Provide(func() datasource.Connector {
        return yourconnector.NewConnector()
    })
    
  4. Add metadata in connector.go

    types.ConnectorTypeYourType: {
        Type: types.ConnectorTypeYourType,
        Name: "Your Platform",
        AuthType: "oauth2",
        // ...
    }
    

Database Schema

data_sources table

  • Stores data source configurations
  • Indexes: tenant_id, knowledge_base_id, type, status
  • Encrypted config field (AES-256-GCM)

sync_logs table

  • Stores sync operation history
  • Indexes: data_source_id, tenant_id, status, started_at
  • Foreign key to data_sources with cascade delete

Key Design Decisions

  1. Adapter Pattern: Each connector is a separate implementation, easy to add new ones
  2. Async Task Queue: Syncs are non-blocking, using asynq (or sync mode in Lite)
  3. Incremental Sync: Supports cursor-based incremental syncs to minimize API calls
  4. Encryption: API keys/tokens stored encrypted with AES-256-GCM
  5. Multi-tenant: All operations are tenant-isolated
  6. Channel Tracking: Knowledge.Channel field tracks data source type for origin tracking

Future Enhancements

  1. Webhook Support: Real-time push from platforms that support webhooks
  2. Conflict Resolution: Advanced merge/conflict strategies for overlapping content
  3. Rate Limiting: Per-connector rate limiting and backoff strategies
  4. Scheduling: Full cron scheduler with time zone support
  5. Monitoring: Metrics, alerting, and sync health dashboards
  6. Filtering: User-defined filters for selective syncing (by title, date, tags, etc)
  7. Transformation: Content transformation pipelines (e.g., extract tables, summarize)
  8. Deduplication: Smart deduplication across multiple sources

Testing

The framework is designed to be testable:

  • Mock connectors can be created for testing
  • No external dependencies required for core logic
  • Database operations isolated and mockable

Error Handling

All operations return detailed error information:

  • Connection failures are captured and status updated
  • Partial failures are logged (some items succeed, some fail)
  • Error messages stored in DataSource for user troubleshooting