* ci(build): add manual docker build workflow Co-Authored-By: Claude <noreply@anthropic.com> * feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects - folder mirror: wiki/drive connectors emit path-qualified FileName so knowledge lands in KB folders mirroring the Feishu tree; shortcut subtrees deduped; sanitized directory names; legacy datasources converge via one-shot resync marker (incremental sync upgrades to full once, then clears) - parse mode: replace FEISHU_DOCX_PARSE_MODE env with per-datasource settings.parse_mode (blocks default, export escape hatch); frontend selector + 5 locales; zero-migration read-time fallback - block-level GFM: extend docx block coverage (callout/grid/iframe/ board/mindnote/agenda/synced blocks), rich text elements (links, inline code, KaTeX equations, mentions), real ordered-list numbering, code language mapping, merged-cell fill, truncation note - embedded objects: images (gated off multimodal toggle) and boards (download_as_image, 403-degraded) stored as knowledge with weknora-img://<n> markers resolved in the document-processing pipeline via knowledge FilePath; document attachments whitelisted (pdf/md/xls(x)/csv/doc(x)/ppt(x)) as standalone knowledge in the parent folder with bidirectional parent/child metadata and filename list in the body; videos skipped with placeholder + source link - golden baseline explicitly re-recorded; docs (.env.example, docker-compose, docs/*, website-docs) synced * fix(datasource): address Feishu deep-adaptation smoke-test findings - folder mapping: drop the space/root-folder name prefix — the selected sync root maps to the knowledge-base root, subdirectories stay relative - docx blocks: render view(33) blocks silently as containers (children only), removing the stray '[飞书块: 视图]' placeholder before file lists - sheet inline tables: fetch merge ranges via sheets/v3 sheets/query and fill the top-left value across the merged region (closed-interval interpretation, flagged for real-environment verification) - knowledge list: hide embedded image/board rows (metadata embedded_image=true) from the document list and folder-tree counts; rows remain for VLM indexing and marker resolution; search untouched - frontend: hydrate protected images (resource:// etc.) in the document preview so embedded Feishu images/boards actually render * fix(datasource): resolve embedded image markers at ingest time The weknora-img://<n> markers were only resolved during document processing, so chunks carried real resource:// URLs while the stored markdown file and source_content kept raw markers — file preview showed blank images. Resolve them in ingestItem before CreateKnowledgeFromFile (children are emitted first, so image rows with FilePath already exist); markers with missing children are kept for the ProcessDocument fallback to degrade. Stored file, source_content and chunks now agree. Existing datasources converge on the next full re-sync (update=delete+recreate). * feat(datasource): resolve Feishu mention users to real names at ingest docx mention_user elements only carry an open_id, so they rendered as a generic '@成员'. Resolve the display name via GET /contact/v3/users/:id (user_id_type=open_id) while generating the markdown during FetchDocxWithBlocks — before storage, per user decision; the frontend performs no lookups. A per-sync closure caches positive and negative lookups; permission errors and empty names silently degrade to '@成员' (debug log only, ingest never blocked). Requires one of the contact read scopes (e.g. contact:user.base:readonly) on the Feishu app. * refactor(datasource): polish round — defect fixes and structural cleanup - attachment placeholder patching now targets the unique '- weknora-att://<token>' marker line rendered per attachment instead of matching by display name, so an identical earlier bullet is never replaced by mistake (regression-tested) - markdown link targets with spaces or parens are percent-escaped (iframe / mention_doc / inline links), keeping them parseable - drive folder-tree walking consolidated into core.WalkDriveTree, deleting the 174-line copy in drive/paths.go - parse mode threading made explicit: DocxFetchInput.ParseMode replaces the private client field (connector entry resolves via ParseFeishuConfig and passes it through ops) - maxFeishuDownloadBytes restored to a const; the oversize-degrade test now declares an over-cap Content-Length and downloadRawBytes rejects early on the header * refactor(datasource): align Feishu image handling with the platform image pipeline Replace the image-as-separate-knowledge-row channel with the platform's orthodox flow: the connector downloads image/board bytes at fetch time and embeds them in the parent markdown as base64 data URIs; the document processing image resolver (image_resolver.ResolveAndStore, the same stage used for parsed PDFs/Word) stores them on the tenant file service, swaps in persistent provider:// URLs and produces OCR/caption child chunks when a VLM is configured. Removed with the marker pipeline: - weknora-img:// ingest resolution in ingestItem - the ProcessDocument Step-1.9 marker hook - the document-list/folder-tree filter for embedded_image rows - the feishu-specific runtime parse_mode backfill; legacy data sources are now stamped by migration 000096 (sqlite 000017) which writes the one-shot resync_required marker — the generic full-upgrade + clear consumption stays Attachment (PDF/Word/md) sub-items are unchanged. * style: gofmt test files * fix(datasource): resolve Drive sub-selection kind from the root listing A single-file Drive selection (rootFolderToken:fileToken) was probed with folder APIs before falling back: folder meta returned 91202 and the folder list 1061002 on every sync, logging API errors for a working configuration. listDriveFilesForResource now walks the root subtree first and reads the selected node's kind off the listing: a file filters straight out of the already-fetched results (zero error calls), a sub-folder still gets its dedicated walk, and only a token missing from the listing (partial listing / permission edge) keeps the legacy probe + fallback. Also pins sub-folder selection semantics with a dedicated test. * fix(datasource): address review findings across the Feishu deep-adaptation branch Four parallel reviewer slices audited the branch; this applies the accepted findings. Content fidelity (blocks rendering): - todo blocks parse style.done — completed items render "- [x]" instead of always unchecked - code fences size past the longest backtick run in the content, so a line of ``` inside code can no longer close the fence early - ordered-list "auto" numbering only resets on a sibling of the last item; child blocks rendered flat between items no longer renumber the run Image pipeline: - inline base64 images are capped at the document-processing resolver's 10 MB budget; larger images degrade to a placeholder instead of leaving a permanent raw-base64 blob in stored markdown - image markers carry a per-document nonce so user-typed look-alike text can never be patched in place of a real marker - attachment marker patching anchors on the token's line end, closing the token-prefix collision - SupportedImageExt recovers image/svg+xml (board exports), which the platform resolver already stores natively Sync correctness: - clearResyncMarker re-reads the row before writing: a long upgraded sync no longer risks reverting concurrent user edits with a stale full-row update - wiki EditTime keys on max(obj_edit_time, node_edit_time): pure moves/renames re-file the document on the next incremental sync - file-typed wiki/drive base names are sanitised; a "/" in a title can no longer fake an extra folder level Docs/tests/comments: - datasources docs rewritten to the v3 image contract (no image rows, no image_map, no parse_mode backfill claim) - DataSourceEditorDialog drops settings.parse_mode when backtracking to a non-Feishu connector type - stale comments corrected (types.go parse-mode transport, repository folder-counts, wiki List partial-error branch, test headers); regression tests added for todo done, fence sizing, ordered numbering, svg sniff * fix(datasource): repair markdown conversion defects found in live smoke test Live re-sync of the all-block-type Feishu test document surfaced three converter defects: - Code blocks: the closing fence reused the opening fence string, so a tagged block ended with ```java — CommonMark requires a bare closing fence, and every renderer left the block unterminated. The closing fence is now backticks only. - Feishu heading7-9 emitted 7-9 hash marks; GFM stops at H6, so those lines rendered as literal '#######' text. They now degrade to bold. - The docx API percent-encodes link targets (http%3A%2F%2F...); escapeURL now reverses that encoding before markdown-escaping, so stored links point at real URLs instead of percent-encoded strings. Verified against the live document via a throwaway API probe: the 'i|g|k' table cells questioned during review are the document's own content (the sheets copy says i|j|k; the two docx tables genuinely contain 'g'), not a converter bug. * fix(frontend): render GFM task lists in document preview Task list items (- [x] / - [ ]) rendered as plain bullets: marked emits <input type="checkbox"> for them, but the preview DOMPurify config's ALLOWED_TAGS allowlist has no 'input', so the element was stripped whole. The preview renderer now overrides checkbox to emit a styled task-checkbox span (class/attr both allowlisted) and the preview styles draw the box with a check mark, hiding the list bullet via :has(). Same gap exists in the chat markdown surface; left untouched there. * fix(datasource): crop blank canvas borders from whiteboard exports The board download_as_image API renders the WHOLE board canvas, not the content bounding box — editors routinely leave the canvas far larger than the drawing, so board/mindnote/diagram exports surfaced with large blank areas below the content. The API has no crop parameter, so the connector now trims near-uniform blank borders after download (bottom-right pixel seeds the background color, 14/255 per-channel tolerance absorbs JPEG noise, 8px margin kept). PNG and JPEG are re-encoded; anything undecodable, fully blank, or already tight passes through unchanged, and oversized images (>64MP) skip the scan to bound memory. * fix(datasource): require corner majority before trimming board exports The blank-border trim seeded the background color from the bottom-right pixel alone. A large solid shape reaching that corner would seed the background with its own fill color, and fully-filled rows could then be trimmed as blank — cutting real content. The background color is now the majority of the four corner pixels (>=3 must agree); on a split vote content reaches the corners and the image passes through untrimmed. Content visible against the background is untouched either way: a row or column survives if even one pixel differs from the background by more than the 14/255 JPEG-noise tolerance. * fix(datasource): widen board trim margin and lower JPEG quality to 80 The 8px margin read as a tight crop — a 32px white frame now visually bounds the content region. JPEG re-encode quality drops 90 -> 80 to keep the re-encoded size closer to the original export. * feat(datasource): timeline widget Markdown degradation + table column alignment Two content-fidelity gaps from the live smoke test: - 文档小组件 (add_ons, block_type 40) carry their full data inline in record; there is no server-side render to download. The timeline component (component_type_id blk_6358a421bca0001c...) now degrades to a Markdown list (- **time · title**:text), honoring contentShow and omitting empty segments; unknown component types and malformed records keep the previous placeholder note. - docx text blocks carry style.align (1 left / 2 center / 3 right, absent unless the author set it). Native-table rendering now collects per-column alignment from the first explicit cell and emits GFM colon separators (:---: / ---:). Embedded sheets/bitable keep the plain separator — their value APIs expose no alignment. * feat(datasource): show document titles for 云文档 mention links The docx renderer emitted mention_doc references as [url](url) because the payload carries no title. Feishu doc URLs now emit a [weknora-docname://token](url) placeholder and FetchDocxWithBlocks resolves the titles in one drive metadata batch call (POST /drive/v1/metas/batch_query, docx/doc/sheet/bitable/wiki/...), patching the link text to [文档名](url). Refs whose metadata cannot be fetched — no permission (970003), deleted/mismatched (970005), API absent — degrade to the previous URL-as-text form with a warn log. This is display-only: referenced docs are not pulled as sub-documents (deferred design lives in the task PRD). * docs(datasource): document mention rendering enhancement and drive rename limitation - Add optional mention scopes (contact:user.base:readonly, drive:drive.metadata:readonly, board:whiteboard:node:read) with degrade behavior to the Feishu permission tables - Describe mention_user real-name lookup, mention_doc title backfill and timeline widget placeholders in the blocks rendering flow - Document the Drive known limitation: folder renames / file moves do not bump modified_time, so incremental sync keeps the old KB folder structure until a full sync; Wiki sources are unaffected - Clarify the directory-mapping wording in website-docs to distinguish Wiki (rename-aware) from Drive (rename-blind) sources * docs(datasource): add blocks vs export tradeoff summary to website-docs * docs(datasource): document docx block-type coverage for blocks mode - Add a full block-type support table (block_type 1-52 grouped) to the datasource dev doc: text/heading/list/task/quote structures, code fences with language aliases, containers (callout/quote/grid/view), native tables with merged-cell splitting (top-left value fill) and column alignment, embedded sheet/bitable converted to in-document Markdown tables, image/board inline pipeline, attachment whitelist, iframe links, timeline add-on degradation and the placeholder set - Summarize the same coverage in the drive user guide, the wiki quick-start doc and website-docs; fix the stale claim that board blocks degrade to placeholders (they ride the image pipeline now) * style: resolve diff-scoped lint findings across feishu packages - Wrap over-length signatures and calls (lll, max 120) - Add doc comments to exported Block* payload types (revive) - Rename unused closure parameters to _ (revive) - Check w.Write / resp.Body.Close error returns (errcheck) - Apply De Morgan rewrite in docname token validation (staticcheck) - gofumpt-format touched files golangci-lint run --new-from-rev=main ./... now reports 0 issues. * ci: drop fork-only docker-build workflow The manual docker-build workflow lives on the fork's main and is not part of the upstream contribution; it entered the branch through the fork-main merge. * fix(migrations): renumber feishu resync migrations to avoid version collision with upstream Upstream added 000017_dingtalk_stream_only (sqlite) and 000096_dingtalk_stream_only (versioned) before our merge, colliding with our 000017/000096 feishu resync migrations. golang-migrate rejects duplicate versions. Renumber to the next free slots (sqlite 000027, versioned 000108) and update comments + test constant. * fix(migrations): renumber feishu resync migrations ahead of upstream 000027/000108 Upstream main now also claims sqlite 000027 and versioned 000108 (session_sandbox_config_tenant), colliding with our just-renumbered resync migrations. Move to 000028 / 000109. * fix(feishu): address review — deprecated env fallback and upgrade notes - Keep FEISHU_DOCX_PARSE_MODE as a one-release deprecated fallback when a data source has no per-source parse_mode (deprecation warning logged); annotate it in .env.example / docker-compose.yml instead of removing. - Point stale migration-number comments at the final numbers (versioned 000110 / sqlite 000030) in datasource_service.go and tests. - website-docs: add Feishu upgrade notes (env deprecation, default mode flip to blocks, one-shot full resync with embedding/VLM recompute); refresh 24-feishu-drive.md parse-mode section to the per-source setting and new blocks semantics. * docs(feishu): document necessary and optional connector permissions Restore the permission matrix lost with the docs/ removal: necessary scopes (wiki/drive/export/docx) with missing-permission symptoms and optional enhancement scopes (embedded sheets/bitable, board export, mention user/doc rendering) with their degrade behavior, in 10-datasource.md's Feishu section; make 24-feishu-drive.md list the three necessary drive scopes and link to the full table. * Revert "docs(feishu): document necessary and optional connector permissions" This reverts commit 3be19cb1b656b3596f2cb6afb2cda8c2a1d12be1. * docs(feishu): permission matrix by connector, parse mode and necessity Necessary-scope tables split by connector (feishu/lark wiki vs feishu_drive/lark_drive) with blocks/export applicability columns and missing-permission symptoms; optional enhancement scopes with their degrade behavior; scope names cross-checked against the Feishu open platform API docs (files/folder list, medias download, export tasks, metas batch_query). Full matrix lives in 10-datasource.md; the drive guide mirrors its own table and links back. * test(feishu): adapt mention_user_test to upstream retryTestServer signature * feat(feishu): mirror the sync root (wiki space / drive folder) as a top-level KB folder So a knowledge base linking several feishu sources can tell them apart at a glance: syncing folder 测试1 yields 测试1/... and syncing wiki space 飞书知识库同步测试 yields 飞书知识库同步测试/.... Root-name resolution failure falls back to the previous root-less behaviour without failing the sync; existing sources converge through the one-shot resync marker. * refactor(feishu): cache wiki space folder prefix per run, refresh stale comments Review follow-ups for d012e593f: cache the sanitized space prefix per spaceID for the whole sync run (K selected spaces now cost one ListWikiSpaces paging pass instead of K), fix the misleading warn wording, and realign the WalkDriveTree / driveFolderName doc comments with root-mirror semantics. * docs(feishu): drop stale and duplicated comments from the deep-adaptation - board.go: trimBoardImage no longer claims a bottom-right seed (the code votes the background over the four corners) - shared.go: rewrite the FetchDocxWithBlocks note (images/boards inline via base64 now; removes a stale-marker splice artefact) - markdown.go: drop an internal task-number leftover - datasource_service.go: collapse the ProcessSync/processSyncStreaming notes duplicated verbatim from resyncRequired's doc comment, keep the call-site why (marker cleared only on success) - datasource_resync_test.go: drop the obsolete 'backfill' wording * feat(i18n): 添加解析模式及其提示信息到繁體中文語言包 * fix(frontend): use radius token for task checkbox; fix simplified chars in zh-TW parse mode CI guard fixes: the PR-added task-checkbox style pushed the radius-literal count to 161 (> baseline 160); the new zh-TW keys missed the locale parity audit (user's dc6fdcff9) — also normalise the simplified 文档 leftovers. * fix(feishu): preserve sync scope, nested content, and legacy settings Cover shortcut selection, ancestor path changes, nested containers and table boards with regression tests. Avoid redundant Drive traversal, repair resync migrations for legacy settings, and preserve inherited parse modes. Bound board image dimensions before decoding. * docs(feishu): clarify shortcut scope and incremental path tracking * Fix Feishu sync reconciliation and embedded media regressions * Bound Feishu table rendering by actual cell data --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: wizardchen <wizardchen@tencent.com> |
||
|---|---|---|
| .. | ||
| connector | ||
| CONNECTOR_IMPLEMENTATION_GUIDE.md | ||
| README.md | ||
| connector.go | ||
| connector_test.go | ||
| errors.go | ||
| filename.go | ||
| filename_test.go | ||
| httpclient.go | ||
| httpclient_test.go | ||
| scheduler.go | ||
| scheduler_test.go | ||
README.md
WeKnora Data Source Sync Framework
Overview
The data source sync framework enables WeKnora to automatically import and synchronize content from external platforms (Feishu, Notion, Confluence, etc.) into knowledge bases. This is the foundational layer upon which all specific connectors are built.
Architecture
Core Components
┌─────────────────────────────────────────────────────────────┐
│ External Data Sources │
│ (Feishu, Notion, Confluence, GitHub, Google Drive...) │
└─────────────┬───────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Connector Registry & Adapters │
│ Each platform (feishu/, notion/, confluence/, etc) │
│ implements the Connector interface │
└─────────────┬───────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ DataSourceService (Business Logic) │
│ ├─ CreateDataSource / UpdateDataSource │
│ ├─ ManualSync / ListAvailableResources │
│ ├─ ValidateConnection / PauseDataSource │
│ └─ ProcessSync (asynq task handler) │
└─────────────┬───────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ HTTP Handler & API Routes (/api/v1/datasource) │
│ REST endpoints for UI/programmatic access │
└─────────────┬───────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Database (MySQL / PostgreSQL) │
│ ├─ data_sources (configuration & state) │
│ └─ sync_logs (history & metadata) │
└─────────────────────────────────────────────────────────────┘
Task Flow
Manual Trigger or Scheduled Job (cron)
↓
Scheduler enqueues asynq Task
↓
ProcessSync (task handler) executes
↓
Connector.Fetch* methods pull data
↓
Diff engine identifies changes (new/update/delete)
↓
KnowledgeService creates/updates Knowledge entries
↓
Documents parsed → chunks → vectors → indexed
File Structure
internal/datasource/
├── connector.go # Connector interface & registry
├── errors.go # Error definitions
└── README.md # This file
internal/types/
├── datasource.go # Data models (DataSource, SyncLog, etc)
└── interfaces/datasource.go # Service interfaces
internal/application/
├── repository/datasource_repo.go # Database access layer
└── service/datasource_service.go # Business logic
internal/handler/
└── datasource.go # HTTP request handlers
internal/router/
├── router.go # Route registration
├── task.go # Asynq task registration
└── sync_task.go # Lite mode task registration
migrations/versioned/
└── 000028_datasource_tables.up/down.sql # Database schema
Data Models
DataSource
Represents a configured external data source:
- ID: Unique identifier
- Type: Connector type (feishu, notion, confluence, etc)
- Config: Encrypted credentials and configuration (JSON)
- SyncSchedule: Cron expression (e.g., "0 */6 * * *")
- SyncMode: "incremental" or "full"
- Status: active | paused | error
- ConflictStrategy: overwrite | skip
- SyncDeletions: Whether to sync deletions from source
- LastSyncAt: Timestamp of last successful sync
- LastSyncCursor: State for incremental sync
- LastSyncResult: Summary of last sync
SyncLog
Tracks execution of each sync operation:
- Status: running | success | partial | failed | canceled
- StartedAt/FinishedAt: Execution timestamps
- Items*: Counters (total, created, updated, deleted, skipped, failed)
- ErrorMessage: Error details if failed
- Result: Detailed sync result (JSON)
Connector Interface
All external data sources implement this interface:
type Connector interface {
Type() string
Validate(ctx, config) error
ListResources(ctx, config) ([]Resource, error)
FetchAll(ctx, config, resourceIDs) ([]FetchedItem, error)
FetchIncremental(ctx, config, cursor) ([]FetchedItem, *SyncCursor, error)
}
API Endpoints
Data Source Management
POST /api/v1/datasource # Create
GET /api/v1/datasource # List (by kb_id)
GET /api/v1/datasource/:id # Get
PUT /api/v1/datasource/:id # Update
DELETE /api/v1/datasource/:id # Delete
Operations
POST /api/v1/datasource/:id/validate # Test connection
GET /api/v1/datasource/:id/resources # List available resources
POST /api/v1/datasource/:id/sync # Trigger manual sync
POST /api/v1/datasource/:id/pause # Pause
POST /api/v1/datasource/:id/resume # Resume
Logs
GET /api/v1/datasource/:id/logs # List sync logs
GET /api/v1/datasource/logs/:log_id # Get specific log
Metadata
GET /api/v1/datasource/types # Available connectors
Implementation Guide
Adding a New Connector
-
Create the connector package in
internal/datasource/connector/<type>/// connector/<type>/client.go - API client // connector/<type>/connector.go - Implements Connector interface // connector/<type>/types.go - Type-specific models -
Implement the Connector interface
type YourConnector struct { // fields } func (c *YourConnector) Type() string { return types.ConnectorTypeYourType } func (c *YourConnector) Validate(ctx, config) error { // Test connectivity and credentials } func (c *YourConnector) ListResources(ctx, config) ([]types.Resource, error) { // List available resources (documents, spaces, etc) } func (c *YourConnector) FetchAll(ctx, config, resourceIDs) ([]types.FetchedItem, error) { // Full sync - fetch all items from specified resources } func (c *YourConnector) FetchIncremental(ctx, config, cursor) ([]types.FetchedItem, *types.SyncCursor, error) { // Incremental sync - fetch only changed items since cursor } -
Register the connector in the container (in container.go)
container.Provide(func() datasource.Connector { return yourconnector.NewConnector() }) -
Add metadata in
connector.gotypes.ConnectorTypeYourType: { Type: types.ConnectorTypeYourType, Name: "Your Platform", AuthType: "oauth2", // ... }
Database Schema
data_sources table
- Stores data source configurations
- Indexes: tenant_id, knowledge_base_id, type, status
- Encrypted config field (AES-256-GCM)
sync_logs table
- Stores sync operation history
- Indexes: data_source_id, tenant_id, status, started_at
- Foreign key to data_sources with cascade delete
Key Design Decisions
- Adapter Pattern: Each connector is a separate implementation, easy to add new ones
- Async Task Queue: Syncs are non-blocking, using asynq (or sync mode in Lite)
- Incremental Sync: Supports cursor-based incremental syncs to minimize API calls
- Encryption: API keys/tokens stored encrypted with AES-256-GCM
- Multi-tenant: All operations are tenant-isolated
- Channel Tracking: Knowledge.Channel field tracks data source type for origin tracking
Feishu request pacing and rate-limit retries
Feishu export creation, export status polling, export downloads, and Wiki node listing each use a separate request budget. Clients using the same API host and App ID share these budgets within one backend process, including concurrent data sources and task retries. Document tokens, export tickets, and query parameters do not create additional budgets.
| Environment variable | Default | Behavior |
|---|---|---|
FEISHU_API_REQUESTS_PER_MINUTE |
80 |
Positive integer; requests to each paced API are evenly spaced (750 ms by default), without accumulating burst capacity. |
FEISHU_RATE_LIMIT_RETRY_WAIT |
10s |
Positive Go duration used when the response supplies no valid reset/retry hint. |
FEISHU_RATE_LIMIT_MAX_WAIT |
5m |
Positive Go duration capping each cooldown, including response-header hints and the fallback. |
FEISHU_EXPORT_MAX_POLLS |
30 |
Positive integer bounding status polls of the same export task; API retries do not consume additional polls. |
FEISHU_EXPORT_POLL_INTERVAL |
2s |
Positive Go duration between status polls while the export is still processing. |
Invalid values use the defaults. Other Feishu APIs retain their existing request frequency, with reactive cooldowns scoped to the HTTP method and normalized API path template. Document tokens, table IDs, sheet ranges, and query parameters do not split a known API's cooldown. Media downloads, file downloads, docx blocks, raw content, bitable fields/record searches, sheet values, folder metadata, Wiki space listing, and Wiki node lookup use independent cooldowns. Unknown endpoints use their service/version/resource prefix. Check the application's actual API quotas before increasing the configured budget.
JSON requests retry HTTP 429 and HTTP 400/200 responses containing Feishu error
code 99991400. Downloads retry HTTP 429 and HTTP 400 with that error code.
The wait is the longer valid value from x-ogw-ratelimit-reset and Retry-After;
Retry-After also accepts an HTTP date. All hints are clamped to the configured
maximum so an unreasonable gateway header cannot indefinitely block later syncs.
Without a valid hint, a rate-limit response imposes a 10-second cooldown by
default. The three retries add a nominal 30 seconds of cooldown; shared pacing
or further rate-limit responses from other clients can extend the total wait.
This replaces the original 2/4/8-second backoff with a shared cooldown scoped
to the affected API.
Only callers of the same API share that cooldown. Requests retain the existing
limit of three retries. Even when retries are exhausted, subsequent documents and other
clients sharing that API wait for the cooldown. Waiting honors task cancellation.
Ordinary permission/validation errors still fail immediately, and download error
bodies retain their existing size bounds.
Export polling uses a bounded number of polls rather than a 60-second wall-clock
deadline. Shared pacing and cooldown waits therefore do not cause false export
timeouts or create replacement tasks within the operation. With the defaults,
an export that remains processing is checked up to 30 times, with 2 seconds
between polls. The caller's context bounds total elapsed time, including waits
and retries. Exhausting the poll budget retains the feishu_timeout failure
classification. Tokens are checked again after gate waits; requests needing a
token refresh rejoin the gate afterwards, so both JSON requests and downloads
remain paced and honor any cooldown imposed during the refresh.
These budgets are process-local. Multiple backend instances or other applications using the same Feishu credentials may still hit the upstream quota; configure their combined traffic accordingly.
Future Enhancements
- Webhook Support: Real-time push from platforms that support webhooks
- Conflict Resolution: Advanced merge/conflict strategies for overlapping content
- Rate Limiting: Extend request pacing to other connectors and coordinate quotas across backend instances
- Scheduling: Full cron scheduler with time zone support
- Monitoring: Metrics, alerting, and sync health dashboards
- Filtering: User-defined filters for selective syncing (by title, date, tags, etc)
- Transformation: Content transformation pipelines (e.g., extract tables, summarize)
- Deduplication: Smart deduplication across multiple sources
Testing
The framework is designed to be testable:
- Mock connectors can be created for testing
- No external dependencies required for core logic
- Database operations isolated and mockable
Error Handling
All operations return detailed error information:
- Connection failures are captured and status updated
- Partial failures are logged (some items succeed, some fail)
- Error messages stored in DataSource for user troubleshooting