WeKnora/docs
Greenplumwine 755243d61c
feat(im): support reverse proxy for feishu websocket long connection (#2550)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

* feat(im): support reverse proxy for feishu websocket long connection

Add api_base_url channel config for Feishu/Lark IM adapters so internal
deployments can route Feishu API calls and the WebSocket long connection
through a reverse proxy (nginx). The config overrides both the SDK
bootstrap domain (WithDomain) and the adapter HTTP API origin (api()),
unified into a single field.

- adapter: NewAdapter gains apiBaseURL + SSRF validation (http(s) scheme,
  ValidateURLForSSRF), mirroring wecom's validateEndpointURL; api() uses
  the override instead of region.OpenBaseURL
- longconn: NewLongConnClient forwards apiBaseURL to larkws.WithDomain
- factory: reads api_base_url once, passes to both adapter and longconn
- frontend: Feishu/Lark channel config gains an API Base URL input

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ci): restore build-test-images workflow dropped by 00d7090f

00d7090f ("docs: 删除误提交的文件") deleted build-test-images.yml
alongside two obsolete design docs, causing the workflow_dispatch
trigger to disappear from the branch. Restore it so the manual
build-test-images action is available again.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(im): support reverse proxy for feishu websocket long connection

飞书 SDK 的 WebSocket 长连接分两阶段:bootstrap HTTP 拿 ws 地址 + ws dial
直连飞书下发的地址。WithDomain 只控制 bootstrap 请求目标,ws dial 的地址
由飞书服务端下发(host 为 msg-frontier.feishu.cn),SDK 不支持改写。

内网部署无法直连飞书 wss,因此新增 api_base_url 字段:让 bootstrap 和
HTTP API 走反代,并通过 nginx sub_filter 改写飞书下发的 wss 地址为明文
ws,使 ws dial 也经反代回连 nginx。

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): add api_base_url config for feishu IM channel

新增 api_base_url 配置入口与 i18n 文案,用于内网反代部署。

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(im): document feishu api_base_url reverse proxy usage

补充 nginx 反代配置说明:三个 location(bootstrap/open.feishu.cn +
sub_filter、ws dial/msg-frontier.feishu.cn、OpenAPI/open.feishu.cn),
以及 sub_filter 改写 wss 地址、rewrite 前缀剥离、SSRF 白名单等要点。

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(ci): add manual docker build workflow

Add a workflow_dispatch-only workflow to build ui/app/docreader/sandbox
images on demand. No auto-trigger on push/tag. Each package is gated by
a boolean input (build_all + per-package), and the branch is picked via
the native workflow_dispatch ref selector. Builds are load-only (push
false) to verify Dockerfiles without needing any registry secret.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ci): sync manual docker build workflow from main (add tar export + unified package job)

* Delete .github/workflows/docker-build.yml

* Delete .github/workflows/build-test-images.yml

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 16:18:27 +08:00
..
api feat(auth): add /auth/oidc/start endpoint for direct 302 OIDC login 2026-08-11 15:41:48 +08:00
cloud-image docs(cloud-image): add cloud-agnostic image packaging scripts 2026-05-11 12:25:19 +08:00
dev fix: validate vector store connection addresses against SSRF policy 2026-06-04 15:26:56 +08:00
images docs(readme): replace showcase screenshots with consistent website-docs captures 2026-08-07 11:37:05 +08:00
wiki feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断 2026-08-11 15:42:02 +08:00
zh refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
BUILTIN_MCP_SERVICES.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
BUILTIN_MODELS.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
CHUNKING.md fix(chunking): post-review polish — error surfacing + doc accuracy 2026-05-06 17:17:07 +08:00
IM集成开发文档.md feat(im): support reverse proxy for feishu websocket long connection (#2550) 2026-08-12 16:18:27 +08:00
KnowledgeGraph.md docs: fix 登陆 typo to 登录 2026-08-07 17:04:49 +08:00
LITE.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
Langfuse集成.md feat(llm): add prompt-cache observability and wiki cache-friendly prompts 2026-07-23 19:17:30 +08:00
MCP功能使用说明.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
OIDC认证调用流程.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
QA.md docs(release): prepare v0.7.2 release notes and version bump 2026-08-07 11:37:05 +08:00
RBAC说明.md docs: clarify workspace owner cardinality 2026-07-23 16:13:35 +08:00
ROADMAP.md feat(roadmap): update progress on document parsing and IM integration 2026-03-31 20:22:34 +08:00
agent-skills.md feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断 2026-08-11 15:42:02 +08:00
docs.go fix(invitation): harden auto-accept follow-ups from code review 2026-08-10 15:47:29 +08:00
embed-secure-mode.md feat(embed): add widget, visitor chat, and management UI (#1657) 2026-06-12 23:10:39 +08:00
embed-subdomain.md feat(embed): add widget, visitor chat, and management UI (#1657) 2026-06-12 23:10:39 +08:00
migration-troubleshooting.md feat(system-info): surface DB migration errors with troubleshooting links 2026-05-14 16:34:50 +08:00
sandbox-cluster.md feat(sandbox): unify remote sandbox on E2B protocol with self-hosted support (#2655) 2026-08-11 20:14:19 +08:00
sandbox-protocol.md feat(sandbox): unify remote sandbox on E2B protocol with self-hosted support (#2655) 2026-08-11 20:14:19 +08:00
swagger.json fix(invitation): harden auto-accept follow-ups from code review 2026-08-10 15:47:29 +08:00
swagger.yaml fix(invitation): harden auto-accept follow-ups from code review 2026-08-10 15:47:29 +08:00
worker-pool-governance.md feat(worker): replace aggregate asynq concurrency with per-pool governance 2026-07-13 15:51:06 +08:00
使用其他向量数据库.md feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
共享空间说明.md refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
开发指南.md fix: update dev script checks and Air docs 2026-07-01 11:52:42 +08:00
开启知识图谱功能.md feat: Update VLM configuration in knowledge base and API documentation 2025-11-20 20:08:24 +08:00
快速开发模式说明.md docs: update Air installation command from cosmtrek/air to air-verse/air 2026-02-04 19:27:44 +08:00
数据源导入开发文档.md feat(feishu-driver): 新增飞书云盘数据源 (#2466) 2026-08-05 19:39:49 +08:00
日志配置.md refactor(logger): support LOG_FORMAT template and harden level coloring 2026-05-22 20:31:54 +08:00
添加新的网络搜索引擎.md feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00