* fix(hybrid-search): normalize MatchCount once instead of at the truncation site
An omitted match_count deserializes to Go's zero value, which reached the
truncation step in HybridSearch and sliced the deduplicated chunk list to
[:0] — a successful retrieval returned an empty result set. A negative
value panicked on that same slice bound and surfaced as a 500.
Resolve the value once at the top of HybridSearch via normalizedMatchCount
so all three consumers agree, rather than patching the truncation site
alone: the over-retrieval floor, the FAQ iterative-retrieval trigger
(which never fired on this path because len(chunks) < 0 is false), and the
truncation itself.
Bound the FAQ iterative path while here, since normalization is what makes
it reachable with a large MatchCount: its TopK seeds from MatchCount*3 and
doubles five times, previously without limit. It now stops at the pool cap
instead of re-issuing an identical query.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(retrieval): share the default retrieval depth as one constant
The value 50 was written three times — the over-retrieval floor in
HybridSearch, the MatchCount fallback added alongside it, and
RetrievalConfig.GetEffectiveEmbeddingTopK — with nothing recording that the
three are meant to agree. Anyone tuning one would silently desync the
others.
Promote it to types.DefaultRetrievalTopK. No behavior change.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: drakezhang <drakezhang@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
AppendWithOverlap deduplicates chunker-prepended table headers by
matching a suffix of acc against a head window of next. When chunks are
strictly contiguous (positionOverlap == 0) there is no overlap to
deduplicate, but the loop still ran. If a sentence at the tail of acc
reappeared as real content inside next's head window (same sentence
written multiple times in the document), the head of next was mistaken
for a prepended header and deleted — silently losing up to hundreds of
bytes from the merged result.
This affects three callers:
- chat_pipeline.mergeOverlappingChunks: RAG search results returned
to the LLM with truncated content
- wiki_ingest.reconstructContent: distillation text fed to the LLM
is truncated, so wiki search returns 0 hits for present content
- frontend doc-content.vue port: the "merged" document view drops
the same segment
Fix: skip text matching when positionOverlap <= 0 and concatenate
verbatim. The chunker's prepended-header case only occurs inside the
overlap region (positionOverlap > 0), which still runs the loop.
Add a regression test covering a compliance-doc scenario where a
logging-requirement sentence recurs in a later Compliance section.
* test(datasource): cover scoped sync deletions
- deleted connector items remove the matching KB knowledge, scoped per data
source via metadata datasource_id + external_id
- SyncDeletions=false neither deletes nor counts
- stream handler classifies deleted/failed items like the batch loop
* fix(datasource): perform real sync deletions scoped per data source
- applyFetchedItem now performs the KB deletion instead of only counting
- lookup is scoped to datasource_id + external_id so identical external IDs
from two data sources cannot collide or overwrite each other
- ingestItem update-path lookup uses the same scoped query
* fix(datasource): note failed deletions retry only on full sync
- SyncResult.DeletionFailed tracks deletion failures (subset of Failed)
- streaming partial message warns that failed deletions only retry on a full
sync, since the connector cursor is checkpointed past the deleted item
* fix(datasource): humanize deletion errors, keep raw detail in logs
- raw lookup/delete errors go to server logs only, with full context
- UI samples carry a humanized "see server logs" fallback
- add datasource.syncError.* i18n keys
Note: ko-KR / ru-RU syncError translations were generated without a
native-speaker review.
* fix(datasource): hard-delete synced rows and tighten deletion scoping
Sync-internal deletions now call HardDeleteKnowledge after the soft-delete
cascade so tombstones cannot block re-sync; subtree sweeps and URL metadata
attach failures are handled with datasource scoping and explicit errors.
* chore: add git hooks aligned with CI and PR checks
Install via scripts/install-git-hooks.sh (sets core.hooksPath). Pre-commit
runs whitespace/gofmt/golangci-lint on staged files; pre-push mirrors
app/frontend/cli workflows on the diff since origin/main.
* fix: gofmt test file; tighten pre-push to match CI app workflow
Pre-push now gofmts the full PR diff and runs go vet on all app packages
(like app.yml), not only Go files in the latest commit. Previous pushes
skipped checks via --no-verify and because hook-only commits had no .go files.
* fix(hooks): align pre-push diff with GitHub PR base via gh
Use three-dot diff against the open PR baseRefOid (same as CI) instead of
a two-dot merge-base range; run full go vet on every push with Go PR changes.
* style: gofumpt datasource_service_test.go
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* ci: add golangci-lint workflow gating new PR code
The repo already ships a .golangci.yml and a `make lint` target, but linting
never ran in CI. A full-tree run currently reports ~5.4k pre-existing findings,
so blocking on it is not viable; instead the workflow enables `only-new-issues`
so a PR is judged solely on the code it introduces.
Runs on pull_request only: `only-new-issues` relies on the PR diff and would
degrade into a full-tree run (and fail) on push. The matrix covers all three Go
modules, since a single run cannot span separate go.mod trees; each resolves the
shared root .golangci.yml via upward config discovery.
* fix: migrate .golangci.yml to the v2 schema
The file declared `version: 2` while still using v1 keys. `golangci-lint run`
tolerates this, but `config verify` (which golangci-lint-action runs before
linting) rejects it: `version` must be a string, and `linters-settings` is not a
valid top-level key in v2.
Quoting the version and moving the block under `linters.settings` also restores
the lll settings, which were silently ignored until now -- applying tab-width 4
raises the lll count in client/ from 32 to 42.
* ci: warm the lint cache on main so pull requests stay fast
Caches written by a pull request run are scoped to that pull request, and the
gating job only runs on `pull_request`, so no shared cache was ever produced.
Every pull request therefore recompiled the whole package graph: the first run
on this branch spent 5m39s of its 6m09s with both the Go build cache and the
golangci-lint analysis cache reporting a miss.
Add a job on main that lints purely for the cache side effect, with
`--issues-exit-code=0` so the existing backlog cannot turn the default branch
red. Manual runs route here as well, giving a full backlog report without the
misleading red status a gating run would produce.
* ci: split the lint cache warmer into its own workflow
A job excluded by a job-level `if` is still reported as a check on every pull
request, and because the matrix is never expanded for a skipped job, GitHub
renders the name literally as "warm lint cache (${{ matrix.module }})".
Separating the triggers into two files removes the stray check instead of
papering over the name. Cache keys are derived from the working directory and
go.sum rather than the workflow, so the caches written on main are still the
ones pull requests restore.
MinerU keys file_parse results by the uploaded filename stem, so preserving
real basenames requires dynamic response lookup instead of results.document.
Also append FileType when the basename has no extension.
- Updated test files to use types.EmbedChannelContextKey instead of middleware.EmbedChannelContextKey for consistency.
- Introduced context cloning decisions for EmbedChannelContextKey to ensure proper context management during background tasks.
- Added tests to verify the correct behavior of context key declarations and their cloning decisions.
- Added a new configuration option for agents to enable or disable long-term memory.
- Implemented UI elements in the Agent Editor for managing long-term memory settings.
- Updated internationalization files for English, Korean, Russian, and Chinese to include new labels and descriptions related to long-term memory.
- Enhanced backend logic to support memory retrieval and management based on user and agent preferences.
- Introduced a new tool for searching long-term memory, ensuring it respects the agent's memory settings.
- Added tests to verify the functionality of the new memory features and their integration with existing systems.
- Introduced a new method to mark memories as forced consolidated, allowing users to request immediate consolidation.
- Added a new message for users indicating that a tidy-up just ran and to try again shortly.
- Updated internationalization files for English, Korean, Russian, and Chinese to include the new message.
- Enhanced memory repository and service logic to support the new forced consolidation functionality.
- Updated database schema to include a new timestamp for forced consolidation.
- Updated the item listing query to ensure deterministic paging by including the item ID in the order clause.
- Added a new test to verify that models used by memory extraction cannot be deleted, ensuring proper error handling.
- Refactored the model deletion logic to check both embedding and extraction models against tenant memory configurations.
- Introduced a new method to handle tombstone records during memory clearing, ensuring that live memories retain their tombstone status when cleared.
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
* fix: support folder drag-and-drop upload with directory structure preserved
Drag-dropping a folder onto the knowledge base page was treated as a
single phantom file (size 0, no extension) and failed with "unsupported
file type" on Chrome/Edge, or silently did nothing on Firefox.
Root cause: collectDroppedFiles checked dataTransfer.files first, which
Chrome/Edge populates with non-traversable directory entries for folder
drops. The function never reached the webkitGetAsEntry branch that can
recursively traverse directory contents.
Fix: reverse the priority — try webkitGetAsEntry first to recursively
traverse dropped directories, setting webkitRelativePath on each File to
preserve the folder structure (same format as <input webkitdirectory>).
Fall back to dataTransfer.files only for plain file drags or browsers
without webkitGetAsEntry support.
Additional details:
- readEntries() is called in a loop until empty (batches may be partial)
- Hidden files/directories (segments starting with .) are skipped,
matching the button-based folder upload behavior
- Top-level dropped files keep webkitRelativePath empty so they upload
as individual files, not folder uploads
- Object.defineProperty is used with try/catch for Safari compatibility
Closes#2676
* style: switch code comments to Chinese for consistency with existing file
* fix: don't fall back to phantom files after directory drag-drop
Once webkitGetAsEntry returns entries, trust the traversal result even
when it is empty (empty or hidden-only folders). Falling back to
dataTransfer.files reintroduced Chrome/Edge's size-0 directory File.
Also isolate per-item failures and cover the collector with unit tests.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* feat(datasource): add Feishu/Lark Drive (云盘) connector
Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.
Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
/ feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
(FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
the wiki connector's three cursor semantics (resume fast-path,
failure-does-not-advance-cursor, toSyncCursor snapshot), plus
ListResources / ResolveResourceAncestors (top-down shared traversal,
best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata
Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".
Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
Drive root folder_token input + load button before the lazy-load tree,
edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
input group
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): refine Feishu Drive connector UI, channel and root folder
A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:
Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
"飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.
Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
switch); the tree area shows a "load a folder first" placeholder until
the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
actionable i18n hints.
Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
to resolve the root folder's human-readable name; the list API only
returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
suffix) so it matches the saved resource_id form and the direct
children's ParentID, which previously broke selection matching on edit.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): sync a single selected Drive file without folder error
When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".
The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.
A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Co-Authored-By: Claude <noreply@anthropic.com>
* ci: add build-test-images workflow for tar.gz artifact export
Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.
- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days
Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks extraction design spec
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu shared-blocks implementation plan
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): sync feishu drive docx via blocks API with export fallback
- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
sub-items), falling back to the export API on blocks errors or empty
renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): label feishu drive knowledge as 飞书云盘 in list
The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): polish feishu drive resource-step UI in datasource editor
- Replace the two stacked t-alert banners under the folder_token input:
shareHint becomes an always-visible form-desc line, rootNotSupportedHint
moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
status + tips on the input (replacing the global MessagePlugin) when
empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
fixed and the resource region (placeholder / loading / empty / tree)
fills the remaining drawer height, with the tree scrolling internally.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): add feishu drive datasource integration guide
User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine
Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.
Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.
Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): split feishu connector into core/wiki/drive packages
Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.
container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).
Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).
Co-Authored-By: Claude <noreply@anthropic.com>
* test(datasource): migrate feishu tests to core/wiki/drive packages
Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.
All tests pass: core 4s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test
Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).
Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.
All tests pass: core 3.5s, wiki 13s, drive 3s.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(datasource): repair feishu cleanup fallout
Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.
All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(frontend): feishu drive data source UI and i18n
Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: add trellis journal merge=union rule
Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor(datasource): rename feishu/drive drive_connector.go to connector.go
Aligns naming with feishu/wiki subpackage (connector.go).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(frontend): allow data:image base64 URIs in sanitized markdown
DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path
The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export
Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.
- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
they don't break under the new default
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout
- 飞书云盘接入说明: add section on docx parse mode (export default vs
blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
add pros/cons table, fix source-file table to core/wiki/drive layout
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: 修复合并main后丢失新增的国际化问题
* fix(i18n): 修复误删的国际化内容
* docs: 删除误提交的文件
* feat(im): support reverse proxy for feishu websocket long connection
Add api_base_url channel config for Feishu/Lark IM adapters so internal
deployments can route Feishu API calls and the WebSocket long connection
through a reverse proxy (nginx). The config overrides both the SDK
bootstrap domain (WithDomain) and the adapter HTTP API origin (api()),
unified into a single field.
- adapter: NewAdapter gains apiBaseURL + SSRF validation (http(s) scheme,
ValidateURLForSSRF), mirroring wecom's validateEndpointURL; api() uses
the override instead of region.OpenBaseURL
- longconn: NewLongConnClient forwards apiBaseURL to larkws.WithDomain
- factory: reads api_base_url once, passes to both adapter and longconn
- frontend: Feishu/Lark channel config gains an API Base URL input
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ci): restore build-test-images workflow dropped by 00d7090f
00d7090f ("docs: 删除误提交的文件") deleted build-test-images.yml
alongside two obsolete design docs, causing the workflow_dispatch
trigger to disappear from the branch. Restore it so the manual
build-test-images action is available again.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(im): support reverse proxy for feishu websocket long connection
飞书 SDK 的 WebSocket 长连接分两阶段:bootstrap HTTP 拿 ws 地址 + ws dial
直连飞书下发的地址。WithDomain 只控制 bootstrap 请求目标,ws dial 的地址
由飞书服务端下发(host 为 msg-frontier.feishu.cn),SDK 不支持改写。
内网部署无法直连飞书 wss,因此新增 api_base_url 字段:让 bootstrap 和
HTTP API 走反代,并通过 nginx sub_filter 改写飞书下发的 wss 地址为明文
ws,使 ws dial 也经反代回连 nginx。
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(frontend): add api_base_url config for feishu IM channel
新增 api_base_url 配置入口与 i18n 文案,用于内网反代部署。
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(im): document feishu api_base_url reverse proxy usage
补充 nginx 反代配置说明:三个 location(bootstrap/open.feishu.cn +
sub_filter、ws dial/msg-frontier.feishu.cn、OpenAPI/open.feishu.cn),
以及 sub_filter 改写 wss 地址、rewrite 前缀剥离、SSRF 白名单等要点。
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(ci): add manual docker build workflow
Add a workflow_dispatch-only workflow to build ui/app/docreader/sandbox
images on demand. No auto-trigger on push/tag. Each package is gated by
a boolean input (build_all + per-package), and the branch is picked via
the native workflow_dispatch ref selector. Builds are load-only (push
false) to verify Dockerfiles without needing any registry secret.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(ci): sync manual docker build workflow from main (add tar export + unified package job)
* Delete .github/workflows/docker-build.yml
* Delete .github/workflows/build-test-images.yml
---------
Co-authored-by: Claude <noreply@anthropic.com>
Notes (media_type=11) were skipped as "unsupported by wiki OpenAPI". The wiki
namespace indeed never exposes a note body, but IMA reads it from a second
namespace: get_media_info reports a notebook_id, which
/openapi/note/v1/get_doc_content turns into the note text. The connector
already parsed that notebook_id into item metadata and then dropped the
record, so this wires up the one call that was missing.
Note bodies arrive as plain text and are ingested as Markdown, since IMA notes
are authored as rich text and the export keeps its heading and list markers.
An unreadable note now takes the same retry path as a failed download instead
of being recorded as synced; empty notes are skipped.
AI sessions (12) and video parses (16) stay skipped, as those genuinely have
no read endpoint, and the connector description now names them specifically
rather than implying notes are unsupported.
Also accept the {retcode, errmsg} spelling of the response envelope alongside
{code, msg}. The reference MCP implementation confirms {code, msg} is correct,
but several published IMA references disagree, and under the wrong spelling
the status code would decode as 0 and turn every API error into a silent
success.
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.
Alongside that:
- Drop media_type from logicalKey. get_knowledge_list omits the field, so
folding it in meant every external_id would shift the day IMA starts
populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.