Commit Graph

2741 Commits (5d3601c4e4d0cfc2c86268f10cc1c3d2a67dcd8e)

Author SHA1 Message Date
koishi514-Z 5d3601c4e4 fix(i18n): add missing tenantless copy 2026-08-20 11:49:26 +08:00
Zhang GH aac4f1586b
Fix/hybrid search match count normalization (#2750)
* fix(hybrid-search): normalize MatchCount once instead of at the truncation site

An omitted match_count deserializes to Go's zero value, which reached the
truncation step in HybridSearch and sliced the deduplicated chunk list to
[:0] — a successful retrieval returned an empty result set. A negative
value panicked on that same slice bound and surfaced as a 500.

Resolve the value once at the top of HybridSearch via normalizedMatchCount
so all three consumers agree, rather than patching the truncation site
alone: the over-retrieval floor, the FAQ iterative-retrieval trigger
(which never fired on this path because len(chunks) < 0 is false), and the
truncation itself.

Bound the FAQ iterative path while here, since normalization is what makes
it reachable with a large MatchCount: its TopK seeds from MatchCount*3 and
doubles five times, previously without limit. It now stops at the pool cap
instead of re-issuing an identical query.

Co-authored-by: Cursor <cursoragent@cursor.com>

* refactor(retrieval): share the default retrieval depth as one constant

The value 50 was written three times — the over-retrieval floor in
HybridSearch, the MatchCount fallback added alongside it, and
RetrievalConfig.GetEffectiveEmbeddingTopK — with nothing recording that the
three are meant to agree. Anyone tuning one would silently desync the
others.

Promote it to types.DefaultRetrievalTopK. No behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: drakezhang <drakezhang@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 11:36:38 +08:00
Zhang GH 478a6521ea
Merge pull request #1156 from toy0116/pr/hybrid-search-matchcount-fix
fix(hybrid-search): guard MatchCount=0 truncating all results
2026-08-19 20:16:23 +08:00
Zhang GH 4da17c5e69
Merge pull request #2731 from Lem0nTea2002/docs/cli-retry-argv-contract
docs(cli): align retry guidance with retry_argv
2026-08-19 15:54:43 +08:00
Zhang GH c42778556c
Merge pull request #2701 from oscarlius/codex/fix-chunk-context-header-nul
fix(chunk): sanitize context headers before persistence
2026-08-19 15:17:31 +08:00
Zhang GH 8f9499b914
Merge pull request #2735 from JohnWick348/docs/update-screenshot-tracker
docs: update website screenshot tracker
2026-08-19 15:05:30 +08:00
Zhang GH 413435475d
Merge pull request #2287 from hbh112233abc/fix/build-images
fix(build): Enable Docker BuildKit in build_images.sh
2026-08-19 15:04:12 +08:00
JohnWick348 37ea8893f5 docs: update website screenshot tracker 2026-08-18 11:28:13 +08:00
Lem0nTea2002 5940700ec4 docs(cli): align retry guidance with retry_argv 2026-08-17 20:04:49 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
lyingbug 7f0e1a91e9
fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界 (#2720)
* build(anydoc): 构建脚本从 Cargo.toml 读取待打补丁的 crate 版本

原先 0.1.8 硬编码在脚本的 6 处,升级要同时改动它们;现在统一从
`anydoc = "=X.Y.Z"` 这一处 pin 推导。

另外补两道拦截:patched-anydoc/ 的复用标记记录已打补丁的版本,避免
版本变更后沿用上一版的副本、最终在 cargo 的 patch 解析里报出难以定位
的错误;构建前校验 version.go 与该 pin 一致,因为这个常量会作为
anydoc_version 写进每一篇已解析文档的元数据。

* fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界

anydoc 0.1.9 只把 pdf-inspector 从 0.1.8 提到 1.14.2,但那一版含 9 个
修复,其中 7 个是给此前完全无上界的解析工作加限额:每页 Form XObject
展开、CID /W 区间、Encoding cidrange 与 ToUnicode bfrange 展开、
分配操作符前的内容流解码、detector 的 Tj/TJ 操作数回溯、以及不相交
矩形的表格聚类。它们全都落在 process_pdf_mem 这条路径上,而这是
anydoc 转换 PDF 的唯一入口,且只由上传文件的字节驱动。

这正是 guarded() 拦不住的那一类问题:耗尽 CPU 不会 panic,分配失败直接
abort。以 detector 回溯为例,缺少操作数的 TJ 会重扫整个内容流,因此
一串裸 `] TJ` 的开销是长度的平方:977 KB 的 PDF 要 26.7 秒,1.9 MB 要
111.9 秒;升级后分别是 5.8 毫秒和 11.0 毫秒。
TestDetectorLookbackStaysLinear 用 15 秒预算把这个上界钉住——它在
0.1.8 上会超时失败,在 0.1.9 上 0.01 秒通过。

另外 2 个修复改善提取质量,直接影响入库内容:Form XObject 内的文字
现在跟踪 text line matrix 并处理 T*/TL/'/"/Tc/Tw,嵌套表单文字不再
丢换行和字间距;小型大写字母(small-caps)的文字段会合并,不再被当成
额外的表格列而产出错位的 markdown 表格。

升级本身不需要改绑定代码:anydoc 0.1.8 与 0.1.9 的 src/ 字节级一致,
C ABI、文档模型和 GFM 序列化器都没变,6 项本地修改原样保留。依赖树也
只动了 3 行版本和 1 个 checksum,lopdf 仍是 0.42,cargo audit 结果与
升级前相同(仅 ttf-parser 未维护这一条已允许的警告)。

* docs(anydoc): 文档与注释中的耗时数字对齐到实测记录

表格与注释里的数字改为与随 PR 附上的实测日志同一次运行的结果,便于核对。
2026-08-16 12:30:55 +08:00
wizardchen ad2f1d1450 fix(docker): copy anydoc-go go.mod before go mod download
go.mod replace-points firecrawl/anydoc to ./third_party/anydoc-go, so the vendored module must exist in the image before dependency download.
2026-08-16 12:29:42 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
Greenplumwine b18ae0f734
【Fix】修复docker-compose.yml中配置的postgres 安全检查问题 (#2704) 2026-08-14 10:53:27 +08:00
shangfeng-liu b6b1148bbb fix(chunk): sanitize context headers before persistence 2026-08-14 00:33:09 +08:00
wizardchen 49010d2526 fix(chat): 兜底回答不再把原始命中数报成检索结果数
检索阶段以 ErrSearchNothing 短路时会走兜底回答,而兜底 prompt 只拿到知识库的
文档标题清单,一条检索片段都没进模型上下文,引用集合同样为空。但进度条的计数
在 MergeResult / RerankResult 都为空时回退到原始 SearchResult,于是时间轴上写着
「找到 10 个结果」——既不是模型看到的上下文数,也没有引用可以点开。

现在兜底路径下 count / doc_count / web_count 归零,原始命中数放进新的
candidate_count,前端相应显示「命中 N 条候选,相关性不足,未用于回答」,与完全
没有候选的「未检索到相关内容」区分开:这个区别指向的是相关性阈值,而不是知识库。
2026-08-13 21:34:21 +08:00
wizardchen 9ed6266264 fix(chat): 持久化快速问答的检索步骤,刷新后不再丢失时间轴
快速问答的检索步骤只存在于 SSE 事件流里,刷新后前端只能靠 knowledge_references
反推时间轴,于是「检索了但没命中」的那一轮回到历史时步骤全部消失,只剩一条
孤立的记忆行。

改为在快速问答模式下订阅 pipeline 的工具事件,把 query_understand /
knowledge_search / attachment_parsing / image_analysis 落到 agent_steps。只在收到
tool_result 时写入,避免被中断的步骤刷新后显示成已完成。

这些调用来自 pipeline 而非模型,ID 加 ragpipe- 前缀并在 Agent 历史回放里过滤掉,
否则会把模型没发起过、甚至没被提供的工具调用塞进请求协议。
2026-08-13 21:34:21 +08:00
wizardchen 9ee9fa0b52 fix(frontend): 修复占位行样式并把记忆行统一收进时间轴
「正在准备回答」占位节点缺少 .tool-event 包装,图标定位与行高规则都挂在该
层下,shimmer 文案因行高不足被裁切。补上包装,与同组件其他步骤结构一致。

Agent 模式的记忆行原先是悬在时间轴上方的独立卡片,现在作为时间轴首个节点
渲染,折叠时随中间步骤一起收起。召回列表、展开状态与「忘记」调用抽成
useChatMemoryRow,两条时间轴共用,避免各写一套后行为分叉。
2026-08-13 21:34:21 +08:00
ttommybot 1d68b4dcd7
fix: filter menus by deployment capabilities (#2674)
* fix: filter menus by deployment capabilities

* fix: address deployment capabilities review feedback

Restore Lite organizations hiding when the capabilities probe fails,
align embed capability detection with route registration, move the
handler into SystemHandler for Swagger/docs/client sync, and add
cross-layer capability key contract tests.

* style: gofmt deployment capabilities files

Fix App CI formatting check so the vet/test step can run.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-13 20:18:48 +08:00
ChenRussell 2a60a7f30a fix(chunk-merge): stop dropping content when contiguous chunks repeat text
AppendWithOverlap deduplicates chunker-prepended table headers by
  matching a suffix of acc against a head window of next. When chunks are
  strictly contiguous (positionOverlap == 0) there is no overlap to
  deduplicate, but the loop still ran. If a sentence at the tail of acc
  reappeared as real content inside next's head window (same sentence
  written multiple times in the document), the head of next was mistaken
  for a prepended header and deleted — silently losing up to hundreds of
  bytes from the merged result.

  This affects three callers:
    - chat_pipeline.mergeOverlappingChunks: RAG search results returned
      to the LLM with truncated content
    - wiki_ingest.reconstructContent: distillation text fed to the LLM
      is truncated, so wiki search returns 0 hits for present content
    - frontend doc-content.vue port: the "merged" document view drops
      the same segment

  Fix: skip text matching when positionOverlap <= 0 and concatenate
  verbatim. The chunker's prepended-header case only occurs inside the
  overlap region (positionOverlap > 0), which still runs the loop.

  Add a regression test covering a compliance-doc scenario where a
  logging-requirement sentence recurs in a later Compliance section.
2026-08-13 19:58:57 +08:00
T3L000 abe96981c7 fix(frontend): 修复 API Playground SSE 终态处理 2026-08-13 19:55:18 +08:00
MidoriKurage 236d8c098f
fix(datasource): perform real sync deletions scoped per data source (#2690)
* test(datasource): cover scoped sync deletions

- deleted connector items remove the matching KB knowledge, scoped per data
  source via metadata datasource_id + external_id
- SyncDeletions=false neither deletes nor counts
- stream handler classifies deleted/failed items like the batch loop

* fix(datasource): perform real sync deletions scoped per data source

- applyFetchedItem now performs the KB deletion instead of only counting
- lookup is scoped to datasource_id + external_id so identical external IDs
  from two data sources cannot collide or overwrite each other
- ingestItem update-path lookup uses the same scoped query

* fix(datasource): note failed deletions retry only on full sync

- SyncResult.DeletionFailed tracks deletion failures (subset of Failed)
- streaming partial message warns that failed deletions only retry on a full
  sync, since the connector cursor is checkpointed past the deleted item

* fix(datasource): humanize deletion errors, keep raw detail in logs

- raw lookup/delete errors go to server logs only, with full context
- UI samples carry a humanized "see server logs" fallback
- add datasource.syncError.* i18n keys

Note: ko-KR / ru-RU syncError translations were generated without a
native-speaker review.

* fix(datasource): hard-delete synced rows and tighten deletion scoping

Sync-internal deletions now call HardDeleteKnowledge after the soft-delete
cascade so tombstones cannot block re-sync; subtree sweeps and URL metadata
attach failures are handled with datasource scoping and explicit errors.

* chore: add git hooks aligned with CI and PR checks

Install via scripts/install-git-hooks.sh (sets core.hooksPath). Pre-commit
runs whitespace/gofmt/golangci-lint on staged files; pre-push mirrors
app/frontend/cli workflows on the diff since origin/main.

* fix: gofmt test file; tighten pre-push to match CI app workflow

Pre-push now gofmts the full PR diff and runs go vet on all app packages
(like app.yml), not only Go files in the latest commit. Previous pushes
skipped checks via --no-verify and because hook-only commits had no .go files.

* fix(hooks): align pre-push diff with GitHub PR base via gh

Use three-dot diff against the open PR baseRefOid (same as CI) instead of
a two-dot merge-base range; run full go vet on every push with Go PR changes.

* style: gofumpt datasource_service_test.go

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-13 19:52:44 +08:00
lyingbug aaad96ef86
ci: add golangci-lint workflow gating new PR code (#2696)
* ci: add golangci-lint workflow gating new PR code

The repo already ships a .golangci.yml and a `make lint` target, but linting
never ran in CI. A full-tree run currently reports ~5.4k pre-existing findings,
so blocking on it is not viable; instead the workflow enables `only-new-issues`
so a PR is judged solely on the code it introduces.

Runs on pull_request only: `only-new-issues` relies on the PR diff and would
degrade into a full-tree run (and fail) on push. The matrix covers all three Go
modules, since a single run cannot span separate go.mod trees; each resolves the
shared root .golangci.yml via upward config discovery.

* fix: migrate .golangci.yml to the v2 schema

The file declared `version: 2` while still using v1 keys. `golangci-lint run`
tolerates this, but `config verify` (which golangci-lint-action runs before
linting) rejects it: `version` must be a string, and `linters-settings` is not a
valid top-level key in v2.

Quoting the version and moving the block under `linters.settings` also restores
the lll settings, which were silently ignored until now -- applying tab-width 4
raises the lll count in client/ from 32 to 42.

* ci: warm the lint cache on main so pull requests stay fast

Caches written by a pull request run are scoped to that pull request, and the
gating job only runs on `pull_request`, so no shared cache was ever produced.
Every pull request therefore recompiled the whole package graph: the first run
on this branch spent 5m39s of its 6m09s with both the Go build cache and the
golangci-lint analysis cache reporting a miss.

Add a job on main that lints purely for the cache side effect, with
`--issues-exit-code=0` so the existing backlog cannot turn the default branch
red. Manual runs route here as well, giving a full backlog report without the
misleading red status a gating run would produce.

* ci: split the lint cache warmer into its own workflow

A job excluded by a job-level `if` is still reported as a check on every pull
request, and because the matrix is never expanded for a skipped job, GitHub
renders the name literally as "warm lint cache (${{ matrix.module }})".

Separating the triggers into two files removes the stray check instead of
papering over the name. Cache keys are derived from the working directory and
go.sum rather than the workflow, so the caches written on main are still the
ones pull requests restore.
2026-08-13 19:26:30 +08:00
wizardchen cb69764223 style(docparser): gofmt mineru converter tests 2026-08-13 18:51:27 +08:00
wizardchen 2308593ca0 fix(docparser): resolve MinerU results by upload filename stem
MinerU keys file_parse results by the uploaded filename stem, so preserving
real basenames requires dynamic response lookup instead of results.document.
Also append FileType when the basename has no extension.
2026-08-13 18:51:27 +08:00
shangfeng-liu 0eac47490b fix(docparser): preserve filename extension in MinerU multipart uploads 2026-08-13 18:51:27 +08:00
T3L000 4e53aa9d77 修复:同步 knowledge-search 的 Swagger 路由 2026-08-13 18:08:58 +08:00
wizardchen 98239de0d2 refactor(context): replace middleware.EmbedChannelContextKey with types.EmbedChannelContextKey
- Updated test files to use types.EmbedChannelContextKey instead of middleware.EmbedChannelContextKey for consistency.
- Introduced context cloning decisions for EmbedChannelContextKey to ensure proper context management during background tasks.
- Added tests to verify the correct behavior of context key declarations and their cloning decisions.
2026-08-13 18:00:02 +08:00
wizardchen 10cc694fed feat(memory): introduce long-term memory feature with i18n updates
- Added a new configuration option for agents to enable or disable long-term memory.
- Implemented UI elements in the Agent Editor for managing long-term memory settings.
- Updated internationalization files for English, Korean, Russian, and Chinese to include new labels and descriptions related to long-term memory.
- Enhanced backend logic to support memory retrieval and management based on user and agent preferences.
- Introduced a new tool for searching long-term memory, ensuring it respects the agent's memory settings.
- Added tests to verify the functionality of the new memory features and their integration with existing systems.
2026-08-13 18:00:02 +08:00
wizardchen fda037e7f3 feat(memory): add forced consolidation feature and related i18n updates
- Introduced a new method to mark memories as forced consolidated, allowing users to request immediate consolidation.
- Added a new message for users indicating that a tidy-up just ran and to try again shortly.
- Updated internationalization files for English, Korean, Russian, and Chinese to include the new message.
- Enhanced memory repository and service logic to support the new forced consolidation functionality.
- Updated database schema to include a new timestamp for forced consolidation.
2026-08-13 18:00:02 +08:00
wizardchen 5b0b34195e refactor(memory): enhance item listing and model deletion logic
- Updated the item listing query to ensure deterministic paging by including the item ID in the order clause.
- Added a new test to verify that models used by memory extraction cannot be deleted, ensuring proper error handling.
- Refactored the model deletion logic to check both embedding and extraction models against tenant memory configurations.
- Introduced a new method to handle tombstone records during memory clearing, ensuring that live memories retain their tombstone status when cleared.
2026-08-13 18:00:02 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
hanxiantao 2e89fdab92 fix(dev): use IPv4 loopback for local services 2026-08-13 11:21:20 +08:00
sandaogouchen 262b172c28
fix(chunking): preserve parent-child text child embeddings (#2680)
* fix(chunking): preserve parent-child text child embeddings

* refactor(chunking): simplify text chunk index selection
2026-08-13 11:20:03 +08:00
liguowei e9f1e87662 fix(data-analysis): 修复 Excel 表格摘要丢失样本数据 2026-08-13 11:18:08 +08:00
ttommybot 11f2b4dcee fix: allow clearing optional model selections 2026-08-13 11:13:03 +08:00
wheatzone 2384fbae4e
fix: support folder drag-and-drop upload with directory structure preserved (#2677)
* fix: support folder drag-and-drop upload with directory structure preserved

Drag-dropping a folder onto the knowledge base page was treated as a
single phantom file (size 0, no extension) and failed with "unsupported
file type" on Chrome/Edge, or silently did nothing on Firefox.

Root cause: collectDroppedFiles checked dataTransfer.files first, which
Chrome/Edge populates with non-traversable directory entries for folder
drops. The function never reached the webkitGetAsEntry branch that can
recursively traverse directory contents.

Fix: reverse the priority — try webkitGetAsEntry first to recursively
traverse dropped directories, setting webkitRelativePath on each File to
preserve the folder structure (same format as <input webkitdirectory>).
Fall back to dataTransfer.files only for plain file drags or browsers
without webkitGetAsEntry support.

Additional details:
- readEntries() is called in a loop until empty (batches may be partial)
- Hidden files/directories (segments starting with .) are skipped,
  matching the button-based folder upload behavior
- Top-level dropped files keep webkitRelativePath empty so they upload
  as individual files, not folder uploads
- Object.defineProperty is used with try/catch for Safari compatibility

Closes #2676

* style: switch code comments to Chinese for consistency with existing file

* fix: don't fall back to phantom files after directory drag-drop

Once webkitGetAsEntry returns entries, trust the traversal result even
when it is empty (empty or hidden-only folders). Falling back to
dataTransfer.files reintroduced Chrome/Edge's size-0 directory File.
Also isolate per-item failures and cover the collector with unit tests.

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-08-13 10:44:16 +08:00
lyingbug 0cc3c33530
fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题 (#2673)
* fix(datasource/gitlab): 修复项目路径编码与连接器单例缓存问题

避免 project_id 二次 URL 编码,并确保每个数据源使用独立 client;
保存时校验 projects 配置,补齐增量同步过滤与相关单测。

* style(datasource/gitlab): gofmt connector sources
2026-08-12 16:22:02 +08:00
Greenplumwine 755243d61c
feat(im): support reverse proxy for feishu websocket long connection (#2550)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

* feat(im): support reverse proxy for feishu websocket long connection

Add api_base_url channel config for Feishu/Lark IM adapters so internal
deployments can route Feishu API calls and the WebSocket long connection
through a reverse proxy (nginx). The config overrides both the SDK
bootstrap domain (WithDomain) and the adapter HTTP API origin (api()),
unified into a single field.

- adapter: NewAdapter gains apiBaseURL + SSRF validation (http(s) scheme,
  ValidateURLForSSRF), mirroring wecom's validateEndpointURL; api() uses
  the override instead of region.OpenBaseURL
- longconn: NewLongConnClient forwards apiBaseURL to larkws.WithDomain
- factory: reads api_base_url once, passes to both adapter and longconn
- frontend: Feishu/Lark channel config gains an API Base URL input

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ci): restore build-test-images workflow dropped by 00d7090f

00d7090f ("docs: 删除误提交的文件") deleted build-test-images.yml
alongside two obsolete design docs, causing the workflow_dispatch
trigger to disappear from the branch. Restore it so the manual
build-test-images action is available again.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(im): support reverse proxy for feishu websocket long connection

飞书 SDK 的 WebSocket 长连接分两阶段:bootstrap HTTP 拿 ws 地址 + ws dial
直连飞书下发的地址。WithDomain 只控制 bootstrap 请求目标,ws dial 的地址
由飞书服务端下发(host 为 msg-frontier.feishu.cn),SDK 不支持改写。

内网部署无法直连飞书 wss,因此新增 api_base_url 字段:让 bootstrap 和
HTTP API 走反代,并通过 nginx sub_filter 改写飞书下发的 wss 地址为明文
ws,使 ws dial 也经反代回连 nginx。

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): add api_base_url config for feishu IM channel

新增 api_base_url 配置入口与 i18n 文案,用于内网反代部署。

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(im): document feishu api_base_url reverse proxy usage

补充 nginx 反代配置说明:三个 location(bootstrap/open.feishu.cn +
sub_filter、ws dial/msg-frontier.feishu.cn、OpenAPI/open.feishu.cn),
以及 sub_filter 改写 wss 地址、rewrite 前缀剥离、SSRF 白名单等要点。

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(ci): add manual docker build workflow

Add a workflow_dispatch-only workflow to build ui/app/docreader/sandbox
images on demand. No auto-trigger on push/tag. Each package is gated by
a boolean input (build_all + per-package), and the branch is picked via
the native workflow_dispatch ref selector. Builds are load-only (push
false) to verify Dockerfiles without needing any registry secret.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(ci): sync manual docker build workflow from main (add tar export + unified package job)

* Delete .github/workflows/docker-build.yml

* Delete .github/workflows/build-test-images.yml

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-12 16:18:27 +08:00
Numblgw d3524d8d04
feat(datasource): 支持 GitLab 数据源文档同步 (#2656)
* 支持gitlab数据源

* 添加单元测试 && 修复gitlab数据源配置多个的UI展示问题

* 补充韩语、俄语文案

* gitlab数据源同步:新增文件类型过滤 && 改为streaming实现

* fix: remove duplicate RSS locale keys

---------

Co-authored-by: liguowei <liguowei@xinye.com>
2026-08-12 16:09:55 +08:00
jacentsao 64d17a0211 feat(im): process file attachments in QA 2026-08-12 16:08:51 +08:00
ShadowSmallBaby c4ce9a95a8 fix(agent): 共享知识库在 runtime_context 中 doc_count 为 0
getKnowledgeBaseInfos 用请求上下文的 tenant 去查文档和 FAQ 列表,
而直接共享过来的知识库属于源 tenant。文档列表是按 tenant 隔离的,
于是共享知识库在 runtime_context 里渲染成 doc_count=0、recent_docs 为空,
模型看不到这些知识库里有内容。

改为按知识库逐个解析查询用的 tenant:tenant 取自 SearchTargets,
buildSearchTargets 已经为每个知识库解析并鉴权过,直接共享的知识库
在那里带着源 tenant。tenant 映射不能只在 SearchTargets 分支里取——
KnowledgeBases 同样会包含共享知识库的 ID(KBSelectionMode="all"
和 @提及两条路径都会写进去),所以两个分支都从 SearchTargets 取。

只放宽 metadata 查询用的 tenant,不改变参与检索的知识库集合;
映射里没有的知识库回落到请求 tenant。

租户解析和 ID 去重均复用 types.SearchTargets 已有的
GetKBTenantMap 和 GetAllKnowledgeBaseIDs。

Shadow
2026-08-12 15:55:51 +08:00
ttommybot 6f8ecc32de fix: restore FAQ batch actions 2026-08-12 15:46:25 +08:00
asmh1989 60d2ffa4fb fix(ui): 统一剪贴板复制为公共方法,HTTP 环境回退 execCommand 2026-08-12 15:38:01 +08:00
wizardchen c79d6b4bc6 fix(frontend): refine user profile change-password popup UX
Move the change-password form into an inline popup anchored to the edit
control so it matches other settings pages and avoids cluttering the row.
2026-08-12 15:34:42 +08:00
wizardchen 8d5540e26d feat(datasource/ima): sync IMA notes via the note OpenAPI
Notes (media_type=11) were skipped as "unsupported by wiki OpenAPI". The wiki
namespace indeed never exposes a note body, but IMA reads it from a second
namespace: get_media_info reports a notebook_id, which
/openapi/note/v1/get_doc_content turns into the note text. The connector
already parsed that notebook_id into item metadata and then dropped the
record, so this wires up the one call that was missing.

Note bodies arrive as plain text and are ingested as Markdown, since IMA notes
are authored as rich text and the export keeps its heading and list markers.
An unreadable note now takes the same retry path as a failed download instead
of being recorded as synced; empty notes are skipped.

AI sessions (12) and video parses (16) stay skipped, as those genuinely have
no read endpoint, and the connector description now names them specifically
rather than implying notes are unsupported.

Also accept the {retcode, errmsg} spelling of the response envelope alongside
{code, msg}. The reference MCP implementation confirms {code, msg} is correct,
but several published IMA references disagree, and under the wrong spelling
the status code would decode as 0 and turn every API error into a silent
success.
2026-08-12 14:56:25 +08:00
wizardchen 3d645d2578 fix(datasource/ima): retry transient sync failures instead of losing documents
The sync cursor was built from the raw IMA listing before any content was
fetched, so an item whose get_media_info or download failed was still recorded
as "seen with this media_id". Every later incremental sync then skipped it as
unchanged and the document was silently missing from the knowledge base for
good. Track the listed keys and the successfully synced keys separately: the
cursor now only remembers what was actually resolved, while delete detection
keeps using the full listing so a failed download is never mistaken for a
document that vanished. Deterministic skips (unsupported media types, items
with no URL) are still recorded so they are not re-probed every cycle.

Alongside that:

- Drop media_type from logicalKey. get_knowledge_list omits the field, so
  folding it in meant every external_id would shift the day IMA starts
  populating it, which reads downstream as "every document deleted".
- Download URL-backed media that carry auth headers rather than handing the
  bare URL to the ingest layer, which cannot replay those headers.
- Derive image extensions from the response Content-Type; IMA reports every
  image as media_type=9, so a JPEG was being written as .png.
- Stop logging API response bodies, which carry pre-signed storage URLs and
  per-request auth headers.
- Validate media URLs against the SSRF policy before the first connection.
- Stop writing the redundant kb_media map into the persisted cursor.
- Say in the connector description that notes and AI sessions are skipped.
2026-08-12 14:56:25 +08:00
joeyczheng 664d193ca7 feat(datasource): add Tencent IMA datasource
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-08-12 14:56:25 +08:00
jiahao6635 cc690bbe7f fix(docreader): avoid chart serialization during xlsx preprocessing 2026-08-12 13:26:19 +08:00
wizardchen c697f6d9e4 fix(frontend): harden document chunk pagination state handling
Align segment labels with loaded page data, surface API failures during
page transitions, and show a loading state while chunk pages are fetching.
2026-08-12 13:23:44 +08:00