Commit Graph

81 Commits (main)

Author SHA1 Message Date
Lukas da3317039b
fix(docparser): keep every row and cell of a CSV upload in its table (#4435) 2026-10-17 23:16:31 +08:00
Lukas e2a58ea3cb
fix(docparser): keep JSON numbers and <, >, & as written in JSON uploads (#4436)
jsonToMarkdown decoded the upload with json.Unmarshal, which turns every
number into a float64. An id above 2^53 therefore changed digits:
12345678901234567890 came out as 12345678901234567000, and
9007199254740993 as 9007199254740992. 1.50 became 1.5 and 1e5 became
100000. The decoder now uses UseNumber, and each number is written back
exactly as it was in the file.

The chunks were encoded with json.Marshal, which escapes <, > and & for
embedding in HTML, so "a < b && c > d" was indexed as
"a \u003c b \u0026\u0026 c \u003e d". They are now encoded with HTML
escaping off, in the size measurement as well, so a chunk is measured as
it is written.
2026-10-17 22:31:39 +08:00
ordinary being 1fe804bd5c
fix(docparser): bound MinerU result packages and their expansion (#4396)
A MinerU result package arrives over the network from a third-party parse API and
was buffered whole before anything about it was validated. The Cloud converter
downloaded it with an unbounded io.ReadAll, even though the V1 client already caps
the same artifact at mineruV1MaxZipBytes - the two implementations of one feature
disagreed about the limit.

Decompression was unbounded on every path. readZipEntry and readZipEntryBytes
called io.ReadAll on a decompressed archive entry, and extractMarkdownZip keeps
every extracted image in memory until the caller hands the slice on, so a package
of many small entries expanded without ever tripping a per-entry check. The
declared UncompressedSize64 is attacker or vendor controlled and may understate
the expansion, so the header is checked and the read is capped as well.

Cap the download at the V1 ceiling, each entry at maxZipEntryBytes, and the
accumulated images at maxZipImagesTotalBytes. The image budget is a parameter on
extractMarkdownZipBounded so a test can exercise it without materialising the
production figure, mirroring the jsonLimit field added in #3985.
2026-10-16 19:37:27 +08:00
Greenplumwine 973d9d051f
feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects (#3497)
* ci(build): add manual docker build workflow

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): deep-adapt Feishu/Lark sync with folder mirror, block parsing and embedded objects

- folder mirror: wiki/drive connectors emit path-qualified FileName so
  knowledge lands in KB folders mirroring the Feishu tree; shortcut
  subtrees deduped; sanitized directory names; legacy datasources
  converge via one-shot resync marker (incremental sync upgrades to
  full once, then clears)
- parse mode: replace FEISHU_DOCX_PARSE_MODE env with per-datasource
  settings.parse_mode (blocks default, export escape hatch); frontend
  selector + 5 locales; zero-migration read-time fallback
- block-level GFM: extend docx block coverage (callout/grid/iframe/
  board/mindnote/agenda/synced blocks), rich text elements (links,
  inline code, KaTeX equations, mentions), real ordered-list numbering,
  code language mapping, merged-cell fill, truncation note
- embedded objects: images (gated off multimodal toggle) and boards
  (download_as_image, 403-degraded) stored as knowledge with
  weknora-img://<n> markers resolved in the document-processing
  pipeline via knowledge FilePath; document attachments whitelisted
  (pdf/md/xls(x)/csv/doc(x)/ppt(x)) as standalone knowledge in the
  parent folder with bidirectional parent/child metadata and filename
  list in the body; videos skipped with placeholder + source link
- golden baseline explicitly re-recorded; docs (.env.example,
  docker-compose, docs/*, website-docs) synced

* fix(datasource): address Feishu deep-adaptation smoke-test findings

- folder mapping: drop the space/root-folder name prefix — the selected
  sync root maps to the knowledge-base root, subdirectories stay relative
- docx blocks: render view(33) blocks silently as containers (children
  only), removing the stray '[飞书块: 视图]' placeholder before file lists
- sheet inline tables: fetch merge ranges via sheets/v3 sheets/query and
  fill the top-left value across the merged region (closed-interval
  interpretation, flagged for real-environment verification)
- knowledge list: hide embedded image/board rows (metadata
  embedded_image=true) from the document list and folder-tree counts;
  rows remain for VLM indexing and marker resolution; search untouched
- frontend: hydrate protected images (resource:// etc.) in the document
  preview so embedded Feishu images/boards actually render

* fix(datasource): resolve embedded image markers at ingest time

The weknora-img://<n> markers were only resolved during document
processing, so chunks carried real resource:// URLs while the stored
markdown file and source_content kept raw markers — file preview
showed blank images. Resolve them in ingestItem before
CreateKnowledgeFromFile (children are emitted first, so image rows
with FilePath already exist); markers with missing children are kept
for the ProcessDocument fallback to degrade. Stored file,
source_content and chunks now agree. Existing datasources converge on
the next full re-sync (update=delete+recreate).

* feat(datasource): resolve Feishu mention users to real names at ingest

docx mention_user elements only carry an open_id, so they rendered as
a generic '@成员'. Resolve the display name via
GET /contact/v3/users/:id (user_id_type=open_id) while generating the
markdown during FetchDocxWithBlocks — before storage, per user
decision; the frontend performs no lookups. A per-sync closure caches
positive and negative lookups; permission errors and empty names
silently degrade to '@成员' (debug log only, ingest never blocked).
Requires one of the contact read scopes (e.g.
contact:user.base:readonly) on the Feishu app.

* refactor(datasource): polish round — defect fixes and structural cleanup

- attachment placeholder patching now targets the unique
  '- weknora-att://<token>' marker line rendered per attachment instead of
  matching by display name, so an identical earlier bullet is never
  replaced by mistake (regression-tested)
- markdown link targets with spaces or parens are percent-escaped
  (iframe / mention_doc / inline links), keeping them parseable
- drive folder-tree walking consolidated into core.WalkDriveTree,
  deleting the 174-line copy in drive/paths.go
- parse mode threading made explicit: DocxFetchInput.ParseMode replaces
  the private client field (connector entry resolves via
  ParseFeishuConfig and passes it through ops)
- maxFeishuDownloadBytes restored to a const; the oversize-degrade test
  now declares an over-cap Content-Length and downloadRawBytes rejects
  early on the header

* refactor(datasource): align Feishu image handling with the platform image pipeline

Replace the image-as-separate-knowledge-row channel with the platform's
orthodox flow: the connector downloads image/board bytes at fetch time and
embeds them in the parent markdown as base64 data URIs; the document
processing image resolver (image_resolver.ResolveAndStore, the same stage
used for parsed PDFs/Word) stores them on the tenant file service, swaps in
persistent provider:// URLs and produces OCR/caption child chunks when a VLM
is configured.

Removed with the marker pipeline:
- weknora-img:// ingest resolution in ingestItem
- the ProcessDocument Step-1.9 marker hook
- the document-list/folder-tree filter for embedded_image rows
- the feishu-specific runtime parse_mode backfill; legacy data sources are
  now stamped by migration 000096 (sqlite 000017) which writes the one-shot
  resync_required marker — the generic full-upgrade + clear consumption stays

Attachment (PDF/Word/md) sub-items are unchanged.

* style: gofmt test files

* fix(datasource): resolve Drive sub-selection kind from the root listing

A single-file Drive selection (rootFolderToken:fileToken) was probed with
folder APIs before falling back: folder meta returned 91202 and the folder
list 1061002 on every sync, logging API errors for a working configuration.

listDriveFilesForResource now walks the root subtree first and reads the
selected node's kind off the listing: a file filters straight out of the
already-fetched results (zero error calls), a sub-folder still gets its
dedicated walk, and only a token missing from the listing (partial listing /
permission edge) keeps the legacy probe + fallback.

Also pins sub-folder selection semantics with a dedicated test.

* fix(datasource): address review findings across the Feishu deep-adaptation branch

Four parallel reviewer slices audited the branch; this applies the accepted
findings.

Content fidelity (blocks rendering):
- todo blocks parse style.done — completed items render "- [x]" instead of
  always unchecked
- code fences size past the longest backtick run in the content, so a line
  of ``` inside code can no longer close the fence early
- ordered-list "auto" numbering only resets on a sibling of the last item;
  child blocks rendered flat between items no longer renumber the run

Image pipeline:
- inline base64 images are capped at the document-processing resolver's
  10 MB budget; larger images degrade to a placeholder instead of leaving a
  permanent raw-base64 blob in stored markdown
- image markers carry a per-document nonce so user-typed look-alike text
  can never be patched in place of a real marker
- attachment marker patching anchors on the token's line end, closing the
  token-prefix collision
- SupportedImageExt recovers image/svg+xml (board exports), which the
  platform resolver already stores natively

Sync correctness:
- clearResyncMarker re-reads the row before writing: a long upgraded sync
  no longer risks reverting concurrent user edits with a stale full-row
  update
- wiki EditTime keys on max(obj_edit_time, node_edit_time): pure
  moves/renames re-file the document on the next incremental sync
- file-typed wiki/drive base names are sanitised; a "/" in a title can no
  longer fake an extra folder level

Docs/tests/comments:
- datasources docs rewritten to the v3 image contract (no image rows,
  no image_map, no parse_mode backfill claim)
- DataSourceEditorDialog drops settings.parse_mode when backtracking to a
  non-Feishu connector type
- stale comments corrected (types.go parse-mode transport, repository
  folder-counts, wiki List partial-error branch, test headers); regression
  tests added for todo done, fence sizing, ordered numbering, svg sniff

* fix(datasource): repair markdown conversion defects found in live smoke test

Live re-sync of the all-block-type Feishu test document surfaced three
converter defects:

- Code blocks: the closing fence reused the opening fence string, so a
  tagged block ended with ```java — CommonMark requires a bare closing
  fence, and every renderer left the block unterminated. The closing
  fence is now backticks only.
- Feishu heading7-9 emitted 7-9 hash marks; GFM stops at H6, so those
  lines rendered as literal '#######' text. They now degrade to bold.
- The docx API percent-encodes link targets (http%3A%2F%2F...); escapeURL
  now reverses that encoding before markdown-escaping, so stored links
  point at real URLs instead of percent-encoded strings.

Verified against the live document via a throwaway API probe: the
'i|g|k' table cells questioned during review are the document's own
content (the sheets copy says i|j|k; the two docx tables genuinely
contain 'g'), not a converter bug.

* fix(frontend): render GFM task lists in document preview

Task list items (- [x] / - [ ]) rendered as plain bullets: marked emits
<input type="checkbox"> for them, but the preview DOMPurify config's
ALLOWED_TAGS allowlist has no 'input', so the element was stripped whole.
The preview renderer now overrides checkbox to emit a styled
task-checkbox span (class/attr both allowlisted) and the preview styles
draw the box with a check mark, hiding the list bullet via :has().

Same gap exists in the chat markdown surface; left untouched there.

* fix(datasource): crop blank canvas borders from whiteboard exports

The board download_as_image API renders the WHOLE board canvas, not the
content bounding box — editors routinely leave the canvas far larger than
the drawing, so board/mindnote/diagram exports surfaced with large blank
areas below the content. The API has no crop parameter, so the connector
now trims near-uniform blank borders after download (bottom-right pixel
seeds the background color, 14/255 per-channel tolerance absorbs JPEG
noise, 8px margin kept). PNG and JPEG are re-encoded; anything
undecodable, fully blank, or already tight passes through unchanged, and
oversized images (>64MP) skip the scan to bound memory.

* fix(datasource): require corner majority before trimming board exports

The blank-border trim seeded the background color from the bottom-right
pixel alone. A large solid shape reaching that corner would seed the
background with its own fill color, and fully-filled rows could then be
trimmed as blank — cutting real content. The background color is now the
majority of the four corner pixels (>=3 must agree); on a split vote
content reaches the corners and the image passes through untrimmed.
Content visible against the background is untouched either way: a row or
column survives if even one pixel differs from the background by more
than the 14/255 JPEG-noise tolerance.

* fix(datasource): widen board trim margin and lower JPEG quality to 80

The 8px margin read as a tight crop — a 32px white frame now visually
bounds the content region. JPEG re-encode quality drops 90 -> 80 to keep
the re-encoded size closer to the original export.

* feat(datasource): timeline widget Markdown degradation + table column alignment

Two content-fidelity gaps from the live smoke test:

- 文档小组件 (add_ons, block_type 40) carry their full data inline in
  record; there is no server-side render to download. The timeline
  component (component_type_id blk_6358a421bca0001c...) now degrades to
  a Markdown list (- **time · title**:text), honoring contentShow and
  omitting empty segments; unknown component types and malformed records
  keep the previous placeholder note.
- docx text blocks carry style.align (1 left / 2 center / 3 right,
  absent unless the author set it). Native-table rendering now collects
  per-column alignment from the first explicit cell and emits GFM colon
  separators (:---: / ---:). Embedded sheets/bitable keep the plain
  separator — their value APIs expose no alignment.

* feat(datasource): show document titles for 云文档 mention links

The docx renderer emitted mention_doc references as [url](url) because
the payload carries no title. Feishu doc URLs now emit a
[weknora-docname://token](url) placeholder and FetchDocxWithBlocks
resolves the titles in one drive metadata batch call
(POST /drive/v1/metas/batch_query, docx/doc/sheet/bitable/wiki/...),
patching the link text to [文档名](url). Refs whose metadata cannot be
fetched — no permission (970003), deleted/mismatched (970005), API
absent — degrade to the previous URL-as-text form with a warn log.

This is display-only: referenced docs are not pulled as sub-documents
(deferred design lives in the task PRD).

* docs(datasource): document mention rendering enhancement and drive rename limitation

- Add optional mention scopes (contact:user.base:readonly,
  drive:drive.metadata:readonly, board:whiteboard:node:read) with
  degrade behavior to the Feishu permission tables
- Describe mention_user real-name lookup, mention_doc title backfill
  and timeline widget placeholders in the blocks rendering flow
- Document the Drive known limitation: folder renames / file moves do
  not bump modified_time, so incremental sync keeps the old KB folder
  structure until a full sync; Wiki sources are unaffected
- Clarify the directory-mapping wording in website-docs to distinguish
  Wiki (rename-aware) from Drive (rename-blind) sources

* docs(datasource): add blocks vs export tradeoff summary to website-docs

* docs(datasource): document docx block-type coverage for blocks mode

- Add a full block-type support table (block_type 1-52 grouped) to the
  datasource dev doc: text/heading/list/task/quote structures, code
  fences with language aliases, containers (callout/quote/grid/view),
  native tables with merged-cell splitting (top-left value fill) and
  column alignment, embedded sheet/bitable converted to in-document
  Markdown tables, image/board inline pipeline, attachment whitelist,
  iframe links, timeline add-on degradation and the placeholder set
- Summarize the same coverage in the drive user guide, the wiki
  quick-start doc and website-docs; fix the stale claim that board
  blocks degrade to placeholders (they ride the image pipeline now)

* style: resolve diff-scoped lint findings across feishu packages

- Wrap over-length signatures and calls (lll, max 120)
- Add doc comments to exported Block* payload types (revive)
- Rename unused closure parameters to _ (revive)
- Check w.Write / resp.Body.Close error returns (errcheck)
- Apply De Morgan rewrite in docname token validation (staticcheck)
- gofumpt-format touched files

golangci-lint run --new-from-rev=main ./... now reports 0 issues.

* ci: drop fork-only docker-build workflow

The manual docker-build workflow lives on the fork's main and is not
part of the upstream contribution; it entered the branch through the
fork-main merge.

* fix(migrations): renumber feishu resync migrations to avoid version collision with upstream

Upstream added 000017_dingtalk_stream_only (sqlite) and 000096_dingtalk_stream_only
(versioned) before our merge, colliding with our 000017/000096 feishu resync
migrations. golang-migrate rejects duplicate versions. Renumber to the next free
slots (sqlite 000027, versioned 000108) and update comments + test constant.

* fix(migrations): renumber feishu resync migrations ahead of upstream 000027/000108

Upstream main now also claims sqlite 000027 and versioned 000108
(session_sandbox_config_tenant), colliding with our just-renumbered
resync migrations. Move to 000028 / 000109.

* fix(feishu): address review — deprecated env fallback and upgrade notes

- Keep FEISHU_DOCX_PARSE_MODE as a one-release deprecated fallback when a
  data source has no per-source parse_mode (deprecation warning logged);
  annotate it in .env.example / docker-compose.yml instead of removing.
- Point stale migration-number comments at the final numbers
  (versioned 000110 / sqlite 000030) in datasource_service.go and tests.
- website-docs: add Feishu upgrade notes (env deprecation, default mode
  flip to blocks, one-shot full resync with embedding/VLM recompute);
  refresh 24-feishu-drive.md parse-mode section to the per-source
  setting and new blocks semantics.

* docs(feishu): document necessary and optional connector permissions

Restore the permission matrix lost with the docs/ removal: necessary
scopes (wiki/drive/export/docx) with missing-permission symptoms and
optional enhancement scopes (embedded sheets/bitable, board export,
mention user/doc rendering) with their degrade behavior, in
10-datasource.md's Feishu section; make 24-feishu-drive.md list the
three necessary drive scopes and link to the full table.

* Revert "docs(feishu): document necessary and optional connector permissions"

This reverts commit 3be19cb1b656b3596f2cb6afb2cda8c2a1d12be1.

* docs(feishu): permission matrix by connector, parse mode and necessity

Necessary-scope tables split by connector (feishu/lark wiki vs
feishu_drive/lark_drive) with blocks/export applicability columns and
missing-permission symptoms; optional enhancement scopes with their
degrade behavior; scope names cross-checked against the Feishu open
platform API docs (files/folder list, medias download, export tasks,
metas batch_query). Full matrix lives in 10-datasource.md; the drive
guide mirrors its own table and links back.

* test(feishu): adapt mention_user_test to upstream retryTestServer signature

* feat(feishu): mirror the sync root (wiki space / drive folder) as a top-level KB folder

So a knowledge base linking several feishu sources can tell them apart at a
glance: syncing folder 测试1 yields 测试1/... and syncing wiki space
飞书知识库同步测试 yields 飞书知识库同步测试/.... Root-name resolution
failure falls back to the previous root-less behaviour without failing the
sync; existing sources converge through the one-shot resync marker.

* refactor(feishu): cache wiki space folder prefix per run, refresh stale comments

Review follow-ups for d012e593f: cache the sanitized space prefix per spaceID
for the whole sync run (K selected spaces now cost one ListWikiSpaces paging
pass instead of K), fix the misleading warn wording, and realign the
WalkDriveTree / driveFolderName doc comments with root-mirror semantics.

* docs(feishu): drop stale and duplicated comments from the deep-adaptation

- board.go: trimBoardImage no longer claims a bottom-right seed (the code
  votes the background over the four corners)
- shared.go: rewrite the FetchDocxWithBlocks note (images/boards inline via
  base64 now; removes a stale-marker splice artefact)
- markdown.go: drop an internal task-number leftover
- datasource_service.go: collapse the ProcessSync/processSyncStreaming notes
  duplicated verbatim from resyncRequired's doc comment, keep the call-site
  why (marker cleared only on success)
- datasource_resync_test.go: drop the obsolete 'backfill' wording

* feat(i18n): 添加解析模式及其提示信息到繁體中文語言包

* fix(frontend): use radius token for task checkbox; fix simplified chars in zh-TW parse mode

CI guard fixes: the PR-added task-checkbox style pushed the radius-literal
count to 161 (> baseline 160); the new zh-TW keys missed the locale parity
audit (user's dc6fdcff9) — also normalise the simplified 文档 leftovers.

* fix(feishu): preserve sync scope, nested content, and legacy settings

Cover shortcut selection, ancestor path changes, nested containers and table boards with regression tests. Avoid redundant Drive traversal, repair resync migrations for legacy settings, and preserve inherited parse modes. Bound board image dimensions before decoding.

* docs(feishu): clarify shortcut scope and incremental path tracking

* Fix Feishu sync reconciliation and embedded media regressions

* Bound Feishu table rendering by actual cell data

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-16 16:49:44 +08:00
lyingbug 194bffbb0a
feat(agent): read chat attachments on demand (#4203)
* feat(agent): read chat attachments on demand

Agent turns pasted query-selected attachment excerpts into the prompt before the model could act. List stored files as a manifest so the agent can page and search them through read_attachment.

* fix(agent): preserve attachment access and image reads

* fix(agent): bound attachment cleanup and image processing

* Remove standalone read_attachment tool

* Read attachments and images through read_file

Merge read_attachment into read_file (attachment:// paths, numbered
document images, staged-copy aliasing, query search), return sandbox
images as images, describe images for non-vision models inside the
tool call with bounded concurrency and model-chosen detail/focus,
render image reads in the UI, and record image metadata in Langfuse.

* fix(agent): make attachment pagination complete and bound image reads

* refactor(agent): separate attachment reads and cache image observations

* fix(attachment): read image batches concurrently and harden description cache

- ReadImages authorizes once and describes images in parallel
- start VLM deadlines after a slot is held; longer timeout for full reads
- detach shared description calls from the first caller's context
- skip rewriting unchanged sandbox projections
- strip the description cache from temporary document API responses
- render upload text before calling the vision model

* fix(attachment): number images in document order and polish attachment UI

- return resolved images in document order instead of reverse
- show attachment file names instead of attachment:// IDs in tool titles
- use the shared file icon and a lighter card for chat attachments

* fix(ci): renumber attachment migrations and satisfy lint

Main already took sqlite 000035 and versioned 000116, so the attachment
lookup indexes move to the next free numbers. Also wrap the ReadImages
signature and drop an unused test parameter.

* fix(ci): expect sqlite migrations to reach version 36
2026-10-13 14:37:17 +08:00
linus-liu-web 1826f46b12
fix(docparser): 云解析超时改为跟随既有配置,不再硬编码 (#3891)
* fix(docparser): 云解析轮询上限接入配置,去掉两处硬编码超时

PaddleOCR-VL Cloud 与 WeKnoraCloud 两个 reader 的整批轮询上限仍是写死的:
paddleocr_vl_cloud_converter.go 的 paddleOCRVLCloudTimeout(600s)直接决定了
pollJob 的 deadline,reader 结构体没有任何 timeout 字段;weknoracloud_http_reader.go
的 pollTimeout 固定 20 分钟。

同族的另外两个 reader 早就可以配置:MinerU Cloud 走
WEKNORA_MINERU_CLOUD_TIMEOUT(mineru_cloud_converter.go),自建 PaddleOCR-VL 走
WEKNORA_PADDLEOCR_VL_TIMEOUT(paddleocr_vl_converter.go)。后果是运维把
WEKNORA_DOCREADER_CALL_TIMEOUT(默认 30m)调大也无效:WeKnoraCloud 的 20 分钟
比它还短,硬编码上限先触发,大体积扫描件固定在第 10/20 分钟失败。

改动:
- PaddleOCRVLCloudReader 增加 timeout / pollInterval 字段(与 MinerUCloudReader
  同形),构造函数用 requestTimeoutFromEnv 读 WEKNORA_PADDLEOCR_VL_CLOUD_TIMEOUT
  (默认仍是 600s),pollJob 用 c.timeout / c.pollInterval;超时文案对齐 MinerU
  的 "task timed out after %s (%d polls)"(旧的 "after %d polls" 是 27b0c84b 已经
  替 MinerU 换掉的写法)。
- WeKnoraCloudSignedDocumentReader 的 pollTimeout 改读
  WEKNORA_WEKNORACLOUD_TIMEOUT(默认仍是 20m)。
- .env.example 补两条说明,位置紧跟 WEKNORA_MINERU_CLOUD_TIMEOUT。

新增用例:两个 reader 的 env→超时值映射(含空/非法/0/负数回退默认)、超时真的会
结束一个厂商持续报 running 的任务(假服务器与假 Transport,带看门狗,回归时 5 秒
内失败而不是轮询 600 秒)。只回退接线(保留字段)时四条用例在 10 秒内全部变红:
PaddleOCR-VL Cloud 的 200ms 配置被忽略、WeKnoraCloud 读回 20m 而非配置值;
把两个源文件整体回退则用例引用的新配置面不存在,包测试无法编译。

* docs(docparser): document PaddleOCR-VL Cloud / WeKnoraCloud poll timeouts

Add the two new env vars to both website-docs timeout tables and pass them
through docker-compose.yml next to WEKNORA_MINERU_CLOUD_TIMEOUT.

* fix(docparser): WeKnoraCloud 轮询上限在调用方已有 deadline 时也生效

入库和临时文档调用 pollTaskResult 时 ctx 都带有 DocReader 调用超时,
原先的 !ok 条件让 WEKNORA_WEKNORACLOUD_TIMEOUT 在生产路径上被跳过。
改为始终叠加 WithTimeout,context 自动取内外较早的 deadline;
回归测试补上父 ctx 带更长 deadline 的场景。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 14:02:42 +08:00
linus-liu-web 9de0946e53
fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938)
anydoc 把内嵌图片就地渲染成 `![alt](images/image-N<ext>)`,`<ext>` 来自
asset_links.rs::extension_for;ImageResolver 再拿这个字符串去一张 map 里查,
键是 anydoc.ImageDir + Asset.Name,而 Name 由 backend_cgo.go::extensionFor 拼出。
查表是精确匹配,两边必须逐字符一致。

EMF/WMF 上两边并不一致:anydoc 对这两类 part 硬编码产出 image/emf 与
image/wmf(shared/assets.rs::media_type_for、shared/officeart.rs),Rust 表
落到 `.bin`,而 Go 侧去查平台 MIME 注册表——在装了 media-types 的 Debian
镜像上拿到 `.emf`/`.wmf`。链接与 ref 于是永远对不上,saveReferencedImage
找不到条目,图片字节从不落盘,且没有任何日志。

- asset_links.rs::extension_for 补 image/emf 与 image/wmf
- backend_cgo.go::extensionFor 补同样两项,并改成纯查表

去掉 mime.ExtensionsByType 是有意的:让扩展名取决于宿主的 /etc/mime.types
正是这个 bug 的成因,何况这张表本来就要和 Rust 的静态表逐字符对齐。未知
类型两边都落到 .bin,仍满足「不写出无扩展名 blob」的原意。

新增 TestEmbeddedVectorImageLinksResolve:在内存里构造一个含一张 EMF 与
一张 WMF 的最小 docx,转换后要求每一条 Markdown 图片引用都能在 resolver
建的 ImageRef 表里命中,并固定扩展名为 .emf/.wmf(避免两边一起退化成
.bin 也能通过)。

Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:57 +08:00
AI即世界 afc6cbd502
fix(milvus): validate connections and preserve gRPC resolver defaults (#3654)
* fix(docreader): preserve gRPC resolver defaults

* fix(milvus): validate connection with SDK handshake

* fix(docparser): preserve gRPC resolver defaults

* fix(milvus): satisfy connection check lint

---------

Co-authored-by: tunan <tunan@ahyk.com>
2026-10-08 00:48:45 +08:00
wizardchen 75fc0ad159 feat(docparser): upgrade AnyDoc to 0.2.4 and parse assets once 2026-09-30 15:25:01 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug ff49f9491d
feat(plugin): Host API with a key-value store, and document parsers from plugins (#3721)
* feat(plugin): Host API with plugin tokens and a key-value store
* feat(plugin): document parsers from code plugins
* docs(pluginsdk): parsers and the Host API
2026-09-26 14:45:39 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
xiaojunxiang 0bf105499b
fix(knowledge): reject invalid JSON uploads before async parse (#3360)
* fix(knowledge): reject invalid JSON uploads before async parse

Validate .json payloads at HTTP upload/replace time with the same gate used
by the simple JSON reader, so JSONC or truncated files fail with HTTP 400
instead of after document-process retries. CreateKnowledgeFromFile itself
skips the gate so datasource sync and IM still create a knowledge row that
fails in async parse.

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>

* fix(knowledge): localize JSON upload validation messages

The upload queue shows the backend message verbatim and the rest of the
upload handler speaks Chinese. Map ValidateJSONContent's errors (now
sentinels with unchanged text, still shared with the async JSON reader)
to Chinese messages at the upload/replace boundary.

---------

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:23:27 +08:00
wizardchen 27b0c84bfd fix(docparser): make the MinerU parse timeouts configurable
Self-hosted MinerU capped every parse at a hardcoded 1000s, which fired
before WEKNORA_DOCREADER_CALL_TIMEOUT and WEKNORA_DOCUMENT_PROCESS_TIMEOUT
could matter, so large PDFs always failed at 16m40s (#3091). MinerU Cloud
had the same problem with a hardcoded 600s polling deadline.

- WEKNORA_MINERU_TIMEOUT (default 1000s) bounds a self-hosted parse: the
  whole V1 job, or the legacy /file_parse request.
- WEKNORA_MINERU_CLOUD_TIMEOUT (default 600s) bounds how long a MinerU
  Cloud batch is polled.
- Both share the env parsing added for WEKNORA_PADDLEOCR_VL_TIMEOUT in
  #3537, now in helpers.go: invalid or non-positive values log a warning
  and fall back to the default.
2026-09-25 18:38:25 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
wizardchen e22a8367ef feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU
MinerU 4.0 removed the synchronous /file_parse endpoint that the mineru
engine relied on, so a self-hosted 4.0 server answered every parse with
404. The reader now probes GET /v1/health before each parse: a 200 selects
the V1 flow, 404/405 keeps the legacy /file_parse path (AMD and domestic
accelerators must stay on mineru<4), and a 503 error envelope surfaces the
server's model-preload failure instead of falling back.

The V1 flow uploads via /v1/uploads (sha256 dedup, API key only sent to
same-origin upload URLs), creates a zip-only parse job, polls with
exponential backoff (2s to 30s), cancels the job on timeout or caller
cancellation, and reads markdown.md plus images/ from the zip artifact.
The zip extraction is shared with the MinerU Cloud reader and now also
resolves <img src> references and percent-encoded paths.

New tenant settings: mineru_server_api_key (redacted in responses,
preserved on update) and mineru_tier (flash/basic/standard/advanced, empty
for the server default). mineru_parse_method maps to V1 ocr_mode. Backend,
vLLM URL, formula/table toggles and language only apply to MinerU 3.x and
are grouped as legacy options in the settings page. The connection check
also verifies the API key on V1 servers, since /v1/health is
unauthenticated.
2026-09-24 20:40:52 +08:00
Leon 402729341a
fix(docparser): make PaddleOCR-VL request timeout configurable (#3537)
* fix(docparser): make PaddleOCR-VL request timeout configurable

* style(docparser): wrap timeout configuration and test lines
2026-09-23 11:58:59 +08:00
Lvcy_cd cc82e75cb5 fix(parser): keep PDFs on builtin instead of defaulting to anydoc
anydoc has no document model for PDF: Convert force-disables asset
extraction, and AnydocReader falls back to builtin only when the whole
file yields no text. A born-digital PDF with figures and tables that
still yields a few characters is therefore reported as parsed, with its
images, tables and layout silently dropped.

35275d56 widened the type-level preference from ppt/pptx to every type
anydoc claims, and pdf is on that list. The predicate asks whether
anydoc can convert a type, not whether it is the better parser for it.
60207997, one commit earlier, recorded the original intent: "Types
without a fallback (pdf, docx, ...) never consult this".

The builtin PDF parser classifies pages individually and routes scanned
ones through OCR/VLM, so it stays the default for PDF. Every other
anydoc format is unchanged, and an explicit pdf -> anydoc engine rule
still wins.
2026-09-21 16:28:02 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
GUO Zhaogong 8d7298fb5d
fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092)
* fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders

KnowledgePostProcessService.Handle gathered "text-like" chunks through
chunkService.ListChunksByKnowledgeID, which is text-only by design, so the
image_ocr / image_caption chunks written by multimodal parsing were never
returned. For a scanned PDF the text chunks contain nothing but markdown
image links, so graph extraction ran on empty input: most calls failed
with "unexpected end of JSON input" and the rest echoed the few-shot
example from the extract config into the graph.

- Fetch the exact chunk types the post-process enriches via
  chunkRepo.ListChunksByKnowledgeIDAndTypes (text, image_ocr,
  image_caption); the service now receives interfaces.ChunkRepository
  through dig like knowledgeService already does
- Build a dedicated graphChunks list for graph extraction: skip
  image_caption chunks (they duplicate the OCR text) and chunks that
  contain only image links (chunkHasExtractableText)
- graphChunkCount, the enqueue loop and the log line use graphChunks;
  summary, wiki, auto-tag and question generation keep using textChunks
- Tests: fake ChunkRepository in the wiki-enqueue test, table test for
  chunkHasExtractableText

* fix(graph): skip figure OCR and harden empty-chunk detection

Reuse the CommonMark image scanner plus extractRealText so titles with
parentheses and <image> wrappers are not treated as prose. Only enqueue
OCR for graph extraction when the parent text chunk has no extractable
text, and skip link-only chunks for question generation as well.

* style(graph): wrap long post-process extract log line

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-08 19:23:13 +08:00
wizardchen 35275d5613 fix(parser): prefer anydoc for every type it converts when linked
ppt/pptx still fall back to markitdown when anydoc is not in the binary.
pdf/docx/xlsx and other anydoc formats now default to the in-process
converter as well; csv/md stay on the Go simple reader.
2026-08-31 14:58:15 +08:00
wizardchen 6020799792 fix(parser): prefer anydoc for ppt/pptx when the binding is linked
Type-level defaults still fall back to markitdown, but a linked anydoc
build should use the in-process converter first instead of always
shipping Office decks through docreader.
2026-08-31 14:58:15 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
lyingbug 7f0e1a91e9
fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界 (#2720)
* build(anydoc): 构建脚本从 Cargo.toml 读取待打补丁的 crate 版本

原先 0.1.8 硬编码在脚本的 6 处,升级要同时改动它们;现在统一从
`anydoc = "=X.Y.Z"` 这一处 pin 推导。

另外补两道拦截:patched-anydoc/ 的复用标记记录已打补丁的版本,避免
版本变更后沿用上一版的副本、最终在 cargo 的 patch 解析里报出难以定位
的错误;构建前校验 version.go 与该 pin 一致,因为这个常量会作为
anydoc_version 写进每一篇已解析文档的元数据。

* fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界

anydoc 0.1.9 只把 pdf-inspector 从 0.1.8 提到 1.14.2,但那一版含 9 个
修复,其中 7 个是给此前完全无上界的解析工作加限额:每页 Form XObject
展开、CID /W 区间、Encoding cidrange 与 ToUnicode bfrange 展开、
分配操作符前的内容流解码、detector 的 Tj/TJ 操作数回溯、以及不相交
矩形的表格聚类。它们全都落在 process_pdf_mem 这条路径上,而这是
anydoc 转换 PDF 的唯一入口,且只由上传文件的字节驱动。

这正是 guarded() 拦不住的那一类问题:耗尽 CPU 不会 panic,分配失败直接
abort。以 detector 回溯为例,缺少操作数的 TJ 会重扫整个内容流,因此
一串裸 `] TJ` 的开销是长度的平方:977 KB 的 PDF 要 26.7 秒,1.9 MB 要
111.9 秒;升级后分别是 5.8 毫秒和 11.0 毫秒。
TestDetectorLookbackStaysLinear 用 15 秒预算把这个上界钉住——它在
0.1.8 上会超时失败,在 0.1.9 上 0.01 秒通过。

另外 2 个修复改善提取质量,直接影响入库内容:Form XObject 内的文字
现在跟踪 text line matrix 并处理 T*/TL/'/"/Tc/Tw,嵌套表单文字不再
丢换行和字间距;小型大写字母(small-caps)的文字段会合并,不再被当成
额外的表格列而产出错位的 markdown 表格。

升级本身不需要改绑定代码:anydoc 0.1.8 与 0.1.9 的 src/ 字节级一致,
C ABI、文档模型和 GFM 序列化器都没变,6 项本地修改原样保留。依赖树也
只动了 3 行版本和 1 个 checksum,lopdf 仍是 0.42,cargo audit 结果与
升级前相同(仅 ttf-parser 未维护这一条已允许的警告)。

* docs(anydoc): 文档与注释中的耗时数字对齐到实测记录

表格与注释里的数字改为与随 PR 附上的实测日志同一次运行的结果,便于核对。
2026-08-16 12:30:55 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
wizardchen cb69764223 style(docparser): gofmt mineru converter tests 2026-08-13 18:51:27 +08:00
wizardchen 2308593ca0 fix(docparser): resolve MinerU results by upload filename stem
MinerU keys file_parse results by the uploaded filename stem, so preserving
real basenames requires dynamic response lookup instead of results.document.
Also append FileType when the basename has no extension.
2026-08-13 18:51:27 +08:00
shangfeng-liu 0eac47490b fix(docparser): preserve filename extension in MinerU multipart uploads 2026-08-13 18:51:27 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
shangfeng-liu 583b87b1b9 fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
zldkr 1168505eef fix(docparser): normalize PaddleOCR-VL Cloud HTML tables 2026-07-30 16:08:24 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
Ochan Kwon c9a342d965
fix(docparser): resolve remote images referenced by HTML img tags (#2355)
Remote http(s) images written as <img src="..."> reached no resolver.
Markdown images went through ResolveRemoteImages, HTML data-URIs and HTML
relative paths each had their own pass, but nothing claimed the remaining
combination, so those images were dropped without a log line and the
multimodal stage was skipped for documents that rely on HTML for layout.

ResolveRemoteImages now scans HTML img tags with a quoted src as a second
pass and rewrites only the src attribute value, leaving the tag intact
because the surrounding markup is why these documents use HTML at all.
The per-image work (whitelist, SSRF, download, icon filter, upload) moves
into one helper so the two syntaxes cannot drift apart on policy. Each
pass carries its own image budget; sharing one would let HTML images eat
into the number of Markdown images a document already gets resolved.

An attribute value is trimmed, HTML-unescaped and its scheme lowercased
before use, because HTML permits padding, spells a query separator &amp;
and tolerates any scheme case, while every fetcher downstream compares the
scheme byte-for-byte. Markdown targets get none of that normalization, so
no Markdown reference resolves differently than it does today. A
whitelisted host keeps serving the image from where it already lives, and
its span is rewritten with the normalized URL rather than the raw attribute
text: later stages both locate the image by searching the document for its
serving URL and fetch that same string, so the two have to agree.

The src attribute must now be preceded by whitespace, so a hyphenated
attribute name is no longer mistaken for it. Among HTML tags this cuts two
ways: one carrying both a hyphenated *-src and a real src now resolves the
real src instead of the placeholder, while one carrying only a hyphenated
*-src is no longer resolved at all. Recovering the latter needs an
attribute parser rather than a regex, and resolving a lazy-loading
placeholder while the real image goes unvisited is the worse of the two
failures. Both halves are pinned by tests.

Image-derived text also has to survive into the answer, which is why this
touches internal/searchutil. The chat enricher and the image_info filters
matched Markdown links only, so an HTML-only passage looked image-free:
every entry was dropped and the model received bare markup with the
caption and OCR text stripped, even though the analysis had run. Both now
recognise HTML img tags through one shared regex, so a tag the parser
stores can always be matched back. Both syntaxes are located against the
original passage and spliced in a single pass; scanning one regex over the
other's output let a metadata block that itself quotes an image reference
be rescanned and duplicated. The chat variant still enriches only images
present in the passage, so orphan entries stay out.

Known limits:

  - An unquoted src and a srcset-only tag are not matched, nor is a tag
    whose attribute values hold a bare > or a quoted src= of their own.
  - imgHTMLDataURI still matches src without a boundary, so a data-src
    holding a base64 placeholder is consumed as if it were the src and the
    whole tag replaced. Worth the same fix, but it changes what an existing
    lazy-loading document resolves, so it belongs on its own.
  - A reference shown as a documentation example is rewritten like any
    other. Recognising code spans needs a Markdown parser; a hand-rolled
    scanner risks the far worse failure of treating a document's real
    screenshots as examples.
  - Rewriting a whitelisted src writes the unescaped URL back into an HTML
    attribute, so a query parameter whose name spells an entity renders
    differently than the author wrote it. Keeping the encoded form would
    mean unescaping on every matcher instead.
  - Image text reaches the model only when the whole reference falls inside
    the retrieved window; an img tag is several times longer than a
    Markdown link and straddles a chunk boundary more often.
  - A reference repeated in one document is fetched once per occurrence,
    which is what gives each chunk its own analysed copy.
  - Nothing bounds how long one document's fetches may take in total.
  - Parent-context pruning removes only Markdown images.

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-07-28 21:08:31 +08:00
wizardchen 7099a7e181 fix(security): redact IM list credentials and harden MinerU SSRF
Stop returning bot credentials from Viewer-accessible IM channel list
endpoints and validate self-hosted MinerU URLs at config save and fetch time.
2026-07-02 17:58:20 +08:00
wizardchen 74c845a1f4 fix(docparser): normalize PaddleOCR-VL HTML tables (#1725)
PaddleOCR-VL renders tables as HTML with per-cell text-align styles.
These waste tokens on layout markup and bypass the chunker's
table-protection logic (which only recognizes Markdown pipe tables),
causing large tables to be split mid-row.

Convert inline HTML <table> blocks to GFM Markdown tables when possible,
and strip presentational attributes (style/class/align/...) for tables
using rowspan/colspan that Markdown cannot express.

Refs: https://github.com/Tencent/WeKnora/issues/1725
2026-06-22 17:31:26 +08:00
wizardchen ecc27e9b0f fix(docparser): rename close variable to avoid shadowing builtin
The angle-bracket image target parser declared a local variable named
`close`, which shadows Go's built-in `close` function and is flagged by
revive's redefines-builtin-id rule, breaking `make lint`. Rename it to
`closeIdx`.
2026-06-22 15:57:41 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
ochan.kwon e9980c6011 fix: deep-copy stored files and images when cloning a knowledge base
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.

Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.

Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.

Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
2026-06-03 14:45:59 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 7aca1017db fix(docparser): address review feedback on PR #1404
Three follow-up fixes on top of the MinerU markdown preservation work:

- Stop applying normalizeMinerUMarkdown inside ResolveAndStore. The
  helper is already called by MinerUReader.Read, and ResolveAndStore is
  shared by every parser (docreader, session attachments, ...). Running
  the heading/image unescape regexes globally would silently rewrite
  content (including inside fenced code blocks) for non-MinerU sources.

- Recognize MinerU image references whose path contains spaces, e.g.
  "images/第 1 页.jpg". The previous regex used in
  extractImageRefsFromContent disallowed whitespace in the URL group,
  so such images were never matched and never persisted. Use a
  whitespace-tolerant pattern aligned with ResolveAndStore's own
  imgPattern.

- Deduplicate uploads when the same MinerU image is referenced under
  multiple path forms (e.g. "images/foo.png" vs "./images/foo.png").
  saveReferencedImage now caches by ref.Filename in addition to the
  raw ref path, so the second variant reuses the previously stored
  ServingURL instead of writing the same bytes to object storage
  again.

Tests added:
- TestProcessImagesMatchesPathsWithSpaces
- TestResolveAndStoreDedupsSameImageRefVariants
2026-05-21 11:42:56 +08:00
MidnightSun 6210f44fb6 fix(docparser): preserve MinerU markdown and persist relative images
MinerU already returns markdown with embedded HTML blocks, but the current\nreader runs the whole document back through html-to-markdown. That\nsecond conversion escapes valid headings and image syntax, so chunk\nprofiling sees plain text instead of markdown structure and relative\nimage references stop matching the storage pipeline.\n\nKeep MinerU output in its original markdown form and only apply narrow\ncompatibility normalization for the specific over-escaped patterns we\nactually need to recover. The converter now matches image refs by the\npaths that are really present in markdown or embedded HTML instead of\nassuming a single images/<name> form.\n\nExtend ImageResolver so relative HTML <img src=...> references share the\nsame storage rewrite path as markdown images, deduplicate repeated saves,\nand keep the frontend sanitizer compatible with MinerU's details/summary\nblocks. Add focused docparser tests that cover escaped markdown repair,\nvariant image path matching, and relative HTML image persistence.
2026-05-21 11:39:57 +08:00
knight 513e589494 feat: add vLLM server URL configuration for MinerU
Add support for configuring vLLM server URL when using vlm-http-client or hybrid-http-client backend in MinerU.

Changes:
- internal/types/tenant.go: Add MinerUVLMServerURL field to ParserEngineConfig
- internal/infrastructure/docparser/mineru_converter.go: Pass server_url to MinerU API when backend is vlm-http-client or hybrid-http-client
- frontend/src/api/system/index.ts: Add TypeScript type definition
- frontend/src/views/settings/ParserEngineSettings.vue: Add vLLM server URL input field
- frontend/src/i18n/locales/*.ts: Add translations (zh-CN, en-US, ko-KR)
2026-05-20 16:33:12 +08:00