Commit Graph

75 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
linus-liu-web 9de0946e53
fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938)
anydoc 把内嵌图片就地渲染成 `![alt](images/image-N<ext>)`,`<ext>` 来自
asset_links.rs::extension_for;ImageResolver 再拿这个字符串去一张 map 里查,
键是 anydoc.ImageDir + Asset.Name,而 Name 由 backend_cgo.go::extensionFor 拼出。
查表是精确匹配,两边必须逐字符一致。

EMF/WMF 上两边并不一致:anydoc 对这两类 part 硬编码产出 image/emf 与
image/wmf(shared/assets.rs::media_type_for、shared/officeart.rs),Rust 表
落到 `.bin`,而 Go 侧去查平台 MIME 注册表——在装了 media-types 的 Debian
镜像上拿到 `.emf`/`.wmf`。链接与 ref 于是永远对不上,saveReferencedImage
找不到条目,图片字节从不落盘,且没有任何日志。

- asset_links.rs::extension_for 补 image/emf 与 image/wmf
- backend_cgo.go::extensionFor 补同样两项,并改成纯查表

去掉 mime.ExtensionsByType 是有意的:让扩展名取决于宿主的 /etc/mime.types
正是这个 bug 的成因,何况这张表本来就要和 Rust 的静态表逐字符对齐。未知
类型两边都落到 .bin,仍满足「不写出无扩展名 blob」的原意。

新增 TestEmbeddedVectorImageLinksResolve:在内存里构造一个含一张 EMF 与
一张 WMF 的最小 docx,转换后要求每一条 Markdown 图片引用都能在 resolver
建的 ImageRef 表里命中,并固定扩展名为 .emf/.wmf(避免两边一起退化成
.bin 也能通过)。

Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:57 +08:00
AI即世界 afc6cbd502
fix(milvus): validate connections and preserve gRPC resolver defaults (#3654)
* fix(docreader): preserve gRPC resolver defaults

* fix(milvus): validate connection with SDK handshake

* fix(docparser): preserve gRPC resolver defaults

* fix(milvus): satisfy connection check lint

---------

Co-authored-by: tunan <tunan@ahyk.com>
2026-10-08 00:48:45 +08:00
wizardchen 75fc0ad159 feat(docparser): upgrade AnyDoc to 0.2.4 and parse assets once 2026-09-30 15:25:01 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug ff49f9491d
feat(plugin): Host API with a key-value store, and document parsers from plugins (#3721)
* feat(plugin): Host API with plugin tokens and a key-value store
* feat(plugin): document parsers from code plugins
* docs(pluginsdk): parsers and the Host API
2026-09-26 14:45:39 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
xiaojunxiang 0bf105499b
fix(knowledge): reject invalid JSON uploads before async parse (#3360)
* fix(knowledge): reject invalid JSON uploads before async parse

Validate .json payloads at HTTP upload/replace time with the same gate used
by the simple JSON reader, so JSONC or truncated files fail with HTTP 400
instead of after document-process retries. CreateKnowledgeFromFile itself
skips the gate so datasource sync and IM still create a knowledge row that
fails in async parse.

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>

* fix(knowledge): localize JSON upload validation messages

The upload queue shows the backend message verbatim and the rest of the
upload handler speaks Chinese. Map ValidateJSONContent's errors (now
sentinels with unchanged text, still shared with the async JSON reader)
to Chinese messages at the upload/replace boundary.

---------

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:23:27 +08:00
wizardchen 27b0c84bfd fix(docparser): make the MinerU parse timeouts configurable
Self-hosted MinerU capped every parse at a hardcoded 1000s, which fired
before WEKNORA_DOCREADER_CALL_TIMEOUT and WEKNORA_DOCUMENT_PROCESS_TIMEOUT
could matter, so large PDFs always failed at 16m40s (#3091). MinerU Cloud
had the same problem with a hardcoded 600s polling deadline.

- WEKNORA_MINERU_TIMEOUT (default 1000s) bounds a self-hosted parse: the
  whole V1 job, or the legacy /file_parse request.
- WEKNORA_MINERU_CLOUD_TIMEOUT (default 600s) bounds how long a MinerU
  Cloud batch is polled.
- Both share the env parsing added for WEKNORA_PADDLEOCR_VL_TIMEOUT in
  #3537, now in helpers.go: invalid or non-positive values log a warning
  and fall back to the default.
2026-09-25 18:38:25 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
wizardchen e22a8367ef feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU
MinerU 4.0 removed the synchronous /file_parse endpoint that the mineru
engine relied on, so a self-hosted 4.0 server answered every parse with
404. The reader now probes GET /v1/health before each parse: a 200 selects
the V1 flow, 404/405 keeps the legacy /file_parse path (AMD and domestic
accelerators must stay on mineru<4), and a 503 error envelope surfaces the
server's model-preload failure instead of falling back.

The V1 flow uploads via /v1/uploads (sha256 dedup, API key only sent to
same-origin upload URLs), creates a zip-only parse job, polls with
exponential backoff (2s to 30s), cancels the job on timeout or caller
cancellation, and reads markdown.md plus images/ from the zip artifact.
The zip extraction is shared with the MinerU Cloud reader and now also
resolves <img src> references and percent-encoded paths.

New tenant settings: mineru_server_api_key (redacted in responses,
preserved on update) and mineru_tier (flash/basic/standard/advanced, empty
for the server default). mineru_parse_method maps to V1 ocr_mode. Backend,
vLLM URL, formula/table toggles and language only apply to MinerU 3.x and
are grouped as legacy options in the settings page. The connection check
also verifies the API key on V1 servers, since /v1/health is
unauthenticated.
2026-09-24 20:40:52 +08:00
Leon 402729341a
fix(docparser): make PaddleOCR-VL request timeout configurable (#3537)
* fix(docparser): make PaddleOCR-VL request timeout configurable

* style(docparser): wrap timeout configuration and test lines
2026-09-23 11:58:59 +08:00
Lvcy_cd cc82e75cb5 fix(parser): keep PDFs on builtin instead of defaulting to anydoc
anydoc has no document model for PDF: Convert force-disables asset
extraction, and AnydocReader falls back to builtin only when the whole
file yields no text. A born-digital PDF with figures and tables that
still yields a few characters is therefore reported as parsed, with its
images, tables and layout silently dropped.

35275d56 widened the type-level preference from ppt/pptx to every type
anydoc claims, and pdf is on that list. The predicate asks whether
anydoc can convert a type, not whether it is the better parser for it.
60207997, one commit earlier, recorded the original intent: "Types
without a fallback (pdf, docx, ...) never consult this".

The builtin PDF parser classifies pages individually and routes scanned
ones through OCR/VLM, so it stays the default for PDF. Every other
anydoc format is unchanged, and an explicit pdf -> anydoc engine rule
still wins.
2026-09-21 16:28:02 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
GUO Zhaogong 8d7298fb5d
fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092)
* fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders

KnowledgePostProcessService.Handle gathered "text-like" chunks through
chunkService.ListChunksByKnowledgeID, which is text-only by design, so the
image_ocr / image_caption chunks written by multimodal parsing were never
returned. For a scanned PDF the text chunks contain nothing but markdown
image links, so graph extraction ran on empty input: most calls failed
with "unexpected end of JSON input" and the rest echoed the few-shot
example from the extract config into the graph.

- Fetch the exact chunk types the post-process enriches via
  chunkRepo.ListChunksByKnowledgeIDAndTypes (text, image_ocr,
  image_caption); the service now receives interfaces.ChunkRepository
  through dig like knowledgeService already does
- Build a dedicated graphChunks list for graph extraction: skip
  image_caption chunks (they duplicate the OCR text) and chunks that
  contain only image links (chunkHasExtractableText)
- graphChunkCount, the enqueue loop and the log line use graphChunks;
  summary, wiki, auto-tag and question generation keep using textChunks
- Tests: fake ChunkRepository in the wiki-enqueue test, table test for
  chunkHasExtractableText

* fix(graph): skip figure OCR and harden empty-chunk detection

Reuse the CommonMark image scanner plus extractRealText so titles with
parentheses and <image> wrappers are not treated as prose. Only enqueue
OCR for graph extraction when the parent text chunk has no extractable
text, and skip link-only chunks for question generation as well.

* style(graph): wrap long post-process extract log line

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-08 19:23:13 +08:00
wizardchen 35275d5613 fix(parser): prefer anydoc for every type it converts when linked
ppt/pptx still fall back to markitdown when anydoc is not in the binary.
pdf/docx/xlsx and other anydoc formats now default to the in-process
converter as well; csv/md stay on the Go simple reader.
2026-08-31 14:58:15 +08:00
wizardchen 6020799792 fix(parser): prefer anydoc for ppt/pptx when the binding is linked
Type-level defaults still fall back to markitdown, but a linked anydoc
build should use the in-process converter first instead of always
shipping Office decks through docreader.
2026-08-31 14:58:15 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
lyingbug 7f0e1a91e9
fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界 (#2720)
* build(anydoc): 构建脚本从 Cargo.toml 读取待打补丁的 crate 版本

原先 0.1.8 硬编码在脚本的 6 处,升级要同时改动它们;现在统一从
`anydoc = "=X.Y.Z"` 这一处 pin 推导。

另外补两道拦截:patched-anydoc/ 的复用标记记录已打补丁的版本,避免
版本变更后沿用上一版的副本、最终在 cargo 的 patch 解析里报出难以定位
的错误;构建前校验 version.go 与该 pin 一致,因为这个常量会作为
anydoc_version 写进每一篇已解析文档的元数据。

* fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界

anydoc 0.1.9 只把 pdf-inspector 从 0.1.8 提到 1.14.2,但那一版含 9 个
修复,其中 7 个是给此前完全无上界的解析工作加限额:每页 Form XObject
展开、CID /W 区间、Encoding cidrange 与 ToUnicode bfrange 展开、
分配操作符前的内容流解码、detector 的 Tj/TJ 操作数回溯、以及不相交
矩形的表格聚类。它们全都落在 process_pdf_mem 这条路径上,而这是
anydoc 转换 PDF 的唯一入口,且只由上传文件的字节驱动。

这正是 guarded() 拦不住的那一类问题:耗尽 CPU 不会 panic,分配失败直接
abort。以 detector 回溯为例,缺少操作数的 TJ 会重扫整个内容流,因此
一串裸 `] TJ` 的开销是长度的平方:977 KB 的 PDF 要 26.7 秒,1.9 MB 要
111.9 秒;升级后分别是 5.8 毫秒和 11.0 毫秒。
TestDetectorLookbackStaysLinear 用 15 秒预算把这个上界钉住——它在
0.1.8 上会超时失败,在 0.1.9 上 0.01 秒通过。

另外 2 个修复改善提取质量,直接影响入库内容:Form XObject 内的文字
现在跟踪 text line matrix 并处理 T*/TL/'/"/Tc/Tw,嵌套表单文字不再
丢换行和字间距;小型大写字母(small-caps)的文字段会合并,不再被当成
额外的表格列而产出错位的 markdown 表格。

升级本身不需要改绑定代码:anydoc 0.1.8 与 0.1.9 的 src/ 字节级一致,
C ABI、文档模型和 GFM 序列化器都没变,6 项本地修改原样保留。依赖树也
只动了 3 行版本和 1 个 checksum,lopdf 仍是 0.42,cargo audit 结果与
升级前相同(仅 ttf-parser 未维护这一条已允许的警告)。

* docs(anydoc): 文档与注释中的耗时数字对齐到实测记录

表格与注释里的数字改为与随 PR 附上的实测日志同一次运行的结果,便于核对。
2026-08-16 12:30:55 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
wizardchen cb69764223 style(docparser): gofmt mineru converter tests 2026-08-13 18:51:27 +08:00
wizardchen 2308593ca0 fix(docparser): resolve MinerU results by upload filename stem
MinerU keys file_parse results by the uploaded filename stem, so preserving
real basenames requires dynamic response lookup instead of results.document.
Also append FileType when the basename has no extension.
2026-08-13 18:51:27 +08:00
shangfeng-liu 0eac47490b fix(docparser): preserve filename extension in MinerU multipart uploads 2026-08-13 18:51:27 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
shangfeng-liu 583b87b1b9 fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
zldkr 1168505eef fix(docparser): normalize PaddleOCR-VL Cloud HTML tables 2026-07-30 16:08:24 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
Ochan Kwon c9a342d965
fix(docparser): resolve remote images referenced by HTML img tags (#2355)
Remote http(s) images written as <img src="..."> reached no resolver.
Markdown images went through ResolveRemoteImages, HTML data-URIs and HTML
relative paths each had their own pass, but nothing claimed the remaining
combination, so those images were dropped without a log line and the
multimodal stage was skipped for documents that rely on HTML for layout.

ResolveRemoteImages now scans HTML img tags with a quoted src as a second
pass and rewrites only the src attribute value, leaving the tag intact
because the surrounding markup is why these documents use HTML at all.
The per-image work (whitelist, SSRF, download, icon filter, upload) moves
into one helper so the two syntaxes cannot drift apart on policy. Each
pass carries its own image budget; sharing one would let HTML images eat
into the number of Markdown images a document already gets resolved.

An attribute value is trimmed, HTML-unescaped and its scheme lowercased
before use, because HTML permits padding, spells a query separator &amp;
and tolerates any scheme case, while every fetcher downstream compares the
scheme byte-for-byte. Markdown targets get none of that normalization, so
no Markdown reference resolves differently than it does today. A
whitelisted host keeps serving the image from where it already lives, and
its span is rewritten with the normalized URL rather than the raw attribute
text: later stages both locate the image by searching the document for its
serving URL and fetch that same string, so the two have to agree.

The src attribute must now be preceded by whitespace, so a hyphenated
attribute name is no longer mistaken for it. Among HTML tags this cuts two
ways: one carrying both a hyphenated *-src and a real src now resolves the
real src instead of the placeholder, while one carrying only a hyphenated
*-src is no longer resolved at all. Recovering the latter needs an
attribute parser rather than a regex, and resolving a lazy-loading
placeholder while the real image goes unvisited is the worse of the two
failures. Both halves are pinned by tests.

Image-derived text also has to survive into the answer, which is why this
touches internal/searchutil. The chat enricher and the image_info filters
matched Markdown links only, so an HTML-only passage looked image-free:
every entry was dropped and the model received bare markup with the
caption and OCR text stripped, even though the analysis had run. Both now
recognise HTML img tags through one shared regex, so a tag the parser
stores can always be matched back. Both syntaxes are located against the
original passage and spliced in a single pass; scanning one regex over the
other's output let a metadata block that itself quotes an image reference
be rescanned and duplicated. The chat variant still enriches only images
present in the passage, so orphan entries stay out.

Known limits:

  - An unquoted src and a srcset-only tag are not matched, nor is a tag
    whose attribute values hold a bare > or a quoted src= of their own.
  - imgHTMLDataURI still matches src without a boundary, so a data-src
    holding a base64 placeholder is consumed as if it were the src and the
    whole tag replaced. Worth the same fix, but it changes what an existing
    lazy-loading document resolves, so it belongs on its own.
  - A reference shown as a documentation example is rewritten like any
    other. Recognising code spans needs a Markdown parser; a hand-rolled
    scanner risks the far worse failure of treating a document's real
    screenshots as examples.
  - Rewriting a whitelisted src writes the unescaped URL back into an HTML
    attribute, so a query parameter whose name spells an entity renders
    differently than the author wrote it. Keeping the encoded form would
    mean unescaping on every matcher instead.
  - Image text reaches the model only when the whole reference falls inside
    the retrieved window; an img tag is several times longer than a
    Markdown link and straddles a chunk boundary more often.
  - A reference repeated in one document is fetched once per occurrence,
    which is what gives each chunk its own analysed copy.
  - Nothing bounds how long one document's fetches may take in total.
  - Parent-context pruning removes only Markdown images.

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-07-28 21:08:31 +08:00
wizardchen 7099a7e181 fix(security): redact IM list credentials and harden MinerU SSRF
Stop returning bot credentials from Viewer-accessible IM channel list
endpoints and validate self-hosted MinerU URLs at config save and fetch time.
2026-07-02 17:58:20 +08:00
wizardchen 74c845a1f4 fix(docparser): normalize PaddleOCR-VL HTML tables (#1725)
PaddleOCR-VL renders tables as HTML with per-cell text-align styles.
These waste tokens on layout markup and bypass the chunker's
table-protection logic (which only recognizes Markdown pipe tables),
causing large tables to be split mid-row.

Convert inline HTML <table> blocks to GFM Markdown tables when possible,
and strip presentational attributes (style/class/align/...) for tables
using rowspan/colspan that Markdown cannot express.

Refs: https://github.com/Tencent/WeKnora/issues/1725
2026-06-22 17:31:26 +08:00
wizardchen ecc27e9b0f fix(docparser): rename close variable to avoid shadowing builtin
The angle-bracket image target parser declared a local variable named
`close`, which shadows Go's built-in `close` function and is flagged by
revive's redefines-builtin-id rule, breaking `make lint`. Rename it to
`closeIdx`.
2026-06-22 15:57:41 +08:00
FFFFFFpy a6a20175e5
fix: preserve MHTML image resolution and Markdown structure (#1743)
* fix(docreader): preserve MHTML blocks and resolve titled images

* fix(docreader): harden MHTML markdown image resolution

* fix(docreader): preserve markdown hard break spaces

---------

Co-authored-by: FFFFFFpy <bushiyan@outlook.com>
2026-06-22 15:53:54 +08:00
FFFFFFpy e21a5da422 feat: add EPUB and MHTML document support 2026-06-16 11:25:27 +08:00
ochan.kwon e9980c6011 fix: deep-copy stored files and images when cloning a knowledge base
Cloning a knowledge base previously copied only the storage path strings
(knowledge.FilePath and chunk.ImageInfo.URL), so the source and the clone
shared the same physical objects in the storage backend. Once the original
file and extracted images are deleted on source removal, the clone is left
with dangling references and its document and images become unreadable —
data loss that occurs even for same-store clones.

Add a CopyFile primitive to the FileService interface and implement it in
every backend: server-side CopyObject on the object stores
(s3/obs/cos/oss/tos/ks3/minio), io.Copy on local, and a no-op on dummy.
Destinations use the knowledge-owned layout and reuse the existing
path/object-key guards; a sentinel ErrCrossBackendCopy is returned when the
source scheme does not match the backend.

Use CopyFile to deep-copy the document file in cloneKnowledge and the
extracted images in CloneChunk and cloneFAQKnowledgeBase via a shared
cloneChunkImageInfo helper that deduplicates identical image URLs per clone
and rewrites them to the new objects. Copied objects are cleaned up
best-effort if a clone fails partway through. A clone-time preflight rejects
cloning into a target bound to a different storage backend when the tenant
pins providers via StorageEngineConfig.

Adds unit tests for local CopyFile (independent copy survives source
deletion, traversal rejection, cross-backend rejection), cloneChunkImageInfo
(empty/multi/dedup/parse-failure/OriginalURL handling), and the storage
provider preflight.
2026-06-03 14:45:59 +08:00
wizardchen bbd3f6324a refactor(parser): reorganize Markdown parser and enhance gRPC document reading
- Moved the _SEPARATOR_CELL regex definition to a more appropriate location in the Markdown parser.
- Implemented a fallback mechanism in the gRPC document reader to handle cases where the ReadStream RPC is unimplemented, ensuring compatibility with older versions.
- Added a readUnary method to maintain backward compatibility with the legacy unary Read RPC.
- Improved cancellation handling in the MinerUCloud and PaddleOCR-VL readers to prevent excessive API calls during context cancellation.
2026-06-03 12:29:13 +08:00
wizardchen ef1047bf67 feat(parser): add OpenDataLoader, PaddleOCR-VL engines, and parser improvements
Introduce opendataloader and PaddleOCR-VL parser engines with tenant-level
settings UI, replace liteparse, and harden Excel/PPT/Markdown parsing.
Optional odl-hybrid sidecar stays local-build only and is excluded from
default dev-start and full profiles.
2026-06-03 12:29:13 +08:00
wizardchen 7b1bb1054f feat(docreader): speed up scanned-PDF parsing, stream image results, isolate heavy async queues
Large scanned PDFs (hundreds of pages) were slow and fragile end-to-end.
This change addresses the parse, transport, and task-scheduling layers:

docreader (parse + transport):
- Parallelize per-page scanned rendering across processes (forkserver/fork),
  with serial fallback. ~4-7x faster on large scanned PDFs; pdfium is not
  thread-safe so we fan out across processes. Configurable via
  DOCREADER_PDF_RENDER_PARALLELISM.
- Add server-streaming ReadStream RPC: emit one meta frame then one frame per
  image, so documents with many page images are no longer capped by the unary
  gRPC message-size limit (a 874-page PDF produced ~193MiB of images, far over
  the 50MB cap) and memory is bounded on both ends. Unary Read is kept for
  backward compatibility; the Go production reader switches to ReadStream.

VLM:
- Make the VLM HTTP timeout configurable (VLM_HTTP_TIMEOUT_SECONDS) and raise
  the default 90s -> 180s so dense scanned-page OCR does not time out with
  "context deadline exceeded".

Async task queues:
- Isolate high-volume, model-heavy fan-out tasks into dedicated asynq queues so
  a single large document cannot saturate the shared worker pool and block
  user-facing document parsing:
    image:multimodal  -> "multimodal"
    chunk:extract     -> "graph"
    question:generation -> "question"
- Register the new queues in the server weight map and the cancel inspector's
  scanned-queue set (so cancelling a knowledge still purges its pending tasks).
2026-06-03 12:29:13 +08:00
wizardchen 7aca1017db fix(docparser): address review feedback on PR #1404
Three follow-up fixes on top of the MinerU markdown preservation work:

- Stop applying normalizeMinerUMarkdown inside ResolveAndStore. The
  helper is already called by MinerUReader.Read, and ResolveAndStore is
  shared by every parser (docreader, session attachments, ...). Running
  the heading/image unescape regexes globally would silently rewrite
  content (including inside fenced code blocks) for non-MinerU sources.

- Recognize MinerU image references whose path contains spaces, e.g.
  "images/第 1 页.jpg". The previous regex used in
  extractImageRefsFromContent disallowed whitespace in the URL group,
  so such images were never matched and never persisted. Use a
  whitespace-tolerant pattern aligned with ResolveAndStore's own
  imgPattern.

- Deduplicate uploads when the same MinerU image is referenced under
  multiple path forms (e.g. "images/foo.png" vs "./images/foo.png").
  saveReferencedImage now caches by ref.Filename in addition to the
  raw ref path, so the second variant reuses the previously stored
  ServingURL instead of writing the same bytes to object storage
  again.

Tests added:
- TestProcessImagesMatchesPathsWithSpaces
- TestResolveAndStoreDedupsSameImageRefVariants
2026-05-21 11:42:56 +08:00
MidnightSun 6210f44fb6 fix(docparser): preserve MinerU markdown and persist relative images
MinerU already returns markdown with embedded HTML blocks, but the current\nreader runs the whole document back through html-to-markdown. That\nsecond conversion escapes valid headings and image syntax, so chunk\nprofiling sees plain text instead of markdown structure and relative\nimage references stop matching the storage pipeline.\n\nKeep MinerU output in its original markdown form and only apply narrow\ncompatibility normalization for the specific over-escaped patterns we\nactually need to recover. The converter now matches image refs by the\npaths that are really present in markdown or embedded HTML instead of\nassuming a single images/<name> form.\n\nExtend ImageResolver so relative HTML <img src=...> references share the\nsame storage rewrite path as markdown images, deduplicate repeated saves,\nand keep the frontend sanitizer compatible with MinerU's details/summary\nblocks. Add focused docparser tests that cover escaped markdown repair,\nvariant image path matching, and relative HTML image persistence.
2026-05-21 11:39:57 +08:00
knight 513e589494 feat: add vLLM server URL configuration for MinerU
Add support for configuring vLLM server URL when using vlm-http-client or hybrid-http-client backend in MinerU.

Changes:
- internal/types/tenant.go: Add MinerUVLMServerURL field to ParserEngineConfig
- internal/infrastructure/docparser/mineru_converter.go: Pass server_url to MinerU API when backend is vlm-http-client or hybrid-http-client
- frontend/src/api/system/index.ts: Add TypeScript type definition
- frontend/src/views/settings/ParserEngineSettings.vue: Add vLLM server URL input field
- frontend/src/i18n/locales/*.ts: Add translations (zh-CN, en-US, ko-KR)
2026-05-20 16:33:12 +08:00
yuheng.huang 00d39e006d fix(docparser): unescape MinerU markdown image syntax html-to-markdown over-escapes to on MinerU'smixed-Markdown+HTML output, breaking downstream image extraction. Restore the canonical image syntax after conversion so images are persisted. 2026-05-18 21:27:53 +08:00
Li Xianggang c19d3543c8 feat(url): 支持docreader不上传替换白名单url的图片 2026-05-18 19:39:30 +08:00
Li Xianggang b3dbf86bf5 feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
wizardchen a3411899cf fix(docreader/auth): harden gRPC TLS/Token rollout from #1359
Follow-up to #1359. Addresses a set of correctness and security gaps in
the initial docreader auth implementation.

- docker-compose: inject GRPC_TLS_*/GRPC_TLS_SERVER_NAME/GRPC_AUTH_TOKEN
  into the WeKnora-app service. Without this the Go client never saw the
  knobs, so enabling token auth on the server broke every RPC.
- client: bind tokenAuth.RequireTransportSecurity() to TLSEnabled so a
  bearer token cannot be sent over an insecure channel once TLS is on.
- server: load_tls_credentials now raises TLSConfigError on misconfig
  (cert/key missing, file unreadable, mTLS without CA); main.py exits 1
  instead of silently downgrading to insecure.
- server: replace endswith("/Check"|"/Watch") health bypass with exact
  match against /grpc.health.v1.Health/{Check,Watch}.
- server: compare tokens with hmac.compare_digest, warn on tokens < 16B.
- server: AuthInterceptor now returns an abort handler matching the
  original RPC kind (unary/stream) and uses context.abort, so streaming
  RPCs surface UNAUTHENTICATED instead of INTERNAL.
- internal/infrastructure/docparser/grpc_parser.go: drop the duplicated
  TLS/tokenAuth block and reuse docreader/client.LoadAuthConfigFromEnv +
  BuildDialOptions. Single source of truth for client-side auth.
- Add GRPC_TLS_SERVER_NAME (client SNI override) and
  GRPC_MTLS_REQUIRE_CLIENT_CERT (server explicit mTLS toggle); document
  the differing CA semantics between client and server in .env*.example.
- Reject half-configured client mTLS (cert XOR key) loudly.
- Fix missing trailing newline in .env.lite.example.

Verified locally: go build ./... and go vet ./... clean; auth.py
fail-fast / token paths smoke-tested.
2026-05-16 21:45:56 +08:00
Li Xianggang 5a02e22f52 feat(auth): 连接docreader支持auth 2026-05-16 21:31:17 +08:00
wizardchen d55b52652c fix(docparser): preserve standalone image uploads from icon filter
When the uploaded file is itself an image, the image reference now carries
an IsOriginal flag so ResolveAndStore skips the small-icon size filter.
Otherwise small standalone images (e.g. avatars below 64x64) were silently
dropped before reaching multimodal OCR/caption processing.
2026-04-29 11:58:02 +08:00