Commit Graph

135 Commits (de2dfd3034a7b506fa269910401bbdcbd4efc71d)

Author SHA1 Message Date
linus-liu-web 9de0946e53
fix(docparser): anydoc 内嵌 EMF/WMF 图片的扩展名与 Rust 侧对齐 (#3938)
anydoc 把内嵌图片就地渲染成 `![alt](images/image-N<ext>)`,`<ext>` 来自
asset_links.rs::extension_for;ImageResolver 再拿这个字符串去一张 map 里查,
键是 anydoc.ImageDir + Asset.Name,而 Name 由 backend_cgo.go::extensionFor 拼出。
查表是精确匹配,两边必须逐字符一致。

EMF/WMF 上两边并不一致:anydoc 对这两类 part 硬编码产出 image/emf 与
image/wmf(shared/assets.rs::media_type_for、shared/officeart.rs),Rust 表
落到 `.bin`,而 Go 侧去查平台 MIME 注册表——在装了 media-types 的 Debian
镜像上拿到 `.emf`/`.wmf`。链接与 ref 于是永远对不上,saveReferencedImage
找不到条目,图片字节从不落盘,且没有任何日志。

- asset_links.rs::extension_for 补 image/emf 与 image/wmf
- backend_cgo.go::extensionFor 补同样两项,并改成纯查表

去掉 mime.ExtensionsByType 是有意的:让扩展名取决于宿主的 /etc/mime.types
正是这个 bug 的成因,何况这张表本来就要和 Rust 的静态表逐字符对齐。未知
类型两边都落到 .bin,仍满足「不写出无扩展名 blob」的原意。

新增 TestEmbeddedVectorImageLinksResolve:在内存里构造一个含一张 EMF 与
一张 WMF 的最小 docx,转换后要求每一条 Markdown 图片引用都能在 resolver
建的 ImageRef 表里命中,并固定扩展名为 .emf/.wmf(避免两边一起退化成
.bin 也能通过)。

Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-08 01:21:57 +08:00
Lukas 98feef38de
fix(chunker): count an escaped pipe as cell text when matching table widths (#3979)
The header tracker compares a row's column count with the active table
header's to tell a new table from more rows of the same one.
tableRowColumnCount split rows on every `|`, so a cell holding an escaped
pipe (`\|`, which GFM writers such as anydoc emit for a pipe in a cell)
counted as one more column. The tracker took that row for the start of a
new table and dropped the header, so the chunks after it no longer began
with the column names.

Split rows on unescaped pipes only. A pipe after an odd run of
backslashes is text inside the cell, as in GFM.
2026-10-08 01:21:16 +08:00
Yeager0204 e915adb147
fix(chunker): preserve source offsets for table child chunks (#3969)
Co-authored-by: kwsYegar <331533331+kwsYegar@users.noreply.github.com>
2026-10-08 01:21:08 +08:00
AI即世界 afc6cbd502
fix(milvus): validate connections and preserve gRPC resolver defaults (#3654)
* fix(docreader): preserve gRPC resolver defaults

* fix(milvus): validate connection with SDK handshake

* fix(docparser): preserve gRPC resolver defaults

* fix(milvus): satisfy connection check lint

---------

Co-authored-by: tunan <tunan@ahyk.com>
2026-10-08 00:48:45 +08:00
wizardchen 75fc0ad159 feat(docparser): upgrade AnyDoc to 0.2.4 and parse assets once 2026-09-30 15:25:01 +08:00
lyingbug 9114e4e4f9
fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794)
* fix(citations): verify source locations and recover document previews

* chore: exclude local research notes from pull request
2026-09-27 20:32:26 +08:00
wizardchen 6c83d7ab47 revert: take the plugin system back out of main
Reverts the eleven plugin system PRs merged on 2026-09-26 so they can be
reviewed before landing:

* a829d7cc9 agent tools served by plugins (#3733)
* 4f5d475c3 Jira example plugin (#3730)
* 01222aac8 dynamic form options, OAuth connections and resource limits (#3729)
* cf502ba46 event delivery and webhooks (#3728)
* 996696057 sandboxed plugin pages (#3727)
* da4ef68d9 standalone plugin hosts (#3725)
* 728007ada remote runtime, Python SDK (#3724)
* ff49f9491 Host API and document parsers (#3721)
* 4328ab26a code plugins (#3720)
* ed63e35cc declarative plugins (#3716)
* c91a8d11f plugin system foundation (#3711)

The tree is back to 47773b5bd, migrations included: no environment ran
the plugin migrations (versioned 000115-000119, sqlite 000034-000038).
2026-09-26 14:58:18 +08:00
lyingbug ff49f9491d
feat(plugin): Host API with a key-value store, and document parsers from plugins (#3721)
* feat(plugin): Host API with plugin tokens and a key-value store
* feat(plugin): document parsers from code plugins
* docs(pluginsdk): parsers and the Host API
2026-09-26 14:45:39 +08:00
lyingbug 4328ab26a9
feat(plugin): code plugins — extension protocol, Go SDK, plugin host, web search and data source adapters (#3720)
* feat(pluginsdk): extension protocol v1, Go SDK and conformance suite
* feat(plugin): run code plugins in an embedded plugin host
* feat(plugin): remote adapters for web search and data sources
* fix(plugin): show plugin errors by their message
* feat(frontend): plugin connectors and web search providers in the UI
* docs(pluginsdk): how to write, package, test and install a plugin
* fix(plugin): gofmt two files and read openapi.yaml line-ending agnostic
* fix(plugin): mark a host plugin ready before Activate returns
2026-09-26 14:45:15 +08:00
lyingbug c91a8d11f8
feat(plugin): plugin system foundation — builtin plugins, config schemas and workspace switches (#3711)
* feat(plugin): add plugin manifest, registry and builtin catalog
* feat(plugin): add config schemas with unified secret handling
* feat(frontend): render config schemas with SchemaForm
* feat(datasource): serve connector credential forms from the backend
* feat(im): describe IM platforms and credentials from the backend
* feat(frontend): render model vendor extra fields from the config schema
* feat(frontend): build the settings modal from an extension registry
* refactor(agent): register builtin tools and their chat views
* feat(plugin): let workspaces enable and disable plugins
* feat(plugin): add the runtime driver interface and gather builtins
2026-09-26 14:43:28 +08:00
xiaojunxiang 0bf105499b
fix(knowledge): reject invalid JSON uploads before async parse (#3360)
* fix(knowledge): reject invalid JSON uploads before async parse

Validate .json payloads at HTTP upload/replace time with the same gate used
by the simple JSON reader, so JSONC or truncated files fail with HTTP 400
instead of after document-process retries. CreateKnowledgeFromFile itself
skips the gate so datasource sync and IM still create a knowledge row that
fails in async parse.

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>

* fix(knowledge): localize JSON upload validation messages

The upload queue shows the backend message verbatim and the rest of the
upload handler speaks Chinese. Map ValidateJSONContent's errors (now
sentinels with unchanged text, still shared with the async JSON reader)
to Chinese messages at the upload/replace boundary.

---------

Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
2026-09-25 22:23:27 +08:00
wizardchen 27b0c84bfd fix(docparser): make the MinerU parse timeouts configurable
Self-hosted MinerU capped every parse at a hardcoded 1000s, which fired
before WEKNORA_DOCREADER_CALL_TIMEOUT and WEKNORA_DOCUMENT_PROCESS_TIMEOUT
could matter, so large PDFs always failed at 16m40s (#3091). MinerU Cloud
had the same problem with a hardcoded 600s polling deadline.

- WEKNORA_MINERU_TIMEOUT (default 1000s) bounds a self-hosted parse: the
  whole V1 job, or the legacy /file_parse request.
- WEKNORA_MINERU_CLOUD_TIMEOUT (default 600s) bounds how long a MinerU
  Cloud batch is polled.
- Both share the env parsing added for WEKNORA_PADDLEOCR_VL_TIMEOUT in
  #3537, now in helpers.go: invalid or non-positive values log a warning
  and fall back to the default.
2026-09-25 18:38:25 +08:00
lyingbug 1bf717a54d
feat(chat): open cited documents at the cited passage (#3700)
* feat(chat): open cited documents at the cited passage

Clicking a citation used to show only the parsed chunk. It now opens the
original file in the references panel, jumps to the cited place and
highlights it, narrowed to the sentence the citation supports.

Parsing records where each chunk sits in the original file
(chunks.source_locators): PDF page and paragraph boxes from the docreader,
MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and
EPUB chapters aligned in Go against any parser engine's Markdown; text
offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites
are followed with a line-diff remapper. Scanned PDF pages get their OCR
text as quotes at search time so a sentence can pick its page.

The frontend renders PDFs with pdf.js and every other previewable type
with the existing preview, then highlights with the CSS Custom Highlight
API in a highlighter yellow that stays visible against the brand green.

* fix(docreader): serialize source blocks outside main so tests import it

test_docx_tables stubs docreader.parser in sys.modules, so importing
docreader.main from a later test failed in the full unittest discover run.
The wire conversion now lives in docreader.source_wire, which needs only
the proto.

* fix(chat): tighten source locators from review

- Remap rewritten-line-for-line stretches by line so scanned PDF pages
  no longer drift onto the wrong page.
- Keep the source preview mounted for citations in the file already open.
- Wait for in-flight PDF page renders before looking for the text layer.
- Count image rune offsets in one pass; stream xlsx rows up to maxUnits.
- Start text locators after a UTF-8 BOM, as viewers decode without it.
- Only re-query image children for scanned-page results.
- Seek audio citations without autoplay and drop stale seek listeners.
- Fix the SQLite migration's cross-reference to versioned/000113.

* fix(migrations): renumber chunk source locators after main's chunk_images

main took versioned 000113 and sqlite 000032 for chunk_images, so the
source_locators column moves to versioned 000114 and sqlite 000033.
2026-09-25 13:26:16 +08:00
wizardchen e22a8367ef feat(docparser): support MinerU 4.0 V1 API for self-hosted MinerU
MinerU 4.0 removed the synchronous /file_parse endpoint that the mineru
engine relied on, so a self-hosted 4.0 server answered every parse with
404. The reader now probes GET /v1/health before each parse: a 200 selects
the V1 flow, 404/405 keeps the legacy /file_parse path (AMD and domestic
accelerators must stay on mineru<4), and a 503 error envelope surfaces the
server's model-preload failure instead of falling back.

The V1 flow uploads via /v1/uploads (sha256 dedup, API key only sent to
same-origin upload URLs), creates a zip-only parse job, polls with
exponential backoff (2s to 30s), cancels the job on timeout or caller
cancellation, and reads markdown.md plus images/ from the zip artifact.
The zip extraction is shared with the MinerU Cloud reader and now also
resolves <img src> references and percent-encoded paths.

New tenant settings: mineru_server_api_key (redacted in responses,
preserved on update) and mineru_tier (flash/basic/standard/advanced, empty
for the server default). mineru_parse_method maps to V1 ocr_mode. Backend,
vLLM URL, formula/table toggles and language only apply to MinerU 3.x and
are grouped as legacy options in the settings page. The connection check
also verifies the API key on V1 servers, since /v1/health is
unauthenticated.
2026-09-24 20:40:52 +08:00
Captain 171097f06e
feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS

A penetration test of a private deployment found that a host outside
SSRF_WHITELIST still produces a DNS query: every outbound guard resolves
the name first and judges the addresses afterwards, which is what makes
it immune to DNS rebinding, but it also means an attacker-chosen name
reaches the resolver and can carry data out of an otherwise offline
network (#3378).

SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and
the check is placed wherever a name is still in hand.

Where a URL is available, CheckSSRFWhitelistOnly refuses a host the
whitelist does not name, IP literals included: URL validation, the
sandbox outbound policy, and the web-fetch target resolver.

Where only a dial address is available, CheckDialWhitelistOnly judges
hostnames and leaves addresses to the checks that already exist, because
an IP arriving at a dialer is either a name something else resolved or a
literal the URL layer already refused. That covers the shared SSRF-safe
dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every
sandbox transport: the HTTP and websocket data planes, the guarded
transport and the Docker engine now share one GuardedDialContext instead
of each building its own dialer.

Where the client resolves its own target, the name is judged before the
client exists. gRPC's resolver runs before any dialer it was given, so
the env-configured qdrant and milvus clients are built through small
constructors that refuse a non-whitelisted host outright; a whitelisted
one is resolved by gRPC as before.

docreader enforces the same switch in its own validator, and compose
passes the variable to both services.

Matching is by name only in this mode: the whitelist's CIDR entries
still match an IP literal, but a hostname is no longer resolved to see
whether its addresses fall inside one, because that lookup is the query
the mode exists to prevent.

The value is parsed with ParseBool, and a non-empty value that parses as
neither enables the mode: silently leaving an egress lockdown off
because an operator wrote "yes" is the one failure mode a security
control must not have.

Still not covered, and documented as such: the Langfuse OTLP exporter
builds its own client and never reaches these guards; and with
HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial
address equals the proxy URL's host exactly, while a mismatch makes the
proxy host itself subject to the whitelist.

The switch defaults to off, so nothing changes for deployments that do
not set it.

* docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs

docs/ has been retired in favour of website-docs/ (#3580), so the notes that
lived in docs/QA.md move to the configuration table: the new switch, what to
whitelist before turning it on, and the three outbound paths it still does
not cover.

The existing SSRF_WHITELIST row was also stale: compose gives only the app its
SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is
empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.
2026-09-23 12:02:06 +08:00
Leon 402729341a
fix(docparser): make PaddleOCR-VL request timeout configurable (#3537)
* fix(docparser): make PaddleOCR-VL request timeout configurable

* style(docparser): wrap timeout configuration and test lines
2026-09-23 11:58:59 +08:00
Lvcy_cd cc82e75cb5 fix(parser): keep PDFs on builtin instead of defaulting to anydoc
anydoc has no document model for PDF: Convert force-disables asset
extraction, and AnydocReader falls back to builtin only when the whole
file yields no text. A born-digital PDF with figures and tables that
still yields a few characters is therefore reported as parsed, with its
images, tables and layout silently dropped.

35275d56 widened the type-level preference from ppt/pptx to every type
anydoc claims, and pdf is on that list. The predicate asks whether
anydoc can convert a type, not whether it is the better parser for it.
60207997, one commit earlier, recorded the original intent: "Types
without a fallback (pdf, docx, ...) never consult this".

The builtin PDF parser classifies pages individually and routes scanned
ones through OCR/VLM, so it stays the default for PDF. Every other
anydoc format is unchanged, and an explicit pdf -> anydoc engine rule
still wins.
2026-09-21 16:28:02 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
serply e80f5dfbc7 feat(web-search): add Serply provider
Serply returns Google results through a keyed JSON API, which fills the
gap between Google Custom Search (needs an engine ID and a restricted
scope) and the scrapers. The adapter follows brave.go: hardcoded
endpoint, redirects refused so the key never leaves the official host,
4MB response cap, and per-call country and freshness filters. country
maps to Google's gl parameter and pd/pw/pm/py map to tbs qdr:d/w/m/y.
Serply has no date-range filter, so that form returns an error instead
of being silently dropped. num is clamped to 10 because the API serves
one results page per call.

Wiring matches the Bocha provider: type constant and metadata, registry
factory, service validation, frontend union, swagger enum entries (edited
by hand; a full swag regen drags in unrelated drift), and the web search
docs table.
2026-09-17 12:25:09 +08:00
GUO Zhaogong 8d7298fb5d
fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders (#3092)
* fix(graph): feed OCR chunks to graph extraction instead of image-link placeholders

KnowledgePostProcessService.Handle gathered "text-like" chunks through
chunkService.ListChunksByKnowledgeID, which is text-only by design, so the
image_ocr / image_caption chunks written by multimodal parsing were never
returned. For a scanned PDF the text chunks contain nothing but markdown
image links, so graph extraction ran on empty input: most calls failed
with "unexpected end of JSON input" and the rest echoed the few-shot
example from the extract config into the graph.

- Fetch the exact chunk types the post-process enriches via
  chunkRepo.ListChunksByKnowledgeIDAndTypes (text, image_ocr,
  image_caption); the service now receives interfaces.ChunkRepository
  through dig like knowledgeService already does
- Build a dedicated graphChunks list for graph extraction: skip
  image_caption chunks (they duplicate the OCR text) and chunks that
  contain only image links (chunkHasExtractableText)
- graphChunkCount, the enqueue loop and the log line use graphChunks;
  summary, wiki, auto-tag and question generation keep using textChunks
- Tests: fake ChunkRepository in the wiki-enqueue test, table test for
  chunkHasExtractableText

* fix(graph): skip figure OCR and harden empty-chunk detection

Reuse the CommonMark image scanner plus extractRealText so titles with
parentheses and <image> wrappers are not treated as prose. Only enqueue
OCR for graph extraction when the parent text chunk has no extractable
text, and skip link-only chunks for question generation as well.

* style(graph): wrap long post-process extract log line

---------

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-08 19:23:13 +08:00
lyingbug 647848f395
fix(agent): improve web search and persistent page reading (#3086)
* fix(agent): improve web search and persistent page reading

* fix(agent): make web fetch snapshots waitable and retryable

Keep in-flight page fetches independent of a caller's short deadline so
content=true and continuations cannot cancel a shared download or miss a
snapshot that is still loading. Mark snapshot_expired as retryable, fetch
only the first three search hits in parallel, and stop forcing Brave's
country to US.
2026-09-07 23:09:27 +08:00
Fei.Chen 15727591a7
feat(web-search): add Bocha AI Search provider (#3036)
* feat(web-search): add Bocha AI Search provider

Add a Bocha (bochaai.com) web search provider following the Metaso
adapter pattern: hardcoded official endpoint (SSRF-safe), Bearer auth,
4MB response cap, error-detail extraction, and RFC3339 date parsing
(datePublished preferred, dateLastCrawled fallback).

Tenant-facing config fields:
- freshness: noLimit/oneDay/oneWeek/oneMonth/oneYear (default noLimit)
- summary: request long text summaries, preferred as snippets (default on)

Wiring: registry factory in container, provider type enum + /types
metadata (dynamic form fields), frontend provider union type, and
swagger docs enum entries.

* fix(web-search): match Bocha's real wire format for codes and error messages

Live endpoint probing showed errors return {"message": .., "code": "401"}
(string code, message field) instead of the documented {msg, code:int}.
Parse the code tolerantly (string or number) and extract the detail from
message/msg so auth and rate-limit failures surface their reason.

* fix(web-search): allow Bocha in provider CRUD validation

In-service integration testing caught a fourth wiring point the adapter,
registry, /types metadata and frontend covered but CRUD did not: the
service-layer whitelist (isValidProviderType) and parameter validation
switches rejected provider type 'bocha', so providers could be tested
via POST /web-search-providers/test but never created or updated.

Add Bocha to both switches (delegating to ValidateBochaParameters like
Metaso/Zhipu) with service-level test coverage.

* fix(web-search): correct Bocha legacy timestamp timezone

---------

Co-authored-by: vicfei <vicfei@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-07 20:06:26 +08:00
wizardchen 35275d5613 fix(parser): prefer anydoc for every type it converts when linked
ppt/pptx still fall back to markitdown when anydoc is not in the binary.
pdf/docx/xlsx and other anydoc formats now default to the in-process
converter as well; csv/md stay on the Go simple reader.
2026-08-31 14:58:15 +08:00
wizardchen 6020799792 fix(parser): prefer anydoc for ppt/pptx when the binding is linked
Type-level defaults still fall back to markitdown, but a linked anydoc
build should use the in-process converter first instead of always
shipping Office decks through docreader.
2026-08-31 14:58:15 +08:00
wizardchen 52182385ec fix(parser): default ppt/pptx to markitdown when no engine rules (#2882)
Empty parser_engine_rules used to resolve to builtin, which historically
did not register ppt/pptx and failed with an unusable ValueError. Route
those types to markitdown by default on both the Go resolver and the
docreader registry, and list engines that actually support the type.

Refs: https://github.com/Tencent/WeKnora/issues/2882
2026-08-31 14:58:15 +08:00
Guiyang Yuan f51a139480
feat(docreader): add XMind outline parsing (#2713)
* feat(docreader): parse modern XMind outlines

* feat(docreader): support classic XMind archives

* feat(docreader): advertise XMind file support

* perf(docreader): avoid scanning XMind archive entries

---------

Co-authored-by: Guiyang Yuan <241028953+guiyangyuan@users.noreply.github.com>
2026-08-20 19:49:48 +08:00
wizardchen 9b4f792a04 fix(docker): 默认在 app 镜像中链接 anydoc 解析引擎
Hub / compose 打包的二进制此前走 stub,设置页会显示引擎未编入。WITH_ANYDOC 改为默认开启,并补上 builder 所需的 curl。
2026-08-16 13:31:51 +08:00
lyingbug 7f0e1a91e9
fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界 (#2720)
* build(anydoc): 构建脚本从 Cargo.toml 读取待打补丁的 crate 版本

原先 0.1.8 硬编码在脚本的 6 处,升级要同时改动它们;现在统一从
`anydoc = "=X.Y.Z"` 这一处 pin 推导。

另外补两道拦截:patched-anydoc/ 的复用标记记录已打补丁的版本,避免
版本变更后沿用上一版的副本、最终在 cargo 的 patch 解析里报出难以定位
的错误;构建前校验 version.go 与该 pin 一致,因为这个常量会作为
anydoc_version 写进每一篇已解析文档的元数据。

* fix(anydoc): 升级到 0.1.9,为恶意 PDF 的解析开销加上上界

anydoc 0.1.9 只把 pdf-inspector 从 0.1.8 提到 1.14.2,但那一版含 9 个
修复,其中 7 个是给此前完全无上界的解析工作加限额:每页 Form XObject
展开、CID /W 区间、Encoding cidrange 与 ToUnicode bfrange 展开、
分配操作符前的内容流解码、detector 的 Tj/TJ 操作数回溯、以及不相交
矩形的表格聚类。它们全都落在 process_pdf_mem 这条路径上,而这是
anydoc 转换 PDF 的唯一入口,且只由上传文件的字节驱动。

这正是 guarded() 拦不住的那一类问题:耗尽 CPU 不会 panic,分配失败直接
abort。以 detector 回溯为例,缺少操作数的 TJ 会重扫整个内容流,因此
一串裸 `] TJ` 的开销是长度的平方:977 KB 的 PDF 要 26.7 秒,1.9 MB 要
111.9 秒;升级后分别是 5.8 毫秒和 11.0 毫秒。
TestDetectorLookbackStaysLinear 用 15 秒预算把这个上界钉住——它在
0.1.8 上会超时失败,在 0.1.9 上 0.01 秒通过。

另外 2 个修复改善提取质量,直接影响入库内容:Form XObject 内的文字
现在跟踪 text line matrix 并处理 T*/TL/'/"/Tc/Tw,嵌套表单文字不再
丢换行和字间距;小型大写字母(small-caps)的文字段会合并,不再被当成
额外的表格列而产出错位的 markdown 表格。

升级本身不需要改绑定代码:anydoc 0.1.8 与 0.1.9 的 src/ 字节级一致,
C ABI、文档模型和 GFM 序列化器都没变,6 项本地修改原样保留。依赖树也
只动了 3 行版本和 1 个 checksum,lopdf 仍是 0.42,cargo audit 结果与
升级前相同(仅 ttf-parser 未维护这一条已允许的警告)。

* docs(anydoc): 文档与注释中的耗时数字对齐到实测记录

表格与注释里的数字改为与随 PR 附上的实测日志同一次运行的结果,便于核对。
2026-08-16 12:30:55 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
wizardchen cb69764223 style(docparser): gofmt mineru converter tests 2026-08-13 18:51:27 +08:00
wizardchen 2308593ca0 fix(docparser): resolve MinerU results by upload filename stem
MinerU keys file_parse results by the uploaded filename stem, so preserving
real basenames requires dynamic response lookup instead of results.document.
Also append FileType when the basename has no extension.
2026-08-13 18:51:27 +08:00
shangfeng-liu 0eac47490b fix(docparser): preserve filename extension in MinerU multipart uploads 2026-08-13 18:51:27 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
sandaogouchen 262b172c28
fix(chunking): preserve parent-child text child embeddings (#2680)
* fix(chunking): preserve parent-child text child embeddings

* refactor(chunking): simplify text chunk index selection
2026-08-13 11:20:03 +08:00
Syt3s 5eaabf6bbb
feat: add Exa WebSearch (#2617)
* feat: add Exa WebSearch

* Fix test for Exa config options validation

* feat: add exa connection test
2026-08-11 17:05:52 +08:00
Binks Riverton 355d161d64 feat: add Metaso web search provider 2026-08-10 13:33:55 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 1985a59b69 fix(chunking): sync preview swagger and normalize splitter config
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
2026-08-05 16:17:48 +08:00
xiaowang d2888e9565 fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
shangfeng-liu 583b87b1b9 fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
zldkr 1168505eef fix(docparser): normalize PaddleOCR-VL Cloud HTML tables 2026-07-30 16:08:24 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen b581094c7b fix(web_fetch): address review follow-ups for structured fetch results
Restore HTTP-only 15s pipeline fetching, reject SPA shell pages when
browser rendering fails, surface per-URL failure fields in the UI, and
keep structured model output when all URLs fail.
2026-07-28 22:17:31 +08:00
langcaiye 75a2eabe20 fix(agent): restore safe rendered web fetch fallback 2026-07-28 22:17:31 +08:00
langcaiye a3da5cb57a fix(agent): preserve web evidence when page fetches fail 2026-07-28 22:17:31 +08:00
wizardchen c193b949aa fix(chunker): require space after English ? and ! in overlap boundaries
Align English question and exclamation marks with period handling so
URL query strings are not split during overlap. Add regression tests for
URL overlap and inline code protection.
2026-07-28 22:02:30 +08:00
ttommybot 21861f4b0f D:\xinchen.ju\Code\WeKnora 2026-07-28 22:02:30 +08:00
ttommybot 2dfc696740 fix: preserve semantic boundaries in chunk overlap 2026-07-28 22:02:30 +08:00