Commit Graph

71 Commits (22cbcdfa5789b33dbb8d47532c21b24da2fdf5aa)

Author SHA1 Message Date
vicfei 2dd36212d6 fix(knowledge): bind extracted images to their document and unscope shared-KB parent lookups
Org-shared KB viewers got 403 for extracted images on current main even
though the retrieving chunk text carried the resource:// handle and the
share grant was valid (#3342, second report after #3022/#3069).

Root cause is a mismatch between ingestion and authorization: document
parses store extracted images through SaveBytes, which registers the
resource but never claims it, while both file proxies (ResolveKBFile and
the message fallback via resourceAccessibleViaSharedKB) authorize through
IsReferencedByKnowledgeBase — which accepts explicit resource_bindings
rows only. The binding-only hardening landed without adding the missing
bindings, so every extracted image became invisible cross-workspace.

Ingestion: bindStoredImages claims each stored image for its knowledge
(ResourceRelationExtractedImage), mirroring bindContentResources — best
effort, same-workspace resources only. Wired into both the file-parse
path (knowledge_process) and manual knowledge (triggerManualProcessing).

Merge: resolveParentChunks fetched parents, grandparents, and image_info
children with the caller's tenant, so shared-KB hits from the sharing
workspace resolved none of them and lost parent context and image_info
(the second defect in the #3342 audit). Those lookups now use the
existing *Only unscoped variants, matching their shared-KB precedent.

Historical documents parsed before this change still carry no bindings;
a backfill is tracked as the follow-up slice.

Verified end-to-end on a fresh stack: publishing a manual document with
an embedded image now records kind=image plus an extracted_image binding
alongside the content-derived attachment binding.
2026-09-22 17:52:49 +08:00
lyingbug 4a9aa06274
refactor(asr): move the wire format into protocol packages, and cover every vendor whose docs offer ASR (#3503) 2026-09-21 13:25:27 +08:00
lyingbug eaca7364cd
fix(knowledge): stop two parse paths from stranding a row in processing (#3445)
* fix(knowledge): stop two parse paths from stranding a row in processing

Both of these left parse_status on "processing" with nothing left to move
it, so the document sat there until the housekeeping sweep failed it an
hour later with "task stuck in processing, recovered by housekeeping" —
reaching the user as a slow, unexplained parse failure.

processChunks returned silently when the embedding model could not be
resolved (deleted model, expired credentials, unreachable provider). It
reports nothing to its callers, so the attempt ended with the row still
marked processing. Persist the failure with the underlying cause and
record the embedding stage as failed, matching how the sibling
GetASRModel branch already handles a model it cannot resolve.

enqueueImageMultimodalTasks seeded the multimodal fan-in counter with
len(images) before the fan-out, then only warned when a payload failed
to marshal or asynq refused the write. Those slots have no task behind
them to drain them, so the counter never reached zero and post-process
was never triggered. Release the shortfall after the fan-out, and drive
post-process directly when the release itself brings the counter to zero
or when nothing reached the queue at all — the same reconciliation
KnowledgePostProcess already performs for its enrichment subtasks.

Seeding the counter before enqueueing is kept deliberately: an image
task that finishes early would otherwise decrement a key the fan-out is
about to overwrite. The release is what settles the resulting gap.

Also collapse the Redis key into multimodalPendingKey so the fan-out and
fan-in cannot drift, and share one enqueueKnowledgePostProcessTask
between the no-multimodal path and the slot release.

* fix(knowledge): keep the embedding-model guard from clobbering abort state

Review follow-up on the two stranded-row fixes.

The embedding-model guard wrote parse_status=failed unconditionally, which
made two cases worse than the silent return it replaced. Resolving a model
reads the database, so the entry guard's verdict is already stale by the
time the error comes back:

  - A user cancel signals asynq CancelProcessing, which cancels the
    handler context. GetEmbeddingModel then fails with context.Canceled
    and the guard overwrote "cancelled" with "failed: ... context
    canceled" — mislabelling an interrupted run as a broken model. Worker
    preemption and graceful shutdown look the same.
  - ReplaceKnowledgeFile keeps the knowledge ID and swaps file_path.
    UpdateKnowledge is a full-row Save and file_path is not among
    omitFieldsOnUpdate, so the stale row rolled the path back over the
    replacement and marked the replacement's live attempt failed.

So the guard now returns without writing when the context is gone or the
error is a context error, re-checks isKnowledgeAborted before writing, and
persists through updateKnowledgeUnlessSourceReplaced like the completion
path already does.

The multimodal slot release gave up on the first Redis error. Since the
release is the only thing that can drain a slot with no task behind it,
that put the row straight back into the stranded state this fix exists to
remove. It now retries a bounded number of times and logs at error level
when it still cannot settle the counter. DECRBY is not idempotent, so a
retry after a lost reply can over-decrement; that finalizes early rather
than stranding, which is the direction to fail in, and post-process
already tolerates repeat deliveries. The nothing-enqueued branch's Del
also moves onto the detached context, so a cancelled parent no longer
leaves the seeded key to expire on its own 24h TTL.

Two reporting corrections while here: the span now carries the same
message as the document list instead of a fixed generic string, because
the timeline renders error_message and withholds error_detail; and the
comment no longer claims expired credentials or an unreachable provider
surface here — resolving a model only reads the row and constructs a
client, so those fail later on the first BatchIndex round-trip.

Adds 6 tests for the paths above. Two of them initially passed against
the unfixed code because the entry guard caught them before the model
call, so the repository fake now serves the cancelled / replaced row only
after the guard's own reads, which is the window that actually exists.
2026-09-20 13:02:05 +08:00
neoczr 473c8726bb
fix(chunking): 修复内联 HTML 表格与保护模式导致的超长 chunk (#3404)
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer

The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.

Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.

* fix(ocr): keep merged-cell HTML tables chunkable

- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
  still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
  pad the block with blank lines so the chunker can split by "\n"
  instead of force-cutting at the absolute max size.

* fix(knowledge): normalize inline HTML tables for all parser engines before chunking

Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.

Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
  before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
  markdown

Converters keep their existing calls (idempotent no-op).

* fix(chunker): bound Markdown link/image protected patterns to one line

The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).

Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.

Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.

* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths

Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.

* style: wrap long lines and avoid shadowing max in table-chunk tests

Pre-push lll/revive failed on the PR diff versus origin/main.

---------

Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-18 21:10:17 +08:00
lyingbug 5574ebc011
feat(knowledge): derive knowledge base descriptions from document profiles (#3370)
* feat(knowledge): derive knowledge base descriptions from document profiles

Knowledge bases only had a hand-written description, which most users
leave empty, so agents bound to several knowledge bases had nothing to
route on and searched all of them. This adds a generated description
that is derived from per-document data instead of maintained by a model.

Document profile: the summary call now returns JSON with a 2-3 sentence
summary plus a gist, 3-5 topic keywords, a document type and one
typical question (knowledges.profile). Input is capped at 8k chars and
output at 1024 tokens; plain-text replies from custom templates still
work and simply carry no profile. The summary chunk embeds the gist and
short summary.

Knowledge base aggregate: a database-only fold over those rows (counts
per file type, tag, normalized topic and doc type; stratified samples of
titles and questions; a hash of the inputs). Deleting, moving or
re-parsing a document changes nothing but the rows, so the next rebuild
is exact without any model judgement.

Description text: one small model call over the aggregate yields a
gist, merged topics and typical questions (knowledge_bases.
generated_profile). The call is skipped when the aggregate hash is
unchanged; failures keep the previous text and record the error; an
empty knowledge base clears it. The manual description is never
touched, and the agent runtime context renders both so the model can
pass knowledge_base_ids instead of fanning out.

Refreshes are requested from summary task exit, post-process completion,
deletion, cross-KB moves and enabling the option, debounced per
knowledge base in 30 s windows via kb:profile on the summary queue.
POST /knowledge-bases/:id/profile/generate regenerates synchronously.
profile_config (enabled, model_id, custom_instructions) opts a document
knowledge base in; the settings dialog shows the generated card with
regenerate and adopt-as-description actions in five locales. Search
results now log hits per chunk type so the value of indexing summary
chunks can be measured.

Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the
SQLite schema test expects the new version and columns.

* fix(knowledge): keep generated descriptions out of duplicates and refresh clones

DuplicateKnowledgeBase copies the source row through JSON, which carried
generated_profile onto a knowledge base that has no documents yet, so
an empty duplicate advertised the source's topics. The duplicate now
starts without a generated description and keeps profile_config so it
derives its own once documents arrive.

CopyKnowledgeBase builds the target field by field and skipped
profile_config, and the clone task never asked for a description after
copying the documents, so a clone stayed undescribed until its next
upload. The target now carries a copy of profile_config and both clone
paths request a refresh on completion.

* fix(knowledge): refresh the loaded clone target and carry document profiles

ProcessKBClone requested the description refresh with the reservation
stub it builds before CopyKnowledgeBase runs. For the main create path
that stub has neither a type nor profile_config, so the refresh helper
rejected it and nothing was enqueued. The refresh now uses the loaded
target returned by CopyKnowledgeBase.

cloneKnowledge copied the summary text but not the structured profile,
so even a scheduled refresh would have aggregated titles without topics
until every clone was re-summarized. Cloned documents now carry an
independent copy of the profile.
2026-09-17 22:21:55 +08:00
wizardchen 38be45c9d1 refactor: remove unused private Go helpers 2026-09-17 15:51:51 +08:00
wizardchen 079e96ae64 fix(knowledge): make in-place file replacement race-safe
Queued ProcessDocument tasks still held the old FilePath, and a crash
between the source UPDATE and reparse could leave completed knowledge
pointing at a new file. Dequeue first, flip pending in the same write,
and skip stale worker Saves when the stored path has moved.
2026-09-16 14:54:47 +08:00
lyingbug 3d81b977f1
refactor(resource-access): unify scoped authorization and transfer recovery (#3097)
* refactor(resource-access): unify scoped authorization and transfer recovery

* fix(resource-access): close reparse and transfer review gaps

* fix: enforce explicit file bindings and recover move cleanup

* fix: align mutation admission and cleanup scope checks
2026-09-08 15:36:33 +08:00
十三分之五六七 786d67609d fix(markdown): sanitize invalid UTF-8 and NUL bytes 2026-08-31 12:24:35 +08:00
lawrence 28dd15ec30 fix(retrieval): 分段中删除的图片不再被检索到
从分段里删掉一张图片后,这张图仍会回到检索结果和模型上下文里。

三个问题串在一起:

1. 删图时 syncEditedChunkImages 会把 image_ocr / image_caption 子块置为
   禁用,但 CollectImageInfoByChunkIDs 查子块时不看 is_enabled,又把它们的
   image_info 挂回了父块。knowledge_search 工具随后把每条 image_info 无条件
   渲染成 markdown 图片写进 LLM 上下文,图片就这样回来了。这里按 is_enabled
   过滤,两级遍历都跳过禁用子块。过滤放在 collector 而不是工具输出层:纯图片
   分块的正文本身就是 OCR 文本、没有 markdown 图片语法,按正文过滤会误杀。
   chat_pipeline 里既有的 fake 仓储 fixture 同步补上 IsEnabled: true:它们不
   走 DB 默认值,零值 false 会被新的过滤当成已禁用子块。

2. UpdateImageInfo 补建 image_ocr / image_caption 子块时没有设置 IsEnabled。
   CreateChunks 用 Select("*") 插入,gorm 的 default:true 不生效,零值 false
   直接落库,这些子块一出生就是禁用的。补上 IsEnabled: true —— 否则第 1 点
   的过滤会把它们永久丢弃。其余 CreateChunks 调用方均已显式设置该字段。

3. 摘要刷新每次都新建一条 summary 分块,旧的留着且仍然启用。原因是它复用了
   ListChunksByKnowledgeID 的结果,而那个查询硬编码 chunk_type = 'text',
   "查找已有 summary 分块"的分支永远匹配不上。残留的旧摘要描述的是编辑前的
   文档,会让用户已经删掉的内容继续被检索到。新增按 chunk_type 过滤的仓储
   方法,刷新时原地更新已有 summary 分块。
2026-08-20 12:01:28 +08:00
wizardchen e123f8247e feat(docparser): 引入 anydoc 引擎,在 Go 进程内解析 office 文档
通过 cgo 链接 anydoc Rust 静态库,把 docx/pptx/xlsx 等转为带原位图片的 Markdown;扫描版 PDF 无文字层时回落到 DocReader OCR。默认构建不链接该库。

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-16 11:37:27 +08:00
sandaogouchen 262b172c28
fix(chunking): preserve parent-child text child embeddings (#2680)
* fix(chunking): preserve parent-child text child embeddings

* refactor(chunking): simplify text chunk index selection
2026-08-13 11:20:03 +08:00
lyingbug 1030832c4b
fix(security): harden outbound URL SSRF protection (#2626)
* fix(security): harden outbound URL SSRF protection

* test(security): add SSRF validation tests for outbound URL caching

Introduce tests for outbound SSRF validation cache to ensure proper handling of URL validation, including cache reuse for the same origin, cache invalidation on whitelist changes, and expiration of cached entries. This enhances the robustness of SSRF protection mechanisms in the application.

* refactor: clean up whitespace in engine_factory.go and ssrf_outbound_cache.go

Removed unnecessary blank lines in engine_factory.go and adjusted variable declarations for consistency in ssrf_outbound_cache.go. This improves code readability without altering functionality.
2026-08-10 13:24:14 +08:00
ttommybot 0827cd6cf0 fix: retry document summaries before fallback 2026-08-05 19:29:32 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 1985a59b69 fix(chunking): sync preview swagger and normalize splitter config
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
2026-08-05 16:17:48 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
xiaowang d2888e9565 fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
xiaowang 51b88bbae3 fix(summary): preserve rows across table chunks / 修复表格分块摘要丢行 2026-08-05 13:12:59 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
lyingbug 82ea91f18e fix(knowledge): clear stale error_message on the remaining reprocess paths
Follow-up to the error_message clearing added for SetFinalizing,
FinalizeSubtask and ProcessDocument. Two routes still persisted a
previous attempt's failure message:

- ProcessManualUpdate flips a row back to 'processing' without clearing
  error_message. A manual knowledge item that failed earlier (for
  example when the knowledge base lookup failed, or after an Asynq
  retry) re-enters processing still carrying that message, so the UI
  shows a failure on a document it also shows as running.

- finalizeIndexedKnowledgeState is the only route to 'completed' that
  bypasses FinalizeSubtask. When a document has no text chunks and no
  pending multimodal work it is completed in place, so the promotion
  that would otherwise clear the column never runs and a successfully
  indexed row keeps reporting the earlier failure.

The 'processing' transition is extracted into markKnowledgeProcessing so
ProcessDocument and ProcessManualUpdate share one definition of what
starting a new attempt means, and a future worker cannot reintroduce the
same omission. The helper also replaces the "processing" string literal
with types.ParseStatusProcessing.

Tests: markKnowledgeProcessing gets direct coverage, and the
finalizeIndexedKnowledgeState table now seeds a stale error_message and
asserts it survives the processing branches (FinalizeSubtask clears it
later) and is cleared on the completed branch.
2026-08-03 20:14:09 +08:00
shangfeng-liu 44206e4ad4 fix(knowledge): clear stale error_message on reprocessing and finalize
A knowledge item that failed once keeps error_message set; when it is
reprocessed or driven through the finalizing pipeline, the stale message
was never cleared, so the UI kept showing an outdated failure on items
that were processing or had completed successfully.

- SetFinalizing and FinalizeSubtask now clear error_message in the same
  atomic UPDATE that transitions parse_status
- ProcessDocument clears knowledge.ErrorMessage when flipping the row
  to processing for a new attempt

Add a SQLite-backed regression test covering both transitions.
2026-08-03 19:38:22 +08:00
Myang_print 5780affdfc fix(knowledge): surface batch reparse failures
Signed-off-by: Myang_print <1182470296@qq.com>
2026-07-31 19:06:16 +08:00
wizardchen 8dd997f0f4 refactor(knowledge): make the supported import extension set the single gate
The xlsx URL-import fix left the accepted extension set reachable through two
indirections and widened ApplyKnowledgeProcessOverrides to gate file types for
callers that are not file imports (web URL fetch, connector sync, reparse).

Collapse the set into one map that both isValidFileType and the extension check
read, move the shared validation next to the other file-type predicates, and
return ApplyKnowledgeProcessOverrides to its override-only scope so
resolveFileImportProcessConfig is the only import gate.

Also drop the unreachable branch in createKnowledgeFromFileURL: getFileType
never yields an empty string, so the resolvedFileType fallback and its else arm
could not run. Normalize the resolved type before it reaches the record and the
task payload, and add an invariant test asserting direct upload and URL import
accept the same extensions so #2447 cannot drift back.
2026-07-31 13:46:56 +08:00
wizardchen 43afa4c1f8 fix(knowledge): align URL file import validation with direct upload
Extract shared file-import checks (type, VLM/ASR, video) into
resolveKnowledgeFileImportConfig and reuse them for file URL import.
Also enqueue DataTableSummaryTask for spreadsheet URL imports and reparse.
2026-07-31 13:46:56 +08:00
wizardchen 48756d13d6 fix(knowledge): allow xlsx and other file types in URL import (#2447)
Align URL-based file import validation with direct file upload by
delegating to isValidFileType instead of a narrower whitelist.
2026-07-31 13:46:56 +08:00
chenjunhong 595d19a806 fix(docreader): restore optional XLSX header context 2026-07-28 22:00:36 +08:00
zhangzherui eb447365cd fix(knowledge): clear stale errors on reparse 2026-07-28 20:44:34 +08:00
wizardchen 497e8312e3 fix(knowledge): harden summary refresh after review feedback
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 20:23:48 +08:00
wizardchen 48b661134b fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
wizardchen 6a439a4c3b fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
wizardchen 850a073c83 fix(summary): restore tenant context for refresh tasks 2026-07-28 20:23:48 +08:00
wizardchen 1b9858ecb8 fix(knowledge): enqueue stale summary refreshes 2026-07-28 20:23:48 +08:00
wizardchen 1320c50bde refactor(knowledge): keep metadata document scoped 2026-07-28 20:23:48 +08:00
wizardchen 95775ae909 fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
wizardchen 6fb858105a feat(llm): add prompt-cache observability and wiki cache-friendly prompts
Normalize provider-specific cache counters into TokenUsage, propagate call
purpose and prefix fingerprints through context for logging and Langfuse,
and restructure wiki page-modify prompts with warmup/singleflight to improve
prefix cache hit rates during reduce batches.
2026-07-23 19:17:30 +08:00
wizardchen 5af8eae81d feat(knowledge): enhance activity logging for FAQ imports
- Added new actions for FAQ import lifecycle: `faq.import_started`, `faq.import_completed`, and `faq.import_failed`.
- Updated activity details to include success, failure, and skipped counts for FAQ imports.
- Enhanced UI to display additional activity information, including filtering options and clearer summaries.
- Implemented tests to validate the new FAQ import outcomes and activity details.

This commit improves the tracking and visibility of FAQ import activities within the knowledge base, ensuring better audit trails and user feedback.
2026-07-23 10:50:32 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
darcy f7ec759cc6 fix(knowledge): propagate chunking strategy to parent-child splitter configs
buildParentChildConfigs only copied ChunkSize/ChunkOverlap/Separators into
the parent and child SplitterConfigs, dropping base.Strategy. Since an empty
Strategy resolves to the legacy tier (resolveChainWithProfile treats "" as
legacy for backwards compatibility), parent-child chunking always ran the
legacy recursive splitter regardless of the configured strategy: no heading
alignment and no ContextHeader breadcrumbs in child embeddings, silently.

The flat path (buildSplitterConfigFromChunking) already propagates Strategy;
parent-child predates the strategy system (b3334990, 2026-03) and was not
updated when the 3-tier strategy landed (ae6fde23, 2026-05). SplitParentChild's
docstring ("Child splitting honours childCfg.Strategy") and mergeBreadcrumbs
both assume the strategy reaches the parent-child path.

Copy base.Strategy into both configs and add a regression test.
2026-07-14 11:56:29 +08:00
wizardchen 07ba01ec93 feat(worker): replace aggregate asynq concurrency with per-pool governance
Split the monolithic upstream worker budget into explicit core, post-process,
enrichment, maintenance, and shared elastic pools, expose live cluster capacity
on the runtime dashboard, and retire the legacy asynq.concurrency setting.
2026-07-13 15:51:06 +08:00
wizardchen 5eefa70e6f Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00
wizardchen c1cddb4d4d Add instruction fields to knowledge base configuration 2026-07-13 12:16:19 +08:00
wizardchen d496d0fc73 feat(task-queue): split upstream workers into isolated core/enrichment/maintenance pools
Route summary/sync/maintenance tasks to dedicated queues and servers so parsing, enrichment, and batch work no longer share one concurrency budget, while extending the runtime dashboard and settings validation to reflect the new topology.
2026-07-10 23:51:29 +08:00
wizardchen fe47ac1f96 feat(system): add runtime task queue observability dashboard for system admins
Expose live asynq queue depths and worker-pool concurrency via a new SystemAdmin API and settings UI, so operators can monitor parsing/wiki backlog without Redis CLI access.
2026-07-10 23:51:29 +08:00
wizardchen 97b7bb9890 fix(knowledge): harden batch reparse UX and async processing
Move batch reparse work to an asynq task to avoid HTTP timeouts, add i18n
for all locales, guard in-flight documents, and prevent concurrent delete/reparse.
2026-06-25 23:57:54 +08:00
杨聪聪 917feceb59 feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
wizardchen 2aadd61fcc feat: implement reparse functionality with process configuration support
- Enhanced the `reparseKnowledge` function to accept optional process configuration overrides, allowing users to adjust parsing settings during reparse operations.
- Updated the frontend components to integrate reparse capabilities, including the `UploadConfirmDialog` and `KnowledgeBase` views, providing a user-friendly interface for confirming reparse actions.
- Added new localization strings for reparse confirmation and process configuration details in multiple languages.
- Modified the backend API to handle reparse requests with optional configuration, ensuring proper validation and persistence of overrides.
- Introduced new types and interfaces to manage reparse-specific data structures in the store, improving type safety and clarity in the codebase.
2026-06-09 20:38:24 +08:00
wizardchen ed9e780c8c feat(client): enhance knowledge file upload with process configuration support
- Updated the `CreateKnowledgeFromFile` function to accept an optional `processConfig` parameter for per-upload parse configuration overrides.
- Introduced the `KnowledgeProcessOverrides` type to define various parsing settings, including parser engine rules and chunking configurations.
- Modified the API client and related components to handle the new process configuration, improving flexibility in knowledge file uploads.
- Updated documentation and API specifications to reflect the new `process_config` parameter in the upload process.
- Added a new `UploadConfirmHost` component to facilitate user confirmation for uploads, integrating with the updated upload logic.
- Enhanced user experience by providing detailed configuration options during the upload process.
2026-06-09 20:38:24 +08:00
wizardchen 1778c5ac7b refactor(tracing): remove Jaeger integration and related components
This commit removes Jaeger tracing support from the project, including the associated Docker configurations, Go modules, and middleware. The changes simplify the codebase by eliminating unused tracing functionality, which was previously integrated for observability. Documentation and scripts have also been updated to reflect the removal of Jaeger references.
2026-06-08 21:01:04 +08:00