Commit Graph

852 Commits (267c847fb86a62c70b4e76ea8db69e8038b7d5ea)

Author SHA1 Message Date
MidoriKurage 0a3f6b0f21
fix(chunks): 流水线重写后的检索结果不用位置匹配 (#2558)
* chore(types): add internal ContentRewritten to types.SearchResult

* test(chunks): pin rewritten results out of the position path

Pipeline-replaced content (parent or neighbor expansion) must not trust
stale coordinates even when the replacement satisfies the length
invariant, and must not extend the merged group's range.

* fix(chunks): keep pipeline-rewritten results off the position path

Parent and neighbor expansion replace Content after retrieval. Without a
marker their stale coordinates could be incorrectly trusted.

Ensure ContentRewritten is false in chunkTrusted.
2026-08-06 14:32:24 +08:00
wizardchen e0ea453e3f fix(chunks): trim trusted overlaps by exact position, not text search
Position-aware merging still routed trimming through AppendWithOverlap,
which searches for the longest suffix match inside a window. On periodic
text (table rows, log lines) that search mistakes the repetition for the
overlap and drops real content, and with a zero overlap there is nothing
to search for in the first place.

When coordinates are trusted the overlap is already known, so verify it
character for character and cut exactly, falling back to the text matcher
only when the bodies disagree inside the overlap.
2026-08-06 12:26:15 +08:00
wizardchen dfe143818c test(chunks): pin exact-overlap contracts for trusted merges
- Overlaps wider than the 400-rune text window must be trimmed once.
- Adjacent chunks built from one repeating row must keep every row.
- A length-preserving text mismatch must fall back and keep both bodies.
2026-08-06 12:26:15 +08:00
MidoriKurage b7b85621fb feat(chunks): restore position-aware merge for trusted pairs
Chunk editing made parser coordinates unreliable, so all merges were
downgraded to pure text matching, including unedited documents.

Restore position-aware merging for pairs whose coordinates are still
trustworthy, and keep the text fallback for edited or stale content so
it is never dropped on coordinates.
2026-08-06 12:14:25 +08:00
MidoriKurage 3abeba31dd test(chunks): add tests to pin contracts for position-aware merge
- Feature contracts: trusted pairs trim overlaps, join adjacent chunks
  seamlessly, and stay separate across position gaps.
- Safety contracts: never drop current content on coordinates alone.
2026-08-06 12:14:25 +08:00
MidoriKurage e2a75db67f chore(types): add internal only ContentRevision to types.SearchResult 2026-08-06 12:14:25 +08:00
PP1 f16a4ede93 fix(wiki): exclude archived pages from stats queries 2026-08-06 11:41:32 +08:00
Greenplumwine 439af21f9b
feat(feishu-driver): 新增飞书云盘数据源 (#2466)
* feat(datasource): add Feishu/Lark Drive (云盘) connector

Add a new Drive (云盘) data source alongside the existing Feishu wiki
connector, letting users sync documents/files under a user-supplied Drive
folder_token. Shares the feishu connector package (Client/Config/Region/
export/download/error-handling) with the wiki connector; only resource
enumeration and fetch dispatch differ.

Backend:
- types/datasource.go: ConnectorTypeFeishuDrive / ConnectorTypeLarkDrive
- region.go: RegionFeishuDrive / RegionLarkDrive + driveFolderURL
- types.go: driveFile / driveFileListResponse / partialDriveFileListError
  / feishuDriveCursor
- client.go: ListDriveFiles / ListDriveFilesRecursiveFrom (shortcut
  expansion via shortcut_info, visited cycle guard, root-folder rejection)
- drive_connector.go: DriveConnector implementing StreamingConnector
  (FetchStream) + FetchAll + FetchIncremental fallback paths, mirroring
  the wiki connector's three cursor semantics (resume fast-path,
  failure-does-not-advance-cursor, toSyncCursor snapshot), plus
  ListResources / ResolveResourceAncestors (top-down shared traversal,
  best-effort) / fetchDriveFileContent / parseDriveResourceID
- container.go + connector.go: register two Drive instances + metadata

Incremental detection uses modified_time returned directly by the list
API (verified) - no batch_query/metas call needed. Shortcuts are expanded
at list-parse time via shortcut_info; target_type is never "folder".

Frontend:
- DataSourceEditorDialog.vue: connectorDefs for feishu_drive/lark_drive,
  Drive root folder_token input + load button before the lazy-load tree,
  edit-mode prefill from resource_ids
- datasourceIcons.ts: reuse feishu/lark icons
- i18n (zh-CN/en-US/ko-KR/ru-RU): connector labels, descriptions, drive
  input group

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): refine Feishu Drive connector UI, channel and root folder

A batch of fixes from end-to-end verification of the Feishu/Lark Drive
connector, all aimed at matching the wiki connector's behavior:

Channel (knowledge "source" label):
- Add ChannelFeishuDrive / ChannelLarkDrive constants so Drive docs show
  "飞书云盘" / "Lark 云盘" instead of "unknown", distinct from the wiki
  connector's "飞书" (ChannelFeishu).
- fetchDriveFileContent / appendDriveFileListFailureItems select the
  channel by region via a new driveChannel() helper.
- ingestItem now prefers metadata["channel"] over ds.Type so the
  connector-supplied channel value reaches the knowledge base.
- doc-content.vue channelLabelMap maps feishu_drive/lark_drive.

Frontend Drive picker UX:
- loadDriveRoot persists the new folder_token in edit mode too (previously
  skipped updates in edit mode, so listResources read the old token).
- The folder_token input is now always visible alongside the tree (not a
  switch); the tree area shows a "load a folder first" placeholder until
  the first successful load.
- extractDriveFolderToken accepts a bare token or a Feishu Drive folder
  URL (https://xxx.feishu.cn/drive/folder/<token>) and extracts the token.
- Edit mode auto-loads when a saved folder_token exists, and reveals
  pre-existing selections via revealExistingSelections.
- Classify Drive list errors (403/forbidden, 401/auth, not-found) into
  actionable i18n hints.

Root folder name & selection:
- Add GetDriveFolderMeta (GET /drive/explorer/v2/folder/:folderToken/meta)
  to resolve the root folder's human-readable name; the list API only
  returns children. Best-effort, falls back to the token.
- Fix root folder ExternalID to be the bare rootFolderToken (no ":token"
  suffix) so it matches the saved resource_id form and the direct
  children's ParentID, which previously broke selection matching on edit.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): sync a single selected Drive file without folder error

When a user selects a single file (not a folder) in the Drive picker, the
resourceID is "rootFolderToken:fileToken". FetchStream/FetchAll/
FetchIncremental previously passed fileToken straight to
ListDriveFilesRecursiveFrom, which expects a folder token - the Feishu
list API returns 1061002 (params error) for a file token, failing the
whole sync with "all fetched items failed".

The wiki connector resolves a single selected node via GetWikiNode; Drive
has no single-file meta API (folder meta only works on folders, and
metas/batch_query requires doc_type as input - a chicken-and-egg).
Instead, listDriveFilesForResource walks the root folder subtree (the
file's parent) and filters to the selected fileToken, yielding the full
driveFile (type/name/modified_time) needed for fetchDriveFileContent.

A sub-folder selection (fileToken is itself a folder) still walks that
sub-folder directly - ListDriveFilesRecursiveFrom accepts a folder token,
so no filtering is needed there.

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for
the current branch and exports them as .tar.gz artifacts downloadable
from the Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Co-Authored-By: Claude <noreply@anthropic.com>

* ci: add build-test-images workflow for tar.gz artifact export

Manual-only workflow_dispatch that builds linux/amd64 Docker images for a
selected branch and exports them as .tar.gz artifacts downloadable from the
Actions run page. No Docker Hub push, no secrets required.

- build-ui: builds frontend dist, builds weknora-ui image, exports tar.gz
- build-app: builds weknora-app image (CGO + duckdb), exports tar.gz
- Input choice: build both / app only / ui only
- Artifacts retained 7 days

Placed on the default branch so the workflow appears in the Actions list;
trigger it with the branch you want to build (e.g. feat/datasource-feishu-drive).

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks extraction design spec

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu shared-blocks implementation plan

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): sync feishu drive docx via blocks API with export fallback

- Extract wiki/drive shared helpers from connector.go into shared.go
- Make fetchDocxWithBlocks a package-level function taking docxFetchInput
- Drive docx now renders via blocks API (Markdown + attachment/image
  sub-items), falling back to the export API on blocks errors or empty
  renders, mirroring the wiki connector
- fetchDriveFileContent returns item slices and takes multimodalEnabled

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): label feishu drive knowledge as 飞书云盘 in list

The knowledge list source column (DocumentListView.getSourceInfo) only
whitelisted feishu/notion/yuque/etc., so feishu_drive/lark_drive docs
fell through to the default "上传" label while the wiki connector's
"feishu" showed "飞书". Map feishu_drive → 飞书云盘 and lark_drive →
Lark 云盘 (i18n keys already existed), and add feishu_drive to the
source filter options in KnowledgeBase.vue.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): polish feishu drive resource-step UI in datasource editor

- Replace the two stacked t-alert banners under the folder_token input:
  shareHint becomes an always-visible form-desc line, rootNotSupportedHint
  moves into a help-circle tooltip next to the label.
- folder_token is now required: red asterisk on the label, inline error
  status + tips on the input (replacing the global MessagePlugin) when
  empty on 加载/下一步, cleared on input.
- Step 2 选择范围 no longer scrolls as a whole: the token input stays
  fixed and the resource region (placeholder / loading / empty / tree)
  fills the remaining drawer height, with the tree scrolling internally.

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): add feishu drive datasource integration guide

User-facing setup guide for feishu_drive/lark_drive: app creation,
required scopes (drive:drive:readonly / drive:export:readonly /
docx:document:readonly), sharing the folder to the app's group, the
four-step wizard, supported file types, sync semantics and FAQ.
Also clarify in extractDriveFolderToken's comment that pasted folder
URLs are matched by path and work for Lark links too.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): unify feishu wiki/drive sync into a single generic streaming engine

Extract a single generic streaming sync engine (engine.go) shared by the wiki Connector and the Drive DriveConnector. Per-connector differences (node type, listing API, edit-time field, cursor wire format, fetch dispatch, log tag) are isolated behind the NodeOps[N] adapter; FetchAll/FetchIncremental become thin shells over the same engine.

Deliberate behaviour changes (design §2.4): the #2136 failure-doesn't-advance-cursor fix now applies to FetchIncremental too (previously it advanced the cursor before fetching, permanently skipping a node on a transient export failure); logs use 'stream progress/summary' uniformly and the FetchIncremental path gains per-100 progress + tally summary logs.

Structural prep for package split (design §3.1): anonymous Data structs in types.go/blocks.go are named so tests can construct wire types across packages.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): split feishu connector into core/wiki/drive packages

Move shared code (Client, blocks, markdown, types, region, engine, shared helpers) into feishu/core; wiki and drive connectors into their own packages. Export core symbols (capitalize) so wiki/drive can reference them; wiki/drive import core and prefix cross-package references.

container.go registration updated to import core/wiki/drive (NewConnector from wiki, NewDriveConnector from drive, Region* from core).

Tests are NOT migrated yet (left in feishu/ root, package feishu) and will fail to compile until moved + adapted in a follow-up commit. Source builds clean (go build ./...feishu/... passes).

Co-Authored-By: Claude <noreply@anthropic.com>

* test(datasource): migrate feishu tests to core/wiki/drive packages

Move 14 test files to their subpackages per design §4. Adapt package declarations, imports, and capitalized core symbol references. Add TestMain (SSRF whitelist) to core and drive. Add local helpers (writeJSON, makeConfig, txt, blk constructors, recordingHandler) where cross-package sharing was not possible. Fix engine.go log wording (summary) clobbered by the capitalization script.

All tests pass: core 4s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): reduce feishu core export surface; add drive cursor round-trip test

Lower-case 87 core symbols that are not referenced cross-package back to unexported (BlocksToMarkdown, CellToString, response types, BlockType constants, etc.), shrinking core's API surface to only what wiki/drive actually use. Restore NodeOps.Fetch capitalization (interface method must match wikiOps/driveOps implementations).

Add TestDriveCursorRoundTrip (design §3.2): driveOps.EncodeCursor/DecodeCursorTimes were new code with no direct test.

All tests pass: core 3.5s, wiki 13s, drive 3s.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(datasource): repair feishu cleanup fallout

Restore core.RegionLarkDrive (cleanup lower-cased it; only container.go references it, outside wiki/drive scan). Restore wiki/connector_realapi_test.go local collectHandler type (cleanup wrongly prefixed core.). Apply gofumpt to helpers_test.go. Add open.feishu.cn/open.larksuite.com to test SSRF whitelist: BaseURL-default tests call ParseFeishuConfig which SSRF-validates the resolved hostname; under a proxy open.feishu.cn resolves to 198.18.0.0/15 restricted range.

All tests pass: core 3.2s, wiki 13s, drive 2.6s. make build-lite passes.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(frontend): feishu drive data source UI and i18n

Add Drive folder/token picker strings and data source editor entries for the feishu_drive/lark_drive connectors across en-US/zh-CN/ko-KR/ru-RU.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: add trellis journal merge=union rule

Append-only developer journals merge cleanly across parallel sessions; task state lives in task.json, not the journal.

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor(datasource): rename feishu/drive drive_connector.go to connector.go

Aligns naming with feishu/wiki subpackage (connector.go).

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(frontend): allow data:image base64 URIs in sanitized markdown

DOMPurify's ALLOWED_URI_REGEXP rejected data: URIs, so <img src="data:image/...">
in uploaded markdown files had their src stripped during sanitizeHTML — the file
preview showed broken images while the chunk view (minio:// via
hydrateProtectedFileImages) worked. Add data:image/ to the allowed URI scheme so
base64 images render in preview and in LLM answers that embed base64 images.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): add FEISHU_DOCX_PARSE_MODE for feishu docx parse path

The blocks API path renders image blocks as empty ![图片]() placeholders and
fans images into separate knowledge items, breaking image↔document association
across retrieval/wiki/agent. FEISHU_DOCX_PARSE_MODE=export forces the async
export API (.docx binary -> docreader), so images are parsed inline and bound to
the parent document via parent_chunk_id (same as a regular docx upload). Default
(unset / "blocks") keeps the existing blocks-first behaviour. Affects both wiki
and drive connectors since they share FetchDocxWithBlocks.

Co-Authored-By: Claude <noreply@anthropic.com>

* feat(datasource): default FEISHU_DOCX_PARSE_MODE to export

Switch the feishu docx parse default from blocks to export so embedded
images are parsed inline and bound to the parent document via
parent_chunk_id (retrieval / wiki / agent all associate images). Operators
who need the blocks behaviour (faster, keeps docx attachments, no
docreader round-trip) set FEISHU_DOCX_PARSE_MODE=blocks.

- shared.go: default unset -> export; update comment to match
- .env.example / docker-compose.yml: default export
- tests: pin FEISHU_DOCX_PARSE_MODE=blocks on blocks-behaviour tests so
  they don't break under the new default

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(datasource): document FEISHU_DOCX_PARSE_MODE and refresh feishu source layout

- 飞书云盘接入说明: add section on docx parse mode (export default vs
  blocks), permission differences, export tradeoffs
- 数据源导入开发: add env var + mode comparison
- 数据源导入开发文档: rewrite content fetch flow (blocks/export paths),
  add pros/cons table, fix source-file table to core/wiki/drive layout

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: 修复合并main后丢失新增的国际化问题

* fix(i18n): 修复误删的国际化内容

* docs: 删除误提交的文件

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-05 19:39:49 +08:00
ttommybot 0827cd6cf0 fix: retry document summaries before fallback 2026-08-05 19:29:32 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 03d6f90c5a fix(faq): prevent duplicate entries when a create is retried
A caller that resends CreateFAQEntry before the first call answers could
create several rows for the same standard question. Indexing runs inline and
the embedding call retries with exponential backoff, so one create can take
far longer than an impatient caller waits; meanwhile the duplicate-question
check only looked at ChunkStatusIndexed, so a sibling create still mid-flight
was invisible and every retry passed the check.

- Serialize concurrent creates of the same question behind a Redis SetNX
  guard, keyed by tenant + knowledge base + question hash, with an in-process
  fallback when Redis is unavailable so an outage cannot block FAQ authoring.
- Widen the duplicate check to every non-deleted status. This also fixes
  ChunkStatusDefault entries, which are listed to users but never blocked
  duplicates.
- Bound the indexing step of an interactive create so a degraded embedding
  service cannot stretch the request out; bulk and background indexing keep
  the full retry budget.
- Report a failed rollback instead of discarding the error, since a leftover
  stored chunk is hidden from the list yet now matches the duplicate check.
2026-08-05 17:05:51 +08:00
wizardchen 1985a59b69 fix(chunking): sync preview swagger and normalize splitter config
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
2026-08-05 16:17:48 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
xiaowang d2888e9565 fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
wizardchen e71c2dce2e fix(api): address resource_urls review follow-ups
Harden public URL mode: stop logging signed URLs at INFO, copy messages
before rewriting history responses, reuse winning grants on insert races,
sync the Go client resource_urls query param, and add knowledge-search
coverage.
2026-08-05 14:42:51 +08:00
wizardchen 8b00cb71a3 perf(storage): reuse a live resource access grant per resource
Resolving a resource:// handle into a public URL inserted one capability row
every time. Rendering an answer resolves the same image across many chunks and
re-reading a message history resolves it again on every call, so a read endpoint
behaved like a write-heavy one.

The table stores only sha256(token), so an existing grant cannot be reused by
looking it up — the plaintext token is unrecoverable. Derive the token instead:
HMAC-SHA256(SYSTEM_AES_KEY, resource id + time window). The same resource in the
same window derives the same token, so the row can be found by its hash and
reused; only the first request in a window inserts. The window is half the TTL,
so a reused grant always has at least TTL/2 of life left.

This keeps the storage properties intact: the table still holds only hashes and
the plaintext cannot be reconstructed from a dump without SYSTEM_AES_KEY, while
authorization stays entirely on the row, so expiry and revocation still decide
whether a token resolves. Without SYSTEM_AES_KEY, grants stay random and
per-request as before. A stable URL per window also lets client and CDN caches
hit.

Stop deleting revoked grants during cleanup. A revoked row has to survive until
its expiry, otherwise a token derived for the same resource and window would
re-create it and revive the access that was just revoked. grant.revoked_at has
no writer yet, so this changes nothing today, but the derived token makes the
tombstone load-bearing.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
xiaowang 51b88bbae3 fix(summary): preserve rows across table chunks / 修复表格分块摘要丢行 2026-08-05 13:12:59 +08:00
qilifan 3507156929
feat(storage): support AWS S3 NoAk default credentials (#2008)
* feat(storage): support AWS S3 default credentials

* fix(storage): allow clearing S3 credentials

* fix static checks and trim locale diff

---------

Co-authored-by: qilifan <4359902+qilifan@users.noreply.github.com>
2026-08-05 11:59:02 +08:00
wizardchen 5723e837f8 fix(chat): cover web partial success and wiki-only embed fallback
Refine embed-failure target filtering so wiki/graph-only KBs and missing
KB metadata still reach HybridSearch instead of hard search_failed.
Add regression tests for KB failure with concurrent web hits and wiki-only
empty recall after embedding errors.
2026-08-04 21:12:51 +08:00
KKtwo 22af9b2f78 fix(chat): preserve retrieval fallback errors 2026-08-04 21:12:51 +08:00
wizardchen 2f2be9f601 refactor(knowledge): improve folder path handling and add tests for underscore scenarios
- Updated SQL queries to properly escape LIKE wildcards in folder path filters.
- Added tests for renaming knowledge folder paths, including cases with underscores in paths.
- Enhanced existing tests to ensure correct folder listing behavior when using underscores.
2026-08-04 20:45:29 +08:00
lyingbug 5fea7a1695 feat(knowledge): browse folders like a file manager and allow re-filing
The scope switch is gone. It existed because "select a folder" was ambiguous
in a list that could not show folders, and turning that ambiguity into a
control just moved the problem onto the user.

The list now shows what a folder contains: its sub-folders as entries first,
then the documents directly inside it. So the mode follows what the user is
already doing instead of a switch - browsing shows one level, and any active
filter searches the whole subtree flat, with a breadcrumb note saying so. A
document uploaded on its own needs no special row either: it sits at the top
level next to the folders, exactly where it is.

Folders can also be adjusted now, which they could not before - a mis-filed
upload previously had to be deleted and re-uploaded. Documents move via the
row menu or the batch bar into an existing or newly typed folder, and a
folder can be renamed in place, carrying its subtree. Both are plain
folder_path updates: nothing is re-parsed, re-chunked or re-embedded.
2026-08-04 20:45:29 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
lyingbug 82ea91f18e fix(knowledge): clear stale error_message on the remaining reprocess paths
Follow-up to the error_message clearing added for SetFinalizing,
FinalizeSubtask and ProcessDocument. Two routes still persisted a
previous attempt's failure message:

- ProcessManualUpdate flips a row back to 'processing' without clearing
  error_message. A manual knowledge item that failed earlier (for
  example when the knowledge base lookup failed, or after an Asynq
  retry) re-enters processing still carrying that message, so the UI
  shows a failure on a document it also shows as running.

- finalizeIndexedKnowledgeState is the only route to 'completed' that
  bypasses FinalizeSubtask. When a document has no text chunks and no
  pending multimodal work it is completed in place, so the promotion
  that would otherwise clear the column never runs and a successfully
  indexed row keeps reporting the earlier failure.

The 'processing' transition is extracted into markKnowledgeProcessing so
ProcessDocument and ProcessManualUpdate share one definition of what
starting a new attempt means, and a future worker cannot reintroduce the
same omission. The helper also replaces the "processing" string literal
with types.ParseStatusProcessing.

Tests: markKnowledgeProcessing gets direct coverage, and the
finalizeIndexedKnowledgeState table now seeds a stale error_message and
asserts it survives the processing branches (FinalizeSubtask clears it
later) and is cleared on the completed branch.
2026-08-03 20:14:09 +08:00
shangfeng-liu 44206e4ad4 fix(knowledge): clear stale error_message on reprocessing and finalize
A knowledge item that failed once keeps error_message set; when it is
reprocessed or driven through the finalizing pipeline, the stale message
was never cleared, so the UI kept showing an outdated failure on items
that were processing or had completed successfully.

- SetFinalizing and FinalizeSubtask now clear error_message in the same
  atomic UPDATE that transitions parse_status
- ProcessDocument clears knowledge.ErrorMessage when flipping the row
  to processing for a new attempt

Add a SQLite-backed regression test covering both transitions.
2026-08-03 19:38:22 +08:00
shiquan_zhan ccb0cee3e6 fix: exclude archived wiki pages from lint cursor 2026-08-03 17:30:41 +08:00
xiaowang 60028cbe0e fix(rerank): preserve code and math passages 2026-08-03 15:10:09 +08:00
Myang_print 5780affdfc fix(knowledge): surface batch reparse failures
Signed-off-by: Myang_print <1182470296@qq.com>
2026-07-31 19:06:16 +08:00
xiaowang 2093fa313a fix(chat): segment Chinese query expansions with jieba 2026-07-31 19:02:16 +08:00
wizardchen 8dd997f0f4 refactor(knowledge): make the supported import extension set the single gate
The xlsx URL-import fix left the accepted extension set reachable through two
indirections and widened ApplyKnowledgeProcessOverrides to gate file types for
callers that are not file imports (web URL fetch, connector sync, reparse).

Collapse the set into one map that both isValidFileType and the extension check
read, move the shared validation next to the other file-type predicates, and
return ApplyKnowledgeProcessOverrides to its override-only scope so
resolveFileImportProcessConfig is the only import gate.

Also drop the unreachable branch in createKnowledgeFromFileURL: getFileType
never yields an empty string, so the resolvedFileType fallback and its else arm
could not run. Normalize the resolved type before it reaches the record and the
task payload, and add an invariant test asserting direct upload and URL import
accept the same extensions so #2447 cannot drift back.
2026-07-31 13:46:56 +08:00
wizardchen 43afa4c1f8 fix(knowledge): align URL file import validation with direct upload
Extract shared file-import checks (type, VLM/ASR, video) into
resolveKnowledgeFileImportConfig and reuse them for file URL import.
Also enqueue DataTableSummaryTask for spreadsheet URL imports and reparse.
2026-07-31 13:46:56 +08:00
wizardchen 48756d13d6 fix(knowledge): allow xlsx and other file types in URL import (#2447)
Align URL-based file import validation with direct file upload by
delegating to isValidFileType instead of a narrower whitelist.
2026-07-31 13:46:56 +08:00
lazyayuan 29c52b0342 fix(evaluation): fix retrieval metrics by mapping chunks back to passage IDs
Two issues caused retrieval metrics (precision, recall, NDCG, MRR, MAP)
to always be 0:

1. recordFinish compared SearchResult.ChunkIndex (the chunk's ordinal
   position in the knowledge base) against QAPair.PIDs (the dataset's
   original passage IDs). These are different ID spaces that never match.
   Fix: use content-based matching to map each retrieved chunk back to
   its source passage, with deduplication for multi-chunk passages.

2. getPassageList had an off-by-one error: make([]string, maxPID) with
   loop i < maxPID dropped the passage with the highest PID.
   Fix: use maxPID+1 and i <= maxPID.

Also adds fallback from rerankResult to searchResult when no rerank
model is configured, matching the pattern in the working copy.
2026-07-31 11:05:13 +08:00
lazyayuan 9e6fee3620 fix(evaluation): use sync passage indexing to fix always-zero retrieval metrics
EvalDataset used CreateKnowledgeFromPassage (async), which returns
immediately while chunking/embedding/vector-indexing runs in the
background. The subsequent RAG queries then hit an empty vector store,
so retrieval metrics (precision, recall, NDCG, MRR, MAP) were always 0.

Switch to CreateKnowledgeFromPassageSync, which waits for indexing to
complete before returning — the same pattern already used in
web_search.go for the identical create-then-query flow.
2026-07-31 11:05:13 +08:00
yuefanxiao 80c2064ce9 fix: 修复 SQLite 下删除 DataSource 失败 2026-07-31 11:00:47 +08:00
wizardchen 3afdc796b0 fix(sqlite): treat zero vector threshold as no filter
Match Milvus/Tencent VectorDB and CLI semantics where threshold 0
means no score floor, and add a regression test for anti-correlated
vectors that would otherwise be dropped.
2026-07-31 10:59:40 +08:00
xiaowang ca19596874 fix(sqlite): filter vector candidates before top-k 2026-07-31 10:59:40 +08:00
sunmh 59cbe58330 fix(sessions): 放行 embed 运行时读取自身会话,修复追问建议 500 与刷新丢对话 2026-07-30 17:51:34 +08:00
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00
wizardchen ab9f705843 fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
wizardchen db9bdc27cd fix(session): scope API runtime admin-console bypass to owned sessions
Replace the broad channel-runtime principal check with owner-scoped rules so
API callers can still read their own api_tenant_key and api_external_user
sessions without opening IM sessions matched via legacy empty user_id scope.
2026-07-30 11:56:05 +08:00
wizardchen 123f2d6349 fix(session): include external users in API session group 2026-07-30 11:56:05 +08:00
xiaowang f3cd3d9724 fix(knowledge): support local HTML uploads
English: Accept .html and .htm knowledge uploads across frontend, Go, and DocReader. Parse static HTML with charset-aware decoding, preserve document structure and local links, and keep scripts/network execution disabled. Add regression coverage for validation, engine registration, encoding, and parser behavior.

中文:打通知识库本地 .html 与 .htm 文件从前端、Go 服务到 DocReader 的完整上传链路。使用可识别字符编码的静态解析,保留文档结构和本地链接,不执行脚本或发起网络请求,并补充文件校验、引擎注册、编码及解析行为的回归测试。
2026-07-29 20:11:17 +08:00
wizardchen 7d2a6311cb fix(agent): exclude disabled FAQ entries 2026-07-29 14:40:54 +08:00
wizardchen c291c6a59f fix(kb): retry KB delete when engine resolution is deferred
ProcessKBDelete must return retryable engine-resolution failures instead
of logging and continuing, which could delete knowledge rows while leaving
embeddings behind. Align the handler with ProcessIndexDelete and cover
cancellation and ErrVectorStoreUnavailable in tests.
2026-07-29 12:25:18 +08:00
ochan.kwon c15c3e972f fix(retriever): rebuild a missing vector store engine on demand
The engine registry is per-process, so a store's engine is missing
whenever this process did not create it. That happens three ways:

  - Startup skips a store whose engine fails to construct and never
    retries it, so that instance cannot serve the store again -- not
    even after a restart, if the backend is slow to come up.
  - A store registered on one instance is absent on every other until
    they restart, because registration is not broadcast.
  - Registration can fail on the instance handling the create, leaving
    the store missing even there.

Each one surfaces as "vector store is currently unavailable" and is
cleared today only by restarting the process.

Add GetOrLoadByStoreID, which falls back to building the engine from the
store row when the lookup misses, and route the two bound-store lookups
through it. The method goes on RetrieveEngineRegistry rather than
StoreRegistry because that is the interface the factory functions
receive; the two declare GetByStoreID separately and neither embeds the
other.

Concurrent callers collapse onto one build. That build runs on a context
detached from the caller that started it: callers share the result, so
letting the first one's cancellation abort it would fail everyone else
waiting. A panic inside the build is contained, because singleflight
re-raises it on a goroutine the HTTP recovery middleware cannot reach
and the build calls third-party client constructors.

Publishing is guarded by a per-store generation sampled before the
build, so an engine that finishes after the entry was registered or
removed by someone else does not overwrite it and leave that engine
orphaned with its connections open. A failed build starts a cooldown:
collapsing only helps concurrent callers, so without it a backend that
stays down would cost a full build timeout on every sequential request.

Separate "the store is gone" from "its engine cannot be built right
now". Async delete handlers answer ErrVectorStoreNotFound with
asynq.SkipRetry, so reporting a database blip or a backend outage that
way would permanently discard knowledge-base and index delete tasks and
orphan the vector data they exist to remove. ErrVectorStoreUnavailable
covers the retryable cases; the repository already distinguishes them,
returning (nil, nil) for a missing row and an error for a failure.

For the same reason a caller's own cancellation is never folded into a
store sentinel. The ownership check runs first and queries the database
with that context, so on a shutdown it is usually where cancellation is
noticed.

A search resolves one engine per distinct store, sequentially, and this
server sets no HTTP read or write timeout, so the resolution loop gets a
budget above a single build. An exhausted budget still leaves the
engines warming, since the build is detached, and is reported as an
unavailable store because retrying is likely to succeed.

Build failures are logged where they happen, with the panic stack. The
cause cannot travel back to the caller -- factory errors embed endpoints
and credentials -- so without this the one thing an operator needs when
self-healing fails to heal would be discarded with the error.

Note a deliberate contract change: an ownership lookup failure used to
be reported as not-found, and a test asserted that on purpose. A
database failure says nothing about whether the store exists, and
treating it as permanent is what discards the task, so that assertion is
inverted rather than preserved.
2026-07-29 12:25:18 +08:00
wizardchen bf68db4653 fix(storage): allow image uploads to use file service fallback 2026-07-29 11:36:24 +08:00
wizardchen 5db186a0fc fix(knowledge): harden file-type scoped duplicate detection
Use case-insensitive file_type matching, scope the filename fallback by type, and document the dedup contract for datasource sweep handling.
2026-07-29 10:36:17 +08:00
koishi514-Z 18b95ebb3a fix(knowledge): distinguish duplicate files by type 2026-07-29 10:36:17 +08:00
wizardchen 2358564bde fix(knowledge): default custom metadata on create 2026-07-28 22:22:59 +08:00