Commit Graph

2578 Commits (fa67dfd44bc1ecef2d80a97841b62d0cd505a1b5)

Author SHA1 Message Date
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
ljl 2b31d3681b fix(embedding): prevent ants worker panic and mutex deadlock on empty/mismatched results
If the embedding provider returns an empty slice of vectors (e.g. length 0)
due to api mismatch or empty input, BatchEmbed will return length 0.
Accessing embedding[i] directly in BatchEmbedWithPool results in an out of bounds panic.
Because the panic occurs inside a mu.Lock() block, the mutex is never unlocked,
leading to a permanent deadlock of the document processing queue.

This fix checks the returned embeddings slice length and returns an error
instead of panicking, releasing the mutex properly.
2026-08-04 16:46:57 +08:00
lyingbug fcc4cd6a9f fix(frontend): announce RAG wait status from a persistent live region
role="status" lived on the wait row itself, so the live region was inserted
together with its own text and screen readers had nothing to announce. Move the
announcement to an sr-only region that stays mounted for the whole turn, which
also covers the pre-pipeline wait row that had no announcement at all.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-04 13:21:23 +08:00
lyingbug 2454eaf253 fix(frontend): cover attachment-only turns and stalled streams in the RAG wait row
The wait row only appeared once a completed knowledge_search step existed, so
attachment-only turns (attachment_parsing / image_analysis with no KB hit) still
showed nothing between the last step and the first answer token. Model waiting is
now a three-state kind: 'model' after retrieval finished, the neutral 'preparing'
row otherwise.

A dropped SSE connection never sets is_completed (the stream layer only raises a
toast), so the row used to promise an answer forever. It now stops claiming
progress after RAG_WAIT_STALL_DELAY_MS and drops the shimmer.

Also share RAG_RETRIEVAL_TOOL_NAMES instead of repeating the knowledge_search /
search_knowledge alias check, name the reveal delay, and move the reveal/stall
timers into a controller with an injectable scheduler so the timing is covered by
tests instead of by source-matching alone.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-04 13:21:23 +08:00
ttommybot 747cbcbc02 fix: show answer generation status after retrieval 2026-08-04 13:21:23 +08:00
ttommybot 91b2d31cd4 fix: unify organization settings modal scrolling 2026-08-03 23:54:10 +08:00
lyingbug 82ea91f18e fix(knowledge): clear stale error_message on the remaining reprocess paths
Follow-up to the error_message clearing added for SetFinalizing,
FinalizeSubtask and ProcessDocument. Two routes still persisted a
previous attempt's failure message:

- ProcessManualUpdate flips a row back to 'processing' without clearing
  error_message. A manual knowledge item that failed earlier (for
  example when the knowledge base lookup failed, or after an Asynq
  retry) re-enters processing still carrying that message, so the UI
  shows a failure on a document it also shows as running.

- finalizeIndexedKnowledgeState is the only route to 'completed' that
  bypasses FinalizeSubtask. When a document has no text chunks and no
  pending multimodal work it is completed in place, so the promotion
  that would otherwise clear the column never runs and a successfully
  indexed row keeps reporting the earlier failure.

The 'processing' transition is extracted into markKnowledgeProcessing so
ProcessDocument and ProcessManualUpdate share one definition of what
starting a new attempt means, and a future worker cannot reintroduce the
same omission. The helper also replaces the "processing" string literal
with types.ParseStatusProcessing.

Tests: markKnowledgeProcessing gets direct coverage, and the
finalizeIndexedKnowledgeState table now seeds a stale error_message and
asserts it survives the processing branches (FinalizeSubtask clears it
later) and is cleared on the completed branch.
2026-08-03 20:14:09 +08:00
shangfeng-liu 44206e4ad4 fix(knowledge): clear stale error_message on reprocessing and finalize
A knowledge item that failed once keeps error_message set; when it is
reprocessed or driven through the finalizing pipeline, the stale message
was never cleared, so the UI kept showing an outdated failure on items
that were processing or had completed successfully.

- SetFinalizing and FinalizeSubtask now clear error_message in the same
  atomic UPDATE that transitions parse_status
- ProcessDocument clears knowledge.ErrorMessage when flipping the row
  to processing for a new attempt

Add a SQLite-backed regression test covering both transitions.
2026-08-03 19:38:22 +08:00
shiquan_zhan ccb0cee3e6 fix: exclude archived wiki pages from lint cursor 2026-08-03 17:30:41 +08:00
wizardchen a8cf3a2b09 fix(mcp-server): normalize shared KB list shape for name resolution
GET /shared-knowledge-bases returns rows with nested knowledge_base
(sharedKBRow), unlike GET /knowledge-bases which returns flat KB objects.
resolve_kb_id and agent_chat preflight were reading top-level id/name,
so shared KB names never matched. Add _normalize_kb_entries and tests.
2026-08-03 15:50:33 +08:00
FincherYuan 1e75a34300 test(mcp-server): bump tool count assertion to 29 for new shared-KB tool
Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-03 15:50:33 +08:00
FincherYuan 71bdecf873 fix(mcp-server): integrate shared KBs into name resolution and tool hints
resolve_kb_id now searches both own and shared knowledge bases when
resolving a name to UUID. Error messages and docstrings in hybrid_search,
chat, and agent_chat now reference list_shared_knowledge_bases as well.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-03 15:50:33 +08:00
FincherYuan 5f4464db8d feat(mcp-server): add list_shared_knowledge_bases tool
Add a new MCP tool to list knowledge bases shared from other workspaces
via the GET /shared-knowledge-bases endpoint, which was previously not
exposed through the MCP server.

- Add WeKnoraClient.list_shared_knowledge_bases() method
- Register list_shared_knowledge_bases() as an @mcp.tool()

Co-Authored-By: Claude <noreply@anthropic.com>
2026-08-03 15:50:33 +08:00
shangfeng-liu 583b87b1b9 fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
ttommybot d3055ae068 fix: count skipped stages in parsing progress 2026-08-03 15:20:16 +08:00
xiaowang 60028cbe0e fix(rerank): preserve code and math passages 2026-08-03 15:10:09 +08:00
xiaowang 31a2358bbf fix(rerank): normalize NVIDIA logit scores 2026-08-03 15:07:15 +08:00
jacentsao cbac06564d fix(chat): keep attachment upload status reactive 2026-08-03 15:01:57 +08:00
xiaowang fe65633f38 fix(rerank): batch LKEAP requests within API limits 2026-08-03 14:59:16 +08:00
Myang_print 5780affdfc fix(knowledge): surface batch reparse failures
Signed-off-by: Myang_print <1182470296@qq.com>
2026-07-31 19:06:16 +08:00
xiaowang 2093fa313a fix(chat): segment Chinese query expansions with jieba 2026-07-31 19:02:16 +08:00
wizardchen 8dd997f0f4 refactor(knowledge): make the supported import extension set the single gate
The xlsx URL-import fix left the accepted extension set reachable through two
indirections and widened ApplyKnowledgeProcessOverrides to gate file types for
callers that are not file imports (web URL fetch, connector sync, reparse).

Collapse the set into one map that both isValidFileType and the extension check
read, move the shared validation next to the other file-type predicates, and
return ApplyKnowledgeProcessOverrides to its override-only scope so
resolveFileImportProcessConfig is the only import gate.

Also drop the unreachable branch in createKnowledgeFromFileURL: getFileType
never yields an empty string, so the resolvedFileType fallback and its else arm
could not run. Normalize the resolved type before it reaches the record and the
task payload, and add an invariant test asserting direct upload and URL import
accept the same extensions so #2447 cannot drift back.
2026-07-31 13:46:56 +08:00
wizardchen 43afa4c1f8 fix(knowledge): align URL file import validation with direct upload
Extract shared file-import checks (type, VLM/ASR, video) into
resolveKnowledgeFileImportConfig and reuse them for file URL import.
Also enqueue DataTableSummaryTask for spreadsheet URL imports and reparse.
2026-07-31 13:46:56 +08:00
wizardchen 436d3e0068 chore: stop tracking local .cursor IDE config
Remove .cursor hooks from the repo and drop gitignore exceptions so
local Cursor rules and hooks stay out of version control.
2026-07-31 13:46:56 +08:00
wizardchen 48756d13d6 fix(knowledge): allow xlsx and other file types in URL import (#2447)
Align URL-based file import validation with direct file upload by
delegating to isValidFileType instead of a narrower whitelist.
2026-07-31 13:46:56 +08:00
wizardchen 0d58af8d82 fix(frontend): route embed protected images through channel file proxy
Embed visitors have no Bearer token, so resource:// images must be fetched
via /api/v1/embed/:channelId/files. Centralize proxy selection in a
protected-file access context and register the embed plane at EmbedPage so
deep renderers like AgentStreamDisplay no longer fall back to /files (401).
2026-07-31 13:29:34 +08:00
wizardchen e99a4dd498 fix(frontend): restore audit and KB activity i18n after locale prune
Re-add flat-key translations for audit log and knowledge-base activity events
that were dropped or nested incorrectly during i18n pruning. Add a shared
action registry, English defaults, and prune safeguards so regenerate-i18n-locales
reads the working tree and cannot strip these runtime keys again.
2026-07-31 11:57:36 +08:00
wizardchen 5443b3814c chore(cursor): run frontend CI before gh pr create
Add scripts/verify_frontend_pr.sh and a beforeShellExecution hook that
blocks PR creation when frontend test, type-check, or build fails.
2026-07-31 11:54:20 +08:00
wizardchen a1a07a52eb fix(frontend): satisfy KB editor type-check after create save
Use a narrowed createdKbId local when reloading data and emitting success
so vue-tsc accepts the post-create flow.
2026-07-31 11:54:20 +08:00
wizardchen 23ae5e179c fix(frontend): close editor on save and guide post-create setup
Align agent and knowledge base editors so edit saves close the modal,
first-time create stays open with contextual footer hints, and follow-up
actions point users to sharing/integration or data-source configuration.
2026-07-31 11:54:20 +08:00
lazyayuan 29c52b0342 fix(evaluation): fix retrieval metrics by mapping chunks back to passage IDs
Two issues caused retrieval metrics (precision, recall, NDCG, MRR, MAP)
to always be 0:

1. recordFinish compared SearchResult.ChunkIndex (the chunk's ordinal
   position in the knowledge base) against QAPair.PIDs (the dataset's
   original passage IDs). These are different ID spaces that never match.
   Fix: use content-based matching to map each retrieved chunk back to
   its source passage, with deduplication for multi-chunk passages.

2. getPassageList had an off-by-one error: make([]string, maxPID) with
   loop i < maxPID dropped the passage with the highest PID.
   Fix: use maxPID+1 and i <= maxPID.

Also adds fallback from rerankResult to searchResult when no rerank
model is configured, matching the pattern in the working copy.
2026-07-31 11:05:13 +08:00
lazyayuan 9e6fee3620 fix(evaluation): use sync passage indexing to fix always-zero retrieval metrics
EvalDataset used CreateKnowledgeFromPassage (async), which returns
immediately while chunking/embedding/vector-indexing runs in the
background. The subsequent RAG queries then hit an empty vector store,
so retrieval metrics (precision, recall, NDCG, MRR, MAP) were always 0.

Switch to CreateKnowledgeFromPassageSync, which waits for indexing to
complete before returning — the same pattern already used in
web_search.go for the identical create-then-query flow.
2026-07-31 11:05:13 +08:00
yuefanxiao 80c2064ce9 fix: 修复 SQLite 下删除 DataSource 失败 2026-07-31 11:00:47 +08:00
wizardchen 3afdc796b0 fix(sqlite): treat zero vector threshold as no filter
Match Milvus/Tencent VectorDB and CLI semantics where threshold 0
means no score floor, and add a regression test for anti-correlated
vectors that would otherwise be dropped.
2026-07-31 10:59:40 +08:00
xiaowang ca19596874 fix(sqlite): filter vector candidates before top-k 2026-07-31 10:59:40 +08:00
ttommybot f7ef782d58 fix: preserve wiki folder path after creation 2026-07-30 21:02:06 +08:00
wizardchen 25f3751138 fix(frontend): prune i18n locales, add audit tooling, and fix UI warnings
Regenerate four locale bundles from static usage analysis (6024 -> 4592 keys),
align translations across locales, escape vue-i18n @ syntax, and add check-i18n
plus gap-scan tooling. Also fix TDesign drawer/textarea teardown warnings and
add missing tenantOthers section labels.
2026-07-30 20:58:00 +08:00
sunmh 59cbe58330 fix(sessions): 放行 embed 运行时读取自身会话,修复追问建议 500 与刷新丢对话 2026-07-30 17:51:34 +08:00
“ZhuangShiHao1207” b67ed27e1d fix(mcp): resolve non-UUID agent IDs 2026-07-30 17:34:37 +08:00
wizardchen 2611babfef ci(frontend): remove docs terminology coupling 2026-07-30 17:32:44 +08:00
wizardchen 0bf9286a2e test(mcp-server): keep thread-local sessions alive 2026-07-30 17:32:44 +08:00
zldkr 1168505eef fix(docparser): normalize PaddleOCR-VL Cloud HTML tables 2026-07-30 16:08:24 +08:00
wizardchen a76c6ef24e fix(mcp-server): adopt official PyPI package tencent-weknora-mcp [v1.1.1]
Rename the distribution to tencent-weknora-mcp, point metadata at
Tencent/WeKnora, bump to 1.1.1 for the transport/session fixes, and
tighten publish to main/tags only via Trusted Publishing.
2026-07-30 15:52:28 +08:00
wizardchen 1e12361cb8 fix(mcp-server): restore HTTP/SSE transport compat after MCPServer migration
Keep stateless HTTP and /sse/messages/ paths aligned with pre-2.x behavior,
use thread-local requests.Session for concurrent tool handlers, and add
transport regression tests.
2026-07-30 15:52:28 +08:00
veater 37837c00a9 fix(mcp-server): migrate to MCPServer high-level API (mcp 2.x) [v1.1.0]
MCP Python SDK 2.0 removed the low-level Server decorator API
(@app.list_tools() / @app.call_tool() / app.get_capabilities()), so the
server crashed at startup with:
  AttributeError: 'Server' object has no attribute 'list_tools'

The released PyPI package weknora-mcp==1.0.1 declared 'mcp>=1.0.0' with
no upper bound, so uvx resolved to the latest 2.0.0 and the plugin's
weknora MCP server could no longer connect.

Fix: rewrite the server on top of the mcp 2.x high-level MCPServer API
(formerly FastMCP), and cap the dependency at 'mcp>=2,<3'.

- weknora_mcp_server.py: replace @app.list_tools()/@app.call_tool() with
  28 @mcp.tool() function-signature tools (input schema auto-generated
  from type hints, descriptions from docstrings, plain return values
  serialized by the framework). Transports use run_stdio_async() /
  sse_app() / streamable_http_app(). WeKnoraClient business logic is
  unchanged.
- deps: mcp>=2,<3 in pyproject.toml / requirements.txt / setup.py;
  commit uv.lock pinning mcp 2.0.0.
- version bump to 1.1.0 (pyproject, setup, __init__, main --version,
  server_version).
- check_imports.py: validate MCPServer / ToolError / mcp_types imports.
- ci(.github/workflows/mcp-server.yml): publish on merge to main +
  manual dispatch, not only on tags; add PyPI version-dedup check so
  re-runs on an unchanged version stay green (absorbed from PR #2399).

Verified: stdio initialize + tools/list returns all 28 tools; no
list_tools AttributeError; tool dispatch + error wrapping confirmed.

Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-07-30 15:52:28 +08:00
wizardchen cbe668c06f fix(ci): scope gofmt check to PR commits and format kb_access_test
Compare base...head for pull_request events instead of the merge commit
so unrelated main-branch files are not checked. Also gofmt
kb_access_test.go, which was unformatted on main.
2026-07-30 14:13:15 +08:00
wizardchen fafa925b26 perf(ci): speed up App workflow Go dependency caching
Use setup-go v6 default go.mod cache key instead of go.sum, warm the
module cache with an explicit go mod download step, and scope vet to the
same package set as tests.
2026-07-30 14:13:15 +08:00
wizardchen 4efc987abb fix(ci): make DocReader client tests runnable in PR checks
Start the DocReader gRPC server in the DocReader workflow, skip client
integration tests when the server is unavailable, exclude docreader
packages from the App workflow test scope, and trigger DocReader checks
on go.mod/go.sum changes.
2026-07-30 14:13:15 +08:00
wizardchen a52137d450 ci: add app frontend and docreader checks 2026-07-30 14:13:15 +08:00
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00