Commit Graph

16 Commits (fc2c12e8ec08efc9b948438fb720db1ff88ca9ca)

Author SHA1 Message Date
wizardchen 17f893865d fix(memory): rank memories by the query, not by importance
Long-term memory chose its candidates before it looked at the question.
Recall listed 400 items by `importance DESC, valid_from DESC`, loaded the
vectors belonging to that list, and scored those; extraction did the same
with a 200-item pool. Importance is uncorrelated with relevance, so the
vector stage could only re-order what importance had already picked, and a
memory below the cut was unreachable no matter how well it matched. The
capacity cap goes up to 2000, so everything past the 400th most important
memory was permanently unrecallable, and `memory_item_embeddings.vector`
being a BYTEA blob meant pgvector was never involved.

The store now answers "which of this person's memories is this question
about":

- Migration 000095 (and the SQLite mirror) add `embedding halfvec` plus a
  `(tenant_id, subject_id, model_id, dims)` index. The column is
  conditional on the vector extension, so a deployment whose retrieval
  driver excludes postgres keeps working. No HNSW: an approximate index is
  filtered after the fact, and one subject holds at most a couple of
  thousand rows, so exact ordering inside the scope is both correct and a
  few milliseconds.
- `SearchItemsByVector` ranks in SQL over every vector the subject has,
  with the status, expiry and kind filters joined in so reaching past the
  candidate list cannot reach past the filters. SQLite and pgvector-less
  PostgreSQL fall back to scoring in process, still over the whole scope.
- Recall fuses the lexical ranking with those hits and grows the candidate
  pool with matches the pool did not hold, which is what lets the search
  escape it. The lexical pool is now the capacity cap, so it can no longer
  be a second, invisible limit under the one that decides what stays active.
- Extraction picks the memories it shows the model the same way. This
  matters more than recall: the memory a new statement contradicts is not
  reliably an important one, so an importance-ordered window hid exactly
  the collision that stops a duplicate being written.
- Vectors are written to the searchable column in the upsert's transaction;
  rows predating the column are moved by `SyncVectorColumn` during
  maintenance, which costs no model calls. The embedding backfill goes from
  50 to 200 per run — at 50, a subject at the capacity cap took over a
  month to become fully searchable.
- Traces and logs report `outside_pool`, the number of semantic matches the
  lexical pool did not contain. It was structurally zero before, so it is
  the number to watch when asking whether semantic recall does anything.

Tests cover whole-scope ranking, the lifecycle and kind filters, pool
growth, the pool-size invariant, and extraction seeing a low-importance
memory it would otherwise duplicate. A pgvector integration test
(WEKNORA_MEMORY_TEST_POSTGRES_DSN) exercises the migration, the halfvec
write, the SQL ranking and the column sync.
2026-09-13 11:43:17 +08:00
lyingbug fd7fa6a7d4
fix(memory): preserve extraction progress and atomic memory updates (#3224)
* fix(memory): preserve extraction progress and atomic memory updates

* fix(memory): bound extraction failures and persist indexed session progress
2026-09-13 00:04:16 +08:00
lyingbug 7e53c31671
feat(browser): connect local Chrome with task controls and display modes (#3180)
* feat(browser): integrate BrowserSkill local task tabs and account pairing

* feat(browser): refine task routing, lifecycle, and display modes

* fix(browser): reclaim temporary task tabs and groups

* test(database): cover browser authorization migration schema
2026-09-11 17:59:27 +08:00
ninesun 1e36e071ec
feat(mcp): add per-tool enable toggle (#2989)
* feat(mcp): add per-tool enable toggle

* style: format migration test

* fix(mcp): keep enable and approval updates from overwriting each other

Partial policy writes now touch only the supplied columns, and a missing
tenant fails closed so a disabled tool cannot slip through.

---------

Co-authored-by: myysophia <myysophia@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-09-05 17:19:20 +08:00
ochan.kwon 894cbec48a feat(agent): report and attribute per-turn LLM token usage
Accumulate token usage across every LLM round of an agent turn and
expose the aggregate everywhere a turn is observable:

- AgentState.TurnUsage accumulates each round's reported usage,
  including the final-answer synthesis call; TokenUsage.Accumulate
  keeps the cache counters as prompt subsets and recomputes the
  combined cache status (unsupported survives only while every call
  was itself unsupported).
- The agent completion event carries the aggregate, and the complete
  stream event exposes it both as data.usage and as a typed
  StreamEvent.Usage that buildStreamResponse promotes into the
  previously-unused StreamResponse.Usage wire field.
- The assistant message persists it as a nullable usage column on
  both migration targets (postgres 000085 jsonb, sqlite 000012 text),
  for web and IM channels alike, so history reads still carry the
  turn's cost after the live stream is gone.
- "[LLM Usage]" log lines gain ", session_id=..., principal=type:id"
  when the context carries them; calls outside sessions stay
  byte-identical.

Usage from failed retry attempts is not accumulated: the per-call
usage log remains the ground truth, while the turn aggregate serves
as the operational signal.
2026-08-24 10:36:19 +08:00
wizardchen 71fe7f31ee fix(migrations): 补齐 SQLite 迁移与 versioned 的索引与数据迁移缺口
对齐 000005/010/011 的邀请索引、tag_id 迁入与 MCP principal 唯一索引,
并扩展测试覆盖分享链接、OAuth upsert 与 v4 标签迁移路径。
2026-08-20 14:35:49 +08:00
icatw b888172dce fix(migrations): 补齐 Lite 模式缺失的 SQLite 版本化迁移
migrations/sqlite 落后于 migrations/versioned,导致 Lite 模式全新数据库
首次建号/建空间失败(tenants.api_principal_config、system_settings、
task_pending_ops 等缺失)。新增 000005-000011 七个 SQLite 迁移,
按 versioned 迁移主题一一补齐:

- 000005  <- 000034/000054: messages.attachments、tenant_invitations token/accepted_count
- 000006  <- 000041: task_pending_ops / task_dead_letters(与 #2433 同一 DDL)
- 000007  <- 000053: system_settings、users.is_system_admin
- 000008  <- 000055/000056: knowledge_processing_spans、knowledges.pending_subtasks_count
- 000009  <- 000060: embed_channels.allow_memory
- 000010  <- 000063: knowledge_tag_relations
- 000011  <- 000064: tenants.api_principal_config、mcp_oauth_tokens principal 列

测试:新增 internal/database 迁移测试,覆盖全新库迁移到 v11 与
v4->v11 升级保数据两条路径;go test ./internal/database 全绿。

关联 #2158;与 #2433 范围重叠(任务表部分 DDL 保持一致)。
2026-08-20 14:35:49 +08:00
lkx 926f1ac582 fix(knowledge): address auto-tag review findings 2026-08-07 15:01:01 +08:00
lkx 9426ee1470 feat: 增加自动添加标签的功能 2026-08-07 15:01:01 +08:00
wizardchen 6f95c75ed2 feat(system-info): surface DB migration errors with troubleshooting links
When a startup database migration fails (e.g. issue #1319: pg_trgm not
available so 000041 cannot build its trigram index), the application
intentionally keeps booting so operators can reach the UI to diagnose.
However, before this change the system info page silently dropped the
"DB Version" row because the value was empty:

  - migration.go only cached the version after a successful m.Up(); the
    error path returned early and left migrationVersionSet=false.
  - system.go used CachedMigrationVersion's ok=false to skip emitting
    db_version, and the JSON tag was already omitempty.
  - SystemInfo.vue gated the entire row on v-if="systemInfo?.db_version".

The end result was the most useful diagnostic surface vanishing in the
exact failure mode that needs it most — Wiki ingest and KG features
would silently produce nothing with no UI hint.

Changes:

  - migration.go: replace sync.Once-based setter with an RWMutex-guarded
    state struct holding {version, dirty, err}. Every failure path now
    calls captureMigrationFailure(m, err), which best-effort reads
    m.Version() so the cached value still reflects the partial state.
  - system.go (handler): always emit db_version (falling back to
    "unknown" when no version could be read), append " (failed)" when an
    error is recorded, and add db_migration_error to the response.
  - swagger / client SDK: keep the API contract in sync with the new
    response field.
  - SystemInfo.vue: render the DB version row whenever either field is
    present, show a "Migration failed" danger tag, and add a full-width
    alert below the row carrying the error message plus two links:
      1. View troubleshooting guide -> new docs/migration-troubleshooting.md
      2. Report an issue -> github.com/Tencent/WeKnora/issues/new,
         prefilled with the captured error and environment metadata.
  - docs/migration-troubleshooting.md: new self-service guide covering
    the common failure modes (missing extension, dirty state, privileges,
    out of disk, schema drift) with concrete psql / make commands.
  - i18n: add the new keys to zh-CN, en-US, ko-KR, ru-RU.

Refs #1319.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-14 16:34:50 +08:00
wizardchen 9a2b54536d fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
wizardchen aefa1c6fe8 feat: enhance system information display with database version
- Added `db_version` field to the `SystemInfo` interface to expose the current database migration version.
- Updated the system information response to include the database version, reflecting its state during application runtime.
- Enhanced the UI in the SystemInfo component to display the database version with appropriate labels and descriptions in multiple languages.

This update improves transparency regarding the database state within the system information settings.
2026-03-02 21:21:49 +08:00
wizardchen 397689d2f3 feat: introduce WeKnora Lite edition with lightweight configuration and deployment
- Added a new `.env.lite.example` file for the Lite version, providing a minimal configuration template.
- Updated `.env.example` to remove deprecated variables and include new Docreader settings.
- Enhanced Docker configurations to support the Lite version, including a new Dockerfile for the Docreader service.
- Introduced a Makefile target for building and running the Lite version, along with packaging capabilities.
- Created GitHub workflows for building and releasing Lite binaries, including Homebrew formula support.
- Implemented a new service file for managing the Lite version as a system service.

This update enables a streamlined, single-binary deployment of WeKnora, reducing external dependencies and simplifying setup.
2026-03-02 21:21:49 +08:00
wizardchen c67a5d3949 feat: 支持数据库迁移脏状态自动恢复和Neo4j连接重试机制 2025-12-08 19:38:16 +08:00
wizardchen 9f6b903ef2 feat: Integrate LLM-based reranking in KnowledgeSearchTool
- Added optional chat model to KnowledgeSearchTool for enhanced reranking using LLM prompt scoring.
- Updated rerankResults method to prioritize LLM-based reranking over traditional rerank model.
- Implemented new functions for LLM scoring and parsing scores from responses, improving search result relevance.
- Enhanced migration script to handle database versioning and dirty state checks more robustly.
2025-11-13 01:22:08 +08:00
wizardchen 77b6133e2a feat: Add MCP services support and enhance migration scripts
- Introduced new MCP services functionality, including management routes and integration with the agent service.
- Updated Docker Compose configuration to include a new SQL migration for MCP services.
- Enhanced Makefile with new migration commands for versioning and creating migrations.
- Improved chat component to handle selected MCP services in message processing.
- Updated migration documentation to reflect new strategies and added MCP services migration files.
2025-11-09 03:27:24 +08:00