Commit Graph

9 Commits (014f95f1394a7bfd1c7bbbdcdc19057a80dcaca5)

Author SHA1 Message Date
lyingbug 6db8fdd601
feat(artifacts): artifacts library page backed by a message_artifacts table (#3384)
* feat(artifacts): add an artifacts library with its own message_artifacts table

Skill-generated files were only reachable from inside the conversation that
produced them. This adds a sidebar entry and a page that lists them across
every conversation the user can see.

Storage: artifacts move out of the messages.artifacts JSONB column into a new
message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on
upgrade. The table is now the only source of truth; the old column is kept and
the down migration writes the table back into it. The message repository
loads artifacts for every message it returns and writes them on create,
update (when non-nil) and fork, so callers still use Message.Artifacts.
mod_time is stored as RFC 3339 text because the collector matches files on
nanosecond mtimes, which PostgreSQL timestamps would truncate.

API: GET /api/v1/artifacts lists the latest version of each file (grouped by
session + source path, with version_count), scoped like the session list,
with keyword / file type filters and paging. Downloads keep using the
per-session endpoint.

Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a
library page with type tabs, search, date groups, infinite loading, preview
drawer, download and jump-to-conversation; strings in all five locales. The
sidebar's top-menu paths are now one shared list.

* fix(artifacts): keep the restore hash guard and count versions by URL

RecordRestoredArtifactMtime again only stamps rows whose content hash matches
the restored content (and ignores empty hashes), and never rewrites the hash,
matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of
v1's content also moved v2's mtime and relabelled v2 with v1's hash.

The library's version_count now counts distinct URLs per (session, source
path). A later answer that references an earlier file stores another row with
the same URL, which COUNT(*) reported as an extra version.

* fix(artifacts): scope the library to web chats and clear the legacy column

The library used the session list's default scope, which counts sessions
without an owner as legacy tenant-level rows. IM sessions are created without
an owner, so every member saw the file names and titles of every IM chat
(downloads then 404 for non-admins). The library now applies the same "web"
bucket predicate as the home sidebar, extracted from QueryPaged so the two
cannot drift: no IM mapping (including cleared ones), no embed marker, no
API-key owner.

Migrations 000103 / SQLite 000023 now clear messages.artifacts after the
backfill. Message loads still SELECT *, so the stale copy was read on every
history load. Only rows that carried artifacts are rewritten; the column stays
for instances on the old binary during a rolling upgrade, and the down
migration still rebuilds it from the table.

* fix(migrations): keep the 000103 backfill alive on malformed artifact JSON

A single file_size that is not an integer ("12.5", "") or a created_at that
does not parse ("garbage", month 13) raised a cast error and failed the whole
migration, blocking startup. Casts now go through pg_temp helpers that return
NULL on bad input, falling back to 0 and the message's created_at; valid rows
are unchanged. SQLite casts never raise; its migration test now covers the
same malformed values.

* docs(artifacts): describe the web-only library scope and the cleared legacy column
2026-09-18 12:10:19 +08:00
wizardchen 17f893865d fix(memory): rank memories by the query, not by importance
Long-term memory chose its candidates before it looked at the question.
Recall listed 400 items by `importance DESC, valid_from DESC`, loaded the
vectors belonging to that list, and scored those; extraction did the same
with a 200-item pool. Importance is uncorrelated with relevance, so the
vector stage could only re-order what importance had already picked, and a
memory below the cut was unreachable no matter how well it matched. The
capacity cap goes up to 2000, so everything past the 400th most important
memory was permanently unrecallable, and `memory_item_embeddings.vector`
being a BYTEA blob meant pgvector was never involved.

The store now answers "which of this person's memories is this question
about":

- Migration 000095 (and the SQLite mirror) add `embedding halfvec` plus a
  `(tenant_id, subject_id, model_id, dims)` index. The column is
  conditional on the vector extension, so a deployment whose retrieval
  driver excludes postgres keeps working. No HNSW: an approximate index is
  filtered after the fact, and one subject holds at most a couple of
  thousand rows, so exact ordering inside the scope is both correct and a
  few milliseconds.
- `SearchItemsByVector` ranks in SQL over every vector the subject has,
  with the status, expiry and kind filters joined in so reaching past the
  candidate list cannot reach past the filters. SQLite and pgvector-less
  PostgreSQL fall back to scoring in process, still over the whole scope.
- Recall fuses the lexical ranking with those hits and grows the candidate
  pool with matches the pool did not hold, which is what lets the search
  escape it. The lexical pool is now the capacity cap, so it can no longer
  be a second, invisible limit under the one that decides what stays active.
- Extraction picks the memories it shows the model the same way. This
  matters more than recall: the memory a new statement contradicts is not
  reliably an important one, so an importance-ordered window hid exactly
  the collision that stops a duplicate being written.
- Vectors are written to the searchable column in the upsert's transaction;
  rows predating the column are moved by `SyncVectorColumn` during
  maintenance, which costs no model calls. The embedding backfill goes from
  50 to 200 per run — at 50, a subject at the capacity cap took over a
  month to become fully searchable.
- Traces and logs report `outside_pool`, the number of semantic matches the
  lexical pool did not contain. It was structurally zero before, so it is
  the number to watch when asking whether semantic recall does anything.

Tests cover whole-scope ranking, the lifecycle and kind filters, pool
growth, the pool-size invariant, and extraction seeing a low-importance
memory it would otherwise duplicate. A pgvector integration test
(WEKNORA_MEMORY_TEST_POSTGRES_DSN) exercises the migration, the halfvec
write, the SQL ranking and the column sync.
2026-09-13 11:43:17 +08:00
lyingbug fd7fa6a7d4
fix(memory): preserve extraction progress and atomic memory updates (#3224)
* fix(memory): preserve extraction progress and atomic memory updates

* fix(memory): bound extraction failures and persist indexed session progress
2026-09-13 00:04:16 +08:00
wizardchen 10cc694fed feat(memory): introduce long-term memory feature with i18n updates
- Added a new configuration option for agents to enable or disable long-term memory.
- Implemented UI elements in the Agent Editor for managing long-term memory settings.
- Updated internationalization files for English, Korean, Russian, and Chinese to include new labels and descriptions related to long-term memory.
- Enhanced backend logic to support memory retrieval and management based on user and agent preferences.
- Introduced a new tool for searching long-term memory, ensuring it respects the agent's memory settings.
- Added tests to verify the functionality of the new memory features and their integration with existing systems.
2026-08-13 18:00:02 +08:00
wizardchen fda037e7f3 feat(memory): add forced consolidation feature and related i18n updates
- Introduced a new method to mark memories as forced consolidated, allowing users to request immediate consolidation.
- Added a new message for users indicating that a tidy-up just ran and to try again shortly.
- Updated internationalization files for English, Korean, Russian, and Chinese to include the new message.
- Enhanced memory repository and service logic to support the new forced consolidation functionality.
- Updated database schema to include a new timestamp for forced consolidation.
2026-08-13 18:00:02 +08:00
wizardchen 5b0b34195e refactor(memory): enhance item listing and model deletion logic
- Updated the item listing query to ensure deterministic paging by including the item ID in the order clause.
- Added a new test to verify that models used by memory extraction cannot be deleted, ensuring proper error handling.
- Refactored the model deletion logic to check both embedding and extraction models against tenant memory configurations.
- Introduced a new method to handle tombstone records during memory clearing, ensuring that live memories retain their tombstone status when cleared.
2026-08-13 18:00:02 +08:00
wizardchen ffcb10f953 feat(memory): add cross-session long-term memory
Persist extracted and explicit memories across sessions with topic
resolution, semantic recall, document affinity, consolidation, workspace
settings, and inbox/interests UI.
2026-08-13 18:00:02 +08:00
wizardchen 380e371c2e refactor: remove Neo4j-based conversation memory feature
Drop the memory pipeline, API fields, settings UI, and embed toggles so chat no longer depends on Neo4j graph storage for episodic recall.
2026-07-18 23:41:48 +08:00
joeyczheng 12baa885d2 feat: implement preview of user level memory graph
Signed-off-by: joeyczheng <joeyczheng@tencent.com>
2026-02-25 14:29:54 +08:00