WeKnora/internal/types
wizardchen 17f893865d fix(memory): rank memories by the query, not by importance
Long-term memory chose its candidates before it looked at the question.
Recall listed 400 items by `importance DESC, valid_from DESC`, loaded the
vectors belonging to that list, and scored those; extraction did the same
with a 200-item pool. Importance is uncorrelated with relevance, so the
vector stage could only re-order what importance had already picked, and a
memory below the cut was unreachable no matter how well it matched. The
capacity cap goes up to 2000, so everything past the 400th most important
memory was permanently unrecallable, and `memory_item_embeddings.vector`
being a BYTEA blob meant pgvector was never involved.

The store now answers "which of this person's memories is this question
about":

- Migration 000095 (and the SQLite mirror) add `embedding halfvec` plus a
  `(tenant_id, subject_id, model_id, dims)` index. The column is
  conditional on the vector extension, so a deployment whose retrieval
  driver excludes postgres keeps working. No HNSW: an approximate index is
  filtered after the fact, and one subject holds at most a couple of
  thousand rows, so exact ordering inside the scope is both correct and a
  few milliseconds.
- `SearchItemsByVector` ranks in SQL over every vector the subject has,
  with the status, expiry and kind filters joined in so reaching past the
  candidate list cannot reach past the filters. SQLite and pgvector-less
  PostgreSQL fall back to scoring in process, still over the whole scope.
- Recall fuses the lexical ranking with those hits and grows the candidate
  pool with matches the pool did not hold, which is what lets the search
  escape it. The lexical pool is now the capacity cap, so it can no longer
  be a second, invisible limit under the one that decides what stays active.
- Extraction picks the memories it shows the model the same way. This
  matters more than recall: the memory a new statement contradicts is not
  reliably an important one, so an importance-ordered window hid exactly
  the collision that stops a duplicate being written.
- Vectors are written to the searchable column in the upsert's transaction;
  rows predating the column are moved by `SyncVectorColumn` during
  maintenance, which costs no model calls. The embedding backfill goes from
  50 to 200 per run — at 50, a subject at the capacity cap took over a
  month to become fully searchable.
- Traces and logs report `outside_pool`, the number of semantic matches the
  lexical pool did not contain. It was structurally zero before, so it is
  the number to watch when asking whether semantic recall does anything.

Tests cover whole-scope ranking, the lifecycle and kind filters, pool
growth, the pool-size invariant, and extraction seeing a low-importance
memory it would otherwise duplicate. A pgvector integration test
(WEKNORA_MEMORY_TEST_POSTGRES_DSN) exercises the migration, the halfvec
write, the SQL ranking and the column sync.
2026-09-13 11:43:17 +08:00
..
interfaces fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
agent.go fix(browser): complete task cancellation, screenshots, and result display (#3206) 2026-09-12 09:32:26 +08:00
agent_share_source.go fix: harden shared agent source selector validation 2026-07-30 12:28:23 +08:00
agent_share_source_test.go fix: harden shared agent source selector validation 2026-07-30 12:28:23 +08:00
agent_test.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
artifact_versions.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
artifact_versions_test.go feat(agent): unify sandbox tools and drop host preloaded skills (#3033) 2026-09-05 16:45:59 +08:00
audit_log.go feat(system-admin): add endpoint for creating users (#2722) 2026-08-20 19:51:10 +08:00
audit_log_test.go feat(system-admin): add endpoint for creating users (#2722) 2026-08-20 19:51:10 +08:00
browser_search.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
builtin_agent_config.go fix(i18n): 修复内置智能体名称与描述不随界面语言切换的问题 (#2828) 2026-08-27 11:53:45 +08:00
builtin_agent_localization_test.go fix(i18n): 修复内置智能体名称与描述不随界面语言切换的问题 (#2828) 2026-08-27 11:53:45 +08:00
builtin_models_config.go feat(models): let system admins edit builtin model config 2026-07-14 15:56:02 +08:00
builtin_models_config_test.go feat(models): let system admins edit builtin model config 2026-07-14 15:56:02 +08:00
caller.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
chat.go feat(sandbox): stream live shell and install command output 2026-09-12 10:16:39 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
chunk.go feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
config_redaction.go fix(sandbox): always require inbound credentials 2026-09-03 13:34:40 +08:00
config_redaction_sandbox_test.go fix(sandbox): always require inbound credentials 2026-09-03 13:34:40 +08:00
config_redaction_test.go feat(storage): support AWS S3 NoAk default credentials (#2008) 2026-08-05 11:59:02 +08:00
const.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
context_clone.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
context_clone_coverage_test.go refactor(context): replace middleware.EmbedChannelContextKey with types.EmbedChannelContextKey 2026-08-13 18:00:02 +08:00
context_helpers.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
context_helpers_test.go fix: retry document summaries before fallback 2026-08-05 19:29:32 +08:00
custom_agent.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
custom_agent_test.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(datasource): persist source created time; Feishu uses content edit time for source_updated_at (#3083) 2026-09-07 19:39:01 +08:00
datasource_test.go feat(feishu): sync docx via blocks API with per-type downdrill (#2087) 2026-07-27 16:04:56 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embed_channel.go feat(i18n): add Japanese (ja-JP) locale support (#3080) 2026-09-07 19:20:41 +08:00
embed_channel_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
embedding.go feat(agent): enhance suggested questions API with tag scopes 2026-07-17 10:19:43 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go fix: support custom jieba dictionary directory 2026-06-08 20:10:37 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go fix: remove GPL converter and bundle third-party license sources 2026-09-08 15:19:10 +08:00
faq_sync.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
faq_sync_test.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
faq_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
file_reference.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
file_reference_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
generated_question_revision_test.go fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
generated_question_source_id_test.go fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_map.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledge_custom_metadata_test.go feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
knowledge_folder.go fix(knowledge): keep long non-ASCII folder names valid UTF-8 2026-09-07 11:50:32 +08:00
knowledge_folder_test.go fix(knowledge): keep long non-ASCII folder names valid UTF-8 2026-09-07 11:50:32 +08:00
knowledge_process.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
knowledge_process_test.go feat(docreader): support forcing PDF scanned-page OCR parsing 2026-06-25 17:30:22 +08:00
knowledge_span.go fix(spans): expand knowledge span name column and truncate long wiki names 2026-07-13 10:35:44 +08:00
knowledgebase.go fix(parser): prefer anydoc for every type it converts when linked 2026-08-31 14:58:15 +08:00
knowledgebase_auto_tag_test.go fix(knowledge): harden auto-tagging and let it respect manual tags 2026-08-07 15:35:58 +08:00
knowledgebase_test.go fix(parser): prefer anydoc for every type it converts when linked 2026-08-31 14:58:15 +08:00
mcp.go feat(mcp)!: streamline setup and use short routing handles 2026-09-09 16:11:52 +08:00
mcp_metadata.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_oauth.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
memory.go fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
memory_extraction.go fix(memory): preserve extraction progress and atomic memory updates (#3224) 2026-09-13 00:04:16 +08:00
memory_test.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
message.go feat(browser): connect local Chrome with task controls and display modes (#3180) 2026-09-11 17:59:27 +08:00
message_attachment_test.go feat(im): process file attachments in QA 2026-08-12 16:08:51 +08:00
message_suggestion.go feat(agent): add question suggestions with after-answer follow-ups 2026-07-14 11:17:25 +08:00
message_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
model.go feat(agent): keep long sandbox turns from truncating, overflowing, or rewriting files 2026-09-02 22:09:09 +08:00
organization.go fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
parser_engine_config_test.go fix(parser): default ppt/pptx to markitdown when no engine rules (#2882) 2026-08-31 14:58:15 +08:00
placeholder.go feat(chat): keep prompt prefixes stable and send provider cache markers 2026-09-03 00:45:52 +08:00
placeholder_time_test.go feat(chat): keep prompt prefixes stable and send provider cache markers 2026-09-03 00:45:52 +08:00
principal.go fix(session): include external users in API session group 2026-07-30 11:56:05 +08:00
principal_test.go test(api-keys): add unit tests for tenant API key validation 2026-07-07 12:00:13 +08:00
prompt_instructions.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
prompt_instructions_test.go Enhance knowledge base configuration by normalizing and validating custom instruction fields. Implemented length checks for prompt instructions and added tests for validation logic. Updated GraphSettings to retain custom instructions when extraction is disabled. 2026-07-13 12:16:19 +08:00
qa_request.go feat(browser): connect local Chrome with task controls and display modes (#3180) 2026-09-11 17:59:27 +08:00
qa_request_test.go fix(chat): preserve steering continuity and composer focus 2026-09-09 20:15:23 +08:00
question_suggestion_test.go feat(agent): add question suggestions with after-answer follow-ups 2026-07-14 11:17:25 +08:00
resource.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
resource_test.go feat(agent): preview skill artifacts and verify installs without running them (#2823) 2026-08-27 10:30:32 +08:00
retrieval_config.go Fix/hybrid search match count normalization (#2750) 2026-08-20 11:36:38 +08:00
retriever.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
runtime_task.go feat(system): add cursor pagination for runtime queue tasks 2026-07-14 16:53:07 +08:00
sandbox_network_policy.go fix(sandbox): always require inbound credentials 2026-09-03 13:34:40 +08:00
sandbox_network_policy_test.go feat(sandbox): add tenant-configurable network policy (#2995) 2026-09-03 10:49:46 +08:00
sandbox_tenant_context_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
search.go fix(chunks): 流水线重写后的检索结果不用位置匹配 (#2558) 2026-08-06 14:32:24 +08:00
search_scope_test.go refactor(agent): introduce modelcontext registry and fix handle-codec bugs 2026-07-27 11:05:43 +08:00
secret.go fix(security): harden secret redaction follow-ups for PR #1884 2026-07-02 14:41:42 +08:00
secret_test.go fix(security): harden secret redaction follow-ups for PR #1884 2026-07-02 14:41:42 +08:00
session.go feat(browser): connect local Chrome with task controls and display modes (#3180) 2026-09-11 17:59:27 +08:00
session_access_test.go feat(sandbox): expose tenant skill management over HTTP and in the console (#2789) 2026-08-25 11:28:12 +08:00
shared_agent_access.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
shared_agent_access_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
storagebackend.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
storagebackend_test.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
sync_item_error_test.go fix(feishu): resilient large-wiki sync — retry, streaming resume, failure visibility 2026-07-21 16:14:22 +08:00
system_setting.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
tag.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
task.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_initiator_test.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_queue_test.go feat: 增加自动添加标签的功能 2026-08-07 15:01:01 +08:00
temporary_document.go fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
tenant.go feat(sandbox): add an interactive terminal and files tab beside chat (#3110) 2026-09-08 17:43:15 +08:00
tenant_api_key.go feat(api-keys): add platform-scoped API keys for control-plane automation 2026-07-21 17:21:35 +08:00
tenant_api_key_scope_test.go Add API key member and space capabilities 2026-07-07 12:00:13 +08:00
tenant_api_key_test.go Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
tenant_env_vars.go fix(sandbox): make skill env capture fill blanks only 2026-08-27 14:32:00 +08:00
tenant_env_vars_test.go fix(sandbox): make skill env capture fill blanks only 2026-08-27 14:32:00 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go Refine scoped API key capabilities 2026-07-07 12:00:13 +08:00
tenant_sandbox_config_entity.go feat(sandbox): 沙箱后端改为按空间配置,并补齐 E2B 模板与检测诊断 2026-08-11 15:42:02 +08:00
tenant_sandbox_config_entity_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
tenant_sandbox_config_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
tenant_skill.go fix(skills): store one catalog zip and cache compressed archives 2026-09-02 15:08:05 +08:00
tenant_test.go fix: preserve MinerU automatic PDF parsing 2026-08-03 15:37:07 +08:00
token_usage_test.go feat(agent): report and attribute per-turn LLM token usage 2026-08-24 10:36:19 +08:00
tracing.go feat(tracing): migrate langfuse client to OTLP/OTel wire protocol 2026-07-17 16:30:45 +08:00
user.go feat(browser): polish connection settings, open tasks in a new window, and streamline prompts (#3193) 2026-09-11 21:28:58 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go refactor(terminology): rename user-facing tenant labels to workspace 2026-07-14 16:14:43 +08:00
vectorstore_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_test.go feat(retriever): configure Tencent VectorDB replicas via env 2026-06-22 20:21:25 +08:00
web_search.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_search_filters.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_search_filters_test.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_search_provider.go fix(agent): improve web search and persistent page reading (#3086) 2026-09-07 23:09:27 +08:00
web_search_provider_test.go feat(web-search): add Bocha AI Search provider (#3036) 2026-09-07 20:06:26 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_page.go fix(wiki): keep folder names that look like page-type labels when placing pages 2026-09-07 17:23:19 +08:00
wiki_page_test.go fix(wiki): keep folder names that look like page-type labels when placing pages 2026-09-07 17:23:19 +08:00