WeKnora/internal/application/service
ochan.kwon 744a367f16 feat(retriever): add OpenSearch type prep ahead of Phase 3 driver
Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.

Changes:

- OpenSearchRetrieverEngineType constant ("opensearch") in
  internal/types/retriever.go. Not added to validEngineTypes /
  GetVectorStoreTypes / retrieverEngineMapping yet (gated).

- ConnectionConfig.InsecureSkipVerify (bool, default false) in
  internal/types/vectorstore.go, placed inside the // Common section
  because it is a cross-driver TLS option. Distinct from the
  Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
  only controls verification of an already-TLS HTTPS connection.
  AES-GCM Value/Scan round-trips the field as plaintext (it travels
  alongside the encrypted Password / APIKey but is not sensitive itself).

- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
  Min/Max (*float64), Enum ([]string). All omitempty so existing UI
  schema entries serialize identically. The fields will drive the UI
  in Phase 3 PR 3 (read-only on Edit + range/enum constraints).

- Six new AuditAction constants under vector_store.* and opensearch.*
  namespaces. Definitions only — emission lands in Phase 3 PR 3.

- EngineAwareNormalizer.Normalize is restructured to group engines by
  the effective score range observed by the normalizer (not the
  theoretical raw cosine range):

    Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
      - Milvus (COSINE metric mode). Milvus docs explicitly state the
        COSINE metric range is [-1, 1].

    Range [0, 1] (passthrough — already on the target scale):
      - Elasticsearch v8 / ElasticFaiss. The driver issues a
        cosineSimilarity(...) script_score script, and Lucene rejects
        negative final scores ("Final relevance scores from the
        script_score query cannot be negative" — ES docs); the
        effective range observed by the normalizer is therefore [0, 1]
        for IR-normalized embeddings. ES was previously grouped with
        Milvus and over-corrected via (score + 1) / 2, which inflated
        every ES result by 50% in mixed-engine RRF fusion.
      - OpenSearch. The k-NN plugin's
        SpaceType.COSINESIMIL.scoreTranslation maps the underlying
        Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
        before the score reaches us. Source:
        github.com/opensearch-project/k-NN at
        src/main/java/org/opensearch/knn/index/SpaceType.java
        (COSINESIMIL enum, scoreTranslation method).
      - Weaviate. The driver requests `certainty`, defined by Weaviate
        as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
      - Postgres pgvector / SQLite sqlite-vec / Qdrant /
        TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
        or normalized inner_product, whose theoretical range is [-1, 1].
        The IR-normalized positive-component unit vectors WeKnora
        targets (BGE / OpenAI text-embedding-3 / Cohere /
        sentence-transformers) keep the observed range in [0, 1];
        negative-cosine embedding models would silently clamp to 0
        downstream — explicitly documented as the IR-normalization
        caveat in the struct godoc.

  Dead enum references (ElasticFaiss, Infinity) are flagged in the
  godoc with a pointer to internal/types/vectorstore.go's existing
  "legacy/experimental, no standalone deployable instance" annotation.
  Their case labels are kept for switch exhaustiveness.

Test coverage:

- OpenSearch constant wire value + collision check against existing
  10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
  field deserializes as false) + round-trip + AES-GCM coexistence
  with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
  + *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
  for vector_store.* and opensearch.*, no wire-string collisions, exact
  wire-value pins for the six new constants.
- EngineAwareNormalizer:
  - CosineRange retains its (score + 1) / 2 coverage but now only for
    Milvus.
  - UnitInterval now covers the full passthrough group (ES /
    ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
    Infinity / TencentVectorDB / Doris).
  - New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
    explicit regression guard for the score-range correction: cos=0.5
    maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
  - OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
    (1.0001) + defensive negative + ±Inf / NaN edges + keyword
    passthrough + nil-ctx safety.

Backward compatibility is preserved at every layer:

- All new struct fields are omitempty / pointer-tagged so existing
  rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
  lands. The ES regrouping changes the post-normalization value for
  every ES vector search result (a correctness fix, not a feature) —
  ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).
2026-05-22 20:43:50 +08:00
..
chat_pipeline feat(observability): expand Langfuse spans across chat pipeline 2026-05-12 21:16:37 +08:00
file feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
memory feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
metric fix: resolve failing test and build errors 2026-04-22 21:17:29 +08:00
retriever feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
agent_history.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_history_test.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_service.go fix(agent): exclude processing docs from prompt 2026-05-18 16:49:14 +08:00
agent_service_test.go fix(agent): exclude processing docs from prompt 2026-05-18 16:49:14 +08:00
agent_share.go fix(rbac): allow source-tenant Admin+ to manage their cross-tenant shares 2026-05-18 17:28:58 +08:00
audit_log.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_retention.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_retention_test.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
audit_log_test.go feat(rbac): per-tenant audit log for RBAC events (PR 6, #1303) 2026-05-18 17:28:58 +08:00
chunk.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
custom_agent.go feat(custom_agent): enhance custom agent service with KB sharing support 2026-05-18 17:28:58 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource_service.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
evaluation.go feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
extract.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
graph.go feat(graph): enhance graph extraction prompt rendering and language context handling 2026-03-25 22:08:29 +08:00
image_multimodal.go feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
kbshare.go fix(rbac): allow source-tenant Admin+ to manage their cross-tenant shares 2026-05-18 17:28:58 +08:00
knowledge.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledge_clone_move.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
knowledge_create.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_delete.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
knowledge_faq.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledge_faq_import.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
knowledge_image_cleanup_test.go fix(knowledge): delete extracted images from storage when knowledge is removed 2026-04-02 12:04:04 +08:00
knowledge_manual_test.go fix: sanitize tab chars and avoid double .md extension in manual download filename 2026-03-16 18:18:23 +08:00
knowledge_owner_chain_test.go feat(rbac): per-KB ownership for knowledge / chunks / wiki pages (#1303) 2026-05-18 17:28:58 +08:00
knowledge_post_process.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
knowledge_process.go fix(summary): preserve image caption/OCR text in document summaries 2026-05-22 17:25:39 +08:00
knowledge_process_status_test.go fix(knowledge): complete indexed documents immediately 2026-05-20 22:40:48 +08:00
knowledge_shared_access_test.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledge_storage_config_test.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
knowledge_summary_test.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
knowledge_util.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledgebase.go feat(knowledge-base): implement per-user pinning for knowledge bases 2026-05-18 17:28:58 +08:00
knowledgebase_pr3_test.go feat(tests): enhance vector store tests with fake Elasticsearch server and add user pinning methods to knowledge base repository 2026-05-18 17:57:46 +08:00
knowledgebase_search.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fanout.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fanout_test.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_faq.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
knowledgebase_search_fusion.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_results.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
knowledgebase_search_shared.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
knowledgebase_search_storegroup.go feat(search): fan-out KB retrieval across bound vector stores 2026-05-20 22:25:39 +08:00
mcp_service.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
mcp_service_test.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
mcp_tool_approval_service.go feat(agent): human-in-the-loop approval for MCP tool calls (#1173) 2026-05-10 22:57:12 +08:00
message.go fix(session): scope session access by user 2026-05-14 15:56:43 +08:00
metric_hook.go refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
model.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
ocr_sanitizer.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
ocr_sanitizer_test.go feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
organization.go feat(organization): transition membership model to tenant-level and enhance invite functionality 2026-05-18 17:28:58 +08:00
session.go feat(session): persist last request state for UI restoration 2026-05-18 22:28:00 +08:00
session_agent_qa.go refactor(api): drop deprecated tenant ConversationConfig field and KV write path 2026-05-18 17:28:58 +08:00
session_knowledge_qa.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
session_knowledge_qa_test.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session_qa_helpers.go feat(agent): support dedicated model for query understanding step 2026-05-12 20:27:20 +08:00
session_user_scope_test.go fix(session): scope session access by user 2026-05-14 15:56:43 +08:00
skill_service.go feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
slug_fuzzy.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
slug_fuzzy_test.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
tag.go feat(rbac): lift org membership from per-user to per-tenant (Plan 3, #1303) 2026-05-18 17:28:58 +08:00
tenant.go refactor(rbac): address PR 1 review feedback 2026-05-18 17:28:58 +08:00
tenant_invitation.go feat(invitations): implement pending invitations feature with polling and dialog 2026-05-18 17:28:58 +08:00
tenant_invitation_test.go feat(invitations): implement pending invitations feature with polling and dialog 2026-05-18 17:28:58 +08:00
tenant_member.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
tenant_member_test.go fix(audit_log): prevent deadlock in Stop() when Start() is never called 2026-05-18 17:28:58 +08:00
user.go feat(tenant): remember user's last-active workspace across logins 2026-05-18 20:37:57 +08:00
user_jwt_test.go fix(auth): honour JWT tenant_id claim in middleware 2026-05-18 17:28:58 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go feat: expose KB ↔ vector store binding in list, editor, and detail UI 2026-05-22 17:40:10 +08:00
vectorstore_healthcheck.go feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
vectorstore_test.go feat(tests): enhance vector store tests with fake Elasticsearch server and add user pinning methods to knowledge base repository 2026-05-18 17:57:46 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
web_search_state.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_ingest.go fix(wiki-ingest,asynqdl): review fixups for PR #1241 2026-05-10 00:01:06 +08:00
wiki_ingest_batch.go fix(wiki-ingest,asynqdl): review fixups for PR #1241 2026-05-10 00:01:06 +08:00
wiki_ingest_cite.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_ingest_cite_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_retry_test.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_ingest_test.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_linkify.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_linkify_test.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_lint.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go perf(wiki): subgraph API and interactive exploration for large graphs 2026-05-08 13:32:54 +08:00