WeKnora/internal/types
ochan.kwon 40b74e2efa feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3)
Phase 3 (#1440) gate flip. PR 1 (#1445) + PR 2a (#1481) + PR 2b (#1482)
laid the type prep + driver skeleton + read/write paths as gated dead
code; this PR wires every activation surface so opensearch becomes a
registerable VectorStore engine.

Activation wiring
- internal/types: validEngineTypes / GetVectorStoreTypes (with HNSW
  bounds + knn_engine enum + Immutable hints) / retrieverEngineMapping /
  buildEnvStoreForDriver — every gated surface now recognises
  "opensearch". IndexConfig grows four omitempty HNSW fields (HNSWM /
  HNSWEFConstruction / HNSWEFSearch / KNNEngine), keeping other engines'
  serialised config byte-identical.
- internal/container: createOpenSearchEngine + the switch case in
  createEngineServiceFromStore; the RETRIEVE_DRIVER=opensearch env path
  in initRetrieveEngineRegistry; NewEngineFactory now closes over the
  AuditLogService (the EngineFactory type itself is unchanged).
- internal/application/service/vectorstore_healthcheck.go: a
  testOpenSearchConnection case so CreateStore's connectivity probe
  accepts opensearch instead of returning 400.
- internal/application/repository/retriever/opensearch/transport.go:
  NewOpenSearchClient is exported so the factory and env path can build
  the TLS-hardened client; healthcheck.go reuses the unexported
  probeVersion / probeKNNPlugin for the service-layer probe.

Service-layer validation
- validateOpenSearchIndexConfig validates the HNSW caps (m 2-100,
  ef_construction 2-4096, ef_search 1-10000, knn_engine ∈ lucene|faiss).
  Shards/replicas continue to be enforced by the flat ValidateIndexConfig.
  Create-only: UpdateStore mutates the name only.
- validateConnectionConfig requires addr for opensearch.

Sync implementations (stubs.go shrinks)
- CopyIndices (copy.go) mirrors the Elasticsearch / Qdrant pattern —
  search → BatchSave with the source_id remap for generated questions —
  so dim/keyword routing and the source_id contract come from BatchSave
  for free. embeddingMap is keyed by the *target* SourceID because
  OpenSearch's BatchSave looks up embeddings by SourceID
  (lookupEmbedding), not by chunk_id (the ES driver's convention).
  Pagination is from/size; copies larger than max_result_window
  (default 10000) need the scroll-based async path that lands later.
- BatchUpdateChunkEnabledStatus / BatchUpdateChunkTagID (bulk_update.go)
  group the input by target value and issue one _update_by_query per
  group over the cross-dim <base>_* pattern. Caller values flow through
  bound script params only — never string-interpolated into the Painless
  source — closing the script-injection surface.
- inspectByQueryResponse (byquery.go) mirrors inspectBulkResponse: the
  full failure reason goes to the debug log only; the returned error
  carries the bounded id + type.
- UpdateByQueryParams.Refresh is *bool in opensearch-go v4.6.0 (the same
  shape as DeleteByQuery's quirk), so refresh=wait_for is not
  expressible; we use refresh=true.

Driver-owned audit (DIP)
- A new opensearch.AuditSink interface (with nopSink + WithAuditSink
  functional option) lets the driver emit opensearch.index_created and
  opensearch.reindex_executed events without importing any service
  package — the service layer implements the interface. NewRepository
  takes opts, so existing 4-arg test call sites keep compiling unchanged.
- internal/container/audit_sink.go bridges AuditSink to AuditLogService.
  When the context carries no tenant (the env-path registration ctx
  during boot, for example) the adapter skips the emit with a warning
  rather than silently writing tenant_id=0, which would collide with the
  system-scope sentinel.

Frontend + polish
- FieldSchema (frontend/src/api/vector-store.ts) gains min/max/enum/
  immutable. VectorStoreSettings.vue is now schema-driven: a closed
  `enum` renders a t-select; number inputs use the schema's `:min`/`:max`
  and fall back to the legacy replica-vs-shard heuristic only when the
  schema does not pin them; a danger-coloured warning fires when
  insecure_skip_verify is toggled on (the switch and warning are wrapped
  in a vertical stack so the warning sits on its own row below the switch).
- i18n: labels for hnsw_m / hnsw_ef_construction / hnsw_ef_search /
  knn_engine / insecure_skip_verify plus the warning copy in en-US,
  ko-KR, zh-CN, ru-RU.
- docker-compose.dev.yml: an opensearch profile (single-node 3.3.2 with
  security plugin disabled for dev only). OpenSearch Dashboards lives in a
  separate, opt-in opensearch-ui profile so the heavy UI container is not
  forced up alongside the cluster (the driver e2e is fully curl-verifiable
  against :9200). The new docs/dev/opensearch-integration-test.md covers the
  end-to-end exercise and the single-node guidance (set replicas=0 to keep
  the cluster Green).

Gating-guard tests flipped
- The "OpenSearch is NOT in validEngineTypes / mapping / types list /
  env builder / stubs" guard tests from PR 1 / PR 2 are replaced by
  their positive counterparts in this PR. The test suite was the
  activation checklist; the activation flip is its diff.

Backward compatibility
- Additive everywhere. IndexConfig's new HNSW fields are omitempty so
  other engines' serialised config is byte-identical. Existing
  Elasticsearch / Qdrant / Milvus / Weaviate / Doris / TencentVectorDB
  stores are untouched. No migrations.

Test plan
- go build ./... clean
- go vet ./... clean
- gofmt -l clean on touched files
- go test ./... — only TestOssEnsureBucket_CreateFails (Aliyun OSS
  endpoint), the docreader gRPC tests, and the doris SQL-shape tests
  fail; all three are pre-existing on upstream/main and untouched by
  this PR.
- New tests across internal/types, opensearch, service and container —
  including a full end-to-end env-path test that exercises
  initRetrieveEngineRegistry with RETRIEVE_DRIVER=opensearch against an
  httptest cluster.
2026-05-29 16:32:27 +08:00
..
interfaces feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
agent.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
audit_log.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
audit_log_test.go test(system): cover system.* audit constants + emitAdminAudit contract 2026-05-26 21:13:56 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
builtin_models_config.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
builtin_models_config_test.go feat(builtin-models): validate YAML entries and align ID length with schema 2026-05-26 11:37:03 +08:00
chat.go docs(chat): clarify cached-token semantics for explicit-cache providers 2026-05-25 16:47:14 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
chunk.go refactor(chunk): remove VideoInfo field from Chunk struct 2026-05-25 19:15:39 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
const.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
context_helpers_test.go fix(rbac): address PR 2 review feedback 2026-05-18 17:28:58 +08:00
custom_agent.go feat(agent): add intent prompt customization in agent editor 2026-05-26 20:41:10 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
datasource_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embedding.go perf: 优化FAQ索引更新和删除性能 2026-01-14 17:30:04 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(types): enhance FAQ metadata normalization and sanitization 2026-03-19 18:20:09 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_map.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go feat: add knowledge parse cancellation with finalizing post-process state 2026-05-28 20:16:02 +08:00
knowledge_span.go refactor(knowledge): replace flat stage table with langfuse-style span tree 2026-05-28 15:14:45 +08:00
knowledgebase.go feat(obs): 支持华为云obs存储 2026-05-18 19:38:23 +08:00
knowledgebase_test.go feat(knowledge-base): validate vector store bindings on create, copy, and delete 2026-05-18 15:58:46 +08:00
mcp.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
memory.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
message.go feat(chat_pipeline): enhance query handling with image and attachment metadata 2026-04-14 18:21:12 +08:00
model.go fix: add model display name 2026-05-28 19:22:17 +08:00
organization.go feat(organization): transition membership model to tenant-level and enhance invite functionality 2026-05-18 17:28:58 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
qa_request.go feat: attachment_support 2026-04-10 12:49:20 +08:00
retrieval_config.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
retriever.go feat(retriever): add OpenSearch type prep ahead of Phase 3 driver 2026-05-22 20:43:50 +08:00
search.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
secret.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session.go feat(session): persist last request state for UI restoration 2026-05-18 22:28:00 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
task.go feat(knowledge): instrument postprocess subspans + polish timeline UI 2026-05-28 15:14:45 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
tenant.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go feat(rbac): tenant member management + invite_only gate (PR 3 of 3, #1303) 2026-05-18 17:28:58 +08:00
tenant_test.go feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
tracing.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
user.go feat(system-admin): implement bootstrap for system admin promotion and enhance system settings management 2026-05-26 21:13:56 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
vectorstore.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_opensearch_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
vectorstore_test.go feat(retriever): activate OpenSearch k-NN driver (PR 3 of 3) 2026-05-29 16:32:27 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
web_search_provider.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00