Lays type-system groundwork for the upcoming OpenSearch k-NN driver
(Phase 3 PR 2, see Tencent/WeKnora#1440), with strict feature-gate:
this PR ships only inert constants, schema extensions, and an
unreachable normalizer case. No path activates an OpenSearch
VectorStore yet — creation continues to fail with "not a valid engine
type" until the activation switch lands in Phase 3 PR 3.
Changes:
- OpenSearchRetrieverEngineType constant ("opensearch") in
internal/types/retriever.go. Not added to validEngineTypes /
GetVectorStoreTypes / retrieverEngineMapping yet (gated).
- ConnectionConfig.InsecureSkipVerify (bool, default false) in
internal/types/vectorstore.go, placed inside the // Common section
because it is a cross-driver TLS option. Distinct from the
Qdrant-specific UseTLS, which enables TLS on gRPC — InsecureSkipVerify
only controls verification of an already-TLS HTTPS connection.
AES-GCM Value/Scan round-trips the field as plaintext (it travels
alongside the encrypted Password / APIKey but is not sensitive itself).
- VectorStoreFieldInfo gains four optional fields: Immutable (bool),
Min/Max (*float64), Enum ([]string). All omitempty so existing UI
schema entries serialize identically. The fields will drive the UI
in Phase 3 PR 3 (read-only on Edit + range/enum constraints).
- Six new AuditAction constants under vector_store.* and opensearch.*
namespaces. Definitions only — emission lands in Phase 3 PR 3.
- EngineAwareNormalizer.Normalize is restructured to group engines by
the effective score range observed by the normalizer (not the
theoretical raw cosine range):
Range [-1, 1] (raw cosine, mapped via (score + 1) / 2):
- Milvus (COSINE metric mode). Milvus docs explicitly state the
COSINE metric range is [-1, 1].
Range [0, 1] (passthrough — already on the target scale):
- Elasticsearch v8 / ElasticFaiss. The driver issues a
cosineSimilarity(...) script_score script, and Lucene rejects
negative final scores ("Final relevance scores from the
script_score query cannot be negative" — ES docs); the
effective range observed by the normalizer is therefore [0, 1]
for IR-normalized embeddings. ES was previously grouped with
Milvus and over-corrected via (score + 1) / 2, which inflated
every ES result by 50% in mixed-engine RRF fusion.
- OpenSearch. The k-NN plugin's
SpaceType.COSINESIMIL.scoreTranslation maps the underlying
Lucene/Faiss distance (1 - cosine) to (1 + cosine) / 2 ∈ [0, 1]
before the score reaches us. Source:
github.com/opensearch-project/k-NN at
src/main/java/org/opensearch/knn/index/SpaceType.java
(COSINESIMIL enum, scoreTranslation method).
- Weaviate. The driver requests `certainty`, defined by Weaviate
as (2 - distance) / 2 = (1 + cosine) / 2, intrinsically [0, 1].
- Postgres pgvector / SQLite sqlite-vec / Qdrant /
TencentVectorDB / Doris. The driver computes (1 - cosine_distance)
or normalized inner_product, whose theoretical range is [-1, 1].
The IR-normalized positive-component unit vectors WeKnora
targets (BGE / OpenAI text-embedding-3 / Cohere /
sentence-transformers) keep the observed range in [0, 1];
negative-cosine embedding models would silently clamp to 0
downstream — explicitly documented as the IR-normalization
caveat in the struct godoc.
Dead enum references (ElasticFaiss, Infinity) are flagged in the
godoc with a pointer to internal/types/vectorstore.go's existing
"legacy/experimental, no standalone deployable instance" annotation.
Their case labels are kept for switch exhaustiveness.
Test coverage:
- OpenSearch constant wire value + collision check against existing
10 engines + gated invariant (NOT in validEngineTypes).
- ConnectionConfig.InsecureSkipVerify backward-compat (missing JSON
field deserializes as false) + round-trip + AES-GCM coexistence
with encrypted Password / APIKey.
- VectorStoreFieldInfo omitempty preservation + new-field serialization
+ *float64 pointer distinction (min=0 vs nil).
- AuditAction dot-namespace convention enforcement, prefix invariants
for vector_store.* and opensearch.*, no wire-string collisions, exact
wire-value pins for the six new constants.
- EngineAwareNormalizer:
- CosineRange retains its (score + 1) / 2 coverage but now only for
Milvus.
- UnitInterval now covers the full passthrough group (ES /
ElasticFaiss / OpenSearch / Weaviate / Postgres / SQLite / Qdrant /
Infinity / TencentVectorDB / Doris).
- New TestEngineAwareNormalizer_ElasticsearchCosinePassthrough is an
explicit regression guard for the score-range correction: cos=0.5
maps to 0.5 (not (0.5 + 1) / 2 = 0.75 as an earlier draft assumed).
- OpenSearch passthrough across (0 / 0.5 / 0.75 / 1) + engine drift
(1.0001) + defensive negative + ±Inf / NaN edges + keyword
passthrough + nil-ctx safety.
Backward compatibility is preserved at every layer:
- All new struct fields are omitempty / pointer-tagged so existing
rows and existing wire formats remain unchanged.
- Normalizer's new OpenSearch case is unreachable until the driver
lands. The ES regrouping changes the post-normalization value for
every ES vector search result (a correctness fix, not a feature) —
ES vector retrieval is currently the only production path affected.
- AuditAction constants emit no audit_log rows in this PR.
- engine_type=opensearch VectorStore creation still rejected (gated).