fix(milvus): support multilingual BM25 retrieval (#3235)
* fix(milvus): support multilingual BM25 retrieval
* fix(milvus): preserve source metric and tighten collection matching
Read the source embedding index metric during BM25 migration, match
collections by {prefix}_{dimension}, and route Japanese/Korean text to
the ICU fallback.
* fix(milvus): satisfy lint on multilingual BM25 changes
---------
Co-authored-by: Jonesxq <239089032+Jonesxq@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
main
parent
fbad5b3730
commit
350b2ab364
|
|
@ -293,6 +293,9 @@ RETRIEVE_DRIVER=postgres
|
|||
# MILVUS_ADDRESS=milvus:19530
|
||||
# MILVUS_COLLECTION=weknora_embeddings
|
||||
# MILVUS_METRIC_TYPE=IP # IP(默认)/COSINE/L2,改后需重建 collection。
|
||||
# 新版 Collection 会为 content 配置 english/chinese/default 多语言分析器;
|
||||
# 旧 Collection 请先运行 go run ./cmd/milvus-migrate(metric 沿用源 Collection),
|
||||
# 再把 MILVUS_COLLECTION 改为目标前缀,不要改 MILVUS_METRIC_TYPE。
|
||||
# MILVUS_USERNAME=your_milvus_username
|
||||
# MILVUS_PASSWORD=your_milvus_password
|
||||
# MILVUS_DB_NAME=your_milvus_db_name
|
||||
|
|
|
|||
|
|
@ -0,0 +1,59 @@
|
|||
# Milvus 多语言 BM25 迁移
|
||||
|
||||
旧版 WeKnora 的 Milvus Collection 只有一个默认文本分析器,中文内容可能没有有效 BM25 关键词。`milvus-migrate` 会保留旧 Collection,并复制已有稠密向量到新的多语言 Collection;Milvus 会依据每行的 `language` 字段重新生成 BM25 稀疏向量。
|
||||
|
||||
稠密向量的 metric(IP / COSINE / L2)默认从源 Collection 的 embedding 索引读取。不要改成与源 Collection 不同的值,否则同一批向量会按错误距离排序。
|
||||
|
||||
## 使用方法
|
||||
|
||||
先确保 Milvus 已启动,再在项目根目录执行:
|
||||
|
||||
```bash
|
||||
go run ./cmd/milvus-migrate --source weknora_embeddings --target weknora_embeddings_multilingual
|
||||
```
|
||||
|
||||
如果当前 shell 已经导入了 `MILVUS_ADDRESS` 和 `MILVUS_COLLECTION`,可以省略对应参数。仅仅把变量写在 `.env.local` 中不会自动注入 `go run` 进程;不确定时建议显式传参:
|
||||
|
||||
```bash
|
||||
go run ./cmd/milvus-migrate \
|
||||
--address 127.0.0.1:19530 \
|
||||
--source weknora_embeddings \
|
||||
--target weknora_embeddings_multilingual
|
||||
```
|
||||
|
||||
若要核对 metric,可传入与源 Collection 相同的 `--metric-type`(以及环境变量 `MILVUS_METRIC_TYPE`)。传入的值必须与源索引一致,否则迁移会失败。
|
||||
|
||||
迁移完成后,把 WeKnora 的 `MILVUS_COLLECTION` 改为目标前缀并重启服务,**保持原来的 `MILVUS_METRIC_TYPE` 不变**:
|
||||
|
||||
```dotenv
|
||||
MILVUS_COLLECTION=weknora_embeddings_multilingual
|
||||
```
|
||||
|
||||
检索列表按 `{前缀}_{维度}` 精确匹配,因此 `weknora_embeddings` 不会误搜到 `weknora_embeddings_multilingual_*`。切换前缀后,新写入和向量/关键词检索才会都走新 Collection。
|
||||
|
||||
迁移程序不会删除旧 Collection。确认中文和英文 BM25 都能召回后,再通过 Milvus 管理工具删除旧 Collection;删除前请先备份。
|
||||
|
||||
迁移默认每批读取 64 行,并且只读取重建目标 Collection 所需的字段,不会读取旧 Collection 中生成的 BM25 稀疏向量。若单个文本块特别长,可以显式降低批大小,例如追加 `--batch-size 32`。
|
||||
|
||||
## Windows PowerShell 注意事项
|
||||
|
||||
WeKnora 的 `internal/utils` 使用 `pg_query_go` 解析 SQL,该依赖需要 CGO。直接执行 `go run` 时,如果当前会话的 `CGO_ENABLED=0`,会出现 `undefined: pg_query.Parse` 或 `undefined: pg_query.Deparse`。
|
||||
|
||||
项目附带了 MSYS2 GCC,可以在当前 PowerShell 会话中临时启用 CGO 后再运行迁移。下面的设置只影响当前窗口,不会修改系统级 Go 配置:
|
||||
|
||||
```powershell
|
||||
# 确认当前目录是包含 go.mod 的项目根目录
|
||||
if (!(Test-Path -LiteralPath '.\go.mod')) { throw '请先切换到 WeKnora 项目根目录' }
|
||||
|
||||
# 使用项目自带的 GCC,避免 Go 找不到 C 编译器
|
||||
$compilerBin = Join-Path (Get-Location) '.local-tools\msys64\ucrt64\bin'
|
||||
$env:CGO_ENABLED = '1'
|
||||
$env:CC = Join-Path $compilerBin 'gcc.exe'
|
||||
$env:CXX = Join-Path $compilerBin 'g++.exe'
|
||||
$env:PATH = "$compilerBin;$env:PATH"
|
||||
|
||||
# 运行迁移;metric 沿用源 Collection,不删除旧 Collection
|
||||
go run ./cmd/milvus-migrate --address 127.0.0.1:19530 --source weknora_embeddings --target weknora_embeddings_multilingual
|
||||
```
|
||||
|
||||
如果项目目录中没有 `.local-tools\msys64\ucrt64\bin`,请先安装可用的 GCC,并将 `$env:CC`、`$env:CXX` 改为对应的 `gcc.exe`、`g++.exe` 绝对路径。
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
// milvus-migrate 将旧版 Milvus Collection 复制为支持中英文 BM25 的新 Collection。
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
milvusRepo "github.com/Tencent/WeKnora/internal/application/repository/retriever/milvus"
|
||||
client "github.com/milvus-io/milvus/client/v2/milvusclient"
|
||||
)
|
||||
|
||||
func main() {
|
||||
sourceDefault := strings.TrimSpace(os.Getenv("MILVUS_COLLECTION"))
|
||||
if sourceDefault == "" {
|
||||
sourceDefault = "weknora_embeddings"
|
||||
}
|
||||
targetDefault := sourceDefault + "_multilingual"
|
||||
|
||||
source := flag.String("source", sourceDefault, "旧 Collection 前缀,例如 weknora_embeddings")
|
||||
target := flag.String("target", targetDefault, "新 Collection 前缀,例如 weknora_embeddings_multilingual")
|
||||
address := flag.String("address", envOr("MILVUS_ADDRESS", "localhost:19530"), "Milvus 地址")
|
||||
username := flag.String("username", os.Getenv("MILVUS_USERNAME"), "Milvus 用户名")
|
||||
password := flag.String("password", os.Getenv("MILVUS_PASSWORD"), "Milvus 密码")
|
||||
database := flag.String("database", os.Getenv("MILVUS_DB_NAME"), "Milvus 数据库名")
|
||||
metric := flag.String(
|
||||
"metric-type",
|
||||
strings.TrimSpace(os.Getenv("MILVUS_METRIC_TYPE")),
|
||||
"稠密向量距离:IP、COSINE 或 L2;省略则沿用源 Collection",
|
||||
)
|
||||
batchSize := flag.Int("batch-size", 64, "每批迁移的行数;文本较长时可进一步调小")
|
||||
flag.Parse()
|
||||
|
||||
metricType, err := milvusRepo.ParseMetricType(*metric)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
if strings.TrimSpace(*source) == strings.TrimSpace(*target) {
|
||||
log.Fatal("--source 和 --target 必须不同;迁移只会复制到新 Collection,不会覆盖旧 Collection")
|
||||
}
|
||||
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
|
||||
defer stop()
|
||||
|
||||
connectCtx, cancelConnect := context.WithTimeout(ctx, 30*time.Second)
|
||||
milvusClient, err := client.New(connectCtx, &client.ClientConfig{
|
||||
Address: *address,
|
||||
Username: *username,
|
||||
Password: *password,
|
||||
DBName: *database,
|
||||
})
|
||||
cancelConnect()
|
||||
if err != nil {
|
||||
log.Fatalf("连接 Milvus 失败:%v", err)
|
||||
}
|
||||
defer func() {
|
||||
closeCtx, cancelClose := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancelClose()
|
||||
if err := milvusClient.Close(closeCtx); err != nil {
|
||||
log.Printf("关闭 Milvus 连接失败:%v", err)
|
||||
}
|
||||
}()
|
||||
|
||||
summary, err := milvusRepo.MigrateLegacyCollections(ctx, milvusClient, milvusRepo.MultilingualMigrationOptions{
|
||||
SourceCollectionBaseName: *source,
|
||||
TargetCollectionBaseName: *target,
|
||||
MetricType: metricType,
|
||||
BatchSize: *batchSize,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf(
|
||||
"迁移失败(已检查 %d 个 Collection,已复制 %d 个 Collection、%d 行):%v",
|
||||
summary.ExaminedCollections,
|
||||
summary.MigratedCollections,
|
||||
summary.MigratedRows,
|
||||
err,
|
||||
)
|
||||
}
|
||||
fmt.Printf(
|
||||
"迁移完成:检查 %d 个 Collection,复制 %d 个 Collection、%d 行。旧 Collection 保留未删除。\n",
|
||||
summary.ExaminedCollections,
|
||||
summary.MigratedCollections,
|
||||
summary.MigratedRows,
|
||||
)
|
||||
}
|
||||
|
||||
func envOr(name, fallback string) string {
|
||||
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
|
||||
return value
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
|
|
@ -0,0 +1,135 @@
|
|||
// Package milvus implements the Milvus retrieve engine, including multilingual BM25.
|
||||
package milvus
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"github.com/milvus-io/milvus/client/v2/entity"
|
||||
)
|
||||
|
||||
const (
|
||||
fieldContent = "content"
|
||||
fieldLanguage = "language"
|
||||
milvusAnalyzerEnglish = "english"
|
||||
milvusAnalyzerChinese = "chinese"
|
||||
milvusAnalyzerDefault = "default"
|
||||
)
|
||||
|
||||
type collectionAnalyzerMode uint8
|
||||
|
||||
const (
|
||||
collectionAnalyzerLegacy collectionAnalyzerMode = iota
|
||||
collectionAnalyzerMulti
|
||||
)
|
||||
|
||||
func analyzerModeFromSchema(schema *entity.Schema) collectionAnalyzerMode {
|
||||
if schema == nil {
|
||||
return collectionAnalyzerLegacy
|
||||
}
|
||||
|
||||
hasLanguageField := false
|
||||
hasMultiAnalyzer := false
|
||||
for _, field := range schema.Fields {
|
||||
if field == nil {
|
||||
continue
|
||||
}
|
||||
if field.Name == fieldLanguage {
|
||||
hasLanguageField = true
|
||||
}
|
||||
if field.Name == fieldContent && field.TypeParams != nil &&
|
||||
field.TypeParams["multi_analyzer_params"] != "" {
|
||||
hasMultiAnalyzer = true
|
||||
}
|
||||
}
|
||||
|
||||
if hasLanguageField && hasMultiAnalyzer {
|
||||
return collectionAnalyzerMulti
|
||||
}
|
||||
return collectionAnalyzerLegacy
|
||||
}
|
||||
|
||||
// multiAnalyzerParams returns the analyzer configuration used by new Milvus
|
||||
// collections. The language field on each row selects the analyzer used for
|
||||
// that row.
|
||||
func multiAnalyzerParams() map[string]any {
|
||||
return map[string]any{
|
||||
"analyzers": map[string]any{
|
||||
milvusAnalyzerEnglish: map[string]string{
|
||||
"type": milvusAnalyzerEnglish,
|
||||
},
|
||||
milvusAnalyzerChinese: map[string]string{
|
||||
"type": milvusAnalyzerChinese,
|
||||
},
|
||||
milvusAnalyzerDefault: map[string]string{
|
||||
"tokenizer": "icu",
|
||||
},
|
||||
},
|
||||
"by_field": fieldLanguage,
|
||||
"alias": map[string]string{
|
||||
"en": milvusAnalyzerEnglish,
|
||||
"zh": milvusAnalyzerChinese,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// detectAnalyzerName selects an analyzer for a document or query. Mixed or
|
||||
// unsupported-language text uses ICU as a safe fallback because Milvus uses
|
||||
// one analyzer name per document and per query.
|
||||
func detectAnalyzerName(text string) string {
|
||||
var hanCount, latinCount int
|
||||
hasJapaneseKana := false
|
||||
hasHangul := false
|
||||
for _, r := range text {
|
||||
switch {
|
||||
case unicode.Is(unicode.Hiragana, r) || unicode.Is(unicode.Katakana, r):
|
||||
hasJapaneseKana = true
|
||||
case unicode.Is(unicode.Hangul, r):
|
||||
hasHangul = true
|
||||
case unicode.Is(unicode.Han, r):
|
||||
hanCount++
|
||||
case r <= unicode.MaxASCII && ((r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z')):
|
||||
latinCount++
|
||||
}
|
||||
}
|
||||
|
||||
// Jieba does not preserve Japanese or Korean word boundaries. Route those
|
||||
// scripts to ICU instead of treating shared Han characters as Chinese.
|
||||
if hasJapaneseKana || hasHangul {
|
||||
return milvusAnalyzerDefault
|
||||
}
|
||||
// Any remaining Han character makes the text Chinese-oriented. This is
|
||||
// important for technical queries such as "AI 注意力": choosing English just
|
||||
// because ASCII characters are more numerous would drop the Chinese terms.
|
||||
if hanCount > 0 {
|
||||
return milvusAnalyzerChinese
|
||||
}
|
||||
if latinCount > 0 {
|
||||
return milvusAnalyzerEnglish
|
||||
}
|
||||
|
||||
return milvusAnalyzerDefault
|
||||
}
|
||||
|
||||
func analyzerNameForUpsert(embedding *MilvusVectorEmbedding) string {
|
||||
if embedding == nil {
|
||||
return milvusAnalyzerDefault
|
||||
}
|
||||
if strings.TrimSpace(embedding.Language) == "" {
|
||||
return detectAnalyzerName(embedding.Content)
|
||||
}
|
||||
return normalizeAnalyzerName(embedding.Language)
|
||||
}
|
||||
|
||||
// normalizeAnalyzerName keeps externally supplied values within the analyzer
|
||||
// names defined by multiAnalyzerParams.
|
||||
func normalizeAnalyzerName(name string) string {
|
||||
switch strings.ToLower(strings.TrimSpace(name)) {
|
||||
case milvusAnalyzerEnglish, "en":
|
||||
return milvusAnalyzerEnglish
|
||||
case milvusAnalyzerChinese, "zh":
|
||||
return milvusAnalyzerChinese
|
||||
default:
|
||||
return milvusAnalyzerDefault
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
package milvus
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/milvus-io/milvus/client/v2/entity"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestDetectAnalyzerName(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
text string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "Chinese",
|
||||
text: "Transformer中多头注意力的三种使用方式及解码器自注意力掩码原因",
|
||||
want: milvusAnalyzerChinese,
|
||||
},
|
||||
{name: "English", text: "How does multi-head attention work", want: milvusAnalyzerEnglish},
|
||||
{name: "ChineseWithTechnicalTerm", text: "AI 注意力", want: milvusAnalyzerChinese},
|
||||
{name: "JapaneseKanjiWithKana", text: "東京都の設定", want: milvusAnalyzerDefault},
|
||||
{name: "KoreanHangul", text: "서울 설정", want: milvusAnalyzerDefault},
|
||||
{name: "Symbols", text: "12345 --", want: milvusAnalyzerDefault},
|
||||
{name: "Empty", text: "", want: milvusAnalyzerDefault},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
require.Equal(t, tt.want, detectAnalyzerName(tt.text))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMultiAnalyzerParams(t *testing.T) {
|
||||
params := multiAnalyzerParams()
|
||||
|
||||
require.Equal(t, fieldLanguage, params["by_field"])
|
||||
require.Equal(t, map[string]string{
|
||||
"type": milvusAnalyzerEnglish,
|
||||
}, params["analyzers"].(map[string]any)[milvusAnalyzerEnglish])
|
||||
require.Equal(t, map[string]string{
|
||||
"type": milvusAnalyzerChinese,
|
||||
}, params["analyzers"].(map[string]any)[milvusAnalyzerChinese])
|
||||
require.Equal(t, map[string]string{
|
||||
"tokenizer": "icu",
|
||||
}, params["analyzers"].(map[string]any)[milvusAnalyzerDefault])
|
||||
}
|
||||
|
||||
func TestNormalizeAnalyzerName(t *testing.T) {
|
||||
require.Equal(t, milvusAnalyzerEnglish, normalizeAnalyzerName(" EN "))
|
||||
require.Equal(t, milvusAnalyzerChinese, normalizeAnalyzerName("zh"))
|
||||
require.Equal(t, milvusAnalyzerDefault, normalizeAnalyzerName("unknown"))
|
||||
}
|
||||
|
||||
func TestAnalyzerNameForUpsert(t *testing.T) {
|
||||
require.Equal(t, milvusAnalyzerDefault, analyzerNameForUpsert(nil))
|
||||
require.Equal(t, milvusAnalyzerChinese, analyzerNameForUpsert(&MilvusVectorEmbedding{
|
||||
Content: "AI 注意力",
|
||||
}))
|
||||
require.Equal(t, milvusAnalyzerEnglish, analyzerNameForUpsert(&MilvusVectorEmbedding{
|
||||
Content: "AI 注意力",
|
||||
Language: "en",
|
||||
}))
|
||||
require.Equal(t, milvusAnalyzerEnglish, analyzerNameForUpsert(&MilvusVectorEmbedding{
|
||||
Content: "hello",
|
||||
Language: " ",
|
||||
}))
|
||||
}
|
||||
|
||||
func TestAnalyzerModeFromSchema(t *testing.T) {
|
||||
legacySchema := entity.NewSchema().WithField(
|
||||
entity.NewField().WithName(fieldContent).WithDataType(entity.FieldTypeVarChar),
|
||||
)
|
||||
require.Equal(t, collectionAnalyzerLegacy, analyzerModeFromSchema(legacySchema))
|
||||
|
||||
multilingualSchema := entity.NewSchema().WithField(
|
||||
entity.NewField().WithName(fieldContent).
|
||||
WithDataType(entity.FieldTypeVarChar).
|
||||
WithMultiAnalyzerParams(multiAnalyzerParams()),
|
||||
).WithField(
|
||||
entity.NewField().WithName(fieldLanguage).WithDataType(entity.FieldTypeVarChar),
|
||||
)
|
||||
require.Equal(t, collectionAnalyzerMulti, analyzerModeFromSchema(multilingualSchema))
|
||||
}
|
||||
|
|
@ -0,0 +1,452 @@
|
|||
package milvus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/milvus-io/milvus/client/v2/column"
|
||||
"github.com/milvus-io/milvus/client/v2/entity"
|
||||
"github.com/milvus-io/milvus/client/v2/index"
|
||||
client "github.com/milvus-io/milvus/client/v2/milvusclient"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
)
|
||||
|
||||
const defaultMultilingualMigrationBatchSize = 64
|
||||
|
||||
// migrationOutputFields deliberately excludes the generated BM25 sparse
|
||||
// vector. The migration only needs the source content, dense vector, and
|
||||
// metadata required to rebuild the target row. Querying "*" can exceed
|
||||
// Milvus's per-query result-size limit when a batch contains long chunks.
|
||||
var migrationOutputFields = []string{
|
||||
fieldID,
|
||||
fieldContent,
|
||||
fieldSourceID,
|
||||
fieldSourceType,
|
||||
fieldChunkID,
|
||||
fieldKnowledgeID,
|
||||
fieldKnowledgeBaseID,
|
||||
fieldTagID,
|
||||
fieldIsEnabled,
|
||||
fieldEmbedding,
|
||||
}
|
||||
|
||||
// MultilingualMigrationOptions describes a safe, copy-only migration from
|
||||
// legacy Milvus collections to collections with multilingual BM25 analyzers.
|
||||
// The source collections are never modified or deleted.
|
||||
type MultilingualMigrationOptions struct {
|
||||
SourceCollectionBaseName string
|
||||
TargetCollectionBaseName string
|
||||
MetricType entity.MetricType
|
||||
ShardsNum int
|
||||
ReplicaNumber int
|
||||
BatchSize int
|
||||
}
|
||||
|
||||
// MigrationSummary reports the result of a copy-only migration.
|
||||
type MigrationSummary struct {
|
||||
ExaminedCollections int
|
||||
MigratedCollections int
|
||||
MigratedRows int
|
||||
}
|
||||
|
||||
// MigrateLegacyCollections copies all legacy WeKnora Milvus collections to a
|
||||
// new multilingual collection base. Existing dense vectors are reused; only
|
||||
// Milvus's BM25 sparse vectors are regenerated from content and language.
|
||||
//
|
||||
// The target collection base must differ from the source base. The operation
|
||||
// is resumable because rows keep their original primary keys and are upserted.
|
||||
func MigrateLegacyCollections(
|
||||
ctx context.Context,
|
||||
milvusClient *client.Client,
|
||||
options MultilingualMigrationOptions,
|
||||
) (MigrationSummary, error) {
|
||||
var summary MigrationSummary
|
||||
if milvusClient == nil {
|
||||
return summary, fmt.Errorf("milvus client is nil")
|
||||
}
|
||||
|
||||
sourceBase := strings.TrimSpace(options.SourceCollectionBaseName)
|
||||
targetBase := strings.TrimSpace(options.TargetCollectionBaseName)
|
||||
if sourceBase == "" || targetBase == "" {
|
||||
return summary, fmt.Errorf("source and target collection base names are required")
|
||||
}
|
||||
if sourceBase == targetBase {
|
||||
return summary, fmt.Errorf("source and target collection base names must differ")
|
||||
}
|
||||
if options.BatchSize <= 0 {
|
||||
options.BatchSize = defaultMultilingualMigrationBatchSize
|
||||
}
|
||||
|
||||
log := logger.GetLogger(ctx)
|
||||
collections, err := milvusClient.ListCollections(ctx, client.NewListCollectionOption())
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("list Milvus collections: %w", err)
|
||||
}
|
||||
sort.Strings(collections)
|
||||
|
||||
targetRepo := &milvusRepository{
|
||||
client: milvusClient,
|
||||
collectionBaseName: targetBase,
|
||||
shardsNum: options.ShardsNum,
|
||||
replicaNumber: options.ReplicaNumber,
|
||||
}
|
||||
|
||||
for _, sourceCollectionName := range collections {
|
||||
dimension, ok := collectionDimensionForBase(sourceCollectionName, sourceBase)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
summary.ExaminedCollections++
|
||||
|
||||
sourceCollection, err := milvusClient.DescribeCollection(
|
||||
ctx,
|
||||
client.NewDescribeCollectionOption(sourceCollectionName),
|
||||
)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("describe source collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
if analyzerModeFromSchema(sourceCollection.Schema) == collectionAnalyzerMulti {
|
||||
log.Infof(
|
||||
"[Milvus] Source collection %s already uses multilingual analyzers, skipping",
|
||||
sourceCollectionName,
|
||||
)
|
||||
continue
|
||||
}
|
||||
schemaDimension, err := dimensionFromSchema(sourceCollection.Schema)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("read dimension from source collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
if schemaDimension != dimension {
|
||||
return summary, fmt.Errorf(
|
||||
"source collection %s name dimension %d does not match schema dimension %d",
|
||||
sourceCollectionName,
|
||||
dimension,
|
||||
schemaDimension,
|
||||
)
|
||||
}
|
||||
|
||||
targetCollectionName := fmt.Sprintf("%s_%d", targetBase, dimension)
|
||||
sourceMetric, err := embeddingMetricFromCollection(ctx, milvusClient, sourceCollectionName)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("read metric from source collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
metricType, err := resolveMigrationMetric(options.MetricType, sourceMetric)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
hasTarget, err := milvusClient.HasCollection(ctx, client.NewHasCollectionOption(targetCollectionName))
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("check target collection %s: %w", targetCollectionName, err)
|
||||
}
|
||||
if hasTarget {
|
||||
targetMetric, err := embeddingMetricFromCollection(ctx, milvusClient, targetCollectionName)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("read metric from target collection %s: %w", targetCollectionName, err)
|
||||
}
|
||||
if targetMetric != metricType {
|
||||
return summary, fmt.Errorf(
|
||||
"target collection %s uses metric %s, source collection %s uses %s; choose a new target base",
|
||||
targetCollectionName,
|
||||
targetMetric,
|
||||
sourceCollectionName,
|
||||
metricType,
|
||||
)
|
||||
}
|
||||
}
|
||||
targetRepo.metricType = metricType
|
||||
if err := targetRepo.ensureCollection(ctx, dimension); err != nil {
|
||||
return summary, fmt.Errorf("prepare target collection %s: %w", targetCollectionName, err)
|
||||
}
|
||||
mode, err := targetRepo.collectionAnalyzerMode(ctx, targetCollectionName)
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("inspect target collection %s: %w", targetCollectionName, err)
|
||||
}
|
||||
if mode != collectionAnalyzerMulti {
|
||||
return summary, fmt.Errorf(
|
||||
"target collection %s is not multilingual; choose a new target base",
|
||||
targetCollectionName,
|
||||
)
|
||||
}
|
||||
|
||||
if err := loadMilvusCollection(ctx, milvusClient, sourceCollectionName); err != nil {
|
||||
return summary, fmt.Errorf("load source collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
copied, err := migrateCollectionRows(
|
||||
ctx,
|
||||
milvusClient,
|
||||
sourceCollectionName,
|
||||
targetCollectionName,
|
||||
dimension,
|
||||
options.BatchSize,
|
||||
)
|
||||
summary.MigratedRows += copied
|
||||
if err != nil {
|
||||
return summary, fmt.Errorf("migrate collection %s: %w", sourceCollectionName, err)
|
||||
}
|
||||
if err := flushMilvusCollection(ctx, milvusClient, targetCollectionName); err != nil {
|
||||
return summary, fmt.Errorf("flush target collection %s: %w", targetCollectionName, err)
|
||||
}
|
||||
|
||||
summary.MigratedCollections++
|
||||
log.Infof(
|
||||
"[Milvus] Migrated %d rows from %s to %s",
|
||||
copied,
|
||||
sourceCollectionName,
|
||||
targetCollectionName,
|
||||
)
|
||||
}
|
||||
|
||||
return summary, nil
|
||||
}
|
||||
|
||||
func collectionDimensionForBase(collectionName, baseName string) (int, bool) {
|
||||
prefix := strings.TrimSpace(baseName) + "_"
|
||||
if !strings.HasPrefix(collectionName, prefix) {
|
||||
return 0, false
|
||||
}
|
||||
dimension, err := strconv.Atoi(strings.TrimPrefix(collectionName, prefix))
|
||||
if err != nil || dimension <= 0 {
|
||||
return 0, false
|
||||
}
|
||||
return dimension, true
|
||||
}
|
||||
|
||||
func matchesDimensionCollection(collectionName, baseName string) bool {
|
||||
_, ok := collectionDimensionForBase(collectionName, baseName)
|
||||
return ok
|
||||
}
|
||||
|
||||
// ParseMetricType accepts IP, COSINE, or L2. An empty value means the caller
|
||||
// wants the source collection's metric.
|
||||
func ParseMetricType(value string) (entity.MetricType, error) {
|
||||
switch strings.ToUpper(strings.TrimSpace(value)) {
|
||||
case "":
|
||||
return "", nil
|
||||
case "IP":
|
||||
return entity.IP, nil
|
||||
case "COSINE":
|
||||
return entity.COSINE, nil
|
||||
case "L2":
|
||||
return entity.L2, nil
|
||||
default:
|
||||
return "", fmt.Errorf("unsupported metric-type %q, must be IP, COSINE, or L2", value)
|
||||
}
|
||||
}
|
||||
|
||||
func resolveMigrationMetric(requested, source entity.MetricType) (entity.MetricType, error) {
|
||||
if source == "" {
|
||||
return "", fmt.Errorf("source collection metric is unknown")
|
||||
}
|
||||
if requested == "" || requested == source {
|
||||
return source, nil
|
||||
}
|
||||
return "", fmt.Errorf(
|
||||
"requested metric %s does not match source collection metric %s",
|
||||
requested,
|
||||
source,
|
||||
)
|
||||
}
|
||||
|
||||
func metricTypeFromIndexParams(params map[string]string) (entity.MetricType, error) {
|
||||
raw := strings.TrimSpace(params[index.MetricTypeKey])
|
||||
if raw == "" {
|
||||
if nested := strings.TrimSpace(params[index.ParamsKey]); nested != "" {
|
||||
var extra map[string]any
|
||||
if err := json.Unmarshal([]byte(nested), &extra); err == nil {
|
||||
if value, ok := extra[index.MetricTypeKey]; ok {
|
||||
raw = fmt.Sprint(value)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if raw == "" {
|
||||
return "", fmt.Errorf("dense vector index is missing metric_type")
|
||||
}
|
||||
return ParseMetricType(raw)
|
||||
}
|
||||
|
||||
func embeddingMetricFromCollection(
|
||||
ctx context.Context,
|
||||
milvusClient *client.Client,
|
||||
collectionName string,
|
||||
) (entity.MetricType, error) {
|
||||
indexNames, err := milvusClient.ListIndexes(
|
||||
ctx,
|
||||
client.NewListIndexOption(collectionName).WithFieldName(fieldEmbedding),
|
||||
)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("list indexes: %w", err)
|
||||
}
|
||||
if len(indexNames) == 0 {
|
||||
return "", fmt.Errorf("no dense vector index found")
|
||||
}
|
||||
description, err := milvusClient.DescribeIndex(
|
||||
ctx,
|
||||
client.NewDescribeIndexOption(collectionName, indexNames[0]),
|
||||
)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("describe index %s: %w", indexNames[0], err)
|
||||
}
|
||||
return metricTypeFromIndexParams(description.Params())
|
||||
}
|
||||
|
||||
func primaryKeyForMigratedRow(
|
||||
document *MilvusVectorEmbeddingWithScore,
|
||||
ids column.Column,
|
||||
rowIndex int,
|
||||
) (string, error) {
|
||||
if document != nil && strings.TrimSpace(document.ID) != "" {
|
||||
return document.ID, nil
|
||||
}
|
||||
if ids != nil {
|
||||
id, err := ids.GetAsString(rowIndex)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read primary key at row %d: %w", rowIndex, err)
|
||||
}
|
||||
if strings.TrimSpace(id) != "" {
|
||||
return id, nil
|
||||
}
|
||||
}
|
||||
return "", fmt.Errorf("missing primary key at row %d", rowIndex)
|
||||
}
|
||||
|
||||
func dimensionFromSchema(schema *entity.Schema) (int, error) {
|
||||
if schema == nil {
|
||||
return 0, fmt.Errorf("collection schema is nil")
|
||||
}
|
||||
for _, field := range schema.Fields {
|
||||
if field == nil || field.Name != fieldEmbedding {
|
||||
continue
|
||||
}
|
||||
rawDimension, ok := field.TypeParams[entity.TypeParamDim]
|
||||
if !ok {
|
||||
return 0, fmt.Errorf("embedding field has no dimension")
|
||||
}
|
||||
dimension, err := strconv.Atoi(rawDimension)
|
||||
if err != nil || dimension <= 0 {
|
||||
return 0, fmt.Errorf("invalid embedding dimension %q", rawDimension)
|
||||
}
|
||||
return dimension, nil
|
||||
}
|
||||
return 0, fmt.Errorf("embedding field %q was not found", fieldEmbedding)
|
||||
}
|
||||
|
||||
func loadMilvusCollection(ctx context.Context, milvusClient *client.Client, collectionName string) error {
|
||||
loadTask, err := milvusClient.LoadCollection(ctx, client.NewLoadCollectionOption(collectionName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return loadTask.Await(ctx)
|
||||
}
|
||||
|
||||
func flushMilvusCollection(ctx context.Context, milvusClient *client.Client, collectionName string) error {
|
||||
flushTask, err := milvusClient.Flush(ctx, client.NewFlushOption(collectionName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return flushTask.Await(ctx)
|
||||
}
|
||||
|
||||
func migrateCollectionRows(
|
||||
ctx context.Context,
|
||||
milvusClient *client.Client,
|
||||
sourceCollectionName string,
|
||||
targetCollectionName string,
|
||||
dimension int,
|
||||
batchSize int,
|
||||
) (int, error) {
|
||||
log := logger.GetLogger(ctx)
|
||||
totalCopied := 0
|
||||
batchNumber := 0
|
||||
queryIterator, err := milvusClient.QueryIterator(
|
||||
ctx,
|
||||
client.NewQueryIteratorOption(sourceCollectionName).
|
||||
WithOutputFields(migrationOutputFields...).
|
||||
WithBatchSize(batchSize),
|
||||
)
|
||||
if err != nil {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"create query iterator for source collection %s: %w",
|
||||
sourceCollectionName,
|
||||
err,
|
||||
)
|
||||
}
|
||||
|
||||
for {
|
||||
resultSet, err := queryIterator.Next(ctx)
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"query source collection %s with iterator: %w",
|
||||
sourceCollectionName,
|
||||
err,
|
||||
)
|
||||
}
|
||||
if resultSet.ResultCount == 0 {
|
||||
break
|
||||
}
|
||||
batchNumber++
|
||||
|
||||
documents, _, err := convertResultSet([]client.ResultSet{resultSet})
|
||||
if err != nil {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"decode source collection %s at iterator batch %d: %w",
|
||||
sourceCollectionName,
|
||||
batchNumber,
|
||||
err,
|
||||
)
|
||||
}
|
||||
if len(documents) == 0 {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"source collection %s returned rows without decodable fields",
|
||||
sourceCollectionName,
|
||||
)
|
||||
}
|
||||
|
||||
embeddings := make([]*MilvusVectorEmbedding, 0, len(documents))
|
||||
for rowIndex, document := range documents {
|
||||
if len(document.Embedding) != dimension {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"source collection %s row %d has vector dimension %d, expected %d",
|
||||
sourceCollectionName,
|
||||
rowIndex,
|
||||
len(document.Embedding),
|
||||
dimension,
|
||||
)
|
||||
}
|
||||
document.ID, err = primaryKeyForMigratedRow(document, resultSet.IDs, rowIndex)
|
||||
if err != nil {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"source collection %s at iterator batch %d: %w",
|
||||
sourceCollectionName,
|
||||
batchNumber,
|
||||
err,
|
||||
)
|
||||
}
|
||||
document.Language = detectAnalyzerName(document.Content)
|
||||
embeddings = append(embeddings, &document.MilvusVectorEmbedding)
|
||||
}
|
||||
|
||||
if _, err := milvusClient.Upsert(ctx, createUpsert(targetCollectionName, embeddings, true)); err != nil {
|
||||
return totalCopied, fmt.Errorf(
|
||||
"upsert target collection %s at iterator batch %d: %w",
|
||||
targetCollectionName,
|
||||
batchNumber,
|
||||
err,
|
||||
)
|
||||
}
|
||||
totalCopied += len(embeddings)
|
||||
log.Infof("[Milvus] Copied %d rows from %s", totalCopied, sourceCollectionName)
|
||||
}
|
||||
|
||||
return totalCopied, nil
|
||||
}
|
||||
|
|
@ -0,0 +1,126 @@
|
|||
package milvus
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/milvus-io/milvus/client/v2/column"
|
||||
"github.com/milvus-io/milvus/client/v2/entity"
|
||||
"github.com/milvus-io/milvus/client/v2/index"
|
||||
client "github.com/milvus-io/milvus/client/v2/milvusclient"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestCollectionDimensionForBase(t *testing.T) {
|
||||
dimension, ok := collectionDimensionForBase("weknora_embeddings_1536", "weknora_embeddings")
|
||||
require.True(t, ok)
|
||||
require.Equal(t, 1536, dimension)
|
||||
|
||||
_, ok = collectionDimensionForBase("weknora_embeddings_multilingual_1536", "weknora_embeddings")
|
||||
require.False(t, ok)
|
||||
_, ok = collectionDimensionForBase("weknora_embeddings", "weknora_embeddings")
|
||||
require.False(t, ok)
|
||||
_, ok = collectionDimensionForBase("weknora_embeddings_1536_backup", "weknora_embeddings")
|
||||
require.False(t, ok)
|
||||
|
||||
require.True(t, matchesDimensionCollection(
|
||||
"weknora_embeddings_multilingual_1536",
|
||||
"weknora_embeddings_multilingual",
|
||||
))
|
||||
require.False(t, matchesDimensionCollection("weknora_embeddings_1536", "weknora_embeddings_multilingual"))
|
||||
}
|
||||
|
||||
func TestParseMetricType(t *testing.T) {
|
||||
metric, err := ParseMetricType("")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.MetricType(""), metric)
|
||||
|
||||
metric, err = ParseMetricType(" cosine ")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.COSINE, metric)
|
||||
|
||||
_, err = ParseMetricType("dot")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestResolveMigrationMetric(t *testing.T) {
|
||||
metric, err := resolveMigrationMetric("", entity.IP)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.IP, metric)
|
||||
|
||||
metric, err = resolveMigrationMetric(entity.IP, entity.IP)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.IP, metric)
|
||||
|
||||
_, err = resolveMigrationMetric(entity.COSINE, entity.IP)
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "does not match")
|
||||
|
||||
_, err = resolveMigrationMetric("", "")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestMetricTypeFromIndexParams(t *testing.T) {
|
||||
metric, err := metricTypeFromIndexParams(map[string]string{
|
||||
index.MetricTypeKey: "IP",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.IP, metric)
|
||||
|
||||
metric, err = metricTypeFromIndexParams(map[string]string{
|
||||
index.ParamsKey: `{"metric_type":"COSINE"}`,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, entity.COSINE, metric)
|
||||
|
||||
_, err = metricTypeFromIndexParams(map[string]string{})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestPrimaryKeyForMigratedRow(t *testing.T) {
|
||||
id, err := primaryKeyForMigratedRow(&MilvusVectorEmbeddingWithScore{
|
||||
MilvusVectorEmbedding: MilvusVectorEmbedding{ID: "pk-1"},
|
||||
}, nil, 0)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "pk-1", id)
|
||||
|
||||
ids := column.NewColumnVarChar(fieldID, []string{"from-ids"})
|
||||
id, err = primaryKeyForMigratedRow(&MilvusVectorEmbeddingWithScore{}, ids, 0)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "from-ids", id)
|
||||
|
||||
_, err = primaryKeyForMigratedRow(&MilvusVectorEmbeddingWithScore{}, nil, 3)
|
||||
require.Error(t, err)
|
||||
require.Contains(t, err.Error(), "missing primary key")
|
||||
}
|
||||
|
||||
func TestConvertResultSetFloatVector(t *testing.T) {
|
||||
embedding := []float32{0.1, 0.2, 0.3}
|
||||
docs, _, err := convertResultSet([]client.ResultSet{{
|
||||
ResultCount: 1,
|
||||
Fields: client.DataSet{
|
||||
column.NewColumnVarChar(fieldID, []string{"pk-1"}),
|
||||
column.NewColumnVarChar(fieldContent, []string{"hello"}),
|
||||
column.NewColumnVarChar(fieldLanguage, []string{"english"}),
|
||||
column.NewColumnFloatVector(fieldEmbedding, 3, [][]float32{embedding}),
|
||||
},
|
||||
}})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, docs, 1)
|
||||
require.Equal(t, "pk-1", docs[0].ID)
|
||||
require.Equal(t, "hello", docs[0].Content)
|
||||
require.Equal(t, "english", docs[0].Language)
|
||||
require.Equal(t, embedding, docs[0].Embedding)
|
||||
}
|
||||
|
||||
func TestConvertResultSetDoubleArray(t *testing.T) {
|
||||
docs, _, err := convertResultSet([]client.ResultSet{{
|
||||
ResultCount: 1,
|
||||
Fields: client.DataSet{
|
||||
column.NewColumnVarChar(fieldID, []string{"pk-1"}),
|
||||
column.NewColumnDoubleArray(fieldEmbedding, [][]float64{{1, 2, 3}}),
|
||||
},
|
||||
}})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, docs, 1)
|
||||
require.Equal(t, []float32{1, 2, 3}, docs[0].Embedding)
|
||||
}
|
||||
|
|
@ -18,11 +18,15 @@ func (m *milvusRepository) MoveKnowledgeIndices(
|
|||
{Field: fieldKnowledgeID, Operator: operatorEqual, Value: knowledgeID},
|
||||
}}
|
||||
size, offset := 100, 0
|
||||
collectionMode, err := m.collectionAnalyzerMode(ctx, collection)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return drainMoveRows(ctx, targetKB, func(ctx context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
|
||||
rows, _, err := m.searchByFilter(ctx, collection, filter, &size, &offset)
|
||||
return rows, err
|
||||
}, func(ctx context.Context, rows []*MilvusVectorEmbedding) error {
|
||||
_, err := m.client.Upsert(ctx, createUpsert(collection, rows))
|
||||
_, err := m.client.Upsert(ctx, createUpsert(collection, rows, collectionMode == collectionAnalyzerMulti))
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,7 +24,6 @@ const (
|
|||
envMilvusCollection = "MILVUS_COLLECTION"
|
||||
envMilvusMetricType = "MILVUS_METRIC_TYPE"
|
||||
defaultCollectionName = "weknora_embeddings"
|
||||
fieldContent = "content"
|
||||
fieldSourceID = "source_id"
|
||||
fieldSourceType = "source_type"
|
||||
fieldChunkID = "chunk_id"
|
||||
|
|
@ -38,7 +37,7 @@ const (
|
|||
)
|
||||
|
||||
var (
|
||||
allFields = []string{fieldID, fieldContent, fieldSourceID, fieldSourceType, fieldChunkID,
|
||||
allFields = []string{fieldID, fieldContent, fieldLanguage, fieldSourceID, fieldSourceType, fieldChunkID,
|
||||
fieldKnowledgeID, fieldKnowledgeBaseID, fieldTagID, fieldIsEnabled, fieldEmbedding}
|
||||
)
|
||||
|
||||
|
|
@ -83,6 +82,42 @@ func (m *milvusRepository) getCollectionName(dimension int) string {
|
|||
return fmt.Sprintf("%s_%d", m.collectionBaseName, dimension)
|
||||
}
|
||||
|
||||
// collectionAnalyzerMode inspects a collection once and caches whether it
|
||||
// uses the multilingual schema. Older WeKnora collections do not have the
|
||||
// language field, so they must continue using the legacy upsert and search
|
||||
// paths until they are migrated.
|
||||
func (m *milvusRepository) collectionAnalyzerMode(
|
||||
ctx context.Context,
|
||||
collectionName string,
|
||||
) (collectionAnalyzerMode, error) {
|
||||
if value, ok := m.collectionAnalyzerModes.Load(collectionName); ok {
|
||||
mode, ok := value.(collectionAnalyzerMode)
|
||||
if ok {
|
||||
return mode, nil
|
||||
}
|
||||
m.collectionAnalyzerModes.Delete(collectionName)
|
||||
}
|
||||
|
||||
collection, err := m.client.DescribeCollection(
|
||||
ctx,
|
||||
client.NewDescribeCollectionOption(collectionName),
|
||||
)
|
||||
if err != nil {
|
||||
return collectionAnalyzerLegacy, fmt.Errorf(
|
||||
"describe Milvus collection %s: %w", collectionName, err,
|
||||
)
|
||||
}
|
||||
if collection == nil || collection.Schema == nil {
|
||||
return collectionAnalyzerLegacy, fmt.Errorf(
|
||||
"collection %s has no schema", collectionName,
|
||||
)
|
||||
}
|
||||
|
||||
mode := analyzerModeFromSchema(collection.Schema)
|
||||
m.collectionAnalyzerModes.Store(collectionName, mode)
|
||||
return mode, nil
|
||||
}
|
||||
|
||||
// ensureCollection ensures the collection exists for the given dimension
|
||||
func (m *milvusRepository) ensureCollection(ctx context.Context, dimension int) error {
|
||||
collectionName := m.getCollectionName(dimension)
|
||||
|
|
@ -124,7 +159,13 @@ func (m *milvusRepository) ensureCollection(ctx context.Context, dimension int)
|
|||
WithDataType(entity.FieldTypeVarChar).
|
||||
WithMaxLength(65535).
|
||||
WithEnableAnalyzer(true).
|
||||
WithEnableMatch(true),
|
||||
// Multi-language analyzer fields are used by the BM25 function;
|
||||
// Milvus does not allow enable_match on the same field.
|
||||
WithMultiAnalyzerParams(multiAnalyzerParams()),
|
||||
entity.NewField().
|
||||
WithName(fieldLanguage).
|
||||
WithDataType(entity.FieldTypeVarChar).
|
||||
WithMaxLength(32),
|
||||
entity.NewField().
|
||||
WithName(fieldContentSparse).
|
||||
WithDataType(entity.FieldTypeSparseVector),
|
||||
|
|
@ -187,6 +228,13 @@ func (m *milvusRepository) ensureCollection(ctx context.Context, dimension int)
|
|||
}
|
||||
|
||||
log.Infof("[Milvus] Successfully created collection %s", collectionName)
|
||||
m.collectionAnalyzerModes.Store(collectionName, collectionAnalyzerMulti)
|
||||
} else {
|
||||
// Existing collections may have been created by an older WeKnora
|
||||
// version. Keep them readable and writable until they are migrated.
|
||||
if _, err := m.collectionAnalyzerMode(ctx, collectionName); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
loadOpt := client.NewLoadCollectionOption(collectionName)
|
||||
|
|
@ -252,11 +300,19 @@ func (m *milvusRepository) Save(ctx context.Context,
|
|||
}
|
||||
|
||||
collectionName := m.getCollectionName(dimension)
|
||||
collectionMode, err := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
embeddingDB.ID = uuid.New().String()
|
||||
opts := createUpsert(collectionName, []*MilvusVectorEmbedding{embeddingDB})
|
||||
opts := createUpsert(
|
||||
collectionName,
|
||||
[]*MilvusVectorEmbedding{embeddingDB},
|
||||
collectionMode == collectionAnalyzerMulti,
|
||||
)
|
||||
|
||||
_, err := m.client.Upsert(ctx, opts)
|
||||
_, err = m.client.Upsert(ctx, opts)
|
||||
if err != nil {
|
||||
log.Errorf("[Milvus] Failed to save index: %v", err)
|
||||
return err
|
||||
|
|
@ -306,6 +362,10 @@ func (m *milvusRepository) BatchSave(ctx context.Context,
|
|||
}
|
||||
|
||||
collectionName := m.getCollectionName(dimension)
|
||||
collectionMode, err := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
n := len(embeddings)
|
||||
embeddingDBList := make([]*MilvusVectorEmbedding, 0, n)
|
||||
|
||||
|
|
@ -314,8 +374,12 @@ func (m *milvusRepository) BatchSave(ctx context.Context,
|
|||
embeddingDB.ID = uuid.New().String()
|
||||
embeddingDBList = append(embeddingDBList, embeddingDB)
|
||||
}
|
||||
opts := createUpsert(collectionName, embeddingDBList)
|
||||
_, err := m.client.Upsert(ctx, opts)
|
||||
opts := createUpsert(
|
||||
collectionName,
|
||||
embeddingDBList,
|
||||
collectionMode == collectionAnalyzerMulti,
|
||||
)
|
||||
_, err = m.client.Upsert(ctx, opts)
|
||||
if err != nil {
|
||||
log.Errorf("[Milvus] Failed to execute batch operation for dimension %d: %v", dimension, err)
|
||||
return fmt.Errorf("failed to batch save (dimension %d): %w", dimension, err)
|
||||
|
|
@ -454,8 +518,7 @@ func updateChunkEnabledStatusInCollections(
|
|||
) error {
|
||||
var updateErrs []error
|
||||
for _, collectionName := range collections {
|
||||
if len(collectionName) <= len(collectionBaseName) ||
|
||||
collectionName[:len(collectionBaseName)] != collectionBaseName {
|
||||
if !matchesDimensionCollection(collectionName, collectionBaseName) {
|
||||
continue
|
||||
}
|
||||
if err := update(ctx, collectionName, enabledChunkIDs, true); err != nil {
|
||||
|
|
@ -496,7 +559,11 @@ func (m *milvusRepository) updateChunkEnabledStatusInCollection(
|
|||
return nil
|
||||
}
|
||||
|
||||
req := createUpsert(collectionName, upsertEmbeddings)
|
||||
collectionMode, err := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req := createUpsert(collectionName, upsertEmbeddings, collectionMode == collectionAnalyzerMulti)
|
||||
if _, err := m.client.Upsert(ctx, req); err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -558,9 +625,7 @@ func (m *milvusRepository) BatchUpdateChunkTagID(ctx context.Context, chunkTagMa
|
|||
|
||||
// Update in all matching collections
|
||||
for _, collectionName := range collections {
|
||||
// Only process collections that start with our base name
|
||||
if len(collectionName) <= len(m.collectionBaseName) ||
|
||||
collectionName[:len(m.collectionBaseName)] != m.collectionBaseName {
|
||||
if !matchesDimensionCollection(collectionName, m.collectionBaseName) {
|
||||
continue
|
||||
}
|
||||
// Update chunks for each tag ID
|
||||
|
|
@ -580,7 +645,12 @@ func (m *milvusRepository) BatchUpdateChunkTagID(ctx context.Context, chunkTagMa
|
|||
upsertEmbeddings = append(upsertEmbeddings, &embedding.MilvusVectorEmbedding)
|
||||
}
|
||||
if len(upsertEmbeddings) > 0 {
|
||||
req := createUpsert(collectionName, upsertEmbeddings)
|
||||
collectionMode, modeErr := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if modeErr != nil {
|
||||
log.Warnf("[Milvus] Failed to inspect collection %s analyzer mode: %v", collectionName, modeErr)
|
||||
continue
|
||||
}
|
||||
req := createUpsert(collectionName, upsertEmbeddings, collectionMode == collectionAnalyzerMulti)
|
||||
_, err := m.client.Upsert(ctx, req)
|
||||
if err != nil {
|
||||
log.Warnf("[Milvus] Failed to update chunks in %s: %v", collectionName, err)
|
||||
|
|
@ -757,9 +827,12 @@ func (m *milvusRepository) KeywordsRetrieve(ctx context.Context,
|
|||
|
||||
// Search in all matching collections
|
||||
for _, collectionName := range collections {
|
||||
// Only process collections that start with our base name
|
||||
if len(collectionName) <= len(m.collectionBaseName) ||
|
||||
collectionName[:len(m.collectionBaseName)] != m.collectionBaseName {
|
||||
if !matchesDimensionCollection(collectionName, m.collectionBaseName) {
|
||||
continue
|
||||
}
|
||||
collectionMode, modeErr := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if modeErr != nil {
|
||||
log.Errorf("[Milvus] Failed to inspect collection %s analyzer mode: %v", collectionName, modeErr)
|
||||
continue
|
||||
}
|
||||
|
||||
|
|
@ -770,6 +843,15 @@ func (m *milvusRepository) KeywordsRetrieve(ctx context.Context,
|
|||
}
|
||||
searchOpt := client.NewSearchOption(collectionName, params.TopK, []entity.Vector{entity.Text(params.Query)})
|
||||
searchOpt.WithANNSField(fieldContentSparse)
|
||||
if collectionMode == collectionAnalyzerMulti {
|
||||
analyzerName := detectAnalyzerName(params.Query)
|
||||
ann := index.NewCustomAnnParam()
|
||||
ann.WithExtraParam("metric_type", "BM25")
|
||||
ann.WithExtraParam("analyzer_name", analyzerName)
|
||||
ann.WithExtraParam("drop_ratio_search", 0)
|
||||
searchOpt.WithAnnParam(&ann)
|
||||
log.Debugf("[Milvus] BM25 analyzer: collection=%s, analyzer=%s", collectionName, analyzerName)
|
||||
}
|
||||
if expr != "" {
|
||||
searchOpt.WithFilter(expr)
|
||||
for k, v := range paramsMap {
|
||||
|
|
@ -832,6 +914,10 @@ func (m *milvusRepository) CopyIndices(ctx context.Context,
|
|||
if err := m.ensureCollection(ctx, dimension); err != nil {
|
||||
return err
|
||||
}
|
||||
collectionMode, err := m.collectionAnalyzerMode(ctx, collectionName)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
batchSize := 64
|
||||
totalCopied := 0
|
||||
|
|
@ -877,6 +963,7 @@ func (m *milvusRepository) CopyIndices(ctx context.Context,
|
|||
targetEmbedding := &MilvusVectorEmbedding{
|
||||
ID: uuid.New().String(),
|
||||
Content: sourceEmbedding.Content,
|
||||
Language: sourceEmbedding.Language,
|
||||
SourceID: targetSourceID,
|
||||
SourceType: sourceEmbedding.SourceType,
|
||||
ChunkID: targetChunkID,
|
||||
|
|
@ -889,7 +976,7 @@ func (m *milvusRepository) CopyIndices(ctx context.Context,
|
|||
targetEmbeddings = append(targetEmbeddings, targetEmbedding)
|
||||
}
|
||||
if len(targetEmbeddings) > 0 {
|
||||
opts := createUpsert(collectionName, targetEmbeddings)
|
||||
opts := createUpsert(collectionName, targetEmbeddings, collectionMode == collectionAnalyzerMulti)
|
||||
_, err := m.client.Upsert(ctx, opts)
|
||||
if err != nil {
|
||||
log.Errorf("[Milvus] Failed to batch upsert target points: %v", err)
|
||||
|
|
@ -928,6 +1015,7 @@ func (m *milvusRepository) calculateStorageSize(embedding *MilvusVectorEmbedding
|
|||
// Payload fields
|
||||
payloadSizeBytes := int64(0)
|
||||
payloadSizeBytes += int64(len(embedding.Content)) // content string
|
||||
payloadSizeBytes += int64(len(embedding.Language)) // language selector
|
||||
payloadSizeBytes += int64(len(embedding.SourceID)) // source_id string
|
||||
payloadSizeBytes += int64(len(embedding.ChunkID)) // chunk_id string
|
||||
payloadSizeBytes += int64(len(embedding.KnowledgeID)) // knowledge_id string
|
||||
|
|
@ -959,6 +1047,7 @@ func (m *milvusRepository) calculateStorageSize(embedding *MilvusVectorEmbedding
|
|||
func toMilvusVectorEmbedding(embedding *types.IndexInfo, additionalParams map[string]interface{}) *MilvusVectorEmbedding {
|
||||
vector := &MilvusVectorEmbedding{
|
||||
Content: embedding.Content,
|
||||
Language: detectAnalyzerName(embedding.Content),
|
||||
SourceID: embedding.SourceID,
|
||||
SourceType: int(embedding.SourceType),
|
||||
ChunkID: embedding.ChunkID,
|
||||
|
|
@ -994,10 +1083,15 @@ func fromMilvusVectorEmbedding(id string,
|
|||
}
|
||||
}
|
||||
|
||||
func createUpsert(collectionName string, embeddings []*MilvusVectorEmbedding) client.UpsertOption {
|
||||
func createUpsert(
|
||||
collectionName string,
|
||||
embeddings []*MilvusVectorEmbedding,
|
||||
includeLanguage bool,
|
||||
) client.UpsertOption {
|
||||
ids := make([]string, 0, len(embeddings))
|
||||
embeddingsData := make([][]float32, 0, len(embeddings))
|
||||
contents := make([]string, 0, len(embeddings))
|
||||
languages := make([]string, 0, len(embeddings))
|
||||
sourceIDs := make([]string, 0, len(embeddings))
|
||||
sourceTypes := make([]int64, 0, len(embeddings))
|
||||
chunkIDs := make([]string, 0, len(embeddings))
|
||||
|
|
@ -1010,6 +1104,9 @@ func createUpsert(collectionName string, embeddings []*MilvusVectorEmbedding) cl
|
|||
ids = append(ids, embedding.ID)
|
||||
embeddingsData = append(embeddingsData, embedding.Embedding)
|
||||
contents = append(contents, embedding.Content)
|
||||
if includeLanguage {
|
||||
languages = append(languages, analyzerNameForUpsert(embedding))
|
||||
}
|
||||
sourceIDs = append(sourceIDs, embedding.SourceID)
|
||||
sourceTypes = append(sourceTypes, int64(embedding.SourceType))
|
||||
chunkIDs = append(chunkIDs, embedding.ChunkID)
|
||||
|
|
@ -1030,6 +1127,10 @@ func createUpsert(collectionName string, embeddings []*MilvusVectorEmbedding) cl
|
|||
WithVarcharColumn(fieldKnowledgeBaseID, knowledgeBaseIDs).
|
||||
WithVarcharColumn(fieldTagID, tagIDs).
|
||||
WithBoolColumn(fieldIsEnabled, isEnableds)
|
||||
if includeLanguage {
|
||||
// New multilingual collections require one analyzer selector per row.
|
||||
opt.WithVarcharColumn(fieldLanguage, languages)
|
||||
}
|
||||
return opt
|
||||
}
|
||||
|
||||
|
|
@ -1075,6 +1176,15 @@ func convertResultSet(resultSet []client.ResultSet) ([]*MilvusVectorEmbeddingWit
|
|||
docs[i].Content = val
|
||||
}
|
||||
}
|
||||
if field == fieldLanguage {
|
||||
for i := 0; i < columns.Len(); i++ {
|
||||
val, err := columns.GetAsString(i)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
docs[i].Language = val
|
||||
}
|
||||
}
|
||||
if field == fieldSourceID {
|
||||
for i := 0; i < columns.Len(); i++ {
|
||||
val, err := columns.GetAsString(i)
|
||||
|
|
@ -1139,20 +1249,31 @@ func convertResultSet(resultSet []client.ResultSet) ([]*MilvusVectorEmbeddingWit
|
|||
}
|
||||
}
|
||||
if field == fieldEmbedding {
|
||||
vectorColumn, ok := columns.(*column.ColumnDoubleArray)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
for i := 0; i < vectorColumn.Len(); i++ {
|
||||
val, err := vectorColumn.Value(i)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("get vector failed: %w", err)
|
||||
switch vectorColumn := columns.(type) {
|
||||
case *column.ColumnFloatVector:
|
||||
for i := 0; i < vectorColumn.Len(); i++ {
|
||||
val, err := vectorColumn.Value(i)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("get float vector failed: %w", err)
|
||||
}
|
||||
embedding := make([]float32, len(val))
|
||||
copy(embedding, val)
|
||||
docs[i].Embedding = embedding
|
||||
}
|
||||
embedding := make([]float32, len(val))
|
||||
for j, v := range val {
|
||||
embedding[j] = float32(v)
|
||||
case *column.ColumnDoubleArray:
|
||||
// Keep compatibility with Milvus responses that expose vectors
|
||||
// through the legacy double-array representation.
|
||||
for i := 0; i < vectorColumn.Len(); i++ {
|
||||
val, err := vectorColumn.Value(i)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("get double vector failed: %w", err)
|
||||
}
|
||||
embedding := make([]float32, len(val))
|
||||
for j, v := range val {
|
||||
embedding[j] = float32(v)
|
||||
}
|
||||
docs[i].Embedding = embedding
|
||||
}
|
||||
docs[i].Embedding = embedding
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -42,3 +42,29 @@ func TestUpdateChunkEnabledStatusInCollectionsPropagatesFailure(t *testing.T) {
|
|||
)
|
||||
require.ErrorIs(t, err, wantErr)
|
||||
}
|
||||
|
||||
func TestUpdateChunkEnabledStatusInCollectionsIgnoresExtendedPrefix(t *testing.T) {
|
||||
var seen []string
|
||||
seenSet := map[string]bool{}
|
||||
err := updateChunkEnabledStatusInCollections(
|
||||
context.Background(),
|
||||
[]string{
|
||||
"other_collection",
|
||||
"weknora_embeddings_1024",
|
||||
"weknora_embeddings_multilingual_1024",
|
||||
"weknora_embeddings_1024_backup",
|
||||
},
|
||||
"weknora_embeddings",
|
||||
[]string{"chunk-1"},
|
||||
nil,
|
||||
func(_ context.Context, collection string, _ []string, _ bool) error {
|
||||
if !seenSet[collection] {
|
||||
seenSet[collection] = true
|
||||
seen = append(seen, collection)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"weknora_embeddings_1024"}, seen)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,11 +16,14 @@ type milvusRepository struct {
|
|||
replicaNumber int // 0 = use Milvus default (1); set at LoadCollection time
|
||||
// Cache for initialized collections (dimension -> true)
|
||||
initializedCollections sync.Map
|
||||
// Cache for collection analyzer modes (collection name -> collectionAnalyzerMode).
|
||||
collectionAnalyzerModes sync.Map
|
||||
}
|
||||
|
||||
type MilvusVectorEmbedding struct {
|
||||
ID string `json:"id"`
|
||||
Content string `json:"content"`
|
||||
Language string `json:"language"`
|
||||
SourceID string `json:"source_id"`
|
||||
SourceType int `json:"source_type"`
|
||||
ChunkID string `json:"chunk_id"`
|
||||
|
|
|
|||
|
|
@ -171,12 +171,12 @@ flowchart TD
|
|||
|
||||
`internal/application/repository/retriever/milvus/repository.go`。
|
||||
|
||||
- **集合管理**:每维度一个 collection(`{MILVUS_COLLECTION|weknora_embeddings}_{dim}`)。schema 含稠密向量 `embedding`(HNSW 索引,M=16 efConstruction=128,metric 由 `MILVUS_METRIC_TYPE` 决定:IP 默认 / COSINE /)与稀疏向量 `content_sparse` —— 通过 **内建 BM25 Function**(`entity.FunctionTypeBM25`)由 content 自动生成,配 `AutoIndex(BM25)`。
|
||||
- **集合管理**:每维度一个 collection(`{MILVUS_COLLECTION|weknora_embeddings}_{dim}`)。schema 含稠密向量 `embedding`(HNSW 索引,M=16 efConstruction=128,metric 由 `MILVUS_METRIC_TYPE` 决定:IP 默认 / COSINE / L2)与稀疏向量 `content_sparse` —— 通过 **内建 BM25 Function**(`entity.FunctionTypeBM25`)由 content 自动生成,配 `AutoIndex(BM25)`。新建 Collection 的 `content` 使用 Milvus 多语言分析器:英文 `english`、中文 `chinese`(内置 Jieba)、未知语言 `default`(ICU),并以 `language` 字段选择分析器。
|
||||
- **向量检索**:`Search` + `WithANNSField(embedding)`,COSINE 模式原始值域 [-1,1],是唯一需要 `(score+1)/2` 归一化的引擎。
|
||||
- **关键词检索**:对 `content_sparse` 做 BM25 稀疏向量检索(Milvus 2.5+ 原生全文检索)。
|
||||
- **关键词检索**:对 `content_sparse` 做 BM25 稀疏向量检索(Milvus 2.5+ 原生全文检索),查询时根据问题文本的语言传入 `analyzer_name`。
|
||||
- **过滤**:`filter.go` 构造布尔表达式(kb/knowledge/tag/is_enabled)。
|
||||
- **启停同步**:`BatchUpdateChunkEnabledStatus` 逐 collection 更新,失败用 `errors.Join` 聚合后**返回错误**而不是只打 warn——主库里已停用的分块绝不能因为索引更新静默失败而继续可被检索到。
|
||||
- 配置:`MILVUS_ADDRESS` / `MILVUS_USERNAME` / `MILVUS_PASSWORD` / `MILVUS_DB_NAME` / `MILVUS_METRIC_TYPE`(改后需重建 collection)。
|
||||
- 配置:`MILVUS_ADDRESS` / `MILVUS_USERNAME` / `MILVUS_PASSWORD` / `MILVUS_DB_NAME` / `MILVUS_METRIC_TYPE`(改后需重建 collection)。旧 Collection 的 schema 不能直接改成多语言分析器;可运行 `go run ./cmd/milvus-migrate --source <旧前缀> --target <新前缀>` 复用已有稠密向量并沿用源 Collection 的 metric,确认检索正常后把 `MILVUS_COLLECTION` 改为新前缀。
|
||||
|
||||
#### Weaviate {#_2-8-weaviate}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue