WeKnora/internal/application/repository
linus-liu-web 04da26024c
fix(weaviate): CopyIndices 补游标防护,并修掉 GraphQL 报错时的 panic (#3881)
* fix(weaviate): 复制索引时游标不前进立即报错,并给分页加跳数上限

CopyIndices 的游标由每页最后一个对象的 id 反推(repository.go:763),循环的
退出条件只有"查询出错"和"这一页为空"。源端只要重复返回同一页,游标就永远
停在同一个 id 上:循环把这一页反复读出、再反复 upsert 进目标库,直到复制任务
撞上 2 小时超时;源端在该页之后的数据永远读不到,而目标库里同一批 chunk 已经
被写成成千上万份候选向量。同包的 move.go 早就记录了 after+where 在 Weaviate
上不可靠,并为此用 seen 集合拦住"游标不前进",CopyIndices 用的正是这个组合,
却没有那层保护。

本次修改:

- 记录每个用过的游标,重复即返回 "copy indices made no progress at cursor",
  不再把"游标不动"当成"读完了"。检测点放在整页解析之后、批量写入之前,所以
  重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页,按每页 64 条合 64 万行,与
  confluence 的 maxPaginationHops 同量级);触顶时打印 Warn 并返回错误,而不是
  把这轮复制当成成功——静默截断会留下一个看起来正常的残缺索引。
- additional["id"] 从无检查断言改为带检查断言,缺 id 或 id 为空时返回错误;
  原先这里会 panic。同包 move.go 的 move index ID 已是这种写法。

回归用例用假 Weaviate 服务端复现两种源端形态:恒定返回同一页的源端在第二次
查询后被拦下(GraphQL 查询 2 次、目标库只写 64 条,而修复前 5s 内同一页被重放
4484 次、写入 286912 条);一直推进但永不结束的源端在第 10000 页被拦下。

* fix(weaviate): 复制索引时 GraphQL 层报错不再被当成"源已读完",也不再 panic

Weaviate 对于"执行不了的查询"返回的是 HTTP 200 + errors 数组 + 没有 data。SDK 的
runGraphQLQuery 只在 HTTP 状态异常时返回 error(weaviate-go-client 的
graphql/gql.go:136-150),所以这种响应到 CopyIndices 手里时 err == nil,而
result.Data 是 nil。

原代码只检查 err,随后一行完成两件事:

    objects, ok := result.Data["Get"].(map[string]interface{})[collectionName].([]interface{})

这里逗号-ok 只作用于最后一个断言,第一个 `.(map[string]interface{})` 是单值断言;
result.Data 为 nil 时该表达式的值是 nil 接口,单值断言直接 panic。实测
(假服务端返回 {"errors":[{"message":"class Weknora_embeddings_64 does not exist"}]}):

    panic: interface conversion: models.JSONObject is nil, not map[string]interface {}
        .../weaviate.(*weaviateRepository).CopyIndices(...)
            repository.go:753

也就是说复制任务在这种响应下不是"静默少拷",而是直接 panic;即便不 panic,把
"没有 data"当成"这一页为空"也会让一次失败的查询以 Index copy completed 收尾。

本次修改:

- 在 err 检查之后补 `len(result.Errors) > 0` → 记录日志并返回
  "weaviate: copy indices query failed: <message>",与同文件 VectorRetrieve
  (:608-611)、KeywordsRetrieve (:677-680) 以及同包 move.go:32-34 的既有写法一致。
- 把那一行的两级断言拆成两次逗号-ok 断言,响应缺 Get 或 class 键时返回
  "weaviate: copy indices invalid response",不再 panic。

回归用例 TestCopyIndicesReportsAQueryError:假服务端对所有 Get 返回上述 errors
响应,断言 CopyIndices 返回带原始 message 的错误(修复前是 panic),且只发一次
查询、没有把任何东西当成复制成功。

* test(weaviate): 补一个"响应读不出 data 时不得 panic"的复制用例

上一个提交把 `result.Data["Get"].(map[string]interface{})` 这一处单值断言拆成了两次
逗号-ok 断言,但只验证了"带 errors 的响应"那条路径:那条路径在 Errors 检查处就返回
了,走不到断言。这里补 TestCopyIndicesRejectsAResponseWithoutData,让假服务端返回
HTTP 200 + `{"data":{}}`(没有 errors,也没有 Get 键),断言 CopyIndices 返回
invalid response 而不是 panic;去掉断言拆分、只保留 Errors 检查时该用例必红。

两个用例都用 recover 把 panic 转成明确的用例失败,避免一个 panic 直接终结整个测试
二进制、连带吞掉同包其它用例的结果。

* fix(retriever/weaviate): CopyIndices 按映射分块读源行,去掉会被服务端拒绝的游标

CopyIndices 用 where + after 遍历源知识库,而 Weaviate 的游标 API 不允许过滤条件
与 after 同时出现:ValidateCursor 直接返回 "where cannot be set with after and
limit parameters",且 client 第一页就会把 WithAfter("") 发成 after: "",所以这个
查询在真实服务端上一页都跑不通。该报错以 HTTP 200 + errors 返回,不检查 errors 的
代码会把它当成"源已读完",返回成功但一条都没复制(本 PR 已补的 errors 检查会把它
变成错误,但那是在暴露一个本就不合法的查询)。即使去掉 where,游标仍然是"跨请求
保持不变的结果顺序"上的位置,服务端不承诺维持该顺序,一旦漂移就会漏行且仍报成功。

改成按映射里的源 chunk ID 分块读:谓词 = knowledge_base_id 相等 AND chunk_id
ContainsAny(本批 64 个),不带 after。读到的行集合由谓词唯一确定,没有游标可漂。

同一个 chunk 会有正文向量 + 多个生成问题向量,64 个 ID 不等于最多 64 行,所以每批
查询显式带 limit(QUERY_MAXIMUM_RESULTS,v1.37.3 默认 10000)。该配置没有客户端
接口(/v1/meta 只有 version/hostname/modules),先按默认值查,被服务端以
"query maximum results exceeded ... QUERY_MAXIMUM_RESULTS 'N'" 拒绝时按 N 降级
重试。一批返回行数达到 limit 说明可能被截断,对该批对半拆分重查直到单个 chunk;
单个 chunk 仍然填满 limit 时明确报错(含 chunk ID 与服务端上限),绝不返回复制成功。
不带 limit 的 GraphQL Get 会落到 QUERY_DEFAULTS_LIMIT_GRAPHQL(100) 静默截断。

页数上限(maxCopyPaginationHops)与重复游标防护随游标一起去掉:读取范围由映射
限定,不存在走不完的分页。GraphQL errors 检查与 data["Get"]、_additional["id"]
的逗号-ok 解包保留。

测试:copy_test.go 的假服务改成按 chunk_id 过滤的真实形状,并像 v1.37.3 一样拒绝
where+after 组合(假服务会断言这种组合从未出现);补同一 chunk 多向量、批次触顶
拆分、单 chunk 超限报错、QUERY_MAXIMUM_RESULTS 降级四类用例。

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: linus-liu-web <linus-liu-web@users.noreply.github.com>
2026-10-09 10:47:30 +08:00
..
retriever fix(weaviate): CopyIndices 补游标防护,并修掉 GraphQL 报错时的 panic (#3881) 2026-10-09 10:47:30 +08:00
agent_share.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
agent_share_source_test.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
audit_log.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
audit_log_scope_test.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
chunk.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
chunk_faq_diff_test.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
chunk_fields_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
chunk_image_assets.go feat(gallery): add an image gallery tab with schema-driven filters (#3694) 2026-09-25 12:33:59 +08:00
chunk_image_assets_test.go feat(gallery): add an image gallery tab with schema-driven filters (#3694) 2026-09-25 12:33:59 +08:00
chunk_revision_test.go fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794) 2026-09-27 20:32:26 +08:00
chunk_sqlite_test.go perf(chunk): speed up chunk ingestion and batch updates (#3369) 2026-09-17 21:38:15 +08:00
custom_agent.go fix(models): show usage details when deletion is blocked (#2975) 2026-09-02 17:20:37 +08:00
datasource_repo.go fix(datasource): validate cron schedules and persist manual-only mode (#3653) 2026-09-25 22:23:18 +08:00
datasource_repo_test.go fix(datasource): restore sync_deletions on the in-memory object after create 2026-08-21 16:37:05 +08:00
embed_channel.go feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
kbshare.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
knowledge.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
knowledge_create_test.go fix(knowledge): default custom metadata on create 2026-07-28 22:22:59 +08:00
knowledge_datasource_external_id_test.go fix(datasource): perform real sync deletions scoped per data source (#2690) 2026-08-13 19:52:44 +08:00
knowledge_duplicate_test.go fix(knowledge): recover stranded deleting rows and align the KB count (#3367) 2026-09-18 00:55:44 +08:00
knowledge_finalize_test.go fix(knowledge): stop parses from getting stuck in processing or finalizing (#3696) 2026-09-25 13:26:38 +08:00
knowledge_folder_move_test.go refactor(knowledge): improve folder path handling and add tests for underscore scenarios 2026-08-04 20:45:29 +08:00
knowledge_folder_test.go refactor(knowledge): improve folder path handling and add tests for underscore scenarios 2026-08-04 20:45:29 +08:00
knowledge_list_filter_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledge_list_sort_test.go feat: add configurable sorting to document and resource lists (#3318) 2026-09-23 12:00:21 +08:00
knowledge_metadata_prefix_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
knowledge_source_schema_test.go fix(schema): expand knowledge source length 2026-06-05 15:12:25 +08:00
knowledge_span_repo.go fix(knowledge): close spans left open on terminal rows and superseded attempts (#3855) 2026-10-08 01:25:38 +08:00
knowledge_span_repo_test.go feat(knowledge): flag documents that stop making progress (#3557) 2026-09-22 21:34:08 +08:00
knowledge_tag.go fix(knowledge): address auto-tag review findings 2026-08-07 15:01:01 +08:00
knowledge_tag_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
knowledge_transfer.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledge_transfer_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledgebase.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
knowledgebase_sqlite_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
like_escape_sqlite_test.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
mcp_endpoint.go feat(mcp): add built-in MCP server with per-workspace endpoints (#3371) 2026-09-17 22:40:05 +08:00
mcp_metadata.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_metadata_test.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_oauth.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_oauth_test.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_service.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_tool_approval_repository.go feat(mcp): add per-tool enable toggle (#2989) 2026-09-05 17:19:20 +08:00
mcp_tool_approval_repository_test.go test(mcp): stabilize concurrent policy updates on SQLite 2026-09-17 17:47:24 +08:00
memory.go fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
memory_extraction.go fix(memory): preserve extraction progress and atomic memory updates (#3224) 2026-09-13 00:04:16 +08:00
memory_lifecycle.go fix(memory): preserve extraction progress and atomic memory updates (#3224) 2026-09-13 00:04:16 +08:00
memory_vector.go fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
message.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_artifact.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_artifact_delete_test.go feat(artifacts): let users delete generated files (#3447) 2026-09-20 13:01:46 +08:00
message_artifact_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_context_checkpoint_test.go feat(agent): size agent history by the window and calibrate token estimates (#3413) 2026-09-18 22:51:31 +08:00
message_fork_boundary_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_owner_scope_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_request_id_test.go feat(session): fork conversations from a historical user message (#3304) 2026-09-17 14:20:18 +08:00
message_rewind_boundary_test.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_search_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_suggestion.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
model.go fix(repository): correct query syntax to get correctly built-in models 2026-05-21 16:11:22 +08:00
model_catalog_config.go feat(models): add system admin model catalog console (#3662) 2026-09-24 16:31:45 +08:00
model_usage.go fix(models): protect auto-tag model references 2026-09-14 19:29:19 +08:00
model_usage_test.go fix(models): protect auto-tag model references 2026-09-14 19:29:19 +08:00
organization.go fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
organization_member_limit_test.go fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
resource.go feat(agent): preview skill artifacts and verify installs without running them (#2823) 2026-08-27 10:30:32 +08:00
resource_references.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
session.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
session_fork_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
session_test.go feat(sandbox): expose tenant skill management over HTTP and in the console (#2789) 2026-08-25 11:28:12 +08:00
storagebackend.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
task_queue.go fix(wiki): bound outstanding follow-up tasks 2026-09-28 22:02:07 +08:00
task_queue_test.go fix(wiki): bound outstanding follow-up tasks 2026-09-28 22:02:07 +08:00
temporary_document.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
tenant.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
tenant_api_key.go feat: support updating API key configuration 2026-08-12 11:43:03 +08:00
tenant_api_key_test.go feat: support updating API key configuration 2026-08-12 11:43:03 +08:00
tenant_disabled_shared_agent.go feat(agent): enhance agent and skill management features 2026-02-09 11:40:25 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
tenant_sandbox_config.go feat(sandbox): expose tenant skill management over HTTP and in the console (#2789) 2026-08-25 11:28:12 +08:00
tenant_sandbox_config_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
tenant_skill.go fix(skills): upgrade outdated sandbox skills without rolling back the catalog or taking them offline (#3406) 2026-09-18 20:27:43 +08:00
tenant_skill_test.go feat(skills): store skills in a tenant catalog and install per sandbox 2026-08-28 15:17:37 +08:00
tenant_test.go feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
user.go feat(sandbox): add an interactive terminal and files tab beside chat (#3110) 2026-09-08 17:43:15 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
user_tenantless_test.go feat(auth): support tenantless provisioning and gated self-service workspace creation 2026-07-13 18:13:51 +08:00
vectorstore.go feat: add VectorStore service layer with types and connection testing 2026-04-13 22:57:17 +08:00
web_search_provider.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
wiki_page.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
wiki_page_test.go feat(wiki): 增加跨知识库 Wiki 搜索 POST /wiki-search (#3535) 2026-09-25 22:22:58 +08:00