WeKnora/internal/application/repository
linus-liu-web 4eb23a8560
fix(qdrant): CopyIndices 用服务端返回的 next_page_offset 翻页,不再按页尾 id 回填 (#3882)
* fix(qdrant): 复制索引改用服务端返回的游标,重复游标报错并给分页加跳数上限

CopyIndices 用"上一页最后一个点的 id"当游标(repository.go:987)。Qdrant 的
scroll offset 是**包含**语义 —— proto 里写的是 "Start with this ID",实测下一页
正是从该 id 本身开始返回,所以每翻一页都会把上一页的最后一个点再读一次、再写一份
到目标库。服务端本来就把下一页该用的游标放在 ScrollResponse.next_page_offset
("Use this offset for the next query")里,而客户端只在 ScrollAndOffset() 中返回它,
本函数调用的 Scroll() 恰好把它丢掉了;终止条件也只能退化成"这一页比 64 短"。

在真实 Qdrant 1.19.1 上实测:150 个源 chunk 复制出 152 个目标点,chunk-63 与
chunk-126 各被复制两次(每个页边界一个),totalCopied 同步虚高。源库只要超过一页
(>64 个 chunk)就会命中,不需要服务端有任何异常行为。

本次修改:

- 改用 q.client.ScrollAndOffset,游标直接取服务端返回的 next_page_offset;终止
  条件改为"服务端没有下一页"(next_page_offset == nil),不再用短页推断读完。
- 记录用过的游标,重复即返回 "copy indices made no progress at cursor ...",
  检测点在整页处理之前,重复的那一页不会再多写一份。
- 分页跳数上限 maxCopyPaginationHops(10000 页);触顶打印 Warn 并返回错误,
  而不是把这一轮复制当成成功。
- 修复后同一份数据在真实 Qdrant 上测得 150 个目标点、0 重复。

回归用例用假 gRPC 服务端(拦截 unary RPC,对齐 repository_test.go 的
newInterceptedQdrantClient)复刻真实服务端的游标语义:150 个源 chunk 必须各复制
一次(修复前 152 个点、2 个重复);游标恒定不动的源端必须报 no progress
(修复前 5 秒写入 3127040 个点仍未返回);一直推进但永不结束的源端在第 10000 页
被拦下。

* fix(qdrant): drop the CopyIndices page cap

10000 pages of 64 points capped a copy at 640k points and failed the
whole clone beyond that. The scroll walks every point of the source
knowledge base (a single-knowledge clone included), so a cap derived from
the chunk map would cut off small clones out of large bases too.
next_page_offset only moves forward, so the repeated-cursor check already
stops a walk that cannot finish, and the task context bounds the rest.

---------

Co-authored-by: linus-liu-web <11640009+linus-liu-web@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
2026-10-08 13:04:43 +08:00
..
retriever fix(qdrant): CopyIndices 用服务端返回的 next_page_offset 翻页,不再按页尾 id 回填 (#3882) 2026-10-08 13:04:43 +08:00
agent_share.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
agent_share_source_test.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
audit_log.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
audit_log_scope_test.go feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
chunk.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
chunk_faq_diff_test.go feat(knowledge): enhance FAQ export functionality 2026-07-23 10:50:32 +08:00
chunk_fields_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
chunk_image_assets.go feat(gallery): add an image gallery tab with schema-driven filters (#3694) 2026-09-25 12:33:59 +08:00
chunk_image_assets_test.go feat(gallery): add an image gallery tab with schema-driven filters (#3694) 2026-09-25 12:33:59 +08:00
chunk_revision_test.go fix(citations): 修复原文定位、高亮范围与 DocReader 启动失败 (#3794) 2026-09-27 20:32:26 +08:00
chunk_sqlite_test.go perf(chunk): speed up chunk ingestion and batch updates (#3369) 2026-09-17 21:38:15 +08:00
custom_agent.go fix(models): show usage details when deletion is blocked (#2975) 2026-09-02 17:20:37 +08:00
datasource_repo.go fix(datasource): validate cron schedules and persist manual-only mode (#3653) 2026-09-25 22:23:18 +08:00
datasource_repo_test.go fix(datasource): restore sync_deletions on the in-memory object after create 2026-08-21 16:37:05 +08:00
embed_channel.go feat(embed): add backend embed channels with auth and rate limiting (#1657) 2026-06-12 23:10:39 +08:00
kbshare.go fix(security): close cross-workspace access paths in shared spaces (#3408) 2026-09-18 22:00:02 +08:00
knowledge.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
knowledge_create_test.go fix(knowledge): default custom metadata on create 2026-07-28 22:22:59 +08:00
knowledge_datasource_external_id_test.go fix(datasource): perform real sync deletions scoped per data source (#2690) 2026-08-13 19:52:44 +08:00
knowledge_duplicate_test.go fix(knowledge): recover stranded deleting rows and align the KB count (#3367) 2026-09-18 00:55:44 +08:00
knowledge_finalize_test.go fix(knowledge): stop parses from getting stuck in processing or finalizing (#3696) 2026-09-25 13:26:38 +08:00
knowledge_folder_move_test.go refactor(knowledge): improve folder path handling and add tests for underscore scenarios 2026-08-04 20:45:29 +08:00
knowledge_folder_test.go refactor(knowledge): improve folder path handling and add tests for underscore scenarios 2026-08-04 20:45:29 +08:00
knowledge_list_filter_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledge_list_sort_test.go feat: add configurable sorting to document and resource lists (#3318) 2026-09-23 12:00:21 +08:00
knowledge_metadata_prefix_test.go feat(memory): add cross-session long-term memory 2026-08-13 18:00:02 +08:00
knowledge_source_schema_test.go fix(schema): expand knowledge source length 2026-06-05 15:12:25 +08:00
knowledge_span_repo.go fix(knowledge): close spans left open on terminal rows and superseded attempts (#3855) 2026-10-08 01:25:38 +08:00
knowledge_span_repo_test.go feat(knowledge): flag documents that stop making progress (#3557) 2026-09-22 21:34:08 +08:00
knowledge_tag.go fix(knowledge): address auto-tag review findings 2026-08-07 15:01:01 +08:00
knowledge_tag_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
knowledge_transfer.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledge_transfer_test.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
knowledgebase.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
knowledgebase_sqlite_test.go feat(knowledge): derive knowledge base descriptions from document profiles (#3370) 2026-09-17 22:21:55 +08:00
like_escape_sqlite_test.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
mcp_endpoint.go feat(mcp): add built-in MCP server with per-workspace endpoints (#3371) 2026-09-17 22:40:05 +08:00
mcp_metadata.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_metadata_test.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_oauth.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_oauth_test.go fix(mcp): coordinate OAuth token refresh and authorization polling 2026-07-23 15:35:24 +08:00
mcp_service.go feat(mcp)!: persist tool catalogs and add guided configuration (#3118) 2026-09-09 13:34:56 +08:00
mcp_tool_approval_repository.go feat(mcp): add per-tool enable toggle (#2989) 2026-09-05 17:19:20 +08:00
mcp_tool_approval_repository_test.go test(mcp): stabilize concurrent policy updates on SQLite 2026-09-17 17:47:24 +08:00
memory.go fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
memory_extraction.go fix(memory): preserve extraction progress and atomic memory updates (#3224) 2026-09-13 00:04:16 +08:00
memory_lifecycle.go fix(memory): preserve extraction progress and atomic memory updates (#3224) 2026-09-13 00:04:16 +08:00
memory_vector.go fix(memory): rank memories by the query, not by importance 2026-09-13 11:43:17 +08:00
message.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_artifact.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_artifact_delete_test.go feat(artifacts): let users delete generated files (#3447) 2026-09-20 13:01:46 +08:00
message_artifact_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_context_checkpoint_test.go feat(agent): size agent history by the window and calibrate token estimates (#3413) 2026-09-18 22:51:31 +08:00
message_fork_boundary_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_owner_scope_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_request_id_test.go feat(session): fork conversations from a historical user message (#3304) 2026-09-17 14:20:18 +08:00
message_rewind_boundary_test.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
message_search_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
message_suggestion.go feat(session): in-place rewind with workspace checkpoint reset (#3469) 2026-09-20 21:01:00 +08:00
model.go fix(repository): correct query syntax to get correctly built-in models 2026-05-21 16:11:22 +08:00
model_catalog_config.go feat(models): add system admin model catalog console (#3662) 2026-09-24 16:31:45 +08:00
model_usage.go fix(models): protect auto-tag model references 2026-09-14 19:29:19 +08:00
model_usage_test.go fix(models): protect auto-tag model references 2026-09-14 19:29:19 +08:00
organization.go fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
organization_member_limit_test.go fix(security): close the low-severity shared-space findings (#3416) 2026-09-18 22:58:46 +08:00
resource.go feat(agent): preview skill artifacts and verify installs without running them (#2823) 2026-08-27 10:30:32 +08:00
resource_references.go refactor(resource-access): unify scoped authorization and transfer recovery (#3097) 2026-09-08 15:36:33 +08:00
session.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
session_fork_test.go feat(artifacts): artifacts library page backed by a message_artifacts table (#3384) 2026-09-18 12:10:19 +08:00
session_test.go feat(sandbox): expose tenant skill management over HTTP and in the console (#2789) 2026-08-25 11:28:12 +08:00
storagebackend.go feat(storage): add multi-instance storage backends per workspace 2026-07-15 14:54:39 +08:00
system_setting.go feat(system): consolidate system admin and settings into one Settings panel 2026-05-26 21:13:56 +08:00
tag.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
task_queue.go fix(wiki): bound outstanding follow-up tasks 2026-09-28 22:02:07 +08:00
task_queue_test.go fix(wiki): bound outstanding follow-up tasks 2026-09-28 22:02:07 +08:00
temporary_document.go feat(chat): add session-scoped temporary attachments with async parsing 2026-07-16 17:41:27 +08:00
tenant.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
tenant_api_key.go feat: support updating API key configuration 2026-08-12 11:43:03 +08:00
tenant_api_key_test.go feat: support updating API key configuration 2026-08-12 11:43:03 +08:00
tenant_disabled_shared_agent.go feat(agent): enhance agent and skill management features 2026-02-09 11:40:25 +08:00
tenant_invitation.go feat(rbac): add multi-use share-link invitations for invite_only mode (#1504) 2026-05-27 19:42:01 +08:00
tenant_member.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
tenant_sandbox_config.go feat(sandbox): expose tenant skill management over HTTP and in the console (#2789) 2026-08-25 11:28:12 +08:00
tenant_sandbox_config_test.go feat(sandbox) (#2301) 2026-08-10 21:28:20 +08:00
tenant_skill.go fix(skills): upgrade outdated sandbox skills without rolling back the catalog or taking them offline (#3406) 2026-09-18 20:27:43 +08:00
tenant_skill_test.go feat(skills): store skills in a tenant catalog and install per sandbox 2026-08-28 15:17:37 +08:00
tenant_test.go feat: add scoped tenant api keys 2026-07-07 12:00:13 +08:00
user.go feat(sandbox): add an interactive terminal and files tab beside chat (#3110) 2026-09-08 17:43:15 +08:00
user_resource_favorite.go feat(favorites): implement user favorites and recents functionality 2026-05-18 17:28:58 +08:00
user_tenantless_test.go feat(auth): support tenantless provisioning and gated self-service workspace creation 2026-07-13 18:13:51 +08:00
vectorstore.go feat: add VectorStore service layer with types and connection testing 2026-04-13 22:57:17 +08:00
web_search_provider.go feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00
wiki_page.go fix(repository): 关键词 LIKE/ILIKE 检索补显式 ESCAPE,修复 Lite 上静默漏结果 2026-09-29 01:31:37 +08:00
wiki_page_test.go feat(wiki): 增加跨知识库 Wiki 搜索 POST /wiki-search (#3535) 2026-09-25 22:22:58 +08:00