refactor(resource-access): unify scoped authorization and transfer recovery (#3097)

* refactor(resource-access): unify scoped authorization and transfer recovery

* fix(resource-access): close reparse and transfer review gaps

* fix: enforce explicit file bindings and recover move cleanup

* fix: align mutation admission and cleanup scope checks
main
lyingbug 2026-09-08 15:36:33 +08:00 committed by GitHub
parent e18cf138a3
commit 3d81b977f1
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
158 changed files with 12746 additions and 3155 deletions

View File

@ -0,0 +1,294 @@
# Resource access boundaries
`access` contains shared permission resolution without depending on Gin or the
application service implementations. HTTP adapters keep parameter parsing, status
codes, RBAC rollout behavior, and response projection at their existing boundaries.
| Entry point | Shared rule | Additional boundary |
| --- | --- | --- |
| Resource mutations with URL/body KB selectors | `CheckOwnershipOrRole`: creator or sufficient tenant role | API-key capabilities, KB scope and tenant visibility remain separate checks |
| KB/document detail and content routes | `ResolveKB`: own tenant, organization share, then read-only shared agent | Tenant role/ownership guards and API-key capabilities |
| Shared documents, chunks, search targets and suggestions | `KBSharePermissions`: organization grants only | Existing user requirements and caller-specific filtering/error behavior |
| FAQ and tag service reads | `resolveKBReadTenant`: execution tenant or organization share | Direct cross-tenant calls require a user; no implicit agent fallback |
| KB list, document search and batch restoration with `agent_id` | Shared-agent lookup, then `SharedAgentKBScope` | API-key intersection; dynamic list/search selections also apply capability filtering |
| Wiki fixer source tenant | Organization Editor permission | Built-in fixer only; exactly one KB |
| Message file shared-KB fallback | Organization Viewer permission | Persisted message reference, exact resource handle, KB/resource owner match |
| KB file proxy | `ResolveKBFile`: exact Viewer grant and current explicit KB binding | Owner tenant and exports namespace; original uploads retain their separate download permission |
| Message file proxy | `ResolveMessageFile`: session ownership and exact persisted output reference | Current shared-agent or organization-KB permission for source-owned resources |
| Message artifact download | `ResolveMessageArtifact`: session ownership and persisted artifact index | Session-owned output remains accessible; source-owned output rechecks current sharing |
| FAQ and tag mutations | `RequireKBWrite`: exact Editor operation grant | API-key ingest capability and KB scope; tenant-role/creator checks remain at the admission boundary |
## Ownership and write roles
URL middleware and body-based handlers use the same lazy ownership decision.
The middleware adapter resolves identity, RBAC rollout configuration and the
configuration-gated superuser flag. API-key principals, sufficient roles,
cross-tenant superusers and disabled enforcement skip the creator lookup.
API keys still require their route capabilities and allowed KB scope.
When a lookup is needed, it must check tenant visibility before returning the
creator. An empty creator never matches an empty user. Middleware passes missing
resources to the handler, reports lookup failures as 503 and audits only actual
policy denials. Body-based handlers retain their existing 404/403/500 mapping.
KB, initialization, Wiki and body-based KB lookups share the same tenant check.
## Caller and resource tenant
`types.Caller` holds the authenticated tenant, user and tenant role. Auth records
it once; `KBRequest.Caller` and `KBAccess.Caller` use that same identity. The
legacy `TenantIDContextKey` continues to scope repository/model execution, so
existing storage APIs do not need a coordinated signature migration.
Use `types.WithExecutionTenant` to switch execution tenants. It captures the
caller first, even when identity is missing, and grants no access by itself.
Use `KBAccess.Context` after authorization to carry both the exact KB grant and
its execution tenant, or `WithGrant` to carry the grant without switching yet.
Grants bind the KB ID, owner tenant, caller and resolved permission; changing
execution tenant cannot turn a Viewer grant into Editor or expose another KB.
HTTP adapters retain Gin's caller keys for existing handlers.
Service reads use `KBPermissions` to check caller ownership, exact upstream
grants, then organization shares for the original caller. This check also applies
to documents/chunks returned by the initial tenant-scoped batch query, not just
rows found during cross-tenant expansion. Search preflight, result enrichment,
FAQ/tag reads, search targets and suggestion scopes follow the same boundary.
General document search enumerates the caller's own/shared KBs; its execution
tenant is not a new identity.
After authorizing a shared agent, `WithSharedAgent` carries its configured,
source-tenant-bound read scope into the pipeline. An arbitrary execution switch
cannot substitute for that grant. `logger.CloneContext` preserves the caller and
resource grants for detached work within the operation. Context grants contain
immutable snapshots; deriving a context does not mutate a sibling search branch.
They are not persisted as authority for subsequent requests. API-key scope is
reapplied when consuming any grant.
## Organization shares
`KBSharePermissions` fixes the caller tenant and role for one operation and caches
the granular decision per KB. Reading many documents/chunks from one KB therefore
queries its membership once. Both denied decisions and errors are cached within
that operation. The next request creates a new resolver and observes revocations.
The resolver is not a process-wide cache and is not shared between goroutines.
The share service remains responsible for the role cap across share permission,
organization membership and caller tenant role. The resolver returns lookup errors
so strict search preflight can report an infrastructure failure while best-effort
result expansion can omit inaccessible rows. `ResolveKB` retains its independent
read-only agent fallback when organization resolution cannot grant access.
## Shared-agent KB scope
A scope is created only after looking up an accessible shared agent. `all` means
all KBs in that agent's tenant; `selected` means its explicit nonempty ID set.
`none`, unknown modes and empty selections authorize no KBs. A nil/empty ID slice
never implicitly grants an entire tenant. Specifying an agent does not permit
falling back to a different shared agent.
These rules do not replace message/session ownership, persisted file-reference
validation, resource binding, Agent tool search-target restrictions, or embed/IM
capability checks. Raw tenant files and message files have different authorization
contracts and must not be admitted by a generic KB grant alone.
## FAQ and tag writes
FAQ/tag service mutations consume an explicit Editor grant; setting an execution
tenant or supplying a KB ID is insufficient. `ResolveKB` keeps the effective
permission used for response projection, but caps its context grant to the
requested operation. Thus an owner read does not mint a reusable write grant.
HTTP routes retain their existing role/creator admission checks. API-key ingest
capability and KB scope are reapplied when consuming a write grant.
Trusted FAQ-import, data-source sync and automatic-tagging workers use
`WithKBTaskWrite` after loading and validating their target KB and child-resource
bindings. This represents continuation of an admitted job, not renewed user
authorization on every retry. It carries exactly one KB, preserves the caller
or its absence, survives in-operation context cloning, and grants no tenant-wide
role. Services use the KB owner's TenantInfo for model/index configuration.
Batch FAQ updates resolve all source/destination tags, entries and exclusions
before mutation. Batch deletion also resolves all parent documents first.
Tag-only batch updates use the same plan. Tag groups execute in ascending seq_id
order, then explicit entry updates override only their specified fields.
Import retains per-entry content-validation results, but out-of-scope resource
references are rejected before admission and checked again in the worker.
This preflight prevents partial writes caused by invalid resource selections;
it does not add a transaction across the database, task queue and search index.
## File authorization and transport
File resolvers return request-local `FileAccess` locators before any storage
reader is opened. The KB catalog accepts only explicit resource bindings to live
documents in the exact live KB. Registered physical aliases resolve to the same
resource ID; chunk/wiki text is never authorization evidence. Historical files
without explicit bindings fail closed and need a trusted migration. A same-tenant
path or handle prefix cannot replace a KB binding. Message evidence comes from
persisted content, references, images, artifacts and tool results; tool arguments
do not prove that the message returned a file. The organization-shared KB
fallback also checks the catalog for an independent live KB/file binding;
retrieval text and a currently shared KB ID alone are insufficient.
Source-owned shared-agent files also require the current live agent share.
KB sources need an explicit live KB binding allowed by the agent's current
`all`/`selected` selection and the caller's API-key KB scope; `none`, unknown
modes, removed KB selections and deleted sources fail closed. Independently
generated artifacts remain accessible when explicitly bound to the exact
authorized message as an artifact. Mentioning a file in message text or the
artifact list does not create that binding. Caller-owned artifacts retain their
session ownership contract.
The storage adapter uses the authorized owner and backend. The shared
`filetransport` package owns response headers, safe inline/download disposition,
streaming and reader closure. Seekable readers use standard HTTP Range and HEAD
handling; streaming-only backends return full content with `Accept-Ranges: none`
without buffering the object (RFC 9110 sections 14.2 and 14.3). Permission-controlled responses use `private, no-store` so a later
request rechecks current access. Tenant-wide and signed-token file routes keep
their own authorization and caching contracts.
## Document and chunk writes
Document metadata, manual content, reparse, document tags/folders, image edits, chunk mutations
and generated-question edits consume explicit Editor grants after loading the
persisted document/KB binding. Full chunk objects cannot reparent existing rows.
Batch inputs are deduplicated and fully validated, including missing IDs and
each KB's grant, before writes. Text edits also check the parent/image-child
relationships that they will modify. Body-based HTTP routes explicitly resolve
an Editor operation; a previously cached owner read is not a write admission.
Document deletion shares a preflight plan and executor for single and batch
calls. The plan validates every selection and KB, loads owner TenantInfo and
routing metadata, and captures image references before marking rows as deleting.
Index/graph failures leave the document and original file available for retry.
Wiki deletion collects all chunk types before deleting chunks, so parent, image
and summary citations are removed along with text citations.
Physical file cleanup remains best effort after database deletion; this does not
introduce a transaction across database, indexes and storage or durable blob GC.
New queued deletion payloads pin `knowledge_base_id` as well as tenant and exact
document IDs. Workers consume a private cleanup scope bound to those references,
not general KB write permission; already deleted IDs are successful no-ops.
A moved document rejects the batch before side effects. Pre-upgrade payloads that
omit the KB are accepted only when remaining rows resolve to one same-tenant KB;
they cannot reconstruct the KB that existed at enqueue time. Failure callbacks
skip rejected scopes and condition status updates on tenant, KB and deleting
state. Scope/identity errors are not retried; infrastructure failures are.
Single/batch delete, clear-content and batch-reparse admission reject already
moving documents with HTTP 409 before enqueue. Workers recheck current state;
a deletion conflict is terminal for that admitted task, so retries cannot
follow a document into a different KB. This does not serialize admission with
a concurrent move. Synchronous metadata/image/chunk/summary handlers preserve
application 403/409 errors, including wrapped errors.
Batch reparse and batch document tags resolve an Editor operation and apply the
same creator-or-Admin admission policy as single-document writes, respecting
RBAC rollout/API-key behavior. Batch reparse payloads likewise carry the admitted
KB ID. Workers validate the whole batch before any submission and use exactly one `WithKBTaskWrite` grant;
each reparse reload consumes that grant and rejects a document still moving.
Missing/mixed/moved selections fail without submitting earlier batch members.
Legacy reparse payloads can reconstruct only a present, single-KB binding.
Session-history, temporary-KB and clone cleanup use exact server-derived
references from their already admitted operations. These cleanup contexts cannot
be used for document/chunk updates or additional IDs. Parsing and enrichment
workers use repositories for their internal persistence and rollback; they must
keep their existing task ownership checks, abort checks and resource bindings.
Repository access is not an incoming-request authorization API.
## Clone and move operations
`WithKBTransfer` admits the exact source/target pair after HTTP role/creator
checks. Clone consumes source Viewer, target Editor and the API-key
`manage_kbs` capability; move consumes Editor on both sides and `ingest`.
Both operations remain within one tenant. Creating a clone reserves its target
ID before enqueue and carries its creator into the worker. The ID of a new
resource is not checked against an existing-resource API-key allow-list.
Workers reload both KBs, validate their owners and consume a private pair grant.
Document/FAQ plans validate all selected resources, chunks, parents and tags
before destructive work. Lifecycle reads include every chunk type and indexing
state. A document's `_knowledge_transfer` metadata records task identity and
progress; retries reuse the reserved target, replace incomplete clones, skip
completed vector moves and retry outstanding wiki cleanup/parser enqueue.
This field is internal operation state, not caller-supplied custom metadata.
For `reparse`, transfer `done` means target binding and parser admission are
complete, not that parsing has finished. `ParseStatus` remains the parser's
state. `reparse_pending` already admits the target parser; extending the
`moving` write guard to that phase would also block legitimate worker writes.
Locking all writes until parsing finishes is a separate processing/concurrency
contract, including ordinary reparses, and is not provided by this transfer marker.
Failed clone chunk/index writes attempt rollback after conditionally claiming
the original destination. Rollback failures are returned, references retained
when cleanup is uncertain, and later retries replace the failed copy. Accounting
is committed only after the clone succeeds. FAQ containers do not inherit the
source document's transfer marker.
Source wiki cleanup starts only after the document ownership CAS succeeds.
The transfer checkpoint preserves source chunk IDs and summary before reparse
removes them. Retract page lists are persisted before page source refs are removed, so queue
failures retain retry evidence. Cleanup errors propagate; redelivery resumes cleanup after a
committed move without repeating vector/chunk relocation. These retries do not
provide a distributed transaction or recovery after retries are exhausted.
Vector reuse relocates KB metadata while preserving physical vector/chunk IDs.
Weaviate and Milvus drain the first source-filtered page until a follow-up query
is empty, avoiding offset limits/skipped rows. Repeated IDs or partial failures
return errors and leave the document checkpoint pending for retry.
OpenSearch selects the entire source KB/document pair, even without DB chunks,
and rejects version conflicts, timeouts and partial failures before DB relocation.
OpenSearch and Elasticsearch 7/8 require present non-negative `total`/`updated`
counts with `updated == total`; their move scripts do not skip/delete documents.
A zero/zero result allows a retry after metadata already reached the target.
Reparse cleanup loads and validates the source KB before touching any backend,
reusing its storage binding throughout; a failed/missing lookup cannot fall
back to the tenant default store.
Vector reuse never copies vectors and then deletes by the same document ID. Source and
target must use the same vector store for reuse, and the same concrete file
storage instance for either mode. Doris ANN tables cannot safely replace rows
without a delete/insert gap, so their moves require `reparse`. Clone still
requires compatible embedding models and vector stores; a move also retains
the existing same-model requirement. This is not a cross-storage migration API.
Conditional document checkpoints bind tenant, KB, status and the stored update
time. They keep storage accounting in the same database transaction and read
back timestamp precision. Delete stops when its checkpoint no longer matches.
Ordinary document/chunk writes reject a document in the `moving` phase. Parser
admission and failure callbacks verify the task's persisted tenant/KB binding;
table-summary failure cleanup also stops if its document changed.
## Follow-up work outside this refactor
These are separate follow-ups, not capabilities provided by this package:
1. **Complete worker contracts and concurrency control.** Audit enrichment,
table-summary, multimodal, wiki and indexing workers end to end. Standardize
payload resource bindings, processing-attempt IDs and conditional writes,
including operations admitted before a concurrent move. Do not replace
these checks with an execution-tenant switch or a blanket task write grant.
2. **Durable recovery across systems.** Add a transactional outbox and explicit
recovery/cancellation for exhausted transfer tasks. Current retries use the
same admitted task identity; a new move request is not a resume API. Database,
queue, vector, graph and blob operations are not one atomic transaction.
Physical file cleanup remains best effort and needs durable garbage collection.
3. **External-backend integration coverage.** Exercise clone/move, partial
backend failures, retries, pagination and search/edit behavior against live
PostgreSQL, Elasticsearch 7/8, OpenSearch, Qdrant, Milvus, Weaviate,
TencentVectorDB and Doris. Local SQLite vector tests and backend compilation
do not establish this coverage. Doris ANN vector reuse remains unsupported.
4. **Finish service-layer migration.** Apply explicit operation grants to the
remaining KB configuration/duplicate/delete, wiki and resource-administration
service methods, preserving their existing route admission policies. Keep
user/organization membership, embed, IM and signed-resource contracts separate.
5. **Trusted legacy file migration.** Backfill historical file bindings from
verified provenance, not arbitrary chunk/wiki mentions. Unbound historical
files now fail closed; request-path text fallback has been removed. Migrate
old queued payloads and transfer records without captured source wiki refs.
A legacy delete task without a KB ID can reconstruct only its present
unambiguous binding, not its original enqueue-time binding.
No public request schema or database column migration is required by this
refactor. Internal service interfaces require operation contexts, queued payloads
add optional scope fields, and malformed/out-of-scope batch selections now fail
before mutation instead of being silently skipped. Callers must also handle a
409 for document edits/deletes while a move is in progress.

View File

@ -0,0 +1,94 @@
package access
import (
"context"
"github.com/Tencent/WeKnora/internal/types"
)
// Context grants contain values, not mutable KB objects. Deriving a context
// copies the grant slice so parallel search branches cannot widen each other.
type kbGrant struct {
caller types.Caller
kbID string
tenantID uint64
permission types.OrgMemberRole
task bool
}
type agentGrant struct {
caller types.Caller
scope types.SharedAgentKBScope
}
// WithSharedAgent scopes execution after the caller has authorized this exact
// agent. Only its configured KBs receive read access; other source-tenant KBs
// and the source tenant's organization memberships are not inherited.
func WithSharedAgent(ctx context.Context, agent *types.CustomAgent) context.Context {
grant := agentGrant{caller: types.CallerFromContext(ctx), scope: types.NewSharedAgentKBScope(agent)}
ctx = context.WithValue(ctx, types.SharedAgentGrantContextKey, grant)
if agent == nil {
return ctx
}
return types.WithExecutionTenant(ctx, agent.TenantID)
}
// HasKBGrant checks only previously resolved resource access. It never infers
// ownership from the execution tenant, and always reapplies API-key scope.
func HasKBGrant(ctx context.Context, kbID string, tenantID uint64, required types.OrgMemberRole) bool {
if !required.IsValid() || types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID) != nil {
return false
}
caller := types.CallerFromContext(ctx)
if kbID == "" || tenantID == 0 {
return false
}
grants, _ := ctx.Value(types.KBGrantsContextKey).([]kbGrant)
for _, grant := range grants {
if (caller.TenantID != 0 || grant.task) && grant.caller == caller && grant.kbID == kbID &&
grant.tenantID == tenantID &&
grant.permission.HasPermission(required) {
return true
}
}
grant, ok := ctx.Value(types.SharedAgentGrantContextKey).(agentGrant)
return ok && caller.TenantID != 0 && required == types.OrgRoleViewer && grant.caller == caller &&
grant.scope.Allows(kbID, tenantID)
}
// KBPermissions combines caller ownership, exact context grants and organization
// shares for one service operation. Pass nil shares when that entry point does
// not permit organization expansion (for example, a caller without a user).
type KBPermissions struct {
ctx context.Context
caller types.Caller
shares *KBSharePermissions
}
// NewKBPermissions resolves reads for one caller and operation.
func NewKBPermissions(ctx context.Context, shares KBShareLookup) *KBPermissions {
caller := types.CallerFromContext(ctx)
return &KBPermissions{
ctx: ctx,
caller: caller,
shares: NewKBSharePermissions(ctx, shares, caller.TenantID, caller.Role),
}
}
// Check combines caller ownership, exact grants and organization sharing.
func (p *KBPermissions) Check(kbID string, ownerTenantID uint64, required types.OrgMemberRole) (bool, error) {
if kbID == "" || ownerTenantID == 0 || !required.IsValid() {
return false, nil
}
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(p.ctx, kbID); err != nil {
return false, err
}
if (p.caller.TenantID == ownerTenantID && required == types.OrgRoleViewer) ||
HasKBGrant(p.ctx, kbID, ownerTenantID, required) {
return true, nil
}
if p.caller.TenantID == 0 || p.caller.TenantID == ownerTenantID {
return false, nil
}
return p.shares.Check(kbID, required)
}

View File

@ -0,0 +1,140 @@
package access
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func callerContext() context.Context {
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
ctx = context.WithValue(ctx, types.UserIDContextKey, "user")
return context.WithValue(ctx, types.TenantRoleContextKey, types.TenantRoleContributor)
}
func TestKBGrantSurvivesExecutionSwitchAndDetachWithoutWidening(t *testing.T) {
ctx := callerContext()
kb := &types.KnowledgeBase{ID: "shared", TenantID: 2}
grant, err := ResolveKB(
ctx,
KBRequest{Caller: types.CallerFromContext(ctx)},
kb,
types.OrgRoleViewer,
&shareLookup{permission: types.OrgRoleViewer},
nil,
)
require.NoError(t, err)
ctx = grant.Context(ctx)
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx))
ctx = logger.CloneContext(types.WithExecutionTenant(ctx, 3))
require.Equal(
t,
types.Caller{TenantID: 1, UserID: "user", Role: types.TenantRoleContributor},
types.CallerFromContext(ctx),
)
require.True(t, HasKBGrant(ctx, "shared", 2, types.OrgRoleViewer))
// A later mutation of the service model cannot mutate the captured grant.
kb.ID = "private"
require.False(t, HasKBGrant(ctx, "private", 2, types.OrgRoleViewer))
require.False(t, HasKBGrant(ctx, "shared", 3, types.OrgRoleViewer))
require.False(t, HasKBGrant(ctx, "shared", 2, types.OrgRoleEditor))
for _, caller := range []types.Caller{
{TenantID: 9, UserID: "user", Role: types.TenantRoleContributor},
{TenantID: 1, UserID: "other", Role: types.TenantRoleContributor},
{TenantID: 1, UserID: "user", Role: types.TenantRoleViewer},
} {
otherCtx := types.WithCaller(ctx, caller)
require.False(t, HasKBGrant(otherCtx, "shared", 2, types.OrgRoleViewer))
require.Equal(
t,
caller,
types.CallerFromContext(grant.Context(otherCtx)),
"reusing a grant cannot impersonate its caller",
)
require.Equal(
t,
uint64(3),
types.MustTenantIDFromContext(grant.Context(otherCtx)),
"a caller mismatch must not change execution scope",
)
}
narrowed := types.WithTenantAPIKeyScope(
ctx,
types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"another"}},
)
require.False(t, HasKBGrant(narrowed, "shared", 2, types.OrgRoleViewer))
}
func TestExecutionTenantDoesNotGrantOwnershipOrChangeShareIdentity(t *testing.T) {
ctx := types.WithExecutionTenant(callerContext(), 2)
shares := &shareLookup{}
allowed, err := NewKBPermissions(ctx, shares).Check("private", 2, types.OrgRoleViewer)
require.NoError(t, err)
require.False(t, allowed)
require.Equal(t, uint64(1), shares.caller)
require.Equal(t, types.TenantRoleContributor, shares.role)
ctx = types.WithExecutionTenant(context.Background(), 2)
allowed, err = NewKBPermissions(ctx, nil).Check("private", 2, types.OrgRoleViewer)
require.NoError(t, err)
require.False(t, allowed, "an absent caller must not become the execution tenant")
}
func TestSharedAgentGrantPreservesConfiguredScopeAcrossDetach(t *testing.T) {
for _, mode := range []string{"all", "selected", "none", "unknown"} {
t.Run(mode, func(t *testing.T) {
agent := &types.CustomAgent{
TenantID: 2,
Config: types.CustomAgentConfig{KBSelectionMode: mode, KnowledgeBases: []string{"selected"}},
}
ctx := logger.CloneContext(WithSharedAgent(callerContext(), agent))
require.Equal(t, uint64(1), types.CallerFromContext(ctx).TenantID)
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx))
require.Equal(t, mode == "all" || mode == "selected", HasKBGrant(ctx, "selected", 2, types.OrgRoleViewer))
require.Equal(t, mode == "all", HasKBGrant(ctx, "other", 2, types.OrgRoleViewer))
require.False(t, HasKBGrant(ctx, "selected", 3, types.OrgRoleViewer))
require.False(t, HasKBGrant(ctx, "selected", 2, types.OrgRoleEditor))
})
}
}
func TestKBGrantBranchesRemainIndependent(t *testing.T) {
base := callerContext()
grant := &KBAccess{Caller: types.CallerFromContext(base), EffectiveTenantID: 2, Permission: types.OrgRoleViewer}
grant.KnowledgeBase = &types.KnowledgeBase{ID: "first", TenantID: 2}
first := grant.WithGrant(base)
grant.KnowledgeBase = &types.KnowledgeBase{ID: "second", TenantID: 2}
second := grant.WithGrant(base)
require.True(t, HasKBGrant(first, "first", 2, types.OrgRoleViewer))
require.False(t, HasKBGrant(first, "second", 2, types.OrgRoleViewer))
require.True(t, HasKBGrant(second, "second", 2, types.OrgRoleViewer))
require.False(t, HasKBGrant(second, "first", 2, types.OrgRoleViewer))
}
func TestKBPermissionsOwnerShortcutOnlyGrantsRead(t *testing.T) {
roles := []types.TenantRole{types.TenantRoleViewer, types.TenantRoleContributor, types.TenantRoleAdmin}
for _, role := range roles {
ctx := context.WithValue(callerContext(), types.TenantRoleContextKey, role)
permissions := NewKBPermissions(ctx, nil)
for _, required := range []types.OrgMemberRole{types.OrgRoleViewer, types.OrgRoleEditor, types.OrgRoleAdmin} {
allowed, err := permissions.Check("kb", 1, required)
require.NoError(t, err)
require.Equal(t, required == types.OrgRoleViewer, allowed)
}
grant := &KBAccess{
Caller: types.CallerFromContext(ctx),
KnowledgeBase: &types.KnowledgeBase{ID: "kb", TenantID: 1},
EffectiveTenantID: 1,
Permission: types.OrgRoleEditor,
}
permissions = NewKBPermissions(grant.Context(ctx), nil)
allowed, err := permissions.Check("kb", 1, types.OrgRoleEditor)
require.NoError(t, err)
require.True(t, allowed)
allowed, err = permissions.Check("kb", 1, types.OrgRoleAdmin)
require.NoError(t, err)
require.False(t, allowed)
}
}

View File

@ -0,0 +1,256 @@
package access
import (
"context"
"encoding/json"
"fmt"
"strings"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
secutils "github.com/Tencent/WeKnora/internal/utils"
)
// FileAccess is a request-local result, produced before storage is opened.
// Physical locators are not a substitute for KB/message authorization.
type FileAccess struct {
OwnerTenantID uint64
Path string
Filename string
StorageBackendID string
}
// FileCatalog resolves logical references to their stored owner and locator.
type FileCatalog interface {
ResolvePath(context.Context, string) (string, *types.StoredResource, error)
}
func resolveFile(
ctx context.Context,
catalog FileCatalog,
reference string,
) (FileAccess, *types.StoredResource, error) {
file := FileAccess{Path: reference, Filename: reference}
if catalog == nil {
if _, resource := types.ParseResourcePath(reference); resource {
return file, nil, ErrNotFound
}
return file, nil, nil
}
path, resource, err := catalog.ResolvePath(ctx, reference)
if err != nil {
return file, nil, ErrNotFound
}
file.Path = path
file.Filename = path
if resource != nil {
file.OwnerTenantID = resource.TenantID
file.StorageBackendID = resource.StorageBackendID
if strings.TrimSpace(resource.OriginalName) != "" {
file.Filename = resource.OriginalName
}
}
return file, resource, nil
}
// ResolveKBFile requires the route's exact KB grant and an independent live
// explicit binding check. Rewriting the execution tenant is insufficient.
func ResolveKBFile(
ctx context.Context,
grant *KBAccess,
kbID, reference string,
catalog FileCatalog,
bindings interfaces.KBResourceLookup,
) (FileAccess, error) {
var empty FileAccess
if grant == nil || grant.KnowledgeBase == nil || grant.KnowledgeBase.ID != kbID ||
grant.Caller != types.CallerFromContext(ctx) || !grant.Permission.HasPermission(types.OrgRoleViewer) {
return empty, ErrForbidden
}
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID); err != nil {
return empty, err
}
file, resource, err := resolveFile(ctx, catalog, reference)
if err != nil {
return empty, err
}
owner := grant.EffectiveTenantID
if owner == 0 || (resource != nil && resource.TenantID != owner) {
return empty, ErrForbidden
}
if err := secutils.ValidateKBScopedStoragePath(file.Path, owner); err != nil {
return empty, ErrForbidden
}
if bindings == nil {
return empty, ErrForbidden
}
bound, err := bindings.IsReferencedByKnowledgeBase(ctx, owner, kbID, reference)
if err != nil {
return empty, fmt.Errorf("check KB file binding: %w", err)
}
if !bound {
return empty, ErrForbidden
}
file.OwnerTenantID = owner
return file, nil
}
// MessageReferencesFile examines only persisted rendering/output fields. Tool
// arguments and request metadata are not evidence of a returned file.
func MessageReferencesFile(message *types.Message, reference string) bool {
if message == nil {
return false
}
if types.ContainsStorageReference(message.Content, reference) {
return true
}
for _, artifact := range message.Artifacts {
if artifact.URL == reference {
return true
}
}
for _, value := range []interface{}{message.KnowledgeReferences, message.Images} {
data, _ := json.Marshal(value)
if types.ContainsStorageReference(string(data), reference) {
return true
}
}
for _, step := range message.AgentSteps {
for _, call := range step.ToolCalls {
if call.Result == nil {
continue
}
data, _ := json.Marshal(call.Result)
if types.ContainsStorageReference(string(data), reference) {
return true
}
}
}
return false
}
// ResolveMessageFile loads the session-authorized message using the original
// caller and rechecks its current sharing relationships on every request.
func ResolveMessageFile(ctx context.Context, sessionID, messageID, reference string, messages MessageFileLookup,
agents SharedAgentFileLookup, catalog FileCatalog, kbShares MessageKBShareAuthorizer,
) (FileAccess, error) {
caller := types.CallerFromContext(ctx)
if caller.TenantID == 0 {
return FileAccess{}, ErrUnauthorized
}
if messages == nil {
return FileAccess{}, ErrNotFound
}
ctx = types.WithExecutionTenant(ctx, caller.TenantID)
message, err := messages.GetMessage(ctx, sessionID, messageID)
if err != nil || message == nil {
return FileAccess{}, ErrNotFound
}
return AuthorizeMessageFile(ctx, message, reference, agents, catalog, kbShares)
}
// AuthorizeMessageFile also serves index-based artifact downloads after their
// session/message lookup. It never accepts an unchecked client message.
func AuthorizeMessageFile(ctx context.Context, message *types.Message, reference string, agents SharedAgentFileLookup,
catalog FileCatalog, kbShares MessageKBShareAuthorizer,
) (FileAccess, error) {
caller := types.CallerFromContext(ctx)
if caller.TenantID == 0 {
return FileAccess{}, ErrUnauthorized
}
if !MessageReferencesFile(message, reference) {
return FileAccess{}, ErrForbidden
}
file, resource, err := resolveFile(ctx, catalog, reference)
if err != nil {
return FileAccess{}, err
}
owner := message.AgentTenantID
if resource != nil {
owner = resource.TenantID
}
if owner == 0 {
return FileAccess{}, ErrForbidden
}
if message.Role == "user" && owner != caller.TenantID {
return FileAccess{}, ErrForbidden
}
if kbShares.Bindings == nil {
kbShares.Bindings, _ = catalog.(interfaces.KBResourceLookup)
}
kbAuthorized := false
if resource != nil && message.AgentTenantID != 0 && message.AgentTenantID != owner {
kbAuthorized = kbShares.resourceAccessibleViaSharedKB(ctx, message, resource, caller.TenantID, caller.Role)
if !kbAuthorized {
return FileAccess{}, ErrForbidden
}
}
if owner != caller.TenantID && !kbAuthorized {
if message.AgentTenantID == 0 {
kbAuthorized = kbShares.resourceAccessibleViaSharedKB(ctx, message, resource, caller.TenantID, caller.Role)
}
if !kbAuthorized {
if message.AgentID == "" || agents == nil {
return FileAccess{}, ErrForbidden
}
agent, err := agents.GetSharedAgentForTenant(ctx, caller.TenantID, caller.Role, message.AgentID, owner)
if err != nil || agent == nil || agent.TenantID != owner {
return FileAccess{}, ErrForbidden
}
bindings, ok := catalog.(interfaces.MessageFileBindingLookup)
if !ok {
return FileAccess{}, ErrForbidden
}
origins, err := bindings.GetMessageFileBindings(ctx, owner, reference, message.ID)
if err != nil || origins == nil {
return FileAccess{}, ErrForbidden
}
allowed := origins.MessageArtifact
scope := types.NewSharedAgentKBScope(agent)
for _, kbID := range origins.KnowledgeBaseIDs {
if scope.Allows(kbID, owner) && types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID) == nil {
allowed = true
}
}
if !allowed {
return FileAccess{}, ErrForbidden
}
}
}
if resource == nil {
if err := secutils.ValidateStoragePathTenant(file.Path, owner); err != nil {
return FileAccess{}, ErrForbidden
}
}
file.OwnerTenantID = owner
return file, nil
}
// ResolveMessageArtifact selects a persisted artifact after session ownership
// was checked. Session-owned output stays downloadable independently of the
// agent; source-owned output still requires its current sharing permission.
func ResolveMessageArtifact(ctx context.Context, message *types.Message, index int, agents SharedAgentFileLookup,
catalog FileCatalog, kbShares MessageKBShareAuthorizer,
) (FileAccess, error) {
caller := types.CallerFromContext(ctx)
if caller.TenantID == 0 {
return FileAccess{}, ErrUnauthorized
}
if message == nil || index < 0 || index >= len(message.Artifacts) {
return FileAccess{}, ErrNotFound
}
artifact := message.Artifacts[index]
file, resource, err := resolveFile(ctx, catalog, artifact.URL)
if err != nil {
return FileAccess{}, err
}
if (resource != nil && resource.TenantID == caller.TenantID) ||
(resource == nil && secutils.ValidateStoragePathTenant(file.Path, caller.TenantID) == nil) {
file.OwnerTenantID = caller.TenantID
file.Filename = artifact.FileName
return file, nil
}
file, err = AuthorizeMessageFile(ctx, message, artifact.URL, agents, catalog, kbShares)
file.Filename = artifact.FileName
return file, err
}

View File

@ -0,0 +1,162 @@
package access
import (
"context"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
type fileCatalog struct{ resource *types.StoredResource }
func (c fileCatalog) ResolvePath(_ context.Context, ref string) (string, *types.StoredResource, error) {
if c.resource == nil {
return ref, nil, nil
}
return c.resource.PhysicalPath, c.resource, nil
}
type fileBinding struct {
allowed bool
err error
kb string
tenant uint64
}
func (b *fileBinding) IsReferencedByKnowledgeBase(_ context.Context, tenant uint64, kb, _ string) (bool, error) {
b.kb, b.tenant = kb, tenant
return b.allowed, b.err
}
type fileMessages struct {
message *types.Message
tenant uint64
}
func (m *fileMessages) GetMessage(ctx context.Context, _, _ string) (*types.Message, error) {
m.tenant = types.MustTenantIDFromContext(ctx)
return m.message, nil
}
func TestKBFilesRequireExactGrantAndBinding(t *testing.T) {
base := callerContext()
grant := &KBAccess{
Caller: types.CallerFromContext(base),
KnowledgeBase: &types.KnowledgeBase{ID: "kb", TenantID: 2},
EffectiveTenantID: 2,
Permission: types.OrgRoleViewer,
}
ctx := grant.Context(base)
bindings := &fileBinding{allowed: true}
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
catalog := fileCatalog{
&types.StoredResource{TenantID: 2, PhysicalPath: "local://2/exports/a.png", OriginalName: "a.png"},
}
file, err := ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
require.NoError(t, err)
require.Equal(t, uint64(2), file.OwnerTenantID)
require.Equal(t, "kb", bindings.kb)
require.Equal(t, uint64(2), bindings.tenant)
bindings.allowed = false
_, err = ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
require.ErrorIs(t, err, ErrForbidden, "a same-tenant file still needs an exact KB binding")
bindings.allowed = true
_, err = ResolveKBFile(ctx, grant, "another", ref, catalog, bindings)
require.ErrorIs(t, err, ErrForbidden)
_, err = ResolveKBFile(ctx, nil, "kb", ref, catalog, bindings)
require.ErrorIs(t, err, ErrForbidden, "execution tenant alone is not a grant")
bindings.err = errors.New("database unavailable")
_, err = ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
require.ErrorIs(t, err, bindings.err)
}
func TestMessageFilesRequireReferenceAndRecheckRevocation(t *testing.T) {
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
messages := &fileMessages{
message: &types.Message{ID: "message", AgentID: "agent", AgentTenantID: 2, Role: "assistant"},
}
agents := &agentLookup{agent: &types.CustomAgent{ID: "agent", TenantID: 2}}
catalog := artifactFileCatalog{
fileCatalog{&types.StoredResource{TenantID: 2, PhysicalPath: "local://2/exports/a.png"}},
}
ctx := types.WithExecutionTenant(callerContext(), 2)
_, err := ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
require.ErrorIs(t, err, ErrForbidden)
messages.message.Content = "![image](" + ref + "x)"
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
require.ErrorIs(t, err, ErrForbidden, "a longer handle is not the requested handle")
messages.message.Content = "![image](" + ref + ")"
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
require.NoError(t, err)
require.Equal(t, uint64(1), messages.tenant, "session lookup must use the caller")
agents.agent = nil
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
require.ErrorIs(t, err, ErrForbidden, "historical references cannot bypass share revocation")
}
func TestMessageArtifactsKeepSessionOwnershipSeparateFromAgentOutput(t *testing.T) {
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
message := &types.Message{
AgentID: "agent",
AgentTenantID: 2,
Artifacts: types.MessageArtifacts{{URL: ref, FileName: "report.pdf"}},
}
catalog := fileCatalog{&types.StoredResource{TenantID: 1, PhysicalPath: "local://1/exports/report.pdf"}}
file, err := ResolveMessageArtifact(callerContext(), message, 0, nil, catalog, MessageKBShareAuthorizer{})
require.NoError(t, err)
require.Equal(t, uint64(1), file.OwnerTenantID)
catalog.resource.TenantID = 2
catalog.resource.PhysicalPath = "local://2/exports/report.pdf"
_, err = ResolveMessageArtifact(callerContext(), message, 0, nil, catalog, MessageKBShareAuthorizer{})
require.ErrorIs(t, err, ErrForbidden)
}
type messageFileKBs struct{ kb *types.KnowledgeBase }
func (k messageFileKBs) GetKnowledgeBasesByIDsOnly(context.Context, []string) ([]*types.KnowledgeBase, error) {
return []*types.KnowledgeBase{k.kb}, nil
}
func TestMessageSharedKBFilesRequireLiveBinding(t *testing.T) {
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
message := &types.Message{
AgentTenantID: 1, Role: "assistant",
KnowledgeReferences: types.References{{KnowledgeBaseID: "shared", Content: ref}},
}
catalog := fileCatalog{&types.StoredResource{
Handle: "AbCdEfGhIjKlMnOpQrStUv", TenantID: 2, PhysicalPath: "local://2/exports/a.png",
}}
binding := &fileBinding{allowed: true}
shares := &shareLookup{permission: types.OrgRoleViewer}
authorizer := MessageKBShareAuthorizer{
ShareGuard: shares, KBs: messageFileKBs{kb: &types.KnowledgeBase{ID: "shared", TenantID: 2}}, Bindings: binding,
}
_, err := AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
require.NoError(t, err)
require.Equal(t, "shared", binding.kb)
require.Equal(t, uint64(2), binding.tenant)
binding.allowed = false
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
require.ErrorIs(t, err, ErrForbidden, "retrieval text alone must not authorize an unrelated same-tenant resource")
binding.allowed = true
binding.err = errors.New("binding lookup unavailable")
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
require.ErrorIs(t, err, ErrForbidden)
binding.err = nil
authorizer.Bindings = nil
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
require.ErrorIs(t, err, ErrForbidden, "missing live lookup must fail closed")
}
// This fixture represents a catalog claim made by the artifact collector.
type artifactFileCatalog struct{ fileCatalog }
func (artifactFileCatalog) GetMessageFileBindings(
_ context.Context,
_ uint64,
_, messageID string,
) (*types.MessageFileBindings, error) {
return &types.MessageFileBindings{MessageArtifact: messageID == "message"}, nil
}

View File

@ -0,0 +1,62 @@
package access
import (
"context"
"github.com/Tencent/WeKnora/internal/types"
)
// KBSharePermissions resolves organization grants for one caller during one
// operation. It deliberately does not grant tenant ownership or shared-agent
// access. Callers retain their own policy for lookup failures (abort or filter).
// Create a new instance for each operation; it is not a cross-request cache and
// must not be shared between goroutines.
type KBSharePermissions struct {
ctx context.Context
lookup KBShareLookup
tenantID uint64
role types.TenantRole
results map[string]kbSharePermission
}
type kbSharePermission struct {
role types.OrgMemberRole
shared bool
err error
}
// NewKBSharePermissions creates a request-local permission cache.
func NewKBSharePermissions(
ctx context.Context,
lookup KBShareLookup,
tenantID uint64,
role types.TenantRole,
) *KBSharePermissions {
return &KBSharePermissions{
ctx: ctx,
lookup: lookup,
tenantID: tenantID,
role: role,
results: make(map[string]kbSharePermission),
}
}
// Permission caches both grants and failures so multiple documents/chunks in a
// KB observe the same decision, without repeating the membership queries.
func (p *KBSharePermissions) Permission(kbID string) (types.OrgMemberRole, bool, error) {
if kbID == "" || p.tenantID == 0 || p.lookup == nil {
return "", false, nil
}
result, ok := p.results[kbID]
if !ok {
result.role, result.shared, result.err = p.lookup.CheckTenantKBPermission(p.ctx, kbID, p.tenantID, p.role)
p.results[kbID] = result
}
return result.role, result.shared, result.err
}
// Check tests the required role against the cached organization permission.
func (p *KBSharePermissions) Check(kbID string, required types.OrgMemberRole) (bool, error) {
role, shared, err := p.Permission(kbID)
return err == nil && shared && role.IsValid() && required.IsValid() && role.HasPermission(required), err
}

View File

@ -0,0 +1,84 @@
package access
import (
"context"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
type permissionLookupFunc func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error)
func (f permissionLookupFunc) CheckTenantKBPermission(
ctx context.Context,
id string,
tenant uint64,
role types.TenantRole,
) (types.OrgMemberRole, bool, error) {
return f(ctx, id, tenant, role)
}
func TestKBSharePermissionsCachePerOperation(t *testing.T) {
calls := map[string]int{}
granted := true
lookup := permissionLookupFunc(
func(_ context.Context, kb string, tenant uint64, role types.TenantRole) (types.OrgMemberRole, bool, error) {
require.Equal(t, uint64(1), tenant)
require.Equal(t, types.TenantRoleContributor, role)
calls[kb]++
return types.OrgRoleEditor, granted, nil
},
)
permissions := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleContributor)
for _, required := range []types.OrgMemberRole{types.OrgRoleViewer, types.OrgRoleEditor, types.OrgRoleAdmin} {
allowed, err := permissions.Check("kb", required)
require.NoError(t, err)
require.Equal(t, required != types.OrgRoleAdmin, allowed)
}
require.Equal(t, 1, calls["kb"], "different minimum roles share the same membership lookup")
_, err := permissions.Check("other", types.OrgRoleViewer)
require.NoError(t, err)
require.Equal(t, 1, calls["other"])
granted = false
fresh := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleContributor)
allowed, err := fresh.Check("kb", types.OrgRoleViewer)
require.NoError(t, err)
require.False(t, allowed, "a new operation must observe share revocation")
require.Equal(t, 2, calls["kb"])
}
func TestKBSharePermissionsCachesFailuresWithoutGranting(t *testing.T) {
failure := errors.New("database unavailable")
calls := 0
lookup := permissionLookupFunc(
func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
calls++
return types.OrgRoleAdmin, true, failure
},
)
permissions := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleAdmin)
for i := 0; i < 3; i++ {
allowed, err := permissions.Check("kb", types.OrgRoleViewer)
require.False(t, allowed)
require.ErrorIs(t, err, failure, "callers must retain the infrastructure error")
}
require.Equal(t, 1, calls)
}
func TestKBSharePermissionsMissingLookupAndInvalidRolesDeny(t *testing.T) {
permissions := NewKBSharePermissions(context.Background(), nil, 1, types.TenantRoleAdmin)
allowed, err := permissions.Check("kb", types.OrgRoleViewer)
require.NoError(t, err)
require.False(t, allowed)
lookup := permissionLookupFunc(
func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
return types.OrgRoleEditor, true, nil
},
)
permissions = NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleAdmin)
allowed, err = permissions.Check("kb", "unknown")
require.NoError(t, err)
require.False(t, allowed)
}

View File

@ -0,0 +1,227 @@
package access
import (
"context"
"fmt"
"github.com/Tencent/WeKnora/internal/types"
)
// KBTransferOperation selects the source and destination permission contract.
type KBTransferOperation string
// Supported knowledge-base transfer operations.
const (
KBTransferClone KBTransferOperation = "clone"
KBTransferMove KBTransferOperation = "move"
)
type kbTransferGrant struct {
caller types.Caller
tenant uint64
source, target, taskID, creatorID string
operation KBTransferOperation
create bool
}
// WithKBTransfer admits a pair of resources after the entry point has resolved
// the source and destination grants and applied its creator/role policy. A new
// clone destination must be a server-reserved ID; it is not an existing KB.
func WithKBTransfer(
ctx context.Context,
source, target *types.KnowledgeBase,
operation KBTransferOperation,
taskID string,
create bool,
) (context.Context, error) {
caller := types.CallerFromContext(ctx)
if taskID == "" {
return ctx, ErrForbidden
}
if err := validateTransferPair(source, target, caller.TenantID, operation, create); err != nil {
return ctx, err
}
if err := transferAPIScope(ctx, source.ID, target.ID, operation, create); err != nil {
return ctx, err
}
required := types.OrgRoleViewer
if operation == KBTransferMove {
required = types.OrgRoleEditor
}
if !HasKBGrant(ctx, source.ID, source.TenantID, required) ||
(!create && !HasKBGrant(ctx, target.ID, target.TenantID, types.OrgRoleEditor)) {
return ctx, ErrForbidden
}
return withTransferGrant(ctx, source, target, operation, taskID, create, false), nil
}
// WithKBTransferTask continues an admitted task. Both KBs must first be loaded
// with their persisted owners (or a reserved new clone destination). This scope
// cannot authorize a different pair or upgrade the worker into a tenant user.
func WithKBTransferTask(
ctx context.Context,
source, target *types.KnowledgeBase,
expectedTenant uint64,
operation KBTransferOperation,
taskID string,
create bool,
) (context.Context, error) {
if taskID == "" {
return ctx, ErrForbidden
}
if err := validateTransferPair(source, target, expectedTenant, operation, create); err != nil {
return ctx, err
}
ctx = types.WithExecutionTenant(ctx, expectedTenant)
return withTransferGrant(ctx, source, target, operation, taskID, create, true), nil
}
func withTransferGrant(
ctx context.Context,
source, target *types.KnowledgeBase,
operation KBTransferOperation,
taskID string,
create, task bool,
) context.Context {
caller := types.CallerFromContext(ctx)
grant := kbTransferGrant{
caller: caller,
tenant: source.TenantID,
source: source.ID,
target: target.ID,
taskID: taskID,
creatorID: target.CreatorID,
operation: operation,
create: create,
}
required := types.OrgRoleViewer
if operation == KBTransferMove {
required = types.OrgRoleEditor
}
ctx = context.WithValue(ctx, types.KBGrantsContextKey, []kbGrant{
{caller: caller, kbID: source.ID, tenantID: source.TenantID, permission: required, task: task},
{caller: caller, kbID: target.ID, tenantID: target.TenantID, permission: types.OrgRoleEditor, task: task},
})
return context.WithValue(ctx, types.KBTransferContextKey, grant)
}
// RequireKBTransfer consumes the exact caller, operation and resource-pair grant.
func RequireKBTransfer(ctx context.Context, source, target *types.KnowledgeBase, operation KBTransferOperation) error {
grant, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
if !ok || grant.caller != types.CallerFromContext(ctx) || grant.operation != operation {
return ErrForbidden
}
if err := validateTransferPair(source, target, grant.tenant, operation, grant.create); err != nil {
return err
}
if grant.source != source.ID || grant.target != target.ID {
return ErrForbidden
}
return transferAPIScope(ctx, source.ID, target.ID, operation, grant.create)
}
// CloneDestination exposes only the destination already admitted by the
// request/worker; CopyKnowledgeBase cannot turn an empty ID into ambient write.
func CloneDestination(ctx context.Context, sourceID string) (id string, create bool, creatorID string, err error) {
g, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
if !ok || g.caller != types.CallerFromContext(ctx) || g.operation != KBTransferClone || g.source != sourceID {
return "", false, "", ErrForbidden
}
return g.target, g.create, g.creatorID, nil
}
// TransferTaskID returns the admitted operation identity for retry checkpoints.
func TransferTaskID(ctx context.Context) string {
g, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
if !ok || g.caller != types.CallerFromContext(ctx) {
return ""
}
return g.taskID
}
func validateTransferPair(
source, target *types.KnowledgeBase,
tenant uint64,
operation KBTransferOperation,
create bool,
) error {
if source == nil || target == nil || tenant == 0 || source.ID == "" || target.ID == "" ||
source.TenantID != tenant ||
target.TenantID != tenant {
return ErrForbidden
}
if source.ID == target.ID {
return fmt.Errorf("source and target knowledge bases must differ")
}
if operation != KBTransferClone && operation != KBTransferMove || create && operation != KBTransferClone {
return ErrForbidden
}
return nil
}
func transferAPIScope(
ctx context.Context,
sourceID, targetID string,
operation KBTransferOperation,
create bool,
) error {
ids := []string{sourceID}
if !create {
ids = append(ids, targetID)
}
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, ids...); err != nil {
return err
}
capability := types.APIKeyCapabilityManageKnowledgeBases
if operation == KBTransferMove {
capability = types.APIKeyCapabilityIngest
}
if scope, ok := types.TenantAPIKeyScopeFromContext(ctx); ok && !scope.FullAccess &&
!scope.HasCapability(capability) {
return ErrForbidden
}
return nil
}
// ValidateKBTransferCompatibility is shared by synchronous admission and task
// execution, so invalid modes/configurations fail before resource mutations.
func ValidateKBTransferCompatibility(
source, target *types.KnowledgeBase,
operation KBTransferOperation,
mode string,
tenant *types.Tenant,
) error {
if source == nil || target == nil {
return ErrNotFound
}
if source.ID == target.ID {
return fmt.Errorf("source and target knowledge bases must differ")
}
if source.Type != target.Type {
return fmt.Errorf("source and target knowledge bases must have the same type")
}
if source.EmbeddingModelID != target.EmbeddingModelID {
return fmt.Errorf("source and target knowledge bases use different embedding models")
}
if operation == KBTransferMove && mode != "reuse_vectors" && mode != "reparse" {
return fmt.Errorf("unknown move mode: %s", mode)
}
if (operation == KBTransferClone || mode == "reuse_vectors") && !source.SharesStoreWith(target) {
return fmt.Errorf("source and target knowledge bases use different vector stores; use reparse mode for moves")
}
// A move retains file paths too, so resolving them through another storage
// instance would make the document unreadable, even when vectors are reparsed.
defaultID, defaultProvider := "", ""
if tenant != nil {
if tenant.DefaultStorageBackendID != nil {
defaultID = *tenant.DefaultStorageBackendID
}
if tenant.StorageEngineConfig != nil {
defaultProvider = tenant.StorageEngineConfig.DefaultProvider
}
}
if !source.SharesStorageBackendWith(target, defaultID, defaultProvider) {
return fmt.Errorf("source and target knowledge bases use different storage instances")
}
return nil
}

View File

@ -0,0 +1,96 @@
package access
import (
"context"
"fmt"
"testing"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestTransferAdmissionRequiresIndependentGrants(t *testing.T) {
source, target := &types.KnowledgeBase{ID: "source", TenantID: 1}, &types.KnowledgeBase{ID: "target", TenantID: 1}
ctx := types.WithCaller(
context.Background(),
types.Caller{TenantID: 1, UserID: "user", Role: types.TenantRoleAdmin},
)
_, err := WithKBTransfer(ctx, source, target, KBTransferMove, "task", false)
require.ErrorIs(t, err, ErrForbidden)
request := KBRequest{Caller: types.CallerFromContext(ctx)}
sourceRead, err := ResolveKB(ctx, request, source, types.OrgRoleViewer, nil, nil)
require.NoError(t, err)
targetRead, err := ResolveKB(ctx, request, target, types.OrgRoleViewer, nil, nil)
require.NoError(t, err)
reads := targetRead.WithGrant(sourceRead.WithGrant(ctx))
_, err = WithKBTransfer(reads, source, target, KBTransferClone, "task", false)
require.ErrorIs(t, err, ErrForbidden, "owner read projection must not authorize target replacement")
targetWrite, err := ResolveKB(ctx, request, target, types.OrgRoleEditor, nil, nil)
require.NoError(t, err)
cloneCtx, err := WithKBTransfer(targetWrite.WithGrant(reads), source, target, KBTransferClone, "task", false)
require.NoError(t, err)
require.NoError(t, RequireKBTransfer(logger.CloneContext(cloneCtx), source, target, KBTransferClone))
require.False(t, HasKBGrant(cloneCtx, source.ID, 1, types.OrgRoleEditor))
_, err = WithKBTransfer(targetWrite.WithGrant(reads), source, target, KBTransferMove, "task", false)
require.ErrorIs(t, err, ErrForbidden)
}
func TestTransferTaskScopeIsExactAndDoesNotCreateCaller(t *testing.T) {
source, target := &types.KnowledgeBase{ID: "source", TenantID: 7}, &types.KnowledgeBase{ID: "target", TenantID: 7}
ctx, err := WithKBTransferTask(context.Background(), source, target, 7, KBTransferMove, "task", false)
require.NoError(t, err)
ctx = logger.CloneContext(ctx)
require.Zero(t, types.CallerFromContext(ctx).TenantID)
require.NoError(t, RequireKBTransfer(ctx, source, target, KBTransferMove))
require.ErrorIs(t, RequireKBTransfer(ctx, source, target, KBTransferClone), ErrForbidden)
require.ErrorIs(
t,
RequireKBTransfer(ctx, source, &types.KnowledgeBase{ID: "third", TenantID: 7}, KBTransferMove),
ErrForbidden,
)
require.False(t, HasKBGrant(ctx, "third", 7, types.OrgRoleViewer))
_, err = WithKBTransferTask(
ctx,
source,
&types.KnowledgeBase{ID: "target", TenantID: 8},
7,
KBTransferMove,
"task",
false,
)
require.ErrorIs(t, err, ErrForbidden)
changed := types.WithCaller(ctx, types.Caller{TenantID: 7, UserID: "different"})
require.ErrorIs(t, RequireKBTransfer(changed, source, target, KBTransferMove), ErrForbidden)
}
func TestTransferAPICapabilitiesRemainSeparate(t *testing.T) {
source, target := &types.KnowledgeBase{ID: "source", TenantID: 1}, &types.KnowledgeBase{ID: "target", TenantID: 1}
for _, tc := range []struct {
operation KBTransferOperation
capability string
create bool
ids []string
allowed bool
}{
{KBTransferClone, "manage_kbs", false, []string{"source", "target"}, true},
{KBTransferClone, "ingest", false, []string{"source", "target"}, false},
{KBTransferMove, "ingest", false, []string{"source", "target"}, true},
{KBTransferMove, "manage_kbs", false, []string{"source", "target"}, false},
{KBTransferMove, "ingest", false, []string{"source"}, false},
{KBTransferClone, "manage_kbs", true, []string{"source"}, true},
} {
t.Run(
fmt.Sprintf("%s/%s/create=%v/ids=%d", tc.operation, tc.capability, tc.create, len(tc.ids)),
func(t *testing.T) {
ctx, err := WithKBTransferTask(context.Background(), source, target, 1, tc.operation, "task", tc.create)
require.NoError(t, err)
ctx = types.WithTenantAPIKeyScope(
ctx,
types.TenantAPIKeyScope{Capabilities: types.StringArray{tc.capability}, KnowledgeBaseIDs: tc.ids},
)
require.Equal(t, tc.allowed, RequireKBTransfer(ctx, source, target, tc.operation) == nil)
},
)
}
}

View File

@ -0,0 +1,45 @@
package access
import (
"context"
"github.com/Tencent/WeKnora/internal/types"
)
// RequireKBWrite consumes an explicit operation grant. Tenant ownership and
// role checks belong to the entry point that resolved the grant; an execution
// tenant alone, or a shared-agent/read grant, never authorizes a mutation.
func RequireKBWrite(ctx context.Context, kb *types.KnowledgeBase) error {
if kb == nil || kb.ID == "" || kb.TenantID == 0 {
return ErrNotFound
}
if scope, ok := types.TenantAPIKeyScopeFromContext(ctx); ok && !scope.FullAccess &&
!scope.HasCapability(types.APIKeyCapabilityIngest) {
return ErrForbidden
}
if !HasKBGrant(ctx, kb.ID, kb.TenantID, types.OrgRoleEditor) {
return ErrForbidden
}
return nil
}
// WithKBTaskWrite is for trusted workers continuing an admitted operation.
// The worker supplies a server-loaded KB and verifies its child-resource
// bindings before any side effects. It grants only this KB, preserves the
// original caller (including an absent caller), and never creates an Admin.
// This is a task execution grant, not a way to authorize an incoming request.
func WithKBTaskWrite(ctx context.Context, kb *types.KnowledgeBase, expectedTenant uint64) (context.Context, error) {
if kb == nil || kb.ID == "" || expectedTenant == 0 || kb.TenantID != expectedTenant {
return ctx, ErrForbidden
}
ctx = types.WithExecutionTenant(ctx, expectedTenant)
grant := kbGrant{
caller: types.CallerFromContext(ctx),
kbID: kb.ID,
tenantID: expectedTenant,
permission: types.OrgRoleEditor,
task: true,
}
// A worker gets exactly one scope, without inheriting another task's KBs.
return context.WithValue(ctx, types.KBGrantsContextKey, []kbGrant{grant}), nil
}

View File

@ -0,0 +1,85 @@
package access
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestKBWriteConsumesOperationGrant(t *testing.T) {
ctx := callerContext()
kb := &types.KnowledgeBase{ID: "kb", TenantID: types.CallerFromContext(ctx).TenantID}
read, err := ResolveKB(ctx, KBRequest{Caller: types.CallerFromContext(ctx)}, kb, types.OrgRoleViewer, nil, nil)
require.NoError(t, err)
require.Equal(t, types.OrgRoleAdmin, read.Permission, "owner permission projection remains unchanged")
require.ErrorIs(t, RequireKBWrite(read.Context(ctx), kb), ErrForbidden, "an owner read does not grant mutation")
require.ErrorIs(t, RequireKBWrite(types.WithExecutionTenant(ctx, kb.TenantID), kb), ErrForbidden)
write, err := ResolveKB(ctx, KBRequest{Caller: types.CallerFromContext(ctx)}, kb, types.OrgRoleEditor, nil, nil)
require.NoError(t, err)
ctx = logger.CloneContext(write.Context(ctx))
require.NoError(t, RequireKBWrite(ctx, kb))
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: "other", TenantID: kb.TenantID}), ErrForbidden)
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: kb.ID, TenantID: kb.TenantID + 1}), ErrForbidden)
for _, tt := range []struct {
name string
scope types.TenantAPIKeyScope
allowed bool
}{
{"retrieve only", types.TenantAPIKeyScope{Capabilities: types.StringArray{"retrieve"}}, false},
{
"ingest",
types.TenantAPIKeyScope{
Capabilities: types.StringArray{"ingest"},
KnowledgeBaseIDs: types.StringArray{"kb"},
},
true,
},
{
"wrong KB",
types.TenantAPIKeyScope{
Capabilities: types.StringArray{"ingest"},
KnowledgeBaseIDs: types.StringArray{"other"},
},
false,
},
{"full access", types.TenantAPIKeyScope{FullAccess: true}, true},
} {
t.Run(tt.name, func(t *testing.T) {
err := RequireKBWrite(types.WithTenantAPIKeyScope(ctx, tt.scope), kb)
require.Equal(t, tt.allowed, err == nil)
})
}
}
func TestKBTaskWriteIsExplicitAndBounded(t *testing.T) {
kb := &types.KnowledgeBase{ID: "kb", TenantID: 7}
base := context.Background()
ctx, err := WithKBTaskWrite(base, kb, 7)
require.NoError(t, err)
ctx = logger.CloneContext(ctx)
require.Zero(t, types.CallerFromContext(ctx).TenantID, "a worker must not impersonate the tenant owner")
require.NoError(t, RequireKBWrite(ctx, kb))
allowed, err := NewKBPermissions(ctx, nil).Check("kb", 7, types.OrgRoleViewer)
require.NoError(t, err)
require.True(t, allowed, "a scoped write task can read its KB")
allowed, err = NewKBPermissions(ctx, nil).Check("other", 7, types.OrgRoleViewer)
require.NoError(t, err)
require.False(t, allowed)
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: "other", TenantID: 7}), ErrForbidden)
_, err = WithKBTaskWrite(base, kb, 8)
require.ErrorIs(t, err, ErrForbidden)
_, err = WithKBTaskWrite(base, kb, 0)
require.ErrorIs(t, err, ErrForbidden)
require.False(t, HasKBGrant(base, "kb", 7, types.OrgRoleEditor), "the parent context stays unmodified")
request := callerContext()
task, err := WithKBTaskWrite(request, kb, 7)
require.NoError(t, err)
require.Equal(t, types.CallerFromContext(request), types.CallerFromContext(task))
changed := types.WithCaller(task, types.Caller{TenantID: 99, UserID: "another"})
require.ErrorIs(t, RequireKBWrite(changed, kb), ErrForbidden)
}

View File

@ -0,0 +1,38 @@
package access
import (
"encoding/json"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
)
// RejectMovingKnowledge is shared by admission and persisted service checks.
// Admission prevents misleading task acceptance; workers must recheck after
// enqueue because a move can claim the document in the intervening window.
func RejectMovingKnowledge(knowledge *types.Knowledge) error {
if knowledge == nil {
return apperrors.NewNotFoundError("knowledge not found")
}
fields := map[string]json.RawMessage{}
if len(knowledge.Metadata) != 0 {
if err := json.Unmarshal(knowledge.Metadata, &fields); err != nil {
return err
}
}
raw := fields[types.KnowledgeTransferMetadataKey]
if len(raw) == 0 {
return nil
}
var state struct {
Operation KBTransferOperation `json:"operation"`
Phase string `json:"phase"`
}
if err := json.Unmarshal(raw, &state); err != nil {
return err
}
if state.Operation == KBTransferMove && state.Phase == "moving" {
return apperrors.NewConflictError("knowledge has an unfinished move; retry the move first")
}
return nil
}

View File

@ -0,0 +1,156 @@
// Package access resolves resource permissions independently of HTTP routing.
// Tenant roles, API-key capabilities, and resource ownership guards remain
// separate checks; a KB grant does not replace them.
package access
import (
"context"
"errors"
"fmt"
"github.com/Tencent/WeKnora/internal/types"
)
// Resource access errors retain distinct adapter response mappings.
var (
ErrUnauthorized = errors.New("kb_access: unauthorized")
ErrNotFound = errors.New("kb_access: not found")
ErrForbidden = errors.New("kb_access: forbidden")
ErrInvalidAgentSource = errors.New("kb_access: invalid agent source")
)
// KBRequest keeps the authenticated caller separate from the resource tenant.
// AgentSourceTenantID is parsed only when the explicit-agent fallback is used.
type KBRequest struct {
Caller types.Caller
AgentID string
AgentSourceTenantID string
}
// KBAccess binds a resolved resource permission to its original caller.
type KBAccess struct {
KnowledgeBase *types.KnowledgeBase
Caller types.Caller
EffectiveTenantID uint64
Permission types.OrgMemberRole
// A read resolution must not mint a write grant even for the KB owner.
operationPermission types.OrgMemberRole
}
// Context scopes resource operations without changing the authenticated caller.
func (a *KBAccess) Context(ctx context.Context) context.Context {
if a == nil || a.KnowledgeBase == nil {
return ctx
}
scoped := a.WithGrant(ctx)
if types.CallerFromContext(scoped) != a.Caller.Normalize() {
return ctx
}
return types.WithExecutionTenant(scoped, a.EffectiveTenantID)
}
// WithGrant carries this authorization without changing execution scope.
func (a *KBAccess) WithGrant(ctx context.Context) context.Context {
if a == nil || a.KnowledgeBase == nil {
return ctx
}
caller := a.Caller.Normalize()
if existing, ok := ctx.Value(types.CallerContextKey).(types.Caller); ok && existing != caller {
return ctx // A grant cannot replace an already authenticated caller.
}
grants, _ := ctx.Value(types.KBGrantsContextKey).([]kbGrant)
permission := a.Permission
if a.operationPermission.IsValid() && permission.HasPermission(a.operationPermission) {
permission = a.operationPermission
}
grants = append(
append([]kbGrant(nil), grants...),
kbGrant{caller: caller, kbID: a.KnowledgeBase.ID, tenantID: a.EffectiveTenantID, permission: permission},
)
return context.WithValue(types.WithCaller(ctx, caller), types.KBGrantsContextKey, grants)
}
// KBShareLookup resolves organization sharing with the caller role cap.
type KBShareLookup interface {
CheckTenantKBPermission(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error)
}
// SharedAgentLookup verifies access to a particular shared agent.
type SharedAgentLookup interface {
GetSharedAgentForTenant(context.Context, uint64, types.TenantRole, string, ...uint64) (*types.CustomAgent, error)
}
// AgentShareLookup also supports read access through any reachable agent.
type AgentShareLookup interface {
SharedAgentLookup
TenantCanAccessKBViaSomeSharedAgent(context.Context, uint64, types.TenantRole, *types.KnowledgeBase) (bool, error)
}
// ResolveKB authorizes a server-loaded KB (or a document's persisted KB/tenant
// reference). Its tenant is authoritative; resolving the owner from a share
// record again would add a lookup and a second source of truth. It never treats
// an effective resource tenant as the caller.
// Share lookup errors do not grant access; an independent agent grant may still
// allow reading. An explicit agent selector never falls back to another agent.
func ResolveKB(ctx context.Context, request KBRequest, kb *types.KnowledgeBase, required types.OrgMemberRole,
shares KBShareLookup, agents AgentShareLookup,
) (*KBAccess, error) {
if request.Caller.TenantID == 0 {
return nil, ErrUnauthorized
}
if kb == nil || kb.ID == "" {
return nil, ErrNotFound
}
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kb.ID); err != nil {
return nil, err
}
request.Caller = request.Caller.Normalize()
if caller, ok := ctx.Value(types.CallerContextKey).(types.Caller); ok && caller != request.Caller {
return nil, ErrForbidden
}
grant := func(permission types.OrgMemberRole) (*KBAccess, error) {
return &KBAccess{
KnowledgeBase: kb, Caller: request.Caller,
EffectiveTenantID: kb.TenantID, Permission: permission, operationPermission: required,
}, nil
}
if kb.TenantID == request.Caller.TenantID {
return grant(types.OrgRoleAdmin)
}
if shares != nil {
permissions := NewKBSharePermissions(ctx, shares, request.Caller.TenantID, request.Caller.Role)
permission, shared, err := permissions.Permission(kb.ID)
if err == nil && shared && permission.HasPermission(required) {
return grant(permission)
}
}
if required != types.OrgRoleViewer || agents == nil {
return nil, ErrForbidden
}
if request.AgentID != "" {
source, err := types.ParseAgentSourceTenantID(request.AgentSourceTenantID)
if err != nil {
return nil, fmt.Errorf("%w: %s", ErrInvalidAgentSource, err)
}
agent, err := agents.GetSharedAgentForTenant(
ctx,
request.Caller.TenantID,
request.Caller.Role,
request.AgentID,
source,
)
if err == nil && types.SharedAgentIncludesKB(agent, kb) {
return grant(types.OrgRoleViewer)
}
} else {
allowed,
err := agents.TenantCanAccessKBViaSomeSharedAgent(ctx,
request.Caller.TenantID,
request.Caller.Role,
kb)
if err == nil && allowed {
return grant(types.OrgRoleViewer)
}
}
return nil, ErrForbidden
}

View File

@ -0,0 +1,282 @@
package access
import (
"context"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
type shareLookup struct {
permission types.OrgMemberRole
err error
caller uint64
role types.TenantRole
}
func (s *shareLookup) CheckTenantKBPermission(
_ context.Context,
_ string,
caller uint64,
role types.TenantRole,
) (types.OrgMemberRole, bool, error) {
s.caller, s.role = caller, role
return s.permission, s.permission != "", s.err
}
type agentLookup struct {
agent *types.CustomAgent
any bool
err error
anyCalls int
source uint64
}
func (s *agentLookup) GetSharedAgentForTenant(
_ context.Context,
_ uint64,
_ types.TenantRole,
_ string,
source ...uint64,
) (*types.CustomAgent, error) {
s.source = source[0]
return s.agent, s.err
}
func (s *agentLookup) TenantCanAccessKBViaSomeSharedAgent(
context.Context,
uint64,
types.TenantRole,
*types.KnowledgeBase,
) (bool, error) {
s.anyCalls++
return s.any, s.err
}
func TestResolveKBPermissionMatrix(t *testing.T) {
for _, tt := range []struct {
name string
own bool
permission types.OrgMemberRole
required types.OrgMemberRole
shareError bool
agentID, source, mode string
selected []string
wrongTenant, missingAgent, agentError, anyAgent bool
want types.OrgMemberRole
wantErr error
}{
{name: "own KB", own: true, required: types.OrgRoleEditor, want: types.OrgRoleAdmin},
{
name: "organization reader",
permission: types.OrgRoleViewer,
required: types.OrgRoleViewer,
want: types.OrgRoleViewer,
},
{
name: "organization editor",
permission: types.OrgRoleEditor,
required: types.OrgRoleEditor,
want: types.OrgRoleEditor,
},
{
name: "reader cannot write",
permission: types.OrgRoleViewer,
required: types.OrgRoleEditor,
wantErr: ErrForbidden,
},
{
name: "share lookup fails closed",
permission: types.OrgRoleEditor,
shareError: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "independent agent grant after share error",
shareError: true,
anyAgent: true,
required: types.OrgRoleViewer,
want: types.OrgRoleViewer,
},
{name: "any shared agent read", anyAgent: true, required: types.OrgRoleViewer, want: types.OrgRoleViewer},
{name: "any agent cannot write", anyAgent: true, required: types.OrgRoleEditor, wantErr: ErrForbidden},
{
name: "agent lookup error",
anyAgent: true,
agentError: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "explicit all",
agentID: "agent",
source: "2",
mode: "all",
required: types.OrgRoleViewer,
want: types.OrgRoleViewer,
},
{
name: "explicit selected",
agentID: "agent",
mode: "selected",
selected: []string{"kb"},
required: types.OrgRoleViewer,
want: types.OrgRoleViewer,
},
{
name: "selection miss does not fall back",
agentID: "agent",
mode: "selected",
selected: []string{"other"},
anyAgent: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "none does not fall back",
agentID: "agent",
mode: "none",
anyAgent: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "unknown mode denied",
agentID: "agent",
mode: "invalid",
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "tenant mismatch denied",
agentID: "agent",
mode: "all",
wrongTenant: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "missing explicit agent does not fall back",
agentID: "agent",
missingAgent: true,
anyAgent: true,
required: types.OrgRoleViewer,
wantErr: ErrForbidden,
},
{
name: "invalid source rejected",
agentID: "agent",
source: "bad",
mode: "all",
required: types.OrgRoleViewer,
wantErr: ErrInvalidAgentSource,
},
{
name: "own grant precedes agent parsing",
own: true,
agentID: "agent",
source: "bad",
required: types.OrgRoleViewer,
want: types.OrgRoleAdmin,
},
{
name: "share grant precedes agent parsing",
permission: types.OrgRoleViewer,
agentID: "agent",
source: "bad",
required: types.OrgRoleViewer,
want: types.OrgRoleViewer,
},
} {
t.Run(tt.name, func(t *testing.T) {
kb := &types.KnowledgeBase{ID: "kb", TenantID: 2}
if tt.own {
kb.TenantID = 1
}
shares := &shareLookup{permission: tt.permission}
if tt.shareError {
shares.err = errors.New("unavailable")
}
agents := &agentLookup{
any: tt.anyAgent,
agent: &types.CustomAgent{
TenantID: 2,
Config: types.CustomAgentConfig{KBSelectionMode: tt.mode, KnowledgeBases: tt.selected},
},
}
if tt.wrongTenant {
agents.agent.TenantID = 3
}
if tt.missingAgent {
agents.agent = nil
}
if tt.agentError {
agents.err = errors.New("unavailable")
}
request := KBRequest{
Caller: types.Caller{TenantID: 1, Role: types.TenantRoleViewer},
AgentID: tt.agentID,
AgentSourceTenantID: tt.source,
}
grant, err := ResolveKB(context.Background(), request, kb, tt.required, shares, agents)
if tt.wantErr != nil {
require.ErrorIs(t, err, tt.wantErr)
require.Nil(t, grant)
} else {
require.NoError(t, err)
require.Equal(t, tt.want, grant.Permission)
require.Equal(t, uint64(1), grant.Caller.TenantID)
require.Equal(t, kb.TenantID, grant.EffectiveTenantID)
}
if !tt.own {
require.Equal(t, uint64(1), shares.caller)
require.Equal(t, types.TenantRoleViewer, shares.role)
}
if tt.agentID != "" {
require.Zero(t, agents.anyCalls)
}
if tt.source == "2" {
require.Equal(t, uint64(2), agents.source)
}
})
}
}
func TestResolveKBMissingIdentityResourceAndAPIKeyScope(t *testing.T) {
kb := &types.KnowledgeBase{ID: "kb", TenantID: 1}
_, err := ResolveKB(context.Background(), KBRequest{}, kb, types.OrgRoleViewer, nil, nil)
require.ErrorIs(t, err, ErrUnauthorized)
_, err = ResolveKB(
context.Background(),
KBRequest{Caller: types.Caller{TenantID: 1}},
nil,
types.OrgRoleViewer,
nil,
nil,
)
require.ErrorIs(t, err, ErrNotFound)
ctx := types.WithTenantAPIKeyScope(
context.Background(),
types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"other"}},
)
_, err = ResolveKB(ctx, KBRequest{Caller: types.Caller{TenantID: 1}}, kb, types.OrgRoleViewer, nil, nil)
require.Error(t, err, "API key scope must apply even to owned KBs")
}

View File

@ -0,0 +1,249 @@
package access
import (
"context"
"strings"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
// MessageFileLookup is the narrow message-service surface needed by the
// message-scoped file proxy. Keeping it small makes the authorization boundary
// independently testable.
type MessageFileLookup interface {
GetMessage(ctx context.Context, sessionID, messageID string) (*types.Message, error)
}
// SharedAgentFileLookup verifies that a source workspace's agent is still
// shared to the caller. Revoking the share therefore also revokes historical
// message-file access.
type SharedAgentFileLookup interface {
GetSharedAgentForTenant(
ctx context.Context,
tenantID uint64,
callerTenantRole types.TenantRole,
agentID string,
sourceTenantID ...uint64,
) (*types.CustomAgent, error)
}
// KBSharePermissionGuard is the org-share surface the message-scoped proxy
// needs for its shared-KB fallback.
type KBSharePermissionGuard = KBShareLookup
// KBTenantLookup resolves knowledge bases without a tenant filter, so a
// message's persisted retrieval evidence can be mapped back to the knowledge
// bases the chunks were retrieved from.
type KBTenantLookup interface {
GetKnowledgeBasesByIDsOnly(ctx context.Context, ids []string) ([]*types.KnowledgeBase, error)
}
// KnowledgeOwnerLookup resolves a knowledge entry to its knowledge base for
// message references that predate the denormalized knowledge_base_id field.
type KnowledgeOwnerLookup interface {
GetKnowledgeByIDOnly(ctx context.Context, id string) (*types.Knowledge, error)
}
// MessageKBShareAuthorizer bundles the read-only lookups behind the
// message proxy's org-shared-KB fallback. A nil ShareGuard or KBs lookup
// disables the fallback. A live Bindings lookup is also required and can
// be supplied by the resource catalog. A nil Knowledges lookup only disables the
// pre-denormalization KnowledgeID path.
type MessageKBShareAuthorizer struct {
ShareGuard KBSharePermissionGuard
KBs KBTenantLookup
Knowledges KnowledgeOwnerLookup
Bindings interfaces.KBResourceLookup
}
// resourceAccessibleViaSharedKB reports whether the message's persisted
// retrieval evidence proves the resource came from an org-shared KB the
// caller may read. This is the fallback for replies whose agent belongs to
// the caller's own workspace while the retrieved resources belong to the
// workspace that shared the knowledge base (#3022).
//
// Evidence required from one retrieval record: a canonical resource://
// handle (not a prefix of a longer token) appears in the chunk text or
// image_info, its knowledge base belongs to the resource's tenant, and
// that KB is org-shared to the caller with at least viewer permission, and
// an independent live resource binding confirms the file still belongs to it.
// Smart-reasoning turns persist that evidence on AgentSteps when
// KnowledgeReferences was never filled. Any lookup failure fails closed.
func (a MessageKBShareAuthorizer) resourceAccessibleViaSharedKB(
ctx context.Context,
message *types.Message,
resource *types.StoredResource,
callerTenantID uint64,
callerTenantRole types.TenantRole,
) bool {
if a.ShareGuard == nil || a.KBs == nil || a.Bindings == nil || message == nil || resource == nil {
return false
}
handle, ok := types.ParseResourcePath(types.BuildResourcePath(resource.Handle))
if !ok {
return false
}
kbIDs := a.collectSharedKBEvidenceIDs(ctx, message, handle)
if len(kbIDs) == 0 {
return false
}
kbs, err := a.KBs.GetKnowledgeBasesByIDsOnly(ctx, kbIDs)
if err != nil {
return false
}
permissions := NewKBSharePermissions(ctx, a.ShareGuard, callerTenantID, callerTenantRole)
for _, kb := range kbs {
if kb == nil || kb.TenantID != resource.TenantID {
continue
}
shared, err := permissions.Check(kb.ID, types.OrgRoleViewer)
if err == nil && shared {
bound, bindingErr := a.Bindings.IsReferencedByKnowledgeBase(
ctx, resource.TenantID, kb.ID, types.BuildResourcePath(handle))
if bindingErr == nil && bound {
return true
}
}
}
return false
}
func (a MessageKBShareAuthorizer) collectSharedKBEvidenceIDs(
ctx context.Context,
message *types.Message,
handle string,
) []string {
seenKB := make(map[string]bool)
seenKnowledge := make(map[string]bool)
var kbIDs, knowledgeIDs []string
addKB := func(id string) {
if id == "" || seenKB[id] {
return
}
seenKB[id] = true
kbIDs = append(kbIDs, id)
}
addKnowledge := func(id string) {
if id == "" || seenKnowledge[id] {
return
}
seenKnowledge[id] = true
knowledgeIDs = append(knowledgeIDs, id)
}
for _, ref := range message.KnowledgeReferences {
if !searchResultHasResourceHandle(ref, handle) {
continue
}
if ref.KnowledgeBaseID != "" {
addKB(ref.KnowledgeBaseID)
continue
}
addKnowledge(ref.KnowledgeID)
}
collectKBEvidenceFromValue(message.AgentSteps, handle, "", "", addKB, addKnowledge)
if a.Knowledges != nil {
for _, knowledgeID := range knowledgeIDs {
knowledge, err := a.Knowledges.GetKnowledgeByIDOnly(ctx, knowledgeID)
if err != nil || knowledge == nil {
continue
}
addKB(knowledge.KnowledgeBaseID)
}
}
return kbIDs
}
func searchResultHasResourceHandle(ref *types.SearchResult, handle string) bool {
if ref == nil {
return false
}
return textHasResourceHandle(ref.Content, handle) ||
textHasResourceHandle(ref.MatchedContent, handle) ||
textHasResourceHandle(ref.ImageInfo, handle)
}
func textHasResourceHandle(text, handle string) bool {
if text == "" || handle == "" {
return false
}
want := types.BuildResourcePath(handle)
for _, ref := range types.ScanResourceReferences(text) {
if ref == want {
return true
}
}
return false
}
func collectKBEvidenceFromValue(
v interface{},
handle, kbID, knowledgeID string,
addKB, addKnowledge func(string),
) {
switch val := v.(type) {
case string:
if !textHasResourceHandle(val, handle) {
return
}
if kbID != "" {
addKB(kbID)
return
}
addKnowledge(knowledgeID)
case types.AgentSteps:
for _, step := range val {
collectKBEvidenceFromValue(step, handle, kbID, knowledgeID, addKB, addKnowledge)
}
case types.AgentStep:
collectKBEvidenceFromValue(val.ToolCalls, handle, kbID, knowledgeID, addKB, addKnowledge)
case []types.ToolCall:
for _, call := range val {
collectKBEvidenceFromValue(call, handle, kbID, knowledgeID, addKB, addKnowledge)
}
case types.ToolCall:
if val.Result != nil {
collectKBEvidenceFromValue(val.Result.Output, handle, kbID, knowledgeID, addKB, addKnowledge)
collectKBEvidenceFromValue(val.Result.Data, handle, kbID, knowledgeID, addKB, addKnowledge)
}
case map[string]interface{}:
nextKB := firstNonEmptyString(
stringFromAnyMap(val, "knowledge_base_id"),
stringFromAnyMap(val, "knowledge_base"),
kbID,
)
nextKnowledge := firstNonEmptyString(stringFromAnyMap(val, "knowledge_id"), knowledgeID)
for _, nested := range val {
collectKBEvidenceFromValue(nested, handle, nextKB, nextKnowledge, addKB, addKnowledge)
}
case []interface{}:
for _, item := range val {
collectKBEvidenceFromValue(item, handle, kbID, knowledgeID, addKB, addKnowledge)
}
case []map[string]interface{}:
for _, item := range val {
collectKBEvidenceFromValue(item, handle, kbID, knowledgeID, addKB, addKnowledge)
}
}
}
func stringFromAnyMap(m map[string]interface{}, key string) string {
if m == nil {
return ""
}
s, _ := m[key].(string)
return strings.TrimSpace(s)
}
func firstNonEmptyString(values ...string) string {
for _, value := range values {
if strings.TrimSpace(value) != "" {
return strings.TrimSpace(value)
}
}
return ""
}

View File

@ -0,0 +1,69 @@
package access
import (
"errors"
"github.com/Tencent/WeKnora/internal/types"
)
var (
// ErrResourceNotFound means the ownership lookup found no resource visible
// in the caller's tenant. HTTP adapters decide how to produce the 404.
ErrResourceNotFound = errors.New("rbac: resource not found")
// ErrOwnershipForbidden indicates neither ownership nor the required role.
ErrOwnershipForbidden = errors.New("rbac: ownership or role insufficient")
)
// OwnershipRequest carries server-resolved identity and rollout policy. The
// cross-tenant bypass must already account for the cluster configuration.
// This gate does not replace API-key capabilities or resource-scope checks.
type OwnershipRequest struct {
UserID string
Role types.TenantRole
APIKey bool
CrossTenantSuperuser bool
Enforce bool
}
// OwnershipDecision contains the details adapters need for logs and audits.
// The error returned with it determines whether access was allowed.
type OwnershipDecision struct {
CreatorID string
EnforcementSkipped bool
LookupFailed bool
}
// CheckOwnershipOrRole performs no resource lookup when API-key authorization,
// role, superuser status, or the RBAC rollout policy already settles the gate.
// The lookup must enforce tenant visibility before returning a creator ID.
func CheckOwnershipOrRole(
request OwnershipRequest,
required types.TenantRole,
lookup func() (string, error),
) (OwnershipDecision, error) {
var decision OwnershipDecision
if request.APIKey || request.Role.HasPermission(required) || request.CrossTenantSuperuser {
return decision, nil
}
if !request.Enforce {
decision.EnforcementSkipped = true
return decision, nil
}
if lookup == nil {
decision.LookupFailed = true
return decision, errors.New("rbac: creator lookup unavailable")
}
creator, err := lookup()
decision.CreatorID = creator
if errors.Is(err, ErrResourceNotFound) {
return decision, ErrResourceNotFound
}
if err != nil {
decision.LookupFailed = true
return decision, err
}
if creator != "" && creator == request.UserID {
return decision, nil
}
return decision, ErrOwnershipForbidden
}

View File

@ -0,0 +1,103 @@
package access
import (
"errors"
"fmt"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestCheckOwnershipOrRoleSkipsLookupWhenGateAlreadyResolved(t *testing.T) {
for _, tt := range []struct {
name string
request OwnershipRequest
skipped bool
}{
{name: "API key", request: OwnershipRequest{APIKey: true, Role: types.TenantRoleViewer, Enforce: true}},
{name: "admin", request: OwnershipRequest{Role: types.TenantRoleAdmin, Enforce: true}},
{name: "owner role", request: OwnershipRequest{Role: types.TenantRoleOwner, Enforce: true}},
{
name: "cross tenant superuser",
request: OwnershipRequest{
Role: types.TenantRoleViewer,
CrossTenantSuperuser: true,
Enforce: true,
},
},
{name: "enforcement disabled", request: OwnershipRequest{Role: types.TenantRoleViewer}, skipped: true},
{name: "role before rollout", request: OwnershipRequest{Role: types.TenantRoleAdmin}},
} {
t.Run(tt.name, func(t *testing.T) {
decision, err := CheckOwnershipOrRole(tt.request, types.TenantRoleAdmin, func() (string, error) {
t.Fatal("ownership lookup must not run on a fast path")
return "", errors.New("unavailable")
})
require.NoError(t, err)
require.Equal(t, tt.skipped, decision.EnforcementSkipped)
require.False(t, decision.LookupFailed)
})
}
}
func TestCheckOwnershipOrRoleResolvesCreator(t *testing.T) {
failure := errors.New("database unavailable")
for _, tt := range []struct {
name, user, creator string
lookupErr, wantErr error
lookupFailed bool
}{
{name: "creator", user: "user", creator: "user"},
{name: "other creator", user: "user", creator: "other", wantErr: ErrOwnershipForbidden},
{name: "tenant owned", user: "user", wantErr: ErrOwnershipForbidden},
{name: "missing identity cannot match empty creator", wantErr: ErrOwnershipForbidden},
{name: "missing resource", user: "user", lookupErr: ErrResourceNotFound, wantErr: ErrResourceNotFound},
{
name: "wrapped missing resource",
user: "user",
lookupErr: fmt.Errorf("load: %w",
ErrResourceNotFound),
wantErr: ErrResourceNotFound,
},
{
name: "lookup failure wins over creator match",
user: "user",
creator: "user",
lookupErr: failure,
wantErr: failure,
lookupFailed: true,
},
{
name: "lookup denial remains a lookup failure",
lookupErr: ErrOwnershipForbidden,
wantErr: ErrOwnershipForbidden,
lookupFailed: true,
},
} {
t.Run(tt.name, func(t *testing.T) {
calls := 0
decision, err := CheckOwnershipOrRole(
OwnershipRequest{UserID: tt.user, Role: types.TenantRoleContributor, Enforce: true},
types.TenantRoleAdmin,
func() (string, error) {
calls++
return tt.creator, tt.lookupErr
},
)
require.Equal(t, 1, calls)
require.ErrorIs(t, err, tt.wantErr)
require.Equal(t, tt.creator, decision.CreatorID)
require.Equal(t, tt.lookupFailed, decision.LookupFailed)
})
}
}
func TestCheckOwnershipOrRoleMissingLookupFailsClosed(t *testing.T) {
decision, err := CheckOwnershipOrRole(OwnershipRequest{Enforce: true}, types.TenantRoleAdmin, nil)
require.Error(t, err)
require.True(t, decision.LookupFailed)
}

View File

@ -575,7 +575,7 @@ func (r *chunkRepository) DeleteByKnowledgeList(ctx context.Context, tenantID ui
func (r *chunkRepository) MoveChunksByKnowledgeID(ctx context.Context, tenantID uint64, knowledgeID string, targetKBID string) error {
return r.db.WithContext(ctx).Model(&types.Chunk{}).
Where("tenant_id = ? AND knowledge_id = ?", tenantID, knowledgeID).
Update("knowledge_base_id", targetKBID).Error
Updates(map[string]any{"knowledge_base_id": targetKBID, "tag_id": ""}).Error
}
// DeleteChunksByTagID deletes all chunks with the specified tag ID
@ -953,30 +953,22 @@ func (r *chunkRepository) UpdateChunkFieldsByTagID(
newTagID *string,
excludeIDs []string,
) ([]string, error) {
// First, get the IDs of chunks that will be affected (for is_enabled sync)
if isEnabled == nil && setFlags == 0 && clearFlags == 0 && newTagID == nil {
return nil, nil
}
// Return every affected entry, including tag-only and flag-only changes.
// Callers use these IDs for index synchronization and subsequent patches.
var affectedIDs []string
if isEnabled != nil {
var chunks []*types.Chunk
query := r.db.WithContext(ctx).
Select("id").
Where("tenant_id = ? AND knowledge_base_id = ? AND chunk_type = ?",
tenantID, kbID, types.ChunkTypeFAQ)
if tagID != "" {
query = query.Where("tag_id = ?", tagID)
}
if len(excludeIDs) > 0 {
query = query.Where("id NOT IN ?", excludeIDs)
}
// Only get chunks that need to change
query = query.Where("is_enabled != ?", *isEnabled)
if err := query.Find(&chunks).Error; err != nil {
return nil, err
}
for _, c := range chunks {
affectedIDs = append(affectedIDs, c.ID)
}
selection := r.db.WithContext(ctx).Model(&types.Chunk{}).
Where("tenant_id = ? AND knowledge_base_id = ? AND chunk_type = ?", tenantID, kbID, types.ChunkTypeFAQ)
if tagID != "" {
selection = selection.Where("tag_id = ?", tagID)
}
if len(excludeIDs) > 0 {
selection = selection.Where("id NOT IN ?", excludeIDs)
}
if err := selection.Pluck("id", &affectedIDs).Error; err != nil {
return nil, err
}
// Build update query
@ -1309,3 +1301,17 @@ func (r *chunkRepository) ListRecentDocumentChunksWithQuestions(
return chunks, nil
}
func (r *chunkRepository) ListAllChunksByKnowledgeID(
ctx context.Context,
tenantID uint64,
knowledgeID string,
) ([]*types.Chunk, error) {
var chunks []*types.Chunk
err := r.db.WithContext(ctx).
Where("tenant_id = ? AND knowledge_id = ?", tenantID, knowledgeID).
Order("id ASC").
Find(&chunks).
Error
return chunks, err
}

View File

@ -0,0 +1,60 @@
package repository
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestTagFieldUpdatesReturnAllAffectedChunks(t *testing.T) {
for _, flagsOnly := range []bool{false, true} {
name := "tag only"
if flagsOnly {
name = "flags only"
}
t.Run(name, func(t *testing.T) {
db := setupChunkTestDB(t)
repo := NewChunkRepository(db)
for _, chunk := range []*types.Chunk{
{ID: "selected", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
{ID: "excluded", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
{ID: "other-kb", TenantID: 7, KnowledgeBaseID: "other", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
{ID: "other-tenant", TenantID: 8, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
{ID: "deleted", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
} {
require.NoError(t, db.Create(chunk).Error)
}
require.NoError(t, db.Delete(&types.Chunk{ID: "deleted"}).Error)
next := "next"
var tag *string
var flags types.ChunkFlags
if flagsOnly {
flags = types.ChunkFlagRecommended
} else {
tag = &next
}
ids, err := repo.UpdateChunkFieldsByTagID(
context.Background(),
7,
"kb",
"tag",
nil,
flags,
0,
tag,
[]string{"excluded"},
)
require.NoError(t, err)
require.Equal(t, []string{"selected"}, ids)
selected, err := repo.GetChunkByID(context.Background(), 7, "selected")
require.NoError(t, err)
if flagsOnly {
require.True(t, selected.Flags.HasFlag(types.ChunkFlagRecommended))
} else {
require.Equal(t, "next", selected.TagID)
}
})
}
}

View File

@ -560,15 +560,20 @@ func (r *knowledgeRepository) UpdateKnowledgeColumns(
// to normal queries and have not moved out of the transient deleting state.
func (r *knowledgeRepository) UpdateActiveDeletingKnowledgeColumns(
ctx context.Context,
id string,
tenantID uint64,
kbID, id string,
values map[string]interface{},
) (bool, error) {
if len(values) == 0 {
if tenantID == 0 || kbID == "" || len(values) == 0 {
return false, nil
}
result := r.db.WithContext(ctx).
Model(&types.Knowledge{}).
Where("id = ? AND parse_status = ?", id, types.ParseStatusDeleting).
Where("tenant_id = ? AND knowledge_base_id = ? AND id = ? AND parse_status = ?",
tenantID,
kbID,
id,
types.ParseStatusDeleting).
Updates(values)
if result.Error != nil {
return false, result.Error

View File

@ -353,22 +353,63 @@ func TestUpdateActiveDeletingKnowledgeColumns_GuardsStateAndSoftDelete(t *testin
activeCompletedID := insertKnowledgeWithStatus(t, db, types.ParseStatusCompleted, false)
deletedDeletingID := insertKnowledgeWithStatus(t, db, types.ParseStatusDeleting, true)
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, activeDeletingID, map[string]interface{}{
"parse_status": types.ParseStatusFailed,
"error_message": "delete task exhausted retries",
})
require.NoError(
t,
db.Exec(
"UPDATE knowledges SET knowledge_base_id = ? WHERE id IN ?",
"delete-kb",
[]string{activeDeletingID, activeCompletedID, deletedDeletingID},
).Error,
)
for _, scope := range []struct {
tenant uint64
kb string
}{{2, "delete-kb"}, {1, "other-kb"}, {0, "delete-kb"}, {1, ""}} {
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(
ctx,
scope.tenant,
scope.kb,
activeDeletingID,
map[string]interface{}{"parse_status": types.ParseStatusFailed},
)
require.NoError(t, err)
require.False(t, updated)
}
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(
ctx,
1,
"delete-kb",
activeDeletingID,
map[string]interface{}{
"parse_status": types.ParseStatusFailed,
"error_message": "delete task exhausted retries",
},
)
require.NoError(t, err)
assert.True(t, updated)
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(ctx, activeCompletedID, map[string]interface{}{
"parse_status": types.ParseStatusFailed,
})
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(
ctx,
1,
"delete-kb",
activeCompletedID,
map[string]interface{}{
"parse_status": types.ParseStatusFailed,
},
)
require.NoError(t, err)
assert.False(t, updated)
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(ctx, deletedDeletingID, map[string]interface{}{
"parse_status": types.ParseStatusFailed,
})
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(
ctx,
1,
"delete-kb",
deletedDeletingID,
map[string]interface{}{
"parse_status": types.ParseStatusFailed,
},
)
require.NoError(t, err)
assert.False(t, updated)

View File

@ -84,7 +84,7 @@ func TestGetKnowledgeBatch_TracksDeletionUntilSoftDelete(t *testing.T) {
assert.Equal(t, id, rows[0].ID)
assert.Equal(t, types.ParseStatusDeleting, rows[0].ParseStatus)
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, id, map[string]interface{}{
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, 1, kbID, id, map[string]interface{}{
"parse_status": types.ParseStatusFailed,
})
require.NoError(t, err)

View File

@ -0,0 +1,74 @@
package repository
import (
"context"
"fmt"
"time"
"github.com/Tencent/WeKnora/internal/types"
"gorm.io/gorm"
)
// UpdateKnowledgeForTransfer is a compare-and-swap checkpoint. Full-row Save
// could resurrect a deleted document or overwrite a concurrently moved row.
// Accounting changes are committed with the checkpoint, making retry cleanup
// safe even if the worker crashes between the database and queue operations.
func (r *knowledgeRepository) UpdateKnowledgeForTransfer(ctx context.Context, before, after *types.Knowledge) error {
if before == nil || after == nil || before.ID == "" || before.ID != after.ID || before.TenantID == 0 ||
before.TenantID != after.TenantID {
return fmt.Errorf("invalid knowledge transfer checkpoint")
}
// PostgreSQL persists timestamps at microsecond precision. Keep the returned
// checkpoint identical to storage so the next CAS in this task can match it.
after.UpdatedAt = time.Now().UTC().Truncate(time.Microsecond)
var storedUpdatedAt time.Time
err := r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
result := tx.Model(&types.Knowledge{}).
Where("id = ? AND tenant_id = ? AND knowledge_base_id = ? AND parse_status = ? AND "+
"updated_at = ?",
before.ID,
before.TenantID,
before.KnowledgeBaseID,
before.ParseStatus,
before.UpdatedAt).
Updates(map[string]any{
"knowledge_base_id": after.KnowledgeBaseID, "parse_status": after.ParseStatus,
"metadata": after.Metadata, "storage_size": after.StorageSize, "updated_at": after.UpdatedAt,
"enable_status": after.EnableStatus, "embedding_model_id": after.EmbeddingModelID,
"description": after.Description,
"processed_at": after.ProcessedAt,
"error_message": after.ErrorMessage,
})
if result.Error != nil {
return result.Error
}
if result.RowsAffected != 1 {
return fmt.Errorf("knowledge changed during transfer")
}
delta := after.StorageSize - before.StorageSize
if delta != 0 {
result := tx.Model(&types.Tenant{}).Where("id = ?", before.TenantID).
UpdateColumn("storage_used",
gorm.Expr("CASE WHEN storage_used + ? < 0 THEN 0 ELSE storage_used + ? END",
delta,
delta))
if result.Error != nil {
return result.Error
}
if result.RowsAffected != 1 {
return fmt.Errorf("transfer tenant not found")
}
}
var stored struct{ UpdatedAt time.Time }
if err := tx.Model(&types.Knowledge{}).Select("updated_at").
Where("id = ? AND tenant_id = ?", after.ID, after.TenantID).Take(&stored).Error; err != nil {
return err
}
storedUpdatedAt = stored.UpdatedAt
return nil
})
if err == nil {
after.UpdatedAt = storedUpdatedAt
}
return err
}

View File

@ -0,0 +1,30 @@
package repository
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestTransferCheckpointReturnsStoredTimestampPrecision(t *testing.T) {
db := setupKnowledgeTestDB(t)
// Emulate a database that stores fewer fractional digits than time.Now.
require.NoError(t, db.Exec(`CREATE TRIGGER truncate_checkpoint_time AFTER UPDATE ON knowledges
BEGIN UPDATE knowledges SET updated_at = strftime('%Y-%m-%d %H:%M:%S', NEW.updated_at) || '+00:00'
WHERE id = NEW.id; END`).Error)
row := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "source", ParseStatus: types.ParseStatusCompleted}
require.NoError(t, db.Omit("custom_metadata").Create(row).Error)
repo := NewKnowledgeRepository(db)
before, err := repo.GetKnowledgeByID(context.Background(), 7, "doc")
require.NoError(t, err)
after := *before
after.ParseStatus = types.ParseStatusProcessing
require.NoError(t, repo.UpdateKnowledgeForTransfer(context.Background(), before, &after))
require.Zero(t, after.UpdatedAt.Nanosecond())
final := after
final.KnowledgeBaseID = "target"
final.ParseStatus = types.ParseStatusCompleted
require.NoError(t, repo.UpdateKnowledgeForTransfer(context.Background(), &after, &final))
}

View File

@ -0,0 +1,57 @@
package repository
import (
"context"
"github.com/Tencent/WeKnora/internal/types"
)
// IsReferencedByKnowledgeBase accepts only explicit bindings to live documents.
// Text mentioning a handle or physical path is never ownership evidence.
func (r *resourceRepository) IsReferencedByKnowledgeBase(
ctx context.Context, tenantID uint64, kbID, resourceID string,
) (bool, error) {
if tenantID == 0 || kbID == "" || resourceID == "" {
return false, nil
}
var count int64
err := r.db.WithContext(ctx).Table("resource_bindings AS b").
Joins("JOIN knowledges AS k ON k.id = b.owner_id AND k.tenant_id = b.tenant_id").
Joins("JOIN knowledge_bases AS kb ON kb.id = k.knowledge_base_id "+
"AND kb.tenant_id = k.tenant_id AND kb.deleted_at IS NULL").
Where("b.resource_id = ? AND b.owner_type = ? AND b.tenant_id = ? "+
"AND k.knowledge_base_id = ? AND k.deleted_at IS NULL",
resourceID, types.ResourceOwnerKnowledge, tenantID, kbID).Count(&count).Error
return count > 0, err
}
func (r *resourceRepository) GetMessageFileBindings(
ctx context.Context, tenantID uint64, resourceID, messageID string,
) (*types.MessageFileBindings, error) {
result := &types.MessageFileBindings{}
if tenantID == 0 || resourceID == "" {
return result, nil
}
err := r.db.WithContext(ctx).Table("resource_bindings AS b").
Joins("JOIN knowledges AS k ON k.id = b.owner_id AND k.tenant_id = b.tenant_id").
Joins("JOIN knowledge_bases AS kb ON kb.id = k.knowledge_base_id "+
"AND kb.tenant_id = k.tenant_id AND kb.deleted_at IS NULL").
Where("b.resource_id = ? AND b.owner_type = ? AND b.tenant_id = ? AND k.deleted_at IS NULL",
resourceID, types.ResourceOwnerKnowledge, tenantID).
Distinct("k.knowledge_base_id").Pluck("k.knowledge_base_id", &result.KnowledgeBaseIDs).Error
if err != nil {
return nil, err
}
if messageID != "" {
var count int64
err = r.db.WithContext(ctx).Model(&types.ResourceBinding{}).
Where("resource_id = ? AND tenant_id = ? AND owner_type = ? AND owner_id = ? AND relation = ?",
resourceID, tenantID, types.ResourceOwnerMessage, messageID, types.ResourceRelationArtifact).
Count(&count).Error
if err != nil {
return nil, err
}
result.MessageArtifact = count > 0
}
return result, nil
}

View File

@ -0,0 +1,41 @@
package doris
import (
"context"
"fmt"
)
func (r *dorisRepository) ValidateKnowledgeIndexMove(ctx context.Context) error {
mode, err := r.resolveCompatMode(ctx)
if err != nil {
return err
}
if mode.usesRewriteChunkUpdates() {
// ANN DUPLICATE KEY tables implement replacement as DELETE + INSERT.
// A failed insert would lose the only vector copy; changing physical IDs
// would break the stable source-ID identity used by subsequent edits.
return fmt.Errorf("reuse_vectors move is not supported by Doris ANN tables; use reparse mode")
}
return nil
}
func (r *dorisRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
dimension int,
_ string,
) error {
if err := r.ValidateKnowledgeIndexMove(ctx); err != nil {
return err
}
if dimension <= 0 {
return fmt.Errorf("invalid embedding dimension")
}
query := fmt.Sprintf(
"UPDATE `%s` SET knowledge_base_id = ?, tag_id = '' WHERE knowledge_base_id = ? AND knowledge_id = ?",
r.getTableName(dimension),
)
_, err := r.db.ExecContext(ctx, query, targetKB, sourceKB, knowledgeID)
return err
}

View File

@ -0,0 +1,36 @@
package doris
import (
"context"
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/require"
)
func TestMoveIndicesRejectsNonAtomicANNReplacement(t *testing.T) {
repo, mock, _, cleanup := newTestRepo(t)
defer cleanup()
primeCompatMode(repo, dorisCompatModeInnerProductDuplicate, nil)
require.ErrorContains(
t,
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
"use reparse mode",
)
require.NoError(t, mock.ExpectationsWereMet(), "rejection must issue no vector delete or insert")
}
func TestMoveIndicesLegacyUpdatesOnlyExactSourceDocument(t *testing.T) {
repo, mock, _, cleanup := newTestRepo(t)
defer cleanup()
primeCompatMode(repo, dorisCompatModeLegacy, nil)
mock.ExpectExec("UPDATE `weknora_embeddings_2` SET knowledge_base_id = \\?, tag_id = '' WHERE "+
"knowledge_base_id = \\? AND knowledge_id = \\?").
WithArgs("target", "source", "doc").
WillReturnResult(sqlmock.NewResult(0, 3))
require.NoError(
t,
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
)
require.NoError(t, mock.ExpectationsWereMet())
}

View File

@ -0,0 +1,59 @@
package v7
import (
"bytes"
"context"
"encoding/json"
"fmt"
)
func (e *elasticsearchRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
_ int,
_ string,
) error {
body, err := json.Marshal(map[string]any{
"query": map[string]any{"bool": map[string]any{"filter": []any{
map[string]any{"term": map[string]any{e.idField("knowledge_base_id"): sourceKB}},
map[string]any{"term": map[string]any{e.idField("knowledge_id"): knowledgeID}},
}}},
"script": map[string]any{
"lang": "painless",
"source": "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = ''",
"params": map[string]any{"target": targetKB},
},
})
if err != nil {
return err
}
resp, err := e.client.UpdateByQuery(
[]string{e.index},
e.client.UpdateByQuery.WithContext(ctx),
e.client.UpdateByQuery.WithBody(bytes.NewReader(body)),
e.client.UpdateByQuery.WithRefresh(true),
)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.IsError() {
return fmt.Errorf("move indices: %s", resp.String())
}
var result struct {
Total *int64 `json:"total"`
Updated *int64 `json:"updated"`
TimedOut bool `json:"timed_out"`
Conflicts int `json:"version_conflicts"`
Failures []json.RawMessage `json:"failures"`
}
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return err
}
if result.Total == nil || result.Updated == nil || *result.Total < 0 || *result.Total != *result.Updated ||
result.TimedOut || result.Conflicts != 0 || len(result.Failures) != 0 {
return fmt.Errorf("move indices was incomplete")
}
return nil
}

View File

@ -0,0 +1,59 @@
package v7
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/elastic/go-elasticsearch/v7"
"github.com/stretchr/testify/require"
)
func TestMoveIndicesRequiresCompleteCounts(t *testing.T) {
for _, tt := range []struct {
name, body string
fails bool
}{
{"complete", `{"total":100,"updated":100}`, false},
{"empty retry", `{"total":0,"updated":0}`, false},
{"partial", `{"total":100,"updated":40}`, true},
{"missing total", `{"updated":40}`, true},
{"missing updated", `{"total":40}`, true},
{"missing counts", `{}`, true},
{"negative", `{"total":-1,"updated":-1}`, true},
{"conflict", `{"total":1,"updated":1,"version_conflicts":1}`, true},
{"timeout", `{"total":1,"updated":1,"timed_out":true}`, true},
} {
t.Run(tt.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Elastic-Product", "Elasticsearch")
if r.URL.Path == "/" {
_, _ = w.Write(
[]byte(
`{"version":{"number":"7.17.0","build_flavor":"default"},"tagline":"You Know, for Search"}`,
),
)
return
}
if r.URL.Path != "/vectors/_update_by_query" {
t.Errorf("unexpected path: %s", r.URL.Path)
}
_, _ = w.Write([]byte(tt.body))
}))
defer server.Close()
client, err := elasticsearch.NewClient(
elasticsearch.Config{Addresses: []string{server.URL}, DisableRetry: true},
)
require.NoError(t, err)
repo := &elasticsearchRepository{client: client, index: "vectors"}
err = repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, "file")
if tt.fails {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}

View File

@ -0,0 +1,48 @@
// Package v8 implements retrieval against its configured vector backend.
package v8
import (
"context"
"encoding/json"
"fmt"
"github.com/elastic/go-elasticsearch/v8/typedapi/types"
)
func (e *elasticsearchRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
_ int,
_ string,
) error {
query := &types.Query{Bool: &types.BoolQuery{Filter: []types.Query{
{
Terms: &types.TermsQuery{
TermsQuery: map[string]types.TermsQueryField{e.idField("knowledge_base_id"): []string{sourceKB}},
},
},
{
Terms: &types.TermsQuery{
TermsQuery: map[string]types.TermsQueryField{e.idField("knowledge_id"): []string{knowledgeID}},
},
},
}}}
source := "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = ''"
target, err := json.Marshal(targetKB)
if err != nil {
return err
}
script := &types.Script{Source: &source, Params: map[string]json.RawMessage{"target": target}}
result, err := e.client.UpdateByQuery(e.index).Query(query).Script(script).Refresh(true).Do(ctx)
if err != nil {
return err
}
if result == nil || result.Total == nil || result.Updated == nil ||
*result.Total < 0 || *result.Total != *result.Updated || (result.TimedOut != nil && *result.TimedOut) ||
(result.VersionConflicts != nil && *result.VersionConflicts != 0) ||
len(result.Failures) != 0 {
return fmt.Errorf("move indices was incomplete")
}
return nil
}

View File

@ -0,0 +1,59 @@
package v8
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/elastic/go-elasticsearch/v8"
"github.com/stretchr/testify/require"
)
func TestMoveIndicesRequiresCompleteCounts(t *testing.T) {
for _, tt := range []struct {
name, body string
fails bool
}{
{"complete", `{"total":100,"updated":100}`, false},
{"empty retry", `{"total":0,"updated":0}`, false},
{"partial", `{"total":100,"updated":40}`, true},
{"missing total", `{"updated":40}`, true},
{"missing updated", `{"total":40}`, true},
{"missing counts", `{}`, true},
{"negative", `{"total":-1,"updated":-1}`, true},
{"conflict", `{"total":1,"updated":1,"version_conflicts":1}`, true},
{"timeout", `{"total":1,"updated":1,"timed_out":true}`, true},
} {
t.Run(tt.name, func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("X-Elastic-Product", "Elasticsearch")
if r.URL.Path == "/" {
_, _ = w.Write(
[]byte(
`{"version":{"number":"7.17.0","build_flavor":"default"},"tagline":"You Know, for Search"}`,
),
)
return
}
if r.URL.Path != "/vectors/_update_by_query" {
t.Errorf("unexpected path: %s", r.URL.Path)
}
_, _ = w.Write([]byte(tt.body))
}))
defer server.Close()
client, err := elasticsearch.NewTypedClient(
elasticsearch.Config{Addresses: []string{server.URL}, DisableRetry: true},
)
require.NoError(t, err)
repo := &elasticsearchRepository{client: client, index: "vectors"}
err = repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, "file")
if tt.fails {
require.Error(t, err)
} else {
require.NoError(t, err)
}
})
}
}

View File

@ -0,0 +1,61 @@
package milvus
import (
"context"
"fmt"
)
func (m *milvusRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
dimension int,
_ string,
) error {
collection := m.getCollectionName(dimension)
filter := &universalFilterCondition{Operator: operatorAnd, Value: []*universalFilterCondition{
{Field: fieldKnowledgeBaseID, Operator: operatorEqual, Value: sourceKB},
{Field: fieldKnowledgeID, Operator: operatorEqual, Value: knowledgeID},
}}
size, offset := 100, 0
return drainMoveRows(ctx, targetKB, func(ctx context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
rows, _, err := m.searchByFilter(ctx, collection, filter, &size, &offset)
return rows, err
}, func(ctx context.Context, rows []*MilvusVectorEmbedding) error {
_, err := m.client.Upsert(ctx, createUpsert(collection, rows))
return err
})
}
// Drain the source predicate without offset windows. A repeated ID means the
// backend has not made the acknowledged update visible; fail for retry instead
// of advancing the document checkpoint or silently omitting remaining vectors.
func drainMoveRows(ctx context.Context, targetKB string,
read func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error),
write func(context.Context, []*MilvusVectorEmbedding) error,
) error {
seen := map[string]bool{}
for {
rows, err := read(ctx)
if err != nil {
return err
}
if len(rows) == 0 {
return nil
}
batch := make([]*MilvusVectorEmbedding, 0, len(rows))
for _, row := range rows {
if row == nil || row.ID == "" || seen[row.ID] {
return fmt.Errorf("invalid or repeated move index")
}
seen[row.ID] = true
copyOfRow := row.MilvusVectorEmbedding
copyOfRow.KnowledgeBaseID = targetKB
copyOfRow.TagID = ""
batch = append(batch, &copyOfRow)
}
if err := write(ctx, batch); err != nil {
return err
}
}
}

View File

@ -0,0 +1,82 @@
package milvus
import (
"context"
"errors"
"fmt"
"testing"
"github.com/stretchr/testify/require"
)
func TestMoveRowsDrainsPagesAndRetriesPartialUpsert(t *testing.T) {
remaining := map[string]*MilvusVectorEmbeddingWithScore{}
moved := map[string]*MilvusVectorEmbedding{}
for i := range 205 {
id := fmt.Sprint(i)
remaining[id] = &MilvusVectorEmbeddingWithScore{MilvusVectorEmbedding: MilvusVectorEmbedding{
ID: id,
KnowledgeID: "doc",
KnowledgeBaseID: "source",
TagID: "old-tag",
ChunkID: "chunk-" + id,
Content: "text",
Embedding: []float32{1, 2, 3},
IsEnabled: i%2 == 0,
}}
}
reads, writes := 0, 0
failure := errors.New("partial upsert")
read := func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
reads++
var rows []*MilvusVectorEmbeddingWithScore
for _, row := range remaining {
rows = append(rows, row)
if len(rows) == 100 {
break
}
}
return rows, nil
}
write := func(_ context.Context, rows []*MilvusVectorEmbedding) error {
writes++
for i, row := range rows {
original := remaining[row.ID]
require.NotNil(t, original)
require.Equal(t, "source", original.KnowledgeBaseID, "read rows must not be mutated before acknowledgement")
require.Equal(t, "target", row.KnowledgeBaseID)
require.Empty(t, row.TagID)
expected := original.MilvusVectorEmbedding
expected.KnowledgeBaseID = "target"
expected.TagID = ""
require.Equal(t, expected, *row)
moved[row.ID] = row
delete(remaining, row.ID)
if writes == 2 && i == 16 {
return failure
}
}
return nil
}
require.ErrorIs(t, drainMoveRows(context.Background(), "target", read, write), failure)
require.Len(t, remaining, 88)
require.NoError(t, drainMoveRows(context.Background(), "target", read, write))
require.Empty(t, remaining)
require.Len(t, moved, 205)
require.Equal(t, 4, reads, "retry must verify an empty source predicate")
}
func TestMoveRowsRejectsUnacknowledgedProgress(t *testing.T) {
row := &MilvusVectorEmbeddingWithScore{
MilvusVectorEmbedding: MilvusVectorEmbedding{ID: "id", KnowledgeBaseID: "source"},
}
err := drainMoveRows(
context.Background(),
"target",
func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
return []*MilvusVectorEmbeddingWithScore{row}, nil
},
func(context.Context, []*MilvusVectorEmbedding) error { return nil },
)
require.ErrorContains(t, err, "repeated")
}

View File

@ -17,8 +17,15 @@ import (
// bounded id + type. A non-zero version_conflicts count with no hard failures
// is logged as a warning but not treated as an error.
func inspectByQueryResponse(body io.Reader) error {
return inspectByQueryResult(body, false)
}
func inspectByQueryResult(body io.Reader, requireComplete bool) error {
var r struct {
VersionConflicts int `json:"version_conflicts"`
Total *int64 `json:"total"`
Updated *int64 `json:"updated"`
TimedOut bool `json:"timed_out"`
VersionConflicts int `json:"version_conflicts"`
Failures []struct {
ID string `json:"id"`
Cause struct {
@ -30,6 +37,13 @@ func inspectByQueryResponse(body io.Reader) error {
if err := json.NewDecoder(body).Decode(&r); err != nil {
return fmt.Errorf("opensearch: parse by-query response: %w", ErrTransport)
}
if requireComplete && (r.TimedOut || r.VersionConflicts != 0) {
return fmt.Errorf("opensearch: incomplete move (timed_out=%t, version_conflicts=%d): %w",
r.TimedOut, r.VersionConflicts, ErrTransport)
}
if requireComplete && (r.Total == nil || r.Updated == nil || *r.Total < 0 || *r.Updated != *r.Total) {
return fmt.Errorf("opensearch: incomplete move document counts: %w", ErrTransport)
}
log := logger.GetLogger(context.Background())
if len(r.Failures) == 0 {
if r.VersionConflicts > 0 {

View File

@ -0,0 +1,49 @@
package opensearch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
)
// MoveKnowledgeIndices changes every vector for the source document, including
// historical vectors whose chunks no longer exist, while preserving vector IDs.
func (r *Repository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
_ int,
_ string,
) error {
body, err := json.Marshal(map[string]any{
"query": map[string]any{"bool": map[string]any{"filter": []any{
map[string]any{"term": map[string]any{"knowledge_base_id": sourceKB}},
map[string]any{"term": map[string]any{"knowledge_id": knowledgeID}},
}}},
"script": map[string]any{
"lang": "painless",
"source": "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = '';",
"params": map[string]any{"target": targetKB},
},
})
if err != nil {
return fmt.Errorf("opensearch: marshal move: %w", err)
}
refresh := true
resp, err := r.client.UpdateByQuery(ctx, osapi.UpdateByQueryReq{
Indices: []string{r.baseIndex + "_*"}, Body: bytes.NewReader(body),
Params: osapi.UpdateByQueryParams{Refresh: &refresh},
})
if err != nil {
return wrapTransport(err)
}
if resp == nil {
return fmt.Errorf("opensearch: missing move response: %w", ErrTransport)
}
defer drainAndClose(resp.Inspect().Response.Body)
return inspectByQueryResult(io.LimitReader(resp.Inspect().Response.Body, 16<<20), true)
}

View File

@ -0,0 +1,61 @@
package opensearch
import (
"context"
"encoding/json"
"net/http"
"testing"
"github.com/stretchr/testify/require"
)
func TestMoveIndicesRequiresCompleteResult(t *testing.T) {
for _, tt := range []struct {
name, response string
fails bool
}{
{"complete", `{"total":2,"updated":2,"version_conflicts":0,"failures":[]}`, false},
{"partial", `{"total":100,"updated":40,"failures":[]}`, true},
{"empty", `{"total":0,"updated":0,"failures":[]}`, false},
{"missing counts", `{"failures":[]}`, true},
{"missing total", `{"updated":2,"failures":[]}`, true},
{"missing updated", `{"total":2,"failures":[]}`, true},
{"negative", `{"total":-1,"updated":-1,"failures":[]}`, true},
{"conflict", `{"updated":1,"version_conflicts":1,"failures":[]}`, true},
{"timeout", `{"timed_out":true,"version_conflicts":0,"failures":[]}`, true},
{"failure", `{"failures":[{"id":"doc","cause":{"type":"unavailable"}}]}`, true},
} {
t.Run(tt.name, func(t *testing.T) {
repo, server := newTestRepo(t, func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(tt.response))
})
defer server.Close()
err := repo.MoveKnowledgeIndices(
context.Background(), "source", "target", "doc", []string{"chunk"}, 1, "file",
)
if tt.fails {
require.ErrorIs(t, err, ErrTransport)
} else {
require.NoError(t, err)
}
})
}
}
func TestMoveIndicesIncludesOrphansEvenWithoutDatabaseChunks(t *testing.T) {
var body map[string]any
repo, server := newTestRepo(t, func(w http.ResponseWriter, r *http.Request) {
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Errorf("decode move request: %v", err)
}
_, _ = w.Write([]byte(`{"total":1,"updated":1,"failures":[]}`))
})
defer server.Close()
require.NoError(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 1, "file"))
want := `{"bool":{"filter":[{"term":{"knowledge_base_id":"source"}},{"term":{"knowledge_id":"doc"}}]}}`
query, err := json.Marshal(body["query"])
require.NoError(t, err)
require.JSONEq(t, want, string(query), "all vectors for this exact source document must be selected")
script := body["script"].(map[string]any)
require.Equal(t, map[string]any{"target": "target"}, script["params"])
}

View File

@ -0,0 +1,16 @@
// Package postgres implements retrieval against its configured vector backend.
package postgres
import "context"
func (g *pgRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
_ int,
_ string,
) error {
return g.db.WithContext(ctx).Model(&pgVector{}).
Where("knowledge_base_id = ? AND knowledge_id = ?", sourceKB, knowledgeID).
Updates(map[string]any{"knowledge_base_id": targetKB, "tag_id": ""}).Error
}

View File

@ -0,0 +1,26 @@
// Package qdrant implements vector retrieval and metadata updates in Qdrant.
package qdrant
import (
"context"
"github.com/qdrant/go-client/qdrant"
)
func (q *qdrantRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
dimension int,
_ string,
) error {
wait := true
_, err := q.client.SetPayload(ctx, &qdrant.SetPayloadPoints{
CollectionName: q.getCollectionName(dimension), Wait: &wait,
Payload: newQdrantValueMap(map[string]any{fieldKnowledgeBaseID: targetKB, fieldTagID: ""}),
PointsSelector: qdrant.NewPointsSelectorFilter(&qdrant.Filter{Must: []*qdrant.Condition{
qdrant.NewMatch(fieldKnowledgeBaseID, sourceKB), qdrant.NewMatch(fieldKnowledgeID, knowledgeID),
}}),
})
return err
}

View File

@ -0,0 +1,17 @@
// Package sqlite implements local retrieval with SQLite FTS and vectors.
package sqlite
import "context"
func (r *sqliteRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
_ int,
_ string,
) error {
// FTS and vector rows keep their IDs. Queries join them to this metadata row.
return r.db.WithContext(ctx).Model(&sqliteEmbedding{}).
Where("knowledge_base_id = ? AND knowledge_id = ?", sourceKB, knowledgeID).
Updates(map[string]any{"knowledge_base_id": targetKB, "tag_id": ""}).Error
}

View File

@ -0,0 +1,72 @@
package sqlite
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
func TestMoveIndicesPreservesVectorsAndExactScopeAcrossRetry(t *testing.T) {
repo := newSQLiteRetrieverTestRepository(t)
ctx := context.Background()
for _, row := range []struct{ chunk, kb, doc string }{
{
"text",
"source",
"doc",
},
{
"image",
"source",
"doc",
},
{
"other",
"source",
"other-doc",
},
{
"foreign",
"third",
"doc",
},
} {
saveSQLiteTestVector(t, repo, sqliteTestIndex(row.chunk, row.kb, row.doc, "old-tag", true), []float32{1, 0})
}
var before []sqliteEmbedding
require.NoError(t, repo.db.Order("id").Find(&before).Error)
for range 2 {
require.NoError(
t,
repo.MoveKnowledgeIndices(ctx, "source", "target", "doc", []string{"text", "image"}, 2, "document"),
)
}
var after []sqliteEmbedding
require.NoError(t, repo.db.Order("id").Find(&after).Error)
require.Len(t, after, len(before))
for i, row := range after {
require.Equal(t, before[i].ID, row.ID)
require.Equal(t, before[i].SourceID, row.SourceID)
if row.KnowledgeID == "doc" && before[i].KnowledgeBaseID == "source" {
require.Equal(t, "target", row.KnowledgeBaseID)
require.Empty(t, row.TagID)
} else {
require.Equal(t, before[i].KnowledgeBaseID, row.KnowledgeBaseID)
require.Equal(t, "old-tag", row.TagID)
}
}
results, err := repo.vectorRetrieve(
ctx,
types.RetrieveParams{
Embedding: []float32{1, 0},
TopK: 10,
KnowledgeBaseIDs: []string{"target"},
RetrieverType: types.VectorRetrieverType,
},
)
require.NoError(t, err)
require.Len(t, results, 1)
require.Len(t, results[0].Results, 2, "target vectors remain searchable after an identity-preserving move")
}

View File

@ -0,0 +1,33 @@
// Package tencentvectordb implements retrieval against its configured vector backend.
package tencentvectordb
import (
"context"
"github.com/tencent/vectordatabase-sdk-go/tcvectordb"
)
func (r *repository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
dimension int,
_ string,
) error {
filter := tcvectordb.NewFilter(
tcvectordb.In(
fieldKnowledgeBaseID,
[]string{sourceKB},
) + " and " + tcvectordb.In(
fieldKnowledgeID,
[]string{knowledgeID},
),
)
_, err := r.client.Database(r.databaseName).
Collection(r.collectionName(dimension)).
Update(ctx, tcvectordb.UpdateDocumentParams{
QueryFilter: filter,
UpdateFields: map[string]tcvectordb.Field{fieldKnowledgeBaseID: {Val: targetKB}, fieldTagID: {Val: ""}},
})
return err
}

View File

@ -0,0 +1,77 @@
// Package weaviate implements retrieval and metadata updates in Weaviate.
package weaviate
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v5/weaviate/filters"
"github.com/weaviate/weaviate-go-client/v5/weaviate/graphql"
)
func (w *weaviateRepository) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
_ []string,
dimension int,
_ string,
) error {
collection := w.getCollectionName(dimension)
where := filters.Where().WithOperator(filters.And).WithOperands([]*filters.WhereBuilder{
filters.Where().WithPath([]string{fieldKnowledgeBaseID}).WithOperator(filters.Equal).WithValueString(sourceKB),
filters.Where().WithPath([]string{fieldKnowledgeID}).WithOperator(filters.Equal).WithValueString(knowledgeID),
})
seen := make(map[string]bool)
for {
result, err := w.client.GraphQL().Get().WithClassName(collection).WithWhere(where).WithLimit(100).
WithFields(graphql.Field{Name: "_additional", Fields: []graphql.Field{{Name: "id"}}}).
Do(ctx)
if err != nil {
return err
}
if result == nil || len(result.Errors) != 0 {
return fmt.Errorf("failed to list move indices")
}
get, ok := result.Data["Get"].(map[string]interface{})
if !ok {
return fmt.Errorf("invalid move index response")
}
rows, ok := get[collection].([]interface{})
if !ok {
return fmt.Errorf("invalid move index collection")
}
if len(rows) == 0 {
return nil
}
var ids []string
for _, row := range rows {
data, ok := row.(map[string]interface{})
if !ok {
return fmt.Errorf("invalid move index row")
}
additional, ok := data["_additional"].(map[string]interface{})
if !ok {
return fmt.Errorf("missing move index ID")
}
id, ok := additional["id"].(string)
if !ok || id == "" {
return fmt.Errorf("invalid move index ID")
}
if seen[id] {
return fmt.Errorf("move indices made no progress for %s", id)
}
seen[id] = true
ids = append(ids, id)
}
// Drain the first filtered page; after+where is unsupported by Weaviate.
// Successful updates remove these IDs from the predicate. Only an empty
// follow-up query proves completion, including after partial-page updates.
for _, id := range ids {
if err := w.client.Data().Updater().WithClassName(collection).WithID(id).WithMerge().
WithProperties(map[string]interface{}{fieldKnowledgeBaseID: targetKB, fieldTagID: ""}).
Do(ctx); err != nil {
return err
}
}
}
}

View File

@ -0,0 +1,130 @@
package weaviate
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"path"
"strings"
"sync"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
sdk "github.com/weaviate/weaviate-go-client/v5/weaviate"
)
func TestMoveKnowledgeIndicesDrainsBeyondOffsetLimitAndRetries(t *testing.T) {
const count = 10005
var mu sync.Mutex
remaining := make(map[string]bool, count)
for i := range count {
remaining[fmt.Sprintf("00000000-0000-0000-0000-%012d", i)] = true
}
failed := false
patches, queries := 0, 0
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
w.Header().Set("Content-Type", "application/json")
switch {
case r.URL.Path == "/v1/graphql":
queries++
var body struct {
Query string `json:"query"`
}
if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&body)) {
w.WriteHeader(400)
return
}
assert.Contains(t, body.Query, "knowledge_base_id")
assert.Contains(t, body.Query, "knowledge_id")
assert.Contains(t, body.Query, "source")
assert.Contains(t, body.Query, "doc")
if strings.Contains(body.Query, "offset") || strings.Contains(body.Query, "after") {
_ = json.NewEncoder(w).
Encode(map[string]any{"errors": []any{map[string]any{"message": "pagination unsupported"}}})
return
}
rows := make([]any, 0, 100)
for id := range remaining {
rows = append(rows, map[string]any{"_additional": map[string]any{"id": id}})
if len(rows) == 100 {
break
}
}
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"Get": map[string]any{"Move_3": rows}}})
case r.Method == http.MethodPatch:
id := path.Base(r.URL.Path)
assert.True(t, remaining[id], "only source-filtered IDs should be moved")
var body struct {
Properties map[string]any `json:"properties"`
Vector []float32 `json:"vector"`
}
if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&body)) {
w.WriteHeader(400)
return
}
assert.Equal(t, map[string]any{fieldKnowledgeBaseID: "target", fieldTagID: ""}, body.Properties)
assert.Empty(t, body.Vector, "metadata merge must preserve existing vectors")
if patches == 137 && !failed {
failed = true
w.WriteHeader(http.StatusServiceUnavailable)
return
}
patches++
delete(remaining, id)
w.WriteHeader(http.StatusNoContent)
default:
// SDK probes the server version before object writes.
if r.URL.Path == "/v1/meta" {
_, _ = w.Write([]byte(`{"version":"1.30.0"}`))
return
}
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
w.WriteHeader(http.StatusNotFound)
}
}))
defer server.Close()
client, err := sdk.NewClient(sdk.Config{Scheme: "http", Host: strings.TrimPrefix(server.URL, "http://")})
require.NoError(t, err)
repo := &weaviateRepository{client: client, collectionBaseName: "Move"}
require.Error(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""))
mu.Lock()
left := len(remaining)
mu.Unlock()
require.Equal(t, count-137, left)
require.NoError(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""))
mu.Lock()
defer mu.Unlock()
require.Empty(t, remaining)
require.Equal(t, count, patches)
require.Greater(t, queries, 100)
}
func TestMoveKnowledgeIndicesRejectsNoProgress(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch r.URL.Path {
case "/v1/graphql":
_, _ = w.Write(
[]byte(`{"data":{"Get":{"Move_3":[{"_additional":{"id":"00000000-0000-0000-0000-000000000001"}}]}}}`),
)
case "/v1/meta":
_, _ = w.Write([]byte(`{"version":"1.30.0"}`))
default:
w.WriteHeader(http.StatusNoContent)
}
}))
defer server.Close()
client, err := sdk.NewClient(sdk.Config{Scheme: "http", Host: strings.TrimPrefix(server.URL, "http://")})
require.NoError(t, err)
repo := &weaviateRepository{client: client, collectionBaseName: "Move"}
require.ErrorContains(
t,
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""),
"no progress",
)
}

View File

@ -512,27 +512,9 @@ func (s *agentShareService) TenantCanAccessKBViaSomeSharedAgent(ctx context.Cont
return false, err
}
for _, info := range list {
if info.Agent == nil {
continue
}
agent := info.Agent
if agent.TenantID != kb.TenantID {
continue
}
mode := agent.Config.KBSelectionMode
if mode == "none" {
continue
}
if mode == "all" {
if types.SharedAgentIncludesKB(info.Agent, kb) {
return true, nil
}
if mode == "selected" {
for _, id := range agent.Config.KnowledgeBases {
if id == kb.ID {
return true, nil
}
}
}
}
return false, nil
}

View File

@ -86,6 +86,9 @@ func (s *chunkService) GetRepository() interfaces.ChunkRepository {
// Returns:
// - error: Any error encountered during chunk creation
func (s *chunkService) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
if err := s.validateChunkWrites(ctx, chunks, true); err != nil {
return err
}
err := s.chunkRepository.CreateChunks(ctx, chunks)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
@ -217,6 +220,9 @@ func (s *chunkService) ListPagedChunksByKnowledgeID(ctx context.Context,
//
// This method handles the actual update logic for a chunk, including updating the vector database representation
func (s *chunkService) UpdateChunk(ctx context.Context, chunk *types.Chunk) error {
if err := s.validateChunkWrites(ctx, []*types.Chunk{chunk}, false); err != nil {
return err
}
logger.Infof(ctx, "Updating chunk, ID: %s, knowledge ID: %s", chunk.ID, chunk.KnowledgeID)
// Update the chunk in the repository
@ -235,6 +241,9 @@ func (s *chunkService) UpdateChunk(ctx context.Context, chunk *types.Chunk) erro
// UpdateChunks updates chunks in batch
func (s *chunkService) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
if err := s.validateChunkWrites(ctx, chunks, false); err != nil {
return err
}
if len(chunks) == 0 {
return nil
}
@ -262,6 +271,9 @@ func (s *chunkService) UpdateChunks(ctx context.Context, chunks []*types.Chunk)
// Returns:
// - error: Any error encountered during deletion
func (s *chunkService) DeleteChunk(ctx context.Context, id string) error {
if _, err := s.writableChunk(ctx, id); err != nil {
return err
}
tenantID := types.MustTenantIDFromContext(ctx)
err := s.chunkRepository.DeleteChunk(ctx, tenantID, id)
if err != nil {
@ -289,10 +301,15 @@ func (s *chunkService) DeleteChunks(ctx context.Context, ids []string) error {
logger.Info(ctx, "Start deleting chunks in batch")
logger.Infof(ctx, "Deleting %d chunks", len(ids))
checkedIDs, err := s.writableChunkIDs(ctx, ids)
if err != nil {
return err
}
ids = checkedIDs
tenantID := types.MustTenantIDFromContext(ctx)
logger.Infof(ctx, "Tenant ID: %d", tenantID)
err := s.chunkRepository.DeleteChunks(ctx, tenantID, ids)
err = s.chunkRepository.DeleteChunks(ctx, tenantID, ids)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
"chunk_ids": ids,
@ -314,6 +331,9 @@ func (s *chunkService) DeleteChunks(ctx context.Context, ids []string) error {
// Returns:
// - error: Any error encountered during bulk deletion
func (s *chunkService) DeleteChunksByKnowledgeID(ctx context.Context, knowledgeID string) error {
if _, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, []string{knowledgeID}); err != nil {
return err
}
logger.Info(ctx, "Start deleting all chunks by knowledge ID")
logger.Infof(ctx, "Knowledge ID: %s", knowledgeID)
@ -334,6 +354,9 @@ func (s *chunkService) DeleteChunksByKnowledgeID(ctx context.Context, knowledgeI
}
func (s *chunkService) DeleteByKnowledgeList(ctx context.Context, ids []string) error {
if _, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, ids); err != nil {
return err
}
logger.Info(ctx, "Start deleting all chunks by knowledge IDs")
logger.Infof(ctx, "Knowledge IDs: %v", ids)
@ -382,14 +405,16 @@ func (s *chunkService) ListChunkByParentID(
func (s *chunkService) UpdateDocumentChunk(
ctx context.Context, chunkID string, content *string, isEnabled *bool, expectedRevision *int,
) (*types.Chunk, error) {
tenantID := types.MustTenantIDFromContext(ctx)
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenantID, chunkID)
chunk, err := s.writableChunk(ctx, chunkID)
if err != nil {
return nil, err
}
if chunk.ChunkType != types.ChunkTypeText {
return nil, fmt.Errorf("only text chunks can be edited")
}
if err := s.validateDocumentChunkRelations(ctx, chunk); err != nil {
return nil, err
}
if expectedRevision != nil && *expectedRevision != chunk.ContentRevision {
return nil, ErrChunkRevisionConflict
}
@ -481,7 +506,7 @@ func (s *chunkService) UpdateDocumentChunk(
}
}
if bodyChanged || newEnabled != revision.IsEnabled {
knowledge, getErr := s.knowledgeRepo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
knowledge, getErr := s.knowledgeRepo.GetKnowledgeByID(ctx, chunk.TenantID, chunk.KnowledgeID)
if getErr == nil {
if err := enqueueSummaryRefresh(
ctx, s.knowledgeRepo, s.task, s.kbRepository, s.spanTracker, knowledge,
@ -700,7 +725,7 @@ func (s *chunkService) UpsertGeneratedQuestion(
if question == "" {
return nil, fmt.Errorf("question cannot be empty")
}
chunk, err := s.chunkRepository.GetChunkByID(ctx, types.MustTenantIDFromContext(ctx), chunkID)
chunk, err := s.writableChunk(ctx, chunkID)
if err != nil {
return nil, err
}
@ -756,7 +781,7 @@ func (s *chunkService) DeleteGeneratedQuestion(ctx context.Context, chunkID stri
tenantID := types.MustTenantIDFromContext(ctx)
// 1. Get the chunk
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenantID, chunkID)
chunk, err := s.writableChunk(ctx, chunkID)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
"chunk_id": chunkID,

View File

@ -3,11 +3,11 @@ package service
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
@ -49,7 +49,7 @@ type editableChunkKBRepo struct {
}
func (editableChunkKBRepo) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
return &types.KnowledgeBase{}, nil
return &types.KnowledgeBase{ID: "kb", TenantID: 1}, nil
}
type editableChunkKnowledgeRepo struct {
@ -57,7 +57,7 @@ type editableChunkKnowledgeRepo struct {
}
func (editableChunkKnowledgeRepo) GetKnowledgeByID(context.Context, uint64, string) (*types.Knowledge, error) {
return nil, errors.New("summary refresh not configured in unit test")
return &types.Knowledge{ID: "knowledge", TenantID: 1, KnowledgeBaseID: "kb"}, nil
}
type imageSyncChunkRepo struct {
@ -165,6 +165,17 @@ func TestUpdateDocumentChunkPreservesGeneratedQuestionsAcrossRevision(t *testing
kbRepository: editableChunkKBRepo{},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb",
TenantID: 1,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 1,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
newContent := "new body"
updated, err := service.UpdateDocumentChunk(ctx, "chunk", &newContent, nil, nil)

View File

@ -0,0 +1,143 @@
package service
import (
"context"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
)
func (s *chunkService) writableChunk(ctx context.Context, id string) (*types.Chunk, error) {
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return nil, err
}
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenant, id)
if err != nil {
return nil, err
}
if chunk == nil || chunk.ID != id || chunk.TenantID != tenant {
return nil, apperrors.NewNotFoundError("chunk not found")
}
knowledge, _, err := loadKnowledgeWrite(ctx, s.knowledgeRepo, s.kbRepository, chunk.KnowledgeID)
if err != nil {
return nil, err
}
if chunk.KnowledgeBaseID != knowledge.KnowledgeBaseID {
return nil, apperrors.NewForbiddenError("chunk does not belong to its knowledge base")
}
copyOfChunk := *chunk
return &copyOfChunk, nil
}
func sameChunkDocument(a, b *types.Chunk) bool {
return a != nil && b != nil && a.TenantID == b.TenantID &&
a.KnowledgeBaseID == b.KnowledgeBaseID && a.KnowledgeID == b.KnowledgeID
}
// Batch validation is completed before calling the repository. Supplied full
// rows cannot reparent an existing chunk to a different authorized document.
func (s *chunkService) validateChunkWrites(ctx context.Context, chunks []*types.Chunk, create bool) error {
ids := make([]string, 0, len(chunks))
for _, chunk := range chunks {
if chunk == nil || chunk.ID == "" {
return apperrors.NewBadRequestError("chunk ID cannot be empty")
}
ids = append(ids, chunk.KnowledgeID)
}
knowledge, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, ids)
if err != nil {
return err
}
parents := make(map[string]*types.Knowledge, len(knowledge))
for _, row := range knowledge {
parents[row.ID] = row
}
storedByID := make(map[string]*types.Chunk)
if !create && len(chunks) > 0 {
chunkIDs := make([]string, 0, len(chunks))
for _, chunk := range chunks {
chunkIDs = append(chunkIDs, chunk.ID)
}
stored, err := s.chunkRepository.ListChunksByID(ctx, knowledge[0].TenantID, chunkIDs)
if err != nil {
return err
}
for _, chunk := range stored {
if chunk != nil {
storedByID[chunk.ID] = chunk
}
}
}
for _, chunk := range chunks {
parent := parents[chunk.KnowledgeID]
if parent == nil || parent.TenantID != chunk.TenantID || parent.KnowledgeBaseID != chunk.KnowledgeBaseID {
return apperrors.NewForbiddenError("chunk does not belong to its knowledge document")
}
if !create {
stored := storedByID[chunk.ID]
if stored == nil || stored.ID != chunk.ID || !sameChunkDocument(stored, chunk) {
return apperrors.NewForbiddenError("chunk ownership cannot be changed")
}
}
}
return nil
}
func (s *chunkService) writableChunkIDs(ctx context.Context, ids []string) ([]string, error) {
ids, err := writeResourceIDs(ids)
if err != nil || len(ids) == 0 {
return ids, err
}
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return nil, err
}
chunks, err := s.chunkRepository.ListChunksByID(ctx, tenant, ids)
if err != nil {
return nil, err
}
byID := make(map[string]*types.Chunk, len(chunks))
for _, chunk := range chunks {
if chunk != nil {
byID[chunk.ID] = chunk
}
}
ordered := make([]*types.Chunk, 0, len(ids))
for _, id := range ids {
if byID[id] == nil {
return nil, apperrors.NewNotFoundError("chunk not found")
}
ordered = append(ordered, byID[id])
}
// Rows were loaded from storage already; only their parent grants remain.
return ids, s.validateChunkWrites(ctx, ordered, true)
}
// Editing a text chunk may also rewrite its parent and image children. Validate
// those persisted relationships before saving the first revision.
func (s *chunkService) validateDocumentChunkRelations(ctx context.Context, chunk *types.Chunk) error {
parents := []*types.Chunk{chunk}
if chunk.ParentChunkID != "" {
parent, err := s.chunkRepository.GetChunkByID(ctx, chunk.TenantID, chunk.ParentChunkID)
if err != nil {
return err
}
if parent == nil || parent.ID != chunk.ParentChunkID || !sameChunkDocument(chunk, parent) {
return apperrors.NewForbiddenError("parent chunk does not belong to its document")
}
parents = append(parents, parent)
}
for _, parent := range parents {
children, err := s.chunkRepository.ListChunkByParentID(ctx, chunk.TenantID, parent.ID)
if err != nil {
return err
}
for _, child := range children {
if !sameChunkDocument(chunk, child) || child.ParentChunkID != parent.ID {
return apperrors.NewForbiddenError("child chunk does not belong to its document")
}
}
}
return nil
}

View File

@ -8,6 +8,7 @@ import (
"time"
"github.com/Tencent/WeKnora/internal/agent/tools"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
@ -579,7 +580,7 @@ func (s *customAgentService) getSuggestedQuestions(
resolvedTags := resolvedSuggestionTagScopes{}
if len(scopeTagIDs) > 0 {
var err error
resolvedTags, err = s.resolveSuggestionTagScopes(ctx, tenantID, tagScopes)
resolvedTags, err = s.resolveSuggestionTagScopes(ctx, tagScopes)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
"agent_id": agentID,
@ -673,7 +674,7 @@ func (s *customAgentService) getSuggestedQuestions(
// querying a KB shared from tenant B would hit `tenant_id = A` and get zero
// rows back — the symptom is "suggested questions never appear for shared KBs".
scopeKBIDs := mergeUniqueStrings(queryKBIDs, resolvedTags.KnowledgeBaseIDs)
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, tenantID, scopeKBIDs)
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, scopeKBIDs)
// Always keep the caller's tenant in the iteration so knowledge_ids-only
// requests (no kbIDs) still execute one query under the caller's tenant.
if len(scopeKBIDs) == 0 {
@ -834,7 +835,6 @@ type resolvedSuggestionTagScopes struct {
// source tenant that owns the tag and chunk rows.
func (s *customAgentService) resolveSuggestionTagScopes(
ctx context.Context,
callerTenantID uint64,
tagScopes []types.TagScope,
) (resolvedSuggestionTagScopes, error) {
result := resolvedSuggestionTagScopes{TagIDsByTenant: make(map[uint64][]string)}
@ -862,7 +862,7 @@ func (s *customAgentService) resolveSuggestionTagScopes(
for kbID := range byKB {
kbIDs = append(kbIDs, kbID)
}
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, callerTenantID, kbIDs)
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, kbIDs)
for tenantID, groupKBIDs := range kbGroups {
for _, kbID := range groupKBIDs {
requested := mergeUniqueStrings(nil, byKB[kbID])
@ -1083,9 +1083,9 @@ func wikiSuggestionFromPage(page *types.WikiPage, locale string) string {
// kbIDs is empty.
func (s *customAgentService) groupKBIDsByEffectiveTenant(
ctx context.Context,
callerTenantID uint64,
kbIDs []string,
) map[uint64][]string {
callerTenantID := types.CallerFromContext(ctx).TenantID
out := make(map[uint64][]string)
if len(kbIDs) == 0 {
return out
@ -1106,20 +1106,13 @@ func (s *customAgentService) groupKBIDsByEffectiveTenant(
kbByID[kb.ID] = kb
}
}
callerRole := types.TenantRoleFromContext(ctx)
permissions := access.NewKBPermissions(ctx, s.kbShareService)
for _, kbID := range kbIDs {
kb := kbByID[kbID]
if kb == nil {
continue
}
if kb.TenantID == callerTenantID {
out[callerTenantID] = append(out[callerTenantID], kbID)
continue
}
if s.kbShareService == nil {
continue
}
ok, err := s.kbShareService.HasTenantKBPermission(ctx, kbID, callerTenantID, callerRole, types.OrgRoleViewer)
ok, err := permissions.Check(kbID, kb.TenantID, types.OrgRoleViewer)
if err != nil || !ok {
continue
}

View File

@ -66,14 +66,13 @@ type suggestionKBShareService struct {
allowed map[string]bool
}
func (s *suggestionKBShareService) HasTenantKBPermission(
func (s *suggestionKBShareService) CheckTenantKBPermission(
_ context.Context,
kbID string,
_ uint64,
_ types.TenantRole,
_ types.OrgMemberRole,
) (bool, error) {
return s.allowed[kbID], nil
) (types.OrgMemberRole, bool, error) {
return types.OrgRoleViewer, s.allowed[kbID], nil
}
func TestResolveSuggestionTagScopes_UsesSourceTenantForSharedKB(t *testing.T) {
@ -97,8 +96,7 @@ func TestResolveSuggestionTagScopes_UsesSourceTenantForSharedKB(t *testing.T) {
}
resolved, err := svc.resolveSuggestionTagScopes(
context.Background(),
callerTenant,
context.WithValue(context.Background(), types.TenantIDContextKey, callerTenant),
[]types.TagScope{{KnowledgeBaseID: kbID, TagIDs: []string{tagID}}},
)
require.NoError(t, err)

View File

@ -13,6 +13,7 @@ import (
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/datasource"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
@ -513,8 +514,10 @@ func (s *DataSourceService) ManualSync(ctx context.Context, dsID string) (*types
logger.Infof(ctx, "sync task enqueued: ds=%s syncLog=%s", dsID, syncLog.ID)
recordKBActivity(ctx, s.audit, ds.TenantID, ds.KnowledgeBaseID, types.AuditActionDataSourceSyncStarted,
"data_source", ds.ID, types.AuditOutcomeAccepted,
map[string]any{"name": ds.Name, "type": ds.Type, "sync_log_id": syncLog.ID,
"task_id": info.ID, "trigger": "manual", "processing_status": "pending"})
map[string]any{
"name": ds.Name, "type": ds.Type, "sync_log_id": syncLog.ID,
"task_id": info.ID, "trigger": "manual", "processing_status": "pending",
})
return syncLog, nil
}
@ -627,6 +630,10 @@ func (s *DataSourceService) ProcessSync(ctx context.Context, task *asynq.Task) e
return nil
}
ctx, err = access.WithKBTaskWrite(ctx, kb, ds.TenantID)
if err != nil {
return fmt.Errorf("%w: data source KB does not belong to its tenant", asynq.SkipRetry)
}
wasPaused := ds.Status == types.DataSourceStatusPaused
// Get connector
@ -719,7 +726,7 @@ func (s *DataSourceService) ProcessSync(ctx context.Context, task *asynq.Task) e
}
// Process fetched items and write to knowledge base
var result = &types.SyncResult{
result := &types.SyncResult{
Total: len(items),
}

View File

@ -254,9 +254,15 @@ func (r *processSyncTenantRepo) GetTenantByID(context.Context, uint64) (*types.T
type processSyncTagService struct {
interfaces.KnowledgeTagService
ctx context.Context
}
func (*processSyncTagService) FindOrCreateTagByName(context.Context, string, string) (*types.KnowledgeTag, error) {
func (s *processSyncTagService) FindOrCreateTagByName(
ctx context.Context,
_ string,
_ string,
) (*types.KnowledgeTag, error) {
s.ctx = ctx
return nil, nil
}

View File

@ -0,0 +1,660 @@
package service
import (
"context"
"encoding/json"
"errors"
"path/filepath"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
type documentKBLookup struct {
interfaces.KnowledgeBaseService
values map[string]*types.KnowledgeBase
err error
}
func (r *documentKBLookup) GetKnowledgeBaseByID(_ context.Context, id string) (*types.KnowledgeBase, error) {
if r.err != nil {
return nil, r.err
}
if r.values[id] == nil {
return nil, repository.ErrKnowledgeBaseNotFound
}
return r.values[id], nil
}
type documentKBRepo struct {
interfaces.KnowledgeBaseRepository
lookup *documentKBLookup
}
func (r documentKBRepo) GetKnowledgeBaseByID(ctx context.Context, id string) (*types.KnowledgeBase, error) {
return r.lookup.GetKnowledgeBaseByID(ctx, id)
}
type documentKnowledgeSpy struct {
interfaces.KnowledgeRepository
writes int
}
func (r *documentKnowledgeSpy) UpdateKnowledge(ctx context.Context, k *types.Knowledge) error {
r.writes++
return r.KnowledgeRepository.UpdateKnowledge(ctx, k)
}
func (r *documentKnowledgeSpy) SetKnowledgeTags(ctx context.Context, id string, tags []string) error {
r.writes++
return r.KnowledgeRepository.SetKnowledgeTags(ctx, id, tags)
}
func (r *documentKnowledgeSpy) DeleteKnowledgeList(ctx context.Context, tenant uint64, ids []string) error {
r.writes++
return r.KnowledgeRepository.DeleteKnowledgeList(ctx, tenant, ids)
}
type documentChunkSpy struct {
interfaces.ChunkRepository
writes int
imageErr error
}
func (r *documentChunkSpy) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
r.writes++
return r.ChunkRepository.CreateChunks(ctx, chunks)
}
func (r *documentChunkSpy) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
r.writes++
return r.ChunkRepository.UpdateChunks(ctx, chunks)
}
func (r *documentChunkSpy) UpdateChunk(ctx context.Context, chunk *types.Chunk) error {
r.writes++
return r.ChunkRepository.UpdateChunk(ctx, chunk)
}
func (r *documentChunkSpy) DeleteChunks(ctx context.Context, tenant uint64, ids []string) error {
r.writes++
return r.ChunkRepository.DeleteChunks(ctx, tenant, ids)
}
func (r *documentChunkSpy) DeleteChunk(ctx context.Context, tenant uint64, id string) error {
r.writes++
return r.ChunkRepository.DeleteChunk(ctx, tenant, id)
}
func (r *documentChunkSpy) DeleteByKnowledgeList(ctx context.Context, tenant uint64, ids []string) error {
r.writes++
return r.ChunkRepository.DeleteByKnowledgeList(ctx, tenant, ids)
}
func (r *documentChunkSpy) DeleteChunksByKnowledgeID(ctx context.Context, tenant uint64, id string) error {
r.writes++
return r.ChunkRepository.DeleteChunksByKnowledgeID(ctx, tenant, id)
}
func (r *documentChunkSpy) SaveChunkRevision(
ctx context.Context,
chunk *types.Chunk,
rev *types.ChunkRevision,
expected int,
) error {
r.writes++
return r.ChunkRepository.SaveChunkRevision(ctx, chunk, rev, expected)
}
func (r *documentChunkSpy) ListImageInfoByKnowledgeIDs(
ctx context.Context,
tenant uint64,
ids []string,
) ([]interfaces.ChunkImageInfo, error) {
if r.imageErr != nil {
return nil, r.imageErr
}
return r.ChunkRepository.ListImageInfoByKnowledgeIDs(ctx, tenant, ids)
}
type documentGraphSpy struct {
interfaces.RetrieveGraphRepository
err error
calls int
}
func (r *documentGraphSpy) DelGraph(context.Context, []types.NameSpace) error {
r.calls++
return r.err
}
type documentTenantSpy struct {
interfaces.TenantRepository
adjustments []int64
}
func (r *documentTenantSpy) GetTenantByID(_ context.Context, id uint64) (*types.Tenant, error) {
return &types.Tenant{ID: id}, nil
}
func (r *documentTenantSpy) AdjustStorageUsed(_ context.Context, _ uint64, delta int64) error {
r.adjustments = append(r.adjustments, delta)
return nil
}
type documentFileSpy struct {
interfaces.FileService
deleted []string
beforeDelete func()
}
func (r *documentFileSpy) DeleteFile(_ context.Context, path string) error {
if r.beforeDelete != nil {
r.beforeDelete()
}
r.deleted = append(r.deleted, path)
return nil
}
type documentWriteFixture struct {
svc *knowledgeService
chunks *chunkService
repo *documentKnowledgeSpy
chunkRepo *documentChunkSpy
kbs *documentKBLookup
graph *documentGraphSpy
files *documentFileSpy
tenants *documentTenantSpy
db *gorm.DB
ctx context.Context
}
func newDocumentWriteFixture(t *testing.T) *documentWriteFixture {
t.Helper()
db, err := gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), "document.db")), &gorm.Config{})
require.NoError(t, err)
sqlDB, err := db.DB()
require.NoError(t, err)
sqlDB.SetMaxOpenConns(1)
t.Cleanup(func() { require.NoError(t, sqlDB.Close()) })
require.NoError(
t,
db.AutoMigrate(
&types.Knowledge{},
&types.Chunk{},
&types.KnowledgeTag{},
&types.KnowledgeTagRelation{},
&types.ChunkRevision{},
),
)
repo := &documentKnowledgeSpy{KnowledgeRepository: repository.NewKnowledgeRepository(db)}
chunkRepo := &documentChunkSpy{ChunkRepository: repository.NewChunkRepository(db)}
kbs := &documentKBLookup{values: map[string]*types.KnowledgeBase{
"kb": {ID: "kb", TenantID: 7}, "other": {ID: "other", TenantID: 7},
}}
for _, row := range []*types.Knowledge{
{
ID: "doc",
TenantID: 7,
KnowledgeBaseID: "kb",
Title: "original",
Type: types.KnowledgeTypeManual,
ParseStatus: types.ManualKnowledgeStatusDraft,
StorageSize: 5,
},
{
ID: "other-doc",
TenantID: 7,
KnowledgeBaseID: "other",
Title: "other",
ParseStatus: types.ParseStatusCompleted,
},
} {
require.NoError(t, db.Create(row).Error)
}
for _, row := range []*types.Chunk{
{
ID: "chunk",
TenantID: 7,
KnowledgeID: "doc",
KnowledgeBaseID: "kb",
ChunkType: types.ChunkTypeText,
Content: "original",
IndexStatus: "ready",
},
{
ID: "other-chunk",
TenantID: 7,
KnowledgeID: "other-doc",
KnowledgeBaseID: "other",
ChunkType: types.ChunkTypeText,
},
} {
require.NoError(t, db.Create(row).Error)
}
require.NoError(t, db.Create(&types.KnowledgeTag{ID: "tag", TenantID: 7, KnowledgeBaseID: "kb", Name: "tag"}).Error)
graph, files, tenants := &documentGraphSpy{}, &documentFileSpy{}, &documentTenantSpy{}
chunks := &chunkService{chunkRepository: chunkRepo, knowledgeRepo: repo, kbRepository: documentKBRepo{lookup: kbs}}
svc := &knowledgeService{
repo: repo,
chunkRepo: chunkRepo,
chunkService: chunks,
kbService: kbs,
graphEngine: graph,
fileSvc: files,
tenantRepo: tenants,
tagRepo: repository.NewKnowledgeTagRepository(db),
}
ctx := types.WithCaller(
context.Background(),
types.Caller{TenantID: 1, UserID: "editor", Role: types.TenantRoleAdmin},
)
grant := &access.KBAccess{
KnowledgeBase: kbs.values["kb"],
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}
ctx = grant.Context(ctx)
ctx = context.WithValue(ctx, types.TenantInfoContextKey, &types.Tenant{ID: 7})
return &documentWriteFixture{
svc: svc,
chunks: chunks,
repo: repo,
chunkRepo: chunkRepo,
kbs: kbs,
graph: graph,
files: files,
tenants: tenants,
db: db,
ctx: ctx,
}
}
func requireForbiddenWrite(t *testing.T, err error) {
t.Helper()
var appErr *apperrors.AppError
require.ErrorAs(t, err, &appErr)
require.Equal(t, 403, appErr.HTTPCode)
}
func (f *documentWriteFixture) requireNoWrites(t *testing.T) {
t.Helper()
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
require.Empty(t, f.files.deleted)
}
func TestDocumentAndChunkWritesRequireOperationGrant(t *testing.T) {
f := newDocumentWriteFixture(t)
ctx := types.WithExecutionTenant(
types.WithCaller(context.Background(), types.Caller{TenantID: 7, Role: types.TenantRoleAdmin}),
7,
)
// Even an owner read resolution projects Admin without authorizing writes.
grant, err := access.ResolveKB(
ctx,
access.KBRequest{Caller: types.CallerFromContext(ctx)},
f.kbs.values["kb"],
types.OrgRoleViewer,
nil,
nil,
)
require.NoError(t, err)
ctx = grant.Context(ctx)
row, _ := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
ops := map[string]func() error{
"set tags": func() error { return f.svc.SetKnowledgeTags(ctx, "doc", nil) },
"move folder": func() error {
_,
err := f.svc.MoveKnowledgeToFolder(ctx,
"kb",
[]string{"doc"},
"folder")
return err
},
"rename folder": func() error {
_,
err := f.svc.RenameKnowledgeFolder(ctx,
"kb",
"folder",
"next")
return err
},
"metadata": func() error { return f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "new"}) },
"manual": func() error {
_, err := f.svc.UpdateManualKnowledge(
ctx,
"doc",
&types.ManualKnowledgePayload{Content: "body", Status: "draft"},
)
return err
},
"tag": func() error { return f.svc.UpdateKnowledgeTag(ctx, "doc", []string{"tag"}) },
"batch tags": func() error {
return f.svc.UpdateKnowledgeTagBatch(ctx,
"",
map[string][]string{"doc": {"tag"}})
},
"delete document": func() error { return f.svc.DeleteKnowledge(ctx, "doc") },
"delete documents": func() error { return f.svc.DeleteKnowledgeList(ctx, []string{"doc"}) },
"create chunks": func() error { return f.chunks.CreateChunks(ctx, []*types.Chunk{row}) },
"update chunk": func() error { return f.chunks.UpdateChunk(ctx, row) },
"update chunks": func() error { return f.chunks.UpdateChunks(ctx, []*types.Chunk{row}) },
"edit chunk": func() error {
body := "new"
_, err := f.chunks.UpdateDocumentChunk(ctx, "chunk", &body, nil, nil)
return err
},
"upsert question": func() error {
_,
err := f.chunks.UpsertGeneratedQuestion(ctx,
"chunk",
"",
"new question")
return err
},
"delete question": func() error { return f.chunks.DeleteGeneratedQuestion(ctx, "chunk", "q") },
"regenerate questions": func() error {
_,
err := f.svc.RegenerateChunkQuestions(ctx,
"chunk")
return err
},
"update image": func() error { return f.svc.UpdateImageInfo(ctx, "doc", "chunk", "[]") },
"delete chunk": func() error { return f.chunks.DeleteChunk(ctx, "chunk") },
"delete chunks": func() error { return f.chunks.DeleteChunks(ctx, []string{"chunk"}) },
"delete document chunks": func() error { return f.chunks.DeleteChunksByKnowledgeID(ctx, "doc") },
"delete batch document chunks": func() error { return f.chunks.DeleteByKnowledgeList(ctx, []string{"doc"}) },
}
for name, op := range ops {
t.Run(name, func(t *testing.T) { requireForbiddenWrite(t, op()); f.requireNoWrites(t) })
}
require.NoError(t, f.svc.UpdateKnowledge(f.ctx, &types.Knowledge{ID: "doc", Title: "shared edit"}))
stored, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "shared edit", stored.Title)
}
func TestDocumentBatchPreflightDoesNotPartiallyMutate(t *testing.T) {
for _, invalid := range []string{"other-doc", "missing", ""} {
t.Run(invalid, func(t *testing.T) {
f := newDocumentWriteFixture(t)
require.Error(t, f.svc.DeleteKnowledgeList(f.ctx, []string{"doc", invalid}))
f.requireNoWrites(t)
require.Error(
t,
f.svc.UpdateKnowledgeTagBatch(f.ctx, "", map[string][]string{"doc": {"tag"}, invalid: nil}),
)
f.requireNoWrites(t)
_, err := f.svc.MoveKnowledgeToFolder(f.ctx, "kb", []string{"doc", invalid}, "next")
require.Error(t, err)
f.requireNoWrites(t)
require.Error(t, f.chunks.DeleteByKnowledgeList(f.ctx, []string{"doc", invalid}))
f.requireNoWrites(t)
})
}
for _, invalid := range []string{"other-chunk", "missing", ""} {
t.Run("chunks/"+invalid, func(t *testing.T) {
f := newDocumentWriteFixture(t)
require.Error(t, f.chunks.DeleteChunks(f.ctx, []string{"chunk", invalid}))
f.requireNoWrites(t)
good, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
require.NoError(t, err)
forged := *good
forged.ID = invalid
require.Error(t, f.chunks.UpdateChunks(f.ctx, []*types.Chunk{good, &forged}))
f.requireNoWrites(t)
})
}
}
func TestDeletePreflightResolvesKBAndImagesBeforeChangingStatus(t *testing.T) {
for _, stage := range []string{"KB lookup", "KB owner", "image lookup"} {
t.Run(stage, func(t *testing.T) {
f := newDocumentWriteFixture(t)
switch stage {
case "KB lookup":
f.kbs.err = errors.New("database unavailable")
case "KB owner":
f.kbs.values["kb"].TenantID = 9
case "image lookup":
f.chunkRepo.imageErr = errors.New("database unavailable")
}
require.Error(t, f.svc.DeleteKnowledge(f.ctx, "doc"))
f.requireNoWrites(t)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, types.ManualKnowledgeStatusDraft, row.ParseStatus)
})
}
}
func deleteDocumentTask(t *testing.T, payload types.KnowledgeListDeletePayload) *asynq.Task {
t.Helper()
data, err := json.Marshal(payload)
require.NoError(t, err)
return asynq.NewTask(types.TypeKnowledgeListDelete, data)
}
func TestDeleteTaskScopeRejectsMovedDocumentsBeforeAnyMutation(t *testing.T) {
f := newDocumentWriteFixture(t)
task := deleteDocumentTask(
t,
types.KnowledgeListDeletePayload{
TenantID: 7,
KnowledgeBaseID: "kb",
KnowledgeIDs: []string{"doc", "other-doc"},
},
)
require.ErrorIs(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task), asynq.SkipRetry)
f.requireNoWrites(t)
task = deleteDocumentTask(
t,
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeIDs: []string{"doc", "other-doc"}},
)
require.ErrorIs(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task), asynq.SkipRetry)
f.requireNoWrites(t)
}
func TestDeleteTaskRetriesAfterPartialCleanupAndAlreadyDeletedRows(t *testing.T) {
f := newDocumentWriteFixture(t)
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("file_path", "local://doc").Error)
task := deleteDocumentTask(
t,
types.KnowledgeListDeletePayload{
TenantID: 7,
KnowledgeBaseID: "kb",
KnowledgeIDs: []string{"doc", "doc", "already-gone"},
},
)
f.graph.err = errors.New("graph unavailable")
err := f.svc.ProcessKnowledgeListDelete(context.Background(), task)
require.ErrorContains(t, err, "graph unavailable")
require.NotErrorIs(t, err, asynq.SkipRetry)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, types.ParseStatusDeleting, row.ParseStatus)
require.Empty(t, f.files.deleted)
require.Empty(t, f.tenants.adjustments)
f.graph.err = nil
f.files.beforeDelete = func() { _, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc"); require.Error(t, err) }
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
require.Equal(t, []string{"local://doc"}, f.files.deleted)
require.Equal(t, []int64{-5}, f.tenants.adjustments)
// A completed task does not need live KB metadata or current user grants.
delete(f.kbs.values, "kb")
writes, calls := f.repo.writes, f.graph.calls
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
require.Equal(t, writes, f.repo.writes)
require.Equal(t, calls, f.graph.calls)
}
func TestKnowledgeCleanupScopeCannotAuthorizeGeneralWritesOrExtraIDs(t *testing.T) {
f := newDocumentWriteFixture(t)
ctx := withKnowledgeCleanup(context.Background(), 7, map[string]string{"doc": "kb"})
require.Zero(t, types.CallerFromContext(ctx).TenantID)
requireForbiddenWrite(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "no"}))
requireForbiddenWrite(t, f.svc.DeleteKnowledgeList(ctx, []string{"doc", "other-doc"}))
requireForbiddenWrite(
t,
f.svc.DeleteKnowledgeList(types.WithCaller(ctx, types.Caller{TenantID: 7}), []string{"doc"}),
)
f.requireNoWrites(t)
require.NoError(t, f.svc.DeleteKnowledge(ctx, "doc"))
}
func TestReferencedKnowledgeCleanupPinsItsIndependentKB(t *testing.T) {
f := newDocumentWriteFixture(t)
requireForbiddenWrite(t, deleteReferencedKnowledge(f.ctx, f.svc, "kb", []string{"doc", "other-doc"}))
f.requireNoWrites(t)
// Session cleanup authority comes from exact persisted references and can
// continue even when the caller has no permission to edit the history KB.
ctx := types.WithExecutionTenant(types.WithCaller(context.Background(), types.Caller{TenantID: 7}), 7)
require.NoError(t, deleteReferencedKnowledge(ctx, f.svc, "kb", []string{"doc", "already-gone"}))
}
func TestChunkEditRejectsCrossDocumentParentBeforeSavingRevision(t *testing.T) {
f := newDocumentWriteFixture(t)
require.NoError(
t,
f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("parent_chunk_id", "other-chunk").Error,
)
body := "edit"
_, err := f.chunks.UpdateDocumentChunk(f.ctx, "chunk", &body, nil, nil)
requireForbiddenWrite(t, err)
f.requireNoWrites(t)
}
func TestChunkEditRejectsCrossDocumentChildBeforeSavingRevision(t *testing.T) {
f := newDocumentWriteFixture(t)
require.NoError(
t,
f.db.Model(&types.Chunk{}).Where("id = ?", "other-chunk").Update("parent_chunk_id", "chunk").Error,
)
body := "edit"
_, err := f.chunks.UpdateDocumentChunk(f.ctx, "chunk", &body, nil, nil)
requireForbiddenWrite(t, err)
f.requireNoWrites(t)
}
func TestChunkBatchWriteAllowsOnlyStoredOwnership(t *testing.T) {
f := newDocumentWriteFixture(t)
row, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
require.NoError(t, err)
row.Content = "updated"
require.NoError(t, f.chunks.UpdateChunks(f.ctx, []*types.Chunk{row}))
stored, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
require.NoError(t, err)
require.Equal(t, "updated", stored.Content)
require.NoError(t, f.chunks.DeleteChunks(f.ctx, []string{"chunk", "chunk"}))
_, err = f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
require.ErrorIs(t, err, repository.ErrChunkNotFound)
}
func TestDocumentWriteTaskAndAPIKeyScopesStayNarrow(t *testing.T) {
f := newDocumentWriteFixture(t)
ctx, err := access.WithKBTaskWrite(context.Background(), f.kbs.values["kb"], 7)
require.NoError(t, err)
require.NoError(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "worker edit"}))
requireForbiddenWrite(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "other-doc", Title: "no"}))
scope := types.TenantAPIKeyScope{
KnowledgeBaseIDs: types.StringArray{"kb"},
Capabilities: types.StringArray{string(types.APIKeyCapabilityRetrieve)},
}
requireForbiddenWrite(
t,
f.svc.UpdateKnowledge(types.WithTenantAPIKeyScope(f.ctx, scope), &types.Knowledge{ID: "doc", Title: "no"}),
)
require.Zero(t, types.CallerFromContext(ctx).TenantID)
}
func TestLegacySingleKBDeleteTaskRemainsExecutable(t *testing.T) {
f := newDocumentWriteFixture(t)
task := deleteDocumentTask(
t,
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeIDs: []string{"doc", "already-gone"}},
)
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
_, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.Error(t, err)
}
func TestMultiKBDeleteRequiresAndAcceptsEveryKBGrant(t *testing.T) {
f := newDocumentWriteFixture(t)
grant := &access.KBAccess{
KnowledgeBase: f.kbs.values["other"],
Caller: types.CallerFromContext(f.ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}
require.NoError(t, f.svc.DeleteKnowledgeList(grant.Context(f.ctx), []string{"doc", "other-doc"}))
rows, err := f.repo.GetKnowledgeBatch(f.ctx, 7, []string{"doc", "other-doc"})
require.NoError(t, err)
require.Empty(t, rows)
}
func TestDeleteCheckpointRejectsChangedOrDeletedDocument(t *testing.T) {
for _, action := range []string{"moved", "deleted"} {
t.Run(action, func(t *testing.T) {
f := newDocumentWriteFixture(t)
plan, err := f.svc.planKnowledgeDelete(f.ctx, []string{"doc"})
require.NoError(t, err)
if action == "moved" {
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error,
)
} else {
require.NoError(t, f.db.Where("id = ?", "doc").Delete(&types.Knowledge{}).Error)
}
require.Error(t, f.svc.executeKnowledgeDelete(plan, true))
require.Zero(t, f.graph.calls)
require.Zero(t, f.chunkRepo.writes)
require.Empty(t, f.files.deleted)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
if action == "moved" {
require.NoError(t, err)
require.Equal(t, "other", row.KnowledgeBaseID)
require.Equal(t, types.ManualKnowledgeStatusDraft, row.ParseStatus)
} else {
require.Error(t, err)
}
})
}
}
func TestDocumentWritesRejectMoveInProgress(t *testing.T) {
f := newDocumentWriteFixture(t)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("metadata", row.Metadata).Error)
for _, op := range []func() error{
func() error { return f.svc.DeleteKnowledge(f.ctx, "doc") },
func() error { return f.svc.UpdateKnowledge(f.ctx, &types.Knowledge{ID: "doc", Title: "changed"}) },
func() error { return f.chunks.DeleteChunk(f.ctx, "chunk") },
func() error { return f.svc.DeleteKnowledgeList(f.ctx, []string{"doc"}) },
} {
var appErr *apperrors.AppError
require.ErrorAs(t, op(), &appErr)
require.Equal(t, 409, appErr.HTTPCode)
}
f.requireNoWrites(t)
}

View File

@ -363,7 +363,10 @@ func (e *EvaluationService) EvalDataset(ctx context.Context, detail *types.Evalu
// Setup cleanup of temporary resources
defer func() {
logger.Infof(ctx, "Cleaning up resources - deleting knowledge: %s", knowledge.ID)
if err := e.knowledgeService.DeleteKnowledge(ctx, knowledge.ID); err != nil {
if err := deleteReferencedKnowledge(ctx,
e.knowledgeService,
knowledgeBaseID,
[]string{knowledge.ID}); err != nil {
logger.Errorf(ctx, "Failed to delete knowledge: %v, knowledge ID: %s", err, knowledge.ID)
}

View File

@ -10,6 +10,7 @@ import (
"time"
"github.com/Tencent/WeKnora/internal/agent/tools"
"github.com/Tencent/WeKnora/internal/application/access"
chatpipeline "github.com/Tencent/WeKnora/internal/application/service/chat_pipeline"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
"github.com/Tencent/WeKnora/internal/config"
@ -468,7 +469,7 @@ func (s *DataTableSummaryService) Handle(ctx context.Context, t *asynq.Task) err
ctx = logger.WithRequestID(ctx, uuid.New().String())
ctx = logger.WithField(ctx, "knowledge", payload.KnowledgeID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
logger.Infof(ctx, "Processing table extraction for knowledge: %s", payload.KnowledgeID)
@ -478,6 +479,12 @@ func (s *DataTableSummaryService) Handle(ctx context.Context, t *asynq.Task) err
return err
}
ctx, err = access.WithKBTaskWrite(ctx, resources.knowledgeBase, payload.TenantID)
if err != nil {
return err
}
ctx = context.WithValue(ctx, types.TenantInfoContextKey, resources.tenant)
// 3. 加载表格数据并生成摘要
chunks, err := s.processTableData(ctx, resources)
if err != nil {
@ -508,12 +515,20 @@ type extractionResources struct {
// 思路:集中加载所有依赖,统一错误处理,避免分散的资源获取逻辑
func (s *DataTableSummaryService) prepareResources(ctx context.Context, payload DataTableSummaryPayload) (*extractionResources, error) {
// 获取并验证知识文件
knowledge, err := s.knowledgeService.GetKnowledgeByID(ctx, payload.KnowledgeID)
knowledge, err := s.knowledgeService.GetRepository().GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
if err != nil {
logger.Errorf(ctx, "failed to get knowledge: %v", err)
return nil, err
}
if knowledge == nil || knowledge.ID != payload.KnowledgeID || knowledge.TenantID != payload.TenantID {
return nil, fmt.Errorf("invalid table summary knowledge scope")
}
kb, err := knowledgeWriteKB(ctx, s.knowledgeBaseService, knowledge)
if err != nil {
return nil, err
}
// 验证文件类型
fileType := strings.ToLower(knowledge.FileType)
if fileType != "csv" && fileType != "xlsx" && fileType != "xls" {
@ -542,13 +557,6 @@ func (s *DataTableSummaryService) prepareResources(ctx context.Context, payload
return nil, err
}
// Load the KB to discover its VectorStoreID binding so the factory can
// route to the bound store (or fall back to tenant engines if unbound).
kb, err := s.knowledgeBaseService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
if err != nil {
logger.Errorf(ctx, "failed to get knowledge base for vector store lookup: %v", err)
return nil, err
}
var vectorStoreID *string
if kb != nil {
vectorStoreID = kb.VectorStoreID
@ -774,12 +782,12 @@ func (s *DataTableSummaryService) cleanupOnFailure(ctx context.Context, resource
logger.Warnf(ctx, "Starting cleanup due to failure: %v", indexErr)
// 1. 更新知识状态为失败
resources.knowledge.ParseStatus = types.ParseStatusFailed
resources.knowledge.ErrorMessage = indexErr.Error()
if err := s.knowledgeService.UpdateKnowledge(ctx, resources.knowledge); err != nil {
logger.Errorf(ctx, "Failed to update knowledge status: %v", err)
} else {
logger.Infof(ctx, "Updated knowledge %s status to failed", resources.knowledge.ID)
before, after := *resources.knowledge, *resources.knowledge
after.ParseStatus = types.ParseStatusFailed
after.ErrorMessage = indexErr.Error()
if err := s.knowledgeService.GetRepository().UpdateKnowledgeForTransfer(ctx, &before, &after); err != nil {
logger.Warnf(ctx, "Table summary cleanup skipped after knowledge changed: %v", err)
return
}
// 提取chunk IDs
@ -790,7 +798,7 @@ func (s *DataTableSummaryService) cleanupOnFailure(ctx context.Context, resource
// 删除已创建的chunks
if len(chunkIDs) > 0 {
if err := s.chunkService.DeleteChunks(ctx, chunkIDs); err != nil {
if err := s.chunkService.GetRepository().DeleteChunks(ctx, resources.knowledge.TenantID, chunkIDs); err != nil {
logger.Errorf(ctx, "Failed to delete chunks: %v", err)
} else {
logger.Infof(ctx, "Deleted %d chunks", len(chunkIDs))

View File

@ -0,0 +1,69 @@
package file
import (
"context"
"net/http"
"os"
"strings"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
// ResolveTenantFileServiceWithFallback applies the storage resolution and
// fallback rule shared by file proxies and artifact downloads: when tenant-level
// resolution fails and the requested provider matches the process-global
// STORAGE_TYPE, serve through the global file service instead of failing.
// Returns ok=false (already logged) when no service can be resolved; the
// caller decides the HTTP status.
func ResolveTenantFileServiceWithFallback(
ctx context.Context,
logTag string,
tenant *types.Tenant,
backendID, provider, absDir string,
storageResolver interfaces.StorageBackendResolver,
globalFileService interfaces.FileService,
) (fileSvc interfaces.FileService, resolvedProvider string, ok bool) {
var err error
if storageResolver != nil {
fileSvc, resolvedProvider, err = storageResolver.ResolveFileService(ctx, tenant, backendID, provider, absDir)
} else if tenant.StorageEngineConfig != nil {
fileSvc, resolvedProvider, err = NewFileServiceFromStorageConfig(provider, tenant.StorageEngineConfig, absDir)
} else {
err = http.ErrMissingFile
}
if err == nil {
return fileSvc, resolvedProvider, true
}
globalStorageType := strings.ToLower(strings.TrimSpace(os.Getenv("STORAGE_TYPE")))
if globalStorageType == "" {
globalStorageType = "local"
}
if provider == globalStorageType && globalFileService != nil {
logger.Warnf(
ctx,
"[Router] %s tenant storage config missing or invalid, fallback to global file "+
"service: tenant_id=%d provider=%s err=%v",
logTag,
tenant.ID,
provider,
err,
)
return globalFileService, globalStorageType, true
}
logger.Warnf(
ctx,
"[Router] %s resolve file service failed without fallback: tenant_id=%d "+
"provider=%s global_storage_type=%s err=%v",
logTag,
tenant.ID,
provider,
globalStorageType,
err,
)
return nil, "", false
}

View File

@ -342,7 +342,7 @@ func (s *ImageMultimodalService) Handle(ctx context.Context, task *asynq.Task) e
}
// Persist chunks
if err := s.chunkService.CreateChunks(ctx, newChunks); err != nil {
if err := s.chunkService.GetRepository().CreateChunks(ctx, newChunks); err != nil {
handleErr = fmt.Errorf("create multimodal chunks: %w", err)
return handleErr
}
@ -446,7 +446,7 @@ func (s *ImageMultimodalService) indexChunks(ctx context.Context, payload types.
continue
}
dbChunk.Status = int(types.ChunkStatusIndexed)
if uerr := s.chunkService.UpdateChunk(ctx, dbChunk); uerr != nil {
if uerr := s.chunkService.GetRepository().UpdateChunk(ctx, dbChunk); uerr != nil {
logger.Warnf(ctx, "[ImageMultimodal] Failed to update chunk %s status to indexed: %v", chunk.ID, uerr)
}
}
@ -503,7 +503,7 @@ func (s *ImageMultimodalService) indexChunks(ctx context.Context, payload types.
continue
}
dbChunk.Status = int(types.ChunkStatusIndexed)
if err := s.chunkService.UpdateChunk(ctx, dbChunk); err != nil {
if err := s.chunkService.GetRepository().UpdateChunk(ctx, dbChunk); err != nil {
logger.Warnf(ctx, "[ImageMultimodal] Failed to update chunk %s status to indexed: %v", chunk.ID, err)
}
}

View File

@ -5,6 +5,7 @@ import (
"errors"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
@ -489,14 +490,7 @@ func (s *kbShareService) CheckTenantKBPermission(ctx context.Context, kbID strin
// HasTenantKBPermission is a thin "do I have at least N" wrapper over
// CheckTenantKBPermission for callers that don't need the granular role.
func (s *kbShareService) HasTenantKBPermission(ctx context.Context, kbID string, callerTenantID uint64, callerTenantRole types.TenantRole, requiredRole types.OrgMemberRole) (bool, error) {
role, isShared, err := s.CheckTenantKBPermission(ctx, kbID, callerTenantID, callerTenantRole)
if err != nil {
return false, err
}
if !isShared {
return false, nil
}
return role.HasPermission(requiredRole), nil
return access.NewKBSharePermissions(ctx, s, callerTenantID, callerTenantRole).Check(kbID, requiredRole)
}
// GetKBSourceTenant gets the source tenant ID for a shared knowledge base

View File

@ -7,6 +7,7 @@ import (
"fmt"
"io"
"reflect"
"sort"
"strings"
"sync"
"time"
@ -592,7 +593,18 @@ func (s *knowledgeService) MoveKnowledgeToFolder(ctx context.Context,
if err != nil {
return 0, err
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
rows, err := loadKnowledgeWriteBatch(ctx, s.repo, s.kbService, ids)
if err != nil {
return 0, err
}
ids = make([]string, 0, len(rows))
for _, row := range rows {
if row.KnowledgeBaseID != kbID {
return 0, werrors.NewForbiddenError("knowledge outside target KB")
}
ids = append(ids, row.ID)
}
tenantID := rows[0].TenantID
affected, err := s.repo.UpdateKnowledgeFolderPath(ctx, tenantID, kbID, ids, normalized)
if err != nil {
logger.Errorf(ctx, "Failed to move knowledge to folder %q: %v", normalized, err)
@ -627,7 +639,18 @@ func (s *knowledgeService) RenameKnowledgeFolder(ctx context.Context,
return 0, werrors.NewBadRequestError("不能将文件夹移动到它自己的子目录下")
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, kbID)
if err != nil {
return 0, err
}
if kb == nil || kb.ID != kbID {
return 0, werrors.NewNotFoundError("knowledge base not found")
}
ctx, err = requireKBWrite(ctx, kb)
if err != nil {
return 0, err
}
tenantID := kb.TenantID
affected, err := s.repo.RenameKnowledgeFolderPath(ctx, tenantID, kbID, source, target)
if err != nil {
logger.Errorf(ctx, "Failed to rename folder %q to %q: %v", source, target, err)
@ -689,7 +712,10 @@ func (s *knowledgeService) GetKnowledgeFile(ctx context.Context, id string) (io.
}
func (s *knowledgeService) UpdateKnowledge(ctx context.Context, knowledge *types.Knowledge) error {
record, err := s.repo.GetKnowledgeByID(ctx, ctx.Value(types.TenantIDContextKey).(uint64), knowledge.ID)
if knowledge == nil {
return werrors.NewBadRequestError("knowledge cannot be nil")
}
record, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledge.ID)
if err != nil {
logger.Errorf(ctx, "Failed to get knowledge record: %v", err)
return err
@ -769,48 +795,48 @@ func (s *knowledgeService) GetKnowledgeBatchWithSharedAccess(ctx context.Context
if len(ids) == 0 {
return nil, nil
}
ownList, err := s.repo.GetKnowledgeBatch(ctx, tenantID, ids)
permissions := kbReadPermissions(ctx, s.kbShareService)
rows, err := s.repo.GetKnowledgeBatch(ctx, tenantID, ids)
if err != nil {
return nil, err
}
ownList := make([]*types.Knowledge, 0, len(rows))
foundSet := make(map[string]bool)
for _, k := range ownList {
appendAllowed := func(k *types.Knowledge) {
if k == nil || foundSet[k.ID] {
return
}
allowed, err := permissions.Check(k.KnowledgeBaseID, k.TenantID, types.OrgRoleViewer)
if err == nil && allowed {
ownList = append(ownList, k)
foundSet[k.ID] = true
}
}
for _, k := range rows {
appendAllowed(k)
if k != nil {
foundSet[k.ID] = true
}
}
userIDVal := ctx.Value(types.UserIDContextKey)
if userIDVal == nil {
return ownList, nil
}
userID, ok := userIDVal.(string)
if !ok || userID == "" {
return ownList, nil
}
// Plan 3: shared-KB permission is keyed on (tenant, tenant_role)
// rather than user. callerTenantRole drives the 3-D cap.
callerTenantRole := types.TenantRoleFromContext(ctx)
for _, id := range ids {
if foundSet[id] {
continue
}
k, err := s.repo.GetKnowledgeByIDOnly(ctx, id)
if err != nil || k == nil || k.KnowledgeBaseID == "" {
continue
if err != nil && !errors.Is(err, repository.ErrKnowledgeNotFound) {
return nil, err
}
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, k.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
if err != nil || !hasPermission {
continue
if err == nil {
appendAllowed(k)
}
foundSet[k.ID] = true
ownList = append(ownList, k)
foundSet[id] = true
}
return ownList, nil
}
// SetKnowledgeTags replaces all tags for a single knowledge entry.
func (s *knowledgeService) SetKnowledgeTags(ctx context.Context, knowledgeID string, tagIDs []string) error {
return s.repo.SetKnowledgeTags(ctx, knowledgeID, tagIDs)
return s.UpdateKnowledgeTag(ctx, knowledgeID, tagIDs)
}
// ListKnowledgeIDsByTagIDs returns document knowledge IDs carrying any of the
@ -909,11 +935,11 @@ func (s *knowledgeService) GetKnowledgeTags(ctx context.Context, knowledgeIDs []
// UpdateKnowledgeTag updates the tags assigned to a knowledge document.
func (s *knowledgeService) UpdateKnowledgeTag(ctx context.Context, knowledgeID string, tagIDs []string) error {
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
knowledge, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
if err != nil {
return err
}
tenantID := knowledge.TenantID
// Validate all tag IDs
if err := s.validateKnowledgeTagIDs(ctx, tenantID, knowledge.KnowledgeBaseID, tagIDs); err != nil {
@ -925,29 +951,25 @@ func (s *knowledgeService) UpdateKnowledgeTag(ctx context.Context, knowledgeID s
// UpdateKnowledgeTagBatch updates tags for document knowledge items in batch.
// authorizedKBID restricts all updates to knowledge items belonging to this KB;
// pass empty string to skip the check (caller must ensure authorization by other means).
func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authorizedKBID string, updates map[string][]string) error {
// an empty value allows multiple KBs only when every KB has an explicit write grant.
func (s *knowledgeService) UpdateKnowledgeTagBatch(
ctx context.Context,
authorizedKBID string,
updates map[string][]string,
) error {
if len(updates) == 0 {
return nil
}
tenantIDVal := ctx.Value(types.TenantIDContextKey)
if tenantIDVal == nil {
return werrors.NewUnauthorizedError("workspace ID not found in context")
}
tenantID, ok := tenantIDVal.(uint64)
if !ok {
return werrors.NewUnauthorizedError("invalid workspace ID in context")
}
// Get all knowledge items in batch
knowledgeIDs := make([]string, 0, len(updates))
for knowledgeID := range updates {
knowledgeIDs = append(knowledgeIDs, knowledgeID)
for id := range updates {
knowledgeIDs = append(knowledgeIDs, id)
}
knowledgeList, err := s.repo.GetKnowledgeBatch(ctx, tenantID, knowledgeIDs)
sort.Strings(knowledgeIDs)
knowledgeList, err := loadKnowledgeWriteBatch(ctx, s.repo, s.kbService, knowledgeIDs)
if err != nil {
return err
}
tenantID := knowledgeList[0].TenantID
// Validate all requested IDs were found and belong to the authorized KB
if authorizedKBID != "" {
@ -1002,15 +1024,15 @@ func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authoriz
if !ok {
return werrors.NewBadRequestError(fmt.Sprintf("标签 %s 不存在", tagID))
}
if tag.KnowledgeBaseID != knowledge.KnowledgeBaseID {
if tag.TenantID != tenantID || tag.KnowledgeBaseID != knowledge.KnowledgeBaseID {
return werrors.NewBadRequestError(fmt.Sprintf("标签 %s 不属于知识库 %s", tagID, knowledge.KnowledgeBaseID))
}
}
}
// Set tags for each knowledge
for knowledgeID, tagIDs := range updates {
if err := s.repo.SetKnowledgeTags(ctx, knowledgeID, tagIDs); err != nil {
for _, knowledgeID := range knowledgeIDs {
if err := s.repo.SetKnowledgeTags(ctx, knowledgeID, updates[knowledgeID]); err != nil {
return err
}
}
@ -1021,10 +1043,12 @@ func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authoriz
// SearchKnowledge searches knowledge items by keyword across the tenant and shared knowledge bases.
// fileTypes: optional list of file extensions to filter by (e.g., ["csv", "xlsx"])
func (s *knowledgeService) SearchKnowledge(ctx context.Context, keyword string, offset, limit int, fileTypes []string) ([]*types.Knowledge, bool, int64, error) {
tenantID, ok := ctx.Value(types.TenantIDContextKey).(uint64)
if !ok {
caller := types.CallerFromContext(ctx)
tenantID := caller.TenantID
if tenantID == 0 {
return nil, false, 0, werrors.NewUnauthorizedError("Workspace ID not found in context")
}
ctx = types.WithExecutionTenant(ctx, tenantID)
scopes := make([]types.KnowledgeSearchScope, 0)
@ -1041,9 +1065,9 @@ func (s *knowledgeService) SearchKnowledge(ctx context.Context, keyword string,
// Shared knowledge bases (document type only). Plan 3 of #1303 keys
// the share lookup on (tenantID, callerTenantRole); userID is no
// longer load-bearing for org-share access.
if userIDVal := ctx.Value(types.UserIDContextKey); userIDVal != nil {
if userID, ok := userIDVal.(string); ok && userID != "" {
callerTenantRole := types.TenantRoleFromContext(ctx)
if s.kbShareService != nil {
if caller.UserID != "" {
callerTenantRole := caller.Role
sharedList, err := s.kbShareService.ListSharedKnowledgeBases(ctx, tenantID, callerTenantRole)
if err == nil {
for _, info := range sharedList {

View File

@ -7,6 +7,7 @@ import (
"sort"
"strings"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/common"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/models/chat"
@ -113,7 +114,7 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
if err := json.Unmarshal(task.Payload(), &payload); err != nil {
return fmt.Errorf("unmarshal knowledge auto tag payload: %w", err)
}
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
@ -153,6 +154,10 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
if kb == nil || kb.TenantID != payload.TenantID || kb.Type != types.KnowledgeBaseTypeDocument {
return skip("knowledge_base_not_eligible", nil)
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
if err != nil {
return err
}
config := kb.AutoTagConfig
if config == nil || !config.Enabled {
return skip("auto_tag_disabled", nil)
@ -251,6 +256,9 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
if len(added) == 0 {
return skip("all_matches_already_attached", types.JSONMap{"matched_tag_count": len(validIDs)})
}
if err := access.RequireKBWrite(ctx, kb); err != nil {
return err
}
if err := s.knowledgeRepo.AddKnowledgeTagRelations(
ctx,
payload.TenantID,

View File

@ -7,6 +7,7 @@ import (
"strings"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/models/chat"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
@ -36,6 +37,7 @@ type autoTagKnowledgeRepo struct {
existing map[string][]*types.KnowledgeTag
added []string
addErr error
writeCtx context.Context
}
func (r *autoTagKnowledgeRepo) GetKnowledgeByIDOnly(context.Context, string) (*types.Knowledge, error) {
@ -49,8 +51,9 @@ func (r *autoTagKnowledgeRepo) GetKnowledgeTags(
}
func (r *autoTagKnowledgeRepo) AddKnowledgeTagRelations(
_ context.Context, _ uint64, _, _ string, tagIDs []string,
ctx context.Context, _ uint64, _, _ string, tagIDs []string,
) error {
r.writeCtx = ctx
if r.addErr != nil {
return r.addErr
}
@ -173,6 +176,9 @@ func TestAutoTagHandleAttachesMatchedTags(t *testing.T) {
fixture := newAutoTagFixture(t, `{"matches":[{"index":1,"confidence":0.9}]}`)
require.NoError(t, fixture.handle(t))
assert.Equal(t, []string{"tag-a"}, fixture.repo.added)
require.Zero(t, types.CallerFromContext(fixture.repo.writeCtx).TenantID)
require.NoError(t, access.RequireKBWrite(fixture.repo.writeCtx, &types.KnowledgeBase{ID: "kb", TenantID: 7}))
require.Error(t, access.RequireKBWrite(fixture.repo.writeCtx, &types.KnowledgeBase{ID: "other", TenantID: 7}))
}
// A missing confidence must not be read as zero: models frequently omit the

View File

@ -5,6 +5,7 @@ import (
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@ -81,18 +82,21 @@ func TestReparseKnowledgeManualEnqueueFailureIsVisible(t *testing.T) {
repo := &reparseFailureKnowledgeRepo{knowledge: knowledge}
svc := &knowledgeService{
repo: repo,
kbService: &reparseFailureKBService{kb: &types.KnowledgeBase{ID: "kb-1"}},
kbService: &reparseFailureKBService{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 7}},
task: failingReparseTaskEnqueuer{err: enqueueErr},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx, grantErr := access.WithKBTaskWrite(ctx, &types.KnowledgeBase{ID: "kb-1", TenantID: 7}, 7)
require.NoError(t, grantErr)
got, err := svc.ReparseKnowledge(ctx, knowledge.ID, nil)
require.Error(t, err)
require.Same(t, knowledge, got)
require.Equal(t, types.ParseStatusFailed, knowledge.ParseStatus)
require.Equal(t, "disabled", knowledge.EnableStatus)
require.Equal(t, "Failed to enqueue processing task", knowledge.ErrorMessage)
require.NotNil(t, got)
require.Equal(t, types.ParseStatusFailed, got.ParseStatus)
require.Equal(t, "disabled", got.EnableStatus)
require.Equal(t, "Failed to enqueue processing task", got.ErrorMessage)
require.GreaterOrEqual(t, repo.updateCalls, 2, "pending and failed states must both be persisted")
}

View File

@ -0,0 +1,186 @@
package service
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/require"
)
type callerScopeShares struct {
interfaces.KBShareService
callers []uint64
}
func (s *callerScopeShares) CheckTenantKBPermission(
_ context.Context,
kbID string,
caller uint64,
_ types.TenantRole,
) (types.OrgMemberRole, bool, error) {
s.callers = append(s.callers, caller)
// Tenant 2 has access to a third tenant's KB; caller 1 does not.
return types.OrgRoleViewer, kbID == "onward" && caller == 2, nil
}
func TestSharedExecutionFiltersDocumentsChunksAndSearchScope(t *testing.T) {
shares := &callerScopeShares{}
svc, db := newKnowledgeSharedAccessService(t, shares)
require.NoError(t, db.AutoMigrate(&types.Chunk{}))
kbs := []*types.KnowledgeBase{
{ID: "own", TenantID: 1},
{ID: "granted", TenantID: 2},
{ID: "private", TenantID: 2},
{ID: "onward", TenantID: 3},
}
var ids []string
for _, kb := range kbs {
ids = append(ids, kb.ID)
seedKnowledge(t, db, &types.Knowledge{ID: kb.ID, TenantID: kb.TenantID, KnowledgeBaseID: kb.ID, Type: "file"})
require.NoError(
t,
db.Create(
&types.Chunk{
ID: kb.ID,
TenantID: kb.TenantID,
KnowledgeBaseID: kb.ID,
KnowledgeID: kb.ID,
Content: kb.ID,
},
).Error,
)
}
search := &knowledgeBaseService{
kgRepo: svc.repo,
chunkRepo: repository.NewChunkRepository(db),
kbShareService: shares,
}
kbService := &suggestionKBService{kbs: make(map[string]*types.KnowledgeBase)}
for _, kb := range kbs {
kbService.kbs[kb.ID] = kb
}
session := &sessionService{knowledgeBaseService: kbService, knowledgeService: svc, kbShareService: shares}
base := newSharedAccessContext()
grant := &access.KBAccess{
KnowledgeBase: kbs[1],
Caller: types.CallerFromContext(base),
EffectiveTenantID: 2,
Permission: types.OrgRoleViewer,
}
for _, execution := range []uint64{1, 2, 3} {
ctx := logger.CloneContext(types.WithExecutionTenant(grant.Context(base), execution))
rows, err := svc.GetKnowledgeBatchWithSharedAccess(ctx, execution, ids)
require.NoError(t, err)
got := make([]string, 0, len(rows))
for _, row := range rows {
got = append(got, row.ID)
}
require.ElementsMatch(t, []string{"own", "granted"}, got)
knowledgeMap, err := search.fetchKnowledgeDataWithShared(ctx, execution, ids)
require.NoError(t, err)
require.Len(t, knowledgeMap, 2)
require.NotNil(t, knowledgeMap["own"])
require.NotNil(t, knowledgeMap["granted"])
chunks, err := search.listChunksByIDWithShared(ctx, execution, ids)
require.NoError(t, err)
got = got[:0]
for _, chunk := range chunks {
got = append(got, chunk.ID)
}
require.ElementsMatch(t, []string{"own", "granted"}, got)
require.NoError(t, search.authorizeKBAccess(ctx, kbs[:2]))
require.Error(t, search.authorizeKBAccess(ctx, kbs[2:3]), "execution tenant is not ownership")
require.Error(t, search.authorizeKBAccess(ctx, kbs[3:]), "source tenant shares are not inherited")
targets, err := session.buildSearchTargets(
ctx,
execution,
ids,
nil,
[]types.TagScope{{KnowledgeBaseID: "private", TagIDs: []string{"tag"}}},
)
require.NoError(t, err)
require.Len(t, targets, 2, "denied KBs must not become search or tag targets")
got = got[:0]
for _, target := range targets {
got = append(got, target.KnowledgeBaseID)
}
require.ElementsMatch(t, []string{"own", "granted"}, got)
_, err = resolveKBReadTenant(ctx, kbs[1], shares)
require.NoError(t, err, "FAQ/tag reads accept the exact upstream grant")
_, err = resolveKBReadTenant(ctx, kbs[2], shares)
require.Error(t, err, "FAQ/tag reads cannot access another KB in the execution tenant")
}
for _, caller := range shares.callers {
require.Equal(t, uint64(1), caller)
}
}
type callerScopeKBList struct {
interfaces.KnowledgeBaseService
tenant uint64
}
func (s *callerScopeKBList) ListKnowledgeBases(ctx context.Context) ([]*types.KnowledgeBase, error) {
s.tenant = types.MustTenantIDFromContext(ctx)
return []*types.KnowledgeBase{{ID: "own", TenantID: s.tenant, Type: types.KnowledgeBaseTypeDocument}}, nil
}
type callerScopeSearchRepo struct {
interfaces.KnowledgeRepository
scopes []types.KnowledgeSearchScope
}
func (r *callerScopeSearchRepo) SearchKnowledgeInScopes(
_ context.Context,
scopes []types.KnowledgeSearchScope,
_ string,
_, _ int,
_ []string,
) ([]*types.Knowledge, bool, int64, error) {
r.scopes = scopes
return nil, false, 0, nil
}
func TestDocumentSearchListsCallerTenantAfterExecutionSwitch(t *testing.T) {
kbs := &callerScopeKBList{}
repo := &callerScopeSearchRepo{}
svc := &knowledgeService{kbService: kbs, repo: repo}
ctx := types.WithExecutionTenant(newSharedAccessContext(), 2)
_, _, _, err := svc.SearchKnowledge(ctx, "query", 0, 10, nil)
require.NoError(t, err)
require.Equal(t, uint64(1), kbs.tenant)
require.Equal(t, []types.KnowledgeSearchScope{{TenantID: 1, KBID: "own"}}, repo.scopes)
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx), "search must not mutate the parent context")
}
func TestSharedAgentBatchUsesGrantInsteadOfSourceTenantOwnership(t *testing.T) {
svc, db := newKnowledgeSharedAccessService(t, &callerScopeShares{})
for _, kbID := range []string{"selected", "private"} {
seedKnowledge(t, db, &types.Knowledge{ID: kbID, TenantID: 2, KnowledgeBaseID: kbID, Type: "file"})
}
agent := &types.CustomAgent{
TenantID: 2,
Config: types.CustomAgentConfig{KBSelectionMode: "selected", KnowledgeBases: []string{"selected"}},
}
ctx := logger.CloneContext(access.WithSharedAgent(newSharedAccessContext(), agent))
rows, err := svc.GetKnowledgeBatchWithSharedAccess(ctx, 2, []string{"selected", "private"})
require.NoError(t, err)
require.Len(t, rows, 1)
require.Equal(t, "selected", rows[0].ID)
ctx = types.WithTenantAPIKeyScope(ctx, types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"private"}})
search := &knowledgeBaseService{}
app, ok := apperrors.IsAppError(
search.authorizeKBAccess(ctx, []*types.KnowledgeBase{{ID: "selected", TenantID: 2}}),
)
require.True(t, ok)
require.Equal(t, apperrors.ErrForbidden, app.Code, "API-key scope denial must not become a lookup failure")
rows, err = svc.GetKnowledgeBatchWithSharedAccess(ctx, 2, []string{"selected", "private"})
require.NoError(t, err)
require.Empty(t, rows)
}

View File

@ -0,0 +1,133 @@
package service
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
"github.com/stretchr/testify/require"
)
func TestDeleteTaskMovingConflictStopsRetries(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
beforeWrites := f.repo.writes
raw, err := json.Marshal(
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc"}},
)
require.NoError(t, err)
err = f.svc.ProcessKnowledgeListDelete(context.Background(), asynq.NewTask(types.TypeKnowledgeListDelete, raw))
require.ErrorIs(t, err, asynq.SkipRetry)
require.ErrorContains(t, err, "unfinished move")
require.Equal(t, beforeWrites, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
}
func TestDeleteWikiRemovesEveryChunkType(t *testing.T) {
f := newDocumentWriteFixture(t)
f.kbs.values["kb"].IndexingStrategy.WikiEnabled = true
refs := types.StringArray{"chunk"}
chunkTypes := []string{
types.ChunkTypeParentText, types.ChunkTypeImageOCR, types.ChunkTypeImageCaption, types.ChunkTypeSummary,
}
for _, typ := range chunkTypes {
require.NoError(
t,
f.db.Create(
&types.Chunk{ID: typ, TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb", ChunkType: typ},
).Error,
)
refs = append(refs, typ)
}
refs = append(refs, "other-chunk")
page := &types.WikiPage{
ID: "page",
Slug: "concept/shared",
SourceRefs: types.StringArray{"doc", "other-doc"},
ChunkRefs: refs,
}
wiki := &moveWikiPageRepo{pages: []*types.WikiPage{page}}
meta := &moveWikiMetaService{}
f.svc.wikiRepo = wiki
f.svc.wikiService = meta
f.svc.taskPendingRepo = &moveWikiPendingRepo{}
f.svc.task = &transferQueue{}
require.NoError(t, f.svc.DeleteKnowledge(f.ctx, "doc"))
require.Len(t, meta.updated, 1)
require.Equal(t, types.StringArray{"other-chunk"}, meta.updated[0].ChunkRefs)
require.Equal(t, types.StringArray{"other-doc"}, meta.updated[0].SourceRefs)
}
type cleanupKBFailure struct {
interfaces.KnowledgeBaseService
value *types.KnowledgeBase
err error
}
func (s *cleanupKBFailure) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
return s.value, s.err
}
func TestCleanupStopsBeforeSideEffectsWhenKBUnavailable(t *testing.T) {
for _, scenario := range []string{"storage error", "nil KB", "wrong KB", "wrong tenant"} {
t.Run(scenario, func(t *testing.T) {
f := newDocumentWriteFixture(t)
lookup := &cleanupKBFailure{}
switch scenario {
case "storage error":
lookup.err = errors.New("database unavailable")
case "wrong KB":
lookup.value = &types.KnowledgeBase{ID: "other", TenantID: 7}
case "wrong tenant":
lookup.value = &types.KnowledgeBase{ID: "kb", TenantID: 8}
}
f.svc.kbService = lookup
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
row.EmbeddingModelID = "bound-model"
// Engine/model dependencies remain absent: touching a fallback backend
// would panic instead of preserving the lookup error and original data.
err = f.svc.cleanupKnowledgeResources(f.ctx, row)
require.Error(t, err)
if lookup.err != nil {
require.ErrorIs(t, err, lookup.err)
}
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
require.Empty(t, f.files.deleted)
require.Empty(t, f.tenants.adjustments)
})
}
}
func TestMoveReparseKBLookupFailurePreservesSourceCheckpoint(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
row.SetManualMetadata(types.NewManualKnowledgeMetadata("content", types.ManualKnowledgeStatusPublish, 1)),
)
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
failure := errors.New("source KB load failed")
f.svc.kbService = &cleanupKBFailure{err: failure}
err = f.svc.moveOneKnowledge(f.ctx, "doc", f.kbs.values["kb"], f.kbs.values["other"], "reparse")
require.ErrorIs(t, err, failure)
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "kb", row.KnowledgeBaseID)
state, err := transferState(row)
require.NoError(t, err)
require.Equal(t, "moving", state.Phase)
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
}

File diff suppressed because it is too large Load Diff

View File

@ -1017,21 +1017,18 @@ func (s *knowledgeService) UpdateManualKnowledge(ctx context.Context,
return nil, werrors.NewValidationError("状态仅支持 draft 或 publish")
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
existing, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
existing, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
if err != nil {
logger.Errorf(ctx, "Failed to load knowledge: %v", err)
return nil, err
}
if !existing.IsManual() {
return nil, werrors.NewBadRequestError("仅支持手工知识的在线编辑")
}
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, existing.KnowledgeBaseID)
ctx, err = withKBWriteTenantInfo(ctx, kb, s.tenantRepo)
if err != nil {
logger.Errorf(ctx, "Failed to get knowledge base for manual update: %v", err)
return nil, err
}
tenantID := existing.TenantID
var version int
if meta, err := existing.ManualMetadata(); err == nil && meta != nil {
@ -1115,7 +1112,7 @@ func (s *knowledgeService) UpdateManualKnowledge(ctx context.Context,
// enqueueManualProcessing enqueues a manual:process Asynq task for async cleanup + re-indexing.
func (s *knowledgeService) enqueueManualProcessing(ctx context.Context,
knowledge *types.Knowledge, content string, needCleanup bool,
knowledge *types.Knowledge, content string, needCleanup bool, options ...asynq.Option,
) (string, error) {
requestID, _ := types.RequestIDFromContext(ctx)
payload := types.ManualProcessPayload{
@ -1134,7 +1131,7 @@ func (s *knowledgeService) enqueueManualProcessing(ctx context.Context,
task := asynq.NewTask(types.TypeManualProcess, payloadBytes,
asynq.Queue(types.QueueDefault), asynq.MaxRetry(3), asynq.Timeout(30*time.Minute))
info, err := s.task.Enqueue(task)
info, err := s.task.Enqueue(task, options...)
if err != nil {
return "", fmt.Errorf("failed to enqueue manual process task: %w", err)
}

View File

@ -4,10 +4,11 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
@ -120,150 +121,11 @@ func deleteExtractedImages(
// DeleteKnowledge deletes a knowledge entry and all related resources
func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error {
// Get the knowledge entry
knowledge, err := s.repo.GetKnowledgeByID(ctx, ctx.Value(types.TenantIDContextKey).(uint64), id)
plan, err := s.planKnowledgeDelete(ctx, []string{id})
if err != nil {
return err
}
// Mark as deleting first to prevent async task conflicts
// This ensures that any running async tasks will detect the deletion and abort
originalStatus := knowledge.ParseStatus
knowledge.ParseStatus = types.ParseStatusDeleting
knowledge.UpdatedAt = time.Now()
if err := s.repo.UpdateKnowledge(ctx, knowledge); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge failed to mark as deleting")
// Continue with deletion even if marking fails
} else {
logger.Infof(ctx, "Marked knowledge %s as deleting (previous status: %s)", id, originalStatus)
}
// Best-effort: purge any queued downstream tasks for this knowledge
// (multimodal / post-process / question / summary / graph extract).
// Worker checkpoints already drop them on the floor, but dequeuing
// here avoids waking workers just to no-op when the parse was still
// in flight at delete time. No-op in Lite mode and on completed rows
// (no queued descendants anyway).
if originalStatus == types.ParseStatusPending ||
originalStatus == types.ParseStatusProcessing {
s.dequeueKnowledgeTasks(ctx, id)
}
// Resolve file service for this KB before spawning goroutines
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
kbFileSvc := s.resolveFileService(ctx, kb)
// Collect image URLs before chunks are deleted (ImageInfo references are lost after deletion)
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
chunkImageInfos, err := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantID, []string{id})
if err != nil {
logger.Errorf(ctx, "Failed to collect image URLs for cleanup: %v", err)
}
var imageInfoStrs []string
for _, ci := range chunkImageInfos {
imageInfoStrs = append(imageInfoStrs, ci.ImageInfo)
}
imageURLs := collectImageURLs(ctx, imageInfoStrs)
wg := errgroup.Group{}
// Delete knowledge embeddings from vector store.
// Skip entirely when the knowledge has no embedding model (e.g. Wiki-only KB):
// nothing was ever written to the vector store, so there is nothing to delete,
// and GetEmbeddingModel would fail with "model ID cannot be empty".
if strings.TrimSpace(knowledge.EmbeddingModelID) != "" {
wg.Go(func() error {
// kb was already loaded above for resolveFileService — reuse its
// VectorStoreID for engine routing.
var boundStoreID *string
if kb != nil {
boundStoreID = kb.VectorStoreID
}
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
ctx,
s.retrieveEngine,
s.ownership,
tenantID,
boundStoreID,
)
if err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
return err
}
embeddingModel, err := s.modelService.GetEmbeddingModel(ctx, knowledge.EmbeddingModelID)
if err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
return err
}
if err := retrieveEngine.DeleteByKnowledgeIDList(ctx, []string{knowledge.ID}, embeddingModel.GetDimensions(), knowledge.Type); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
return err
}
return nil
})
} else {
logger.Infof(ctx, "Knowledge %s has no embedding model, skipping vector store cleanup", knowledge.ID)
}
// Clean wiki pages before deleting chunks so cleanup can still identify
// which chunk_refs belonged to this source document.
if kb != nil && kb.IsWikiEnabled() {
s.cleanupWikiOnKnowledgeDelete(ctx, knowledge)
}
// Delete all chunks associated with this knowledge
wg.Go(func() error {
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete chunks failed")
return err
}
return nil
})
// Delete the knowledge graph
wg.Go(func() error {
namespace := types.NameSpace{KnowledgeBase: knowledge.KnowledgeBaseID, Knowledge: knowledge.ID}
if err := s.graphEngine.DelGraph(ctx, []types.NameSpace{namespace}); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge graph failed")
return err
}
return nil
})
if err = wg.Wait(); err != nil {
return err
}
if err := s.repo.DeleteKnowledgeTagRelations(ctx, id); err != nil {
logger.Warnf(ctx, "Failed to delete tag relations for knowledge %s: %v", id, err)
}
// Delete the knowledge row FIRST, then drop its physical file. Physical
// cleanup is deliberately deferred until the row is gone: if any of the
// index/chunk/graph cleanups above failed we already returned early with the
// row (and its file) intact, so the queued retry — or a user-triggered
// reparse — can still read the original file. Deleting the file before the
// row could leave a "file missing but row present" zombie that can neither be
// reparsed nor cleanly re-deleted (issue #2192). Orphaning a file after the
// row is gone is the tolerable failure mode instead.
if err := s.repo.DeleteKnowledge(ctx, tenantID, id); err != nil {
return err
}
// Best-effort physical cleanup. Errors here only leak storage; they must not
// fail the delete now that the row is already gone.
if knowledge.FilePath != "" {
if err := kbFileSvc.DeleteFile(ctx, knowledge.FilePath); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete file failed")
}
}
deleteExtractedImages(ctx, kbFileSvc, knowledgeResourceOwners(s.resourceCatalog, knowledge.ID), imageURLs)
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
tenantInfo.StorageUsed -= knowledge.StorageSize
if err := s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, -knowledge.StorageSize); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge update tenant storage used failed")
}
recordKBActivity(ctx, s.audit, tenantID, knowledge.KnowledgeBaseID, types.AuditActionKnowledgeDeleted,
"knowledge", knowledge.ID, types.AuditOutcomeSuccess,
map[string]any{"title": knowledge.Title, "type": knowledge.Type})
return nil
return s.executeKnowledgeDelete(plan, true)
}
// cleanupWikiOnKnowledgeDelete handles wiki pages when a source document is deleted.
@ -277,9 +139,9 @@ func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error
// - It writes a tombstone + scrubs pending ingest ops so new pages cannot be
// born with a stale source_ref (guards (a) queued ingest and (b) ingest
// tasks mid-LLM call — both consult the tombstone before writing).
// - It immediately reconciles any pages already present (delete-if-only-ref
// or strip-ref-if-multi).
// - It *unconditionally* enqueues a retract task. Crucially we DO NOT gate
// - It persists a retract task before reconciling existing pages (delete
// if only source, or strip the source if shared), preserving retry evidence.
// Crucially we DO NOT gate
// enqueue on "pages currently exist": in the ingest/delete race the
// knowledge may have pages that exist only after this function returns
// (the ingest task fires later and, absent the tombstone, would have
@ -287,20 +149,42 @@ func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error
// run time, so even with an empty PageSlugs it will do the right thing —
// and at worst it's a cheap no-op.
func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, knowledge *types.Knowledge) {
if err := s.cleanupWikiReferences(ctx, knowledge, s.wikiChunkRefsForKnowledge(ctx, knowledge)); err != nil {
logger.Warnf(ctx, "wiki cleanup failed: %v", err)
}
}
func (s *knowledgeService) cleanupWikiReferences(
ctx context.Context,
knowledge *types.Knowledge,
sourceChunkRefs map[string]bool,
) error {
if knowledge == nil {
return
return nil
}
kbID := knowledge.KnowledgeBaseID
knowledgeID := knowledge.ID
if kbID == "" || knowledgeID == "" {
return
return nil
}
var cleanupErr error
// (1) Tombstone + scrub pending ingest — must happen first so any
// wiki_ingest task that wakes up between here and the retract enqueue
// below sees "knowledge gone" and bails out.
s.markKnowledgeDeletedForWiki(ctx, kbID, knowledgeID)
s.scrubWikiPendingIngest(ctx, kbID, knowledgeID, "cleanup")
if s.redisClient != nil {
key := WikiDeletedTombstoneKey(kbID, knowledgeID)
if err := s.redisClient.Set(ctx, key, "1", wikiDeletedTTL).Err(); err != nil {
cleanupErr = errors.Join(cleanupErr, err)
}
}
if s.taskPendingRepo != nil {
err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest)
if err != nil {
cleanupErr = errors.Join(cleanupErr, err)
}
}
// Pull title/summary from the knowledge itself — do NOT read them from
// existing wiki pages. In the race window wiki pages may not exist yet,
@ -322,9 +206,8 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
pages, err := s.wikiRepo.ListBySourceRef(ctx, kbID, knowledgeID)
if err != nil {
logger.Warnf(ctx, "wiki cleanup: failed to list pages by source ref %s: %v", knowledgeID, err)
pages = nil
return errors.Join(cleanupErr, err)
}
sourceChunkRefs := s.wikiChunkRefsForKnowledge(ctx, knowledge)
// Prefer the on-disk summary if the summary page already exists (it's
// richer than the raw user-provided description). Leave docSummary
@ -336,22 +219,37 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
}
}
// Persist the retract page set before removing source refs. Otherwise an
// enqueue failure would leave the retry unable to rediscover those pages.
var pageSlugs, folderIDs []string
for _, page := range pages {
if page.PageType != types.WikiPageTypeIndex {
pageSlugs = append(pageSlugs, page.Slug)
folderIDs = append(folderIDs, page.FolderID)
}
}
lang := types.LanguageFromContextOrDefault(ctx)
tenantID, _ := types.TenantIDFromContext(ctx)
if err := enqueueWikiRetract(ctx, s.task, s.taskPendingRepo, WikiRetractPayload{
TenantID: tenantID, KnowledgeBaseID: kbID, KnowledgeID: knowledgeID,
DocTitle: docTitle, DocSummary: docSummary, Language: lang,
PageSlugs: pageSlugs, FolderIDs: uniqueWikiFolderIDs(folderIDs),
}); err != nil {
return errors.Join(cleanupErr, err)
}
var deletedSlugs []string
var retractSlugs []string
var affectedFolderIDs []string
for _, page := range pages {
if page.PageType == types.WikiPageTypeIndex {
continue
}
if page.FolderID != "" {
affectedFolderIDs = append(affectedFolderIDs, page.FolderID)
}
remaining := removeSourceRef(page.SourceRefs, knowledgeID)
if len(remaining) == 0 {
if err := s.wikiService.DeletePage(ctx, kbID, page.Slug); err != nil {
logger.Warnf(ctx, "wiki cleanup: failed to delete page %s: %v", page.Slug, err)
cleanupErr = errors.Join(cleanupErr, err)
} else {
deletedSlugs = append(deletedSlugs, page.Slug)
}
@ -360,8 +258,7 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
page.ChunkRefs = removeChunkRefs(page.ChunkRefs, sourceChunkRefs)
if err := s.wikiService.UpdatePageMeta(ctx, page); err != nil {
logger.Warnf(ctx, "wiki cleanup: failed to update source refs for page %s: %v", page.Slug, err)
} else {
retractSlugs = append(retractSlugs, page.Slug)
cleanupErr = errors.Join(cleanupErr, err)
}
}
}
@ -371,34 +268,14 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
len(deletedSlugs), knowledgeID, deletedSlugs)
}
allAffectedSlugs := append(retractSlugs, deletedSlugs...)
// (3) Unconditionally enqueue the retract task. See function comment —
// an empty PageSlugs is not a bug, it's the signal "re-query at run
// time". The handler will ListPagesBySourceRef again, pick up any
// pages that materialised after we looked, and also rebuild the index
// so the knowledge's disappearance is reflected in the UI.
lang := types.LanguageFromContextOrDefault(ctx)
tenantID, _ := types.TenantIDFromContext(ctx)
EnqueueWikiRetract(ctx, s.task, s.taskPendingRepo, WikiRetractPayload{
TenantID: tenantID,
KnowledgeBaseID: kbID,
KnowledgeID: knowledgeID,
DocTitle: docTitle,
DocSummary: docSummary,
Language: lang,
PageSlugs: allAffectedSlugs,
FolderIDs: uniqueWikiFolderIDs(affectedFolderIDs),
})
logger.Infof(ctx, "wiki cleanup: enqueued retract task for knowledge %s (%d known slugs: %v)",
knowledgeID, len(allAffectedSlugs), allAffectedSlugs)
return cleanupErr
}
func (s *knowledgeService) wikiChunkRefsForKnowledge(ctx context.Context, knowledge *types.Knowledge) map[string]bool {
if knowledge == nil || s.chunkRepo == nil {
return nil
}
chunks, err := s.chunkRepo.ListChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID)
chunks, err := s.chunkRepo.ListAllChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID)
if err != nil {
logger.Warnf(ctx, "wiki cleanup: failed to list chunks for knowledge %s: %v", knowledge.ID, err)
return nil
@ -413,19 +290,6 @@ func (s *knowledgeService) wikiChunkRefsForKnowledge(ctx context.Context, knowle
return refs
}
// markKnowledgeDeletedForWiki writes a short-TTL tombstone so any wiki_ingest
// task still running or queued for this knowledge can short-circuit before
// resurrecting a page with a stale source_ref. No-op when Redis is absent.
func (s *knowledgeService) markKnowledgeDeletedForWiki(ctx context.Context, kbID, knowledgeID string) {
if s.redisClient == nil || kbID == "" || knowledgeID == "" {
return
}
key := WikiDeletedTombstoneKey(kbID, knowledgeID)
if err := s.redisClient.Set(ctx, key, "1", wikiDeletedTTL).Err(); err != nil {
logger.Warnf(ctx, "wiki cleanup: failed to write tombstone %s: %v", key, err)
}
}
// scrubWikiPendingIngest removes queued WikiOpIngest entries for a knowledge
// from task_pending_ops. Used by both the delete path (we're about to
// soft-delete the doc, no point ingesting it) and the reparse path (the
@ -441,7 +305,8 @@ func (s *knowledgeService) scrubWikiPendingIngest(ctx context.Context, kbID, kno
if s.taskPendingRepo == nil || kbID == "" || knowledgeID == "" {
return
}
if err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest); err != nil {
err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest)
if err != nil {
logger.Warnf(ctx, "wiki %s: failed to scrub pending ingest ops for knowledge %s: %v", reason, knowledgeID, err)
return
}
@ -552,15 +417,20 @@ func buildKnowledgeVectorDeleteGroups(
// DeleteKnowledgeList deletes a knowledge entry and all related resources
func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string) error {
if len(ids) == 0 {
return nil
}
// 1. Get the knowledge entry
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
knowledgeList, err := s.repo.GetKnowledgeBatch(ctx, tenantInfo.ID, ids)
plan, err := s.planKnowledgeDelete(ctx, ids)
if err != nil {
return err
}
return s.executeKnowledgeDelete(plan, false)
}
func (s *knowledgeService) executeKnowledgeDelete(plan *knowledgeDeletePlan, single bool) error {
if len(plan.ids) == 0 {
return nil
}
ctx, ids, knowledgeList := plan.ctx, plan.ids, plan.knowledge
knowledgeBases, kbFileServices := plan.kbs, plan.files
tenantInfo, _ := types.TenantInfoFromContext(ctx)
// Mark all as deleting first to prevent async task conflicts.
// Remember which entries still had queued / in-flight downstream tasks
@ -568,12 +438,12 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
var inFlightIDs []string
for _, knowledge := range knowledgeList {
prev := knowledge.ParseStatus
before := *knowledge
knowledge.ParseStatus = types.ParseStatusDeleting
knowledge.UpdatedAt = time.Now()
if err := s.repo.UpdateKnowledge(ctx, knowledge); err != nil {
if err := s.repo.UpdateKnowledgeForTransfer(ctx, &before, knowledge); err != nil {
logger.GetLogger(ctx).WithField("error", err).WithField("knowledge_id", knowledge.ID).
Errorf("DeleteKnowledgeList failed to mark as deleting")
// Continue with deletion even if marking fails
return err
}
if prev == types.ParseStatusPending || prev == types.ParseStatusProcessing {
inFlightIDs = append(inFlightIDs, knowledge.ID)
@ -582,28 +452,14 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
logger.Infof(ctx, "Marked %d knowledge entries as deleting", len(knowledgeList))
// Best-effort dequeue of downstream tasks for in-flight entries.
// See DeleteKnowledge for the rationale; loop is per-knowledge because
// Workers also check deletion state; this avoids waking them unnecessarily.
// The loop is per-knowledge because
// the inspector only filters by knowledge_id, not by ID set.
for _, kid := range inFlightIDs {
s.dequeueKnowledgeTasks(ctx, kid)
}
// Pre-resolve KB metadata and file services so goroutines don't need DB access.
knowledgeBases := make(map[string]*types.KnowledgeBase)
kbFileServices := make(map[string]interfaces.FileService)
for _, knowledge := range knowledgeList {
if _, ok := kbFileServices[knowledge.KnowledgeBaseID]; !ok {
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
knowledgeBases[knowledge.KnowledgeBaseID] = kb
kbFileServices[knowledge.KnowledgeBaseID] = s.resolveFileService(ctx, kb)
}
}
// Collect image URLs before chunks are deleted
chunkImageInfos, err := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantInfo.ID, ids)
if err != nil {
logger.Errorf(ctx, "Failed to collect image URLs for batch cleanup: %v", err)
}
chunkImageInfos := plan.imageInfo
knowledgeToKB := make(map[string]string)
for _, k := range knowledgeList {
knowledgeToKB[k.ID] = k.KnowledgeBaseID
@ -664,7 +520,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
// 3. Clean wiki pages before deleting chunks so cleanup can still identify
// which chunk_refs belonged to each source document.
for _, knowledge := range knowledgeList {
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
kb := knowledgeBases[knowledge.KnowledgeBaseID]
if kb != nil && kb.IsWikiEnabled() {
s.cleanupWikiOnKnowledgeDelete(ctx, knowledge)
}
@ -672,7 +528,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
// 4. Delete all chunks associated with this knowledge
wg.Go(func() error {
if err := s.chunkService.DeleteByKnowledgeList(ctx, ids); err != nil {
if err := s.chunkRepo.DeleteByKnowledgeList(ctx, tenantInfo.ID, ids); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete chunks failed")
return err
}
@ -695,7 +551,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
return nil
})
if err = wg.Wait(); err != nil {
if err := wg.Wait(); err != nil {
return err
}
for _, knowledgeID := range ids {
@ -704,7 +560,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
}
}
// 6. Delete the knowledge rows FIRST, then drop their physical files. See
// DeleteKnowledge for the rationale: deferring file removal until the rows are
// Deferring file removal until the rows are
// gone avoids "file missing but row present" zombies that break reparse /
// re-delete when an earlier cleanup step failed (issue #2192). A failure below
// only orphans storage.
@ -731,7 +587,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
}
deleteExtractedImages(ctx, fSvc, knowledgeResourceOwners(s.resourceCatalog, kbKnowledgeIDs[kbID]...), urls)
}
tenantInfo.StorageUsed += storageAdjust
// TenantInfo can be shared by concurrent cleanup branches; update only storage accounting in the repository.
if err := s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, storageAdjust); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge update tenant storage used failed")
}
@ -752,6 +608,13 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
details["knowledge_ids"] = knowledgeIDs
}
kbActivityAppendSampleTitles(details, titles...)
if single {
knowledge := knowledges[0]
recordKBActivity(ctx, s.audit, tenantInfo.ID, kbID, types.AuditActionKnowledgeDeleted,
"knowledge", knowledge.ID, types.AuditOutcomeSuccess,
map[string]any{"title": knowledge.Title, "type": knowledge.Type})
continue
}
recordKBActivity(ctx, s.audit, tenantInfo.ID, kbID, types.AuditActionKnowledgeBatchDeleted,
"knowledge", "", types.AuditOutcomeSuccess, details)
}
@ -768,23 +631,14 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
return nil
}
kb, err := knowledgeWriteKB(ctx, s.kbService, knowledge)
if err != nil {
return fmt.Errorf("resolve cleanup knowledge base: %w", err)
}
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
if knowledge.EmbeddingModelID != "" {
// Load KB to discover its VectorStoreID binding. Falls back to tenant
// effective engines if the KB has no binding or the load fails.
//
// Silent fallback risk: if a bound KB fails to load here due to a
// transient DB error, the cleanup will delete from env engines and
// leave orphan vectors in the bound store. Warn so operators can spot it.
var boundStoreID *string
if kb, loadErr := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID); loadErr == nil && kb != nil {
boundStoreID = kb.VectorStoreID
} else if loadErr != nil {
logger.GetLogger(ctx).WithField("error", loadErr).WithField("knowledge_base_id", knowledge.KnowledgeBaseID).
Warnf("cleanupKnowledgeResources: failed to load KB for vector store resolution; falling back to tenant effective engines")
}
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
ctx, s.retrieveEngine, s.ownership, tenantInfo.ID, boundStoreID)
ctx, s.retrieveEngine, s.ownership, tenantInfo.ID, kb.VectorStoreID)
if err != nil {
logger.GetLogger(ctx).WithField("error", err).Error("Failed to init retrieve engine during cleanup")
cleanupErr = errors.Join(cleanupErr, err)
@ -803,7 +657,6 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
}
// Collect image URLs before chunks are deleted
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
fileSvc := s.resolveFileService(ctx, kb)
chunkImageInfos, imgErr := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantInfo.ID, []string{knowledge.ID})
if imgErr != nil {
@ -816,7 +669,7 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
}
imageURLs := collectImageURLs(ctx, imageInfoStrs)
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
logger.GetLogger(ctx).WithField("error", err).Error("Failed to delete manual knowledge chunks")
cleanupErr = errors.Join(cleanupErr, err)
}
@ -860,20 +713,46 @@ func (s *knowledgeService) ProcessKnowledgeListDelete(ctx context.Context, t *as
logger.Infof(ctx, "Processing knowledge list delete task for %d knowledge items", len(payload.KnowledgeIDs))
// Get tenant info
tenant, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
if err != nil {
logger.Errorf(ctx, "Failed to get tenant %d: %v", payload.TenantID, err)
return err
if payload.TenantID == 0 {
return fmt.Errorf("invalid delete task tenant: %w", asynq.SkipRetry)
}
// Set context values
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = context.WithValue(ctx, types.TenantInfoContextKey, tenant)
// Delete knowledge list
if err := s.DeleteKnowledgeList(ctx, payload.KnowledgeIDs); err != nil {
logger.Errorf(ctx, "Failed to delete knowledge list: %v", err)
ids, err := writeResourceIDs(payload.KnowledgeIDs)
if err != nil {
return fmt.Errorf("invalid delete task IDs: %w", asynq.SkipRetry)
}
if len(ids) == 0 {
return nil
}
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
kbID := payload.KnowledgeBaseID
if kbID == "" {
// Pre-upgrade tasks were admitted by single-KB endpoints but omitted
// the KB ID. Reconstruct only an unambiguous, same-tenant scope.
rows, err := s.repo.GetKnowledgeBatch(ctx, payload.TenantID, ids)
if err != nil {
return err
}
if len(rows) == 0 {
return nil
}
for _, row := range rows {
if row == nil || row.TenantID != payload.TenantID || row.KnowledgeBaseID == "" ||
(kbID != "" && row.KnowledgeBaseID != kbID) {
return fmt.Errorf("ambiguous legacy delete scope: %w", asynq.SkipRetry)
}
kbID = row.KnowledgeBaseID
}
}
bindings := make(map[string]string, len(ids))
for _, id := range ids {
bindings[id] = kbID
}
ctx = withKnowledgeCleanup(ctx, payload.TenantID, bindings)
if err := s.DeleteKnowledgeList(ctx, ids); err != nil {
var appErr *apperrors.AppError
if errors.As(err, &appErr) && (appErr.HTTPCode == 403 || appErr.HTTPCode == 400 || appErr.HTTPCode == 409) {
return fmt.Errorf("invalid delete task scope: %v: %w", err, asynq.SkipRetry)
}
return err
}

View File

@ -0,0 +1,169 @@
package service
import (
"context"
"github.com/Tencent/WeKnora/internal/application/access"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
type knowledgeCleanupKey struct{}
// Cleanup continues a previously admitted operation on exact persisted
// references. It never grants general KB write access to a session owner.
type knowledgeCleanupScope struct {
caller types.Caller
tenant uint64
bindings map[string]string // knowledge ID -> KB ID, copied when admitted
}
func withKnowledgeCleanup(ctx context.Context, tenant uint64, bindings map[string]string) context.Context {
ctx = types.WithExecutionTenant(ctx, tenant)
copyOfBindings := make(map[string]string, len(bindings))
for id, kb := range bindings {
copyOfBindings[id] = kb
}
return context.WithValue(ctx, knowledgeCleanupKey{}, knowledgeCleanupScope{
caller: types.CallerFromContext(ctx), tenant: tenant, bindings: copyOfBindings,
})
}
// Only call with IDs obtained from an already authorized persisted relationship
// (session messages, a temporary KB, or a clone diff), never a request body.
// expectedKB additionally pins workflows with an independently known KB.
func deleteReferencedKnowledge(
ctx context.Context,
svc interfaces.KnowledgeService,
expectedKB string,
ids []string,
) error {
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return err
}
ids, err = writeResourceIDs(ids)
if err != nil || len(ids) == 0 {
return err
}
rows, err := svc.GetRepository().GetKnowledgeBatch(ctx, tenant, ids)
if err != nil {
return err
}
wanted := make(map[string]bool, len(ids))
for _, id := range ids {
wanted[id] = true
}
bindings := make(map[string]string, len(rows))
remaining := make([]string, 0, len(rows))
for _, row := range rows {
if row == nil || !wanted[row.ID] || row.TenantID != tenant || row.KnowledgeBaseID == "" ||
(expectedKB != "" && row.KnowledgeBaseID != expectedKB) {
return apperrors.NewForbiddenError("cleanup resource binding changed")
}
bindings[row.ID] = row.KnowledgeBaseID
remaining = append(remaining, row.ID)
}
if len(remaining) == 0 {
return nil
}
return svc.DeleteKnowledgeList(withKnowledgeCleanup(ctx, tenant, bindings), remaining)
}
type knowledgeDeletePlan struct {
ctx context.Context
ids []string
knowledge []*types.Knowledge
kbs map[string]*types.KnowledgeBase
files map[string]interfaces.FileService
imageInfo []interfaces.ChunkImageInfo
}
func (s *knowledgeService) planKnowledgeDelete(ctx context.Context, ids []string) (*knowledgeDeletePlan, error) {
ids, err := writeResourceIDs(ids)
if err != nil {
return nil, err
}
plan := &knowledgeDeletePlan{
ctx: ctx,
kbs: make(map[string]*types.KnowledgeBase),
files: make(map[string]interfaces.FileService),
}
if len(ids) == 0 {
return plan, nil
}
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return nil, err
}
cleanup, isCleanup := ctx.Value(knowledgeCleanupKey{}).(knowledgeCleanupScope)
if isCleanup {
if cleanup.tenant != tenant || cleanup.caller != types.CallerFromContext(ctx) {
return nil, apperrors.NewForbiddenError("invalid cleanup scope")
}
for _, id := range ids {
if cleanup.bindings[id] == "" {
return nil, apperrors.NewForbiddenError("knowledge outside cleanup scope")
}
}
}
rows, err := s.repo.GetKnowledgeBatch(ctx, tenant, ids)
if err != nil {
return nil, err
}
byID := make(map[string]*types.Knowledge, len(rows))
for _, row := range rows {
if row != nil && row.TenantID == tenant {
byID[row.ID] = row
}
}
for _, id := range ids {
row := byID[id]
if row == nil {
if isCleanup {
// An earlier attempt already removed this exact resource.
continue
}
return nil, apperrors.NewNotFoundError("knowledge not found")
}
if isCleanup && cleanup.bindings[id] != row.KnowledgeBaseID {
return nil, apperrors.NewForbiddenError("cleanup resource binding changed")
}
if err := access.RejectMovingKnowledge(row); err != nil {
return nil, err
}
if _, ok := plan.kbs[row.KnowledgeBaseID]; !ok {
kb, err := knowledgeWriteKB(ctx, s.kbService, row)
if err != nil {
return nil, err
}
if !isCleanup {
if _, err := requireKBWrite(ctx, kb); err != nil {
return nil, err
}
}
plan.kbs[kb.ID] = kb
}
copyOfRow := *row
plan.knowledge = append(plan.knowledge, &copyOfRow)
plan.ids = append(plan.ids, id)
}
if len(plan.ids) == 0 {
return plan, nil
}
// Resolve owner storage/index context and all KB routing before mutations.
ctx, err = withKBWriteTenantInfo(ctx, plan.kbs[plan.knowledge[0].KnowledgeBaseID], s.tenantRepo)
if err != nil {
return nil, err
}
plan.ctx = ctx
for id, kb := range plan.kbs {
plan.files[id] = s.resolveFileService(ctx, kb)
}
plan.imageInfo, err = s.chunkRepo.ListImageInfoByKnowledgeIDs(ctx, tenant, plan.ids)
if err != nil {
return nil, err
}
return plan, nil
}

View File

@ -32,32 +32,9 @@ func (s *knowledgeService) ListFAQEntries(ctx context.Context,
return nil, err
}
// Check if this is a shared knowledge base access
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
effectiveTenantID := tenantID
// If the kb belongs to a different tenant, check for shared access
if kb.TenantID != tenantID {
// Get user ID from context
userIDVal := ctx.Value(types.UserIDContextKey)
if userIDVal == nil {
return nil, werrors.NewForbiddenError("无权访问该知识库")
}
_ = userIDVal.(string) // userID retained only for legacy log fields
callerTenantRole := types.TenantRoleFromContext(ctx)
// Check if the caller's tenant has at least viewer permission via org sharing.
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, kbID, tenantID, callerTenantRole, types.OrgRoleViewer)
if err != nil || !hasPermission {
return nil, werrors.NewForbiddenError("无权访问该知识库")
}
// Use the source tenant ID for data access
sourceTenantID, err := s.kbShareService.GetKBSourceTenant(ctx, kbID)
if err != nil {
return nil, werrors.NewForbiddenError("无权访问该知识库")
}
effectiveTenantID = sourceTenantID
effectiveTenantID, err := resolveKBReadTenant(ctx, kb, s.kbShareService)
if err != nil {
return nil, err
}
faqKnowledge, err := s.findFAQKnowledge(ctx, effectiveTenantID, kb.ID)
@ -141,7 +118,7 @@ func (s *knowledgeService) CreateFAQEntry(ctx context.Context,
return nil, werrors.NewBadRequestError("请求体不能为空")
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return nil, err
}
@ -342,7 +319,7 @@ func (s *knowledgeService) UpdateFAQEntry(ctx context.Context,
if payload == nil {
return nil, werrors.NewBadRequestError("请求体不能为空")
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return nil, err
}
@ -510,7 +487,7 @@ func (s *knowledgeService) AddSimilarQuestions(ctx context.Context,
return nil, werrors.NewBadRequestError("相似问列表不能为空")
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return nil, err
}
@ -655,7 +632,7 @@ func (s *knowledgeService) AddSimilarQuestions(ctx context.Context,
func (s *knowledgeService) UpdateFAQEntryStatus(ctx context.Context,
kbID string, entryID string, isEnabled bool,
) error {
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
@ -704,7 +681,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
if req == nil || (len(req.ByID) == 0 && len(req.ByTag) == 0) {
return nil
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
@ -713,25 +690,17 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
enabledUpdates := make(map[string]bool)
tagUpdates := make(map[string]string)
// Convert exclude seq_ids to UUIDs
excludeUUIDs := make([]string, 0, len(req.ExcludeIDs))
if len(req.ExcludeIDs) > 0 {
excludeChunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, req.ExcludeIDs)
if err == nil {
for _, c := range excludeChunks {
excludeUUIDs = append(excludeUUIDs, c.ID)
}
}
plan, err := s.planFAQFields(ctx, kb, req)
if err != nil {
return err
}
excludeUUIDs := plan.excludeIDs
// Handle ByTag updates first (by tag seq_id)
if len(req.ByTag) > 0 {
for tagSeqID, update := range req.ByTag {
// Convert tag seq_id to UUID
tag, err := s.tagRepo.GetBySeqID(ctx, tenantID, tagSeqID)
if err != nil {
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", tagSeqID))
}
for _, tagSeqID := range sortedFAQIDs(req.ByTag) {
update := req.ByTag[tagSeqID]
tag := plan.tags[tagSeqID]
var setFlags, clearFlags types.ChunkFlags
@ -748,10 +717,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
var newTagUUID *string
if update.TagID != nil {
if *update.TagID > 0 {
newTag, err := s.tagRepo.GetBySeqID(ctx, tenantID, *update.TagID)
if err != nil {
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", *update.TagID))
}
newTag := plan.tags[*update.TagID]
newTagUUID = &newTag.ID
} else {
emptyStr := ""
@ -768,6 +734,19 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
return err
}
// Preserve group changes when an explicit ByID patch follows it.
for _, id := range affectedIDs {
if chunk := plan.chunksByID[id]; chunk != nil {
if update.IsEnabled != nil {
chunk.IsEnabled = *update.IsEnabled
}
chunk.Flags = (chunk.Flags | setFlags) &^ clearFlags
if newTagUUID != nil {
chunk.TagID = *newTagUUID
}
}
}
// Collect affected IDs for retriever sync
if len(affectedIDs) > 0 {
if update.IsEnabled != nil {
@ -786,33 +765,15 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
// Handle ByID updates (by entry seq_id)
if len(req.ByID) > 0 {
entrySeqIDs := make([]int64, 0, len(req.ByID))
for entrySeqID := range req.ByID {
entrySeqIDs = append(entrySeqIDs, entrySeqID)
}
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, entrySeqIDs)
if err != nil {
return err
}
// Build chunk seq_id to chunk map
chunkBySeqID := make(map[int64]*types.Chunk)
for _, chunk := range chunks {
chunkBySeqID[chunk.SeqID] = chunk
}
chunkBySeqID := plan.chunks
setFlags := make(map[string]types.ChunkFlags)
clearFlags := make(map[string]types.ChunkFlags)
chunksToUpdate := make([]*types.Chunk, 0)
for entrySeqID, update := range req.ByID {
chunk, exists := chunkBySeqID[entrySeqID]
if !exists {
continue
}
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
continue
}
for _, entrySeqID := range sortedFAQIDs(req.ByID) {
update := req.ByID[entrySeqID]
chunk := chunkBySeqID[entrySeqID]
needUpdate := false
@ -839,10 +800,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
if update.TagID != nil {
var newTagID string
if *update.TagID > 0 {
newTag, err := s.tagRepo.GetBySeqID(ctx, tenantID, *update.TagID)
if err != nil {
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", *update.TagID))
}
newTag := plan.tags[*update.TagID]
newTagID = newTag.ID
}
if chunk.TagID != newTagID {
@ -900,7 +858,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
// UpdateFAQEntryTag updates the tag assigned to an FAQ entry.
func (s *knowledgeService) UpdateFAQEntryTag(ctx context.Context, kbID string, entryID string, tagID *string) error {
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
@ -948,104 +906,15 @@ func (s *knowledgeService) UpdateFAQEntryTag(ctx context.Context, kbID string, e
// UpdateFAQEntryTagBatch updates tags for FAQ entries in batch.
// Key: entry seq_id, Value: tag seq_id (nil to remove tag)
func (s *knowledgeService) UpdateFAQEntryTagBatch(ctx context.Context, kbID string, updates map[int64]*int64) error {
if len(updates) == 0 {
return nil
req := &types.FAQEntryFieldsBatchUpdate{ByID: make(map[int64]types.FAQEntryFieldsUpdate, len(updates))}
for id, tag := range updates {
value := int64(0) // nil in the tag API means remove the tag.
if tag != nil {
value = *tag
}
req.ByID[id] = types.FAQEntryFieldsUpdate{TagID: &value}
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
// Get all chunks in batch by seq_id
entrySeqIDs := make([]int64, 0, len(updates))
for entrySeqID := range updates {
entrySeqIDs = append(entrySeqIDs, entrySeqID)
}
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, entrySeqIDs)
if err != nil {
return err
}
// Build chunk seq_id to chunk map
chunkBySeqID := make(map[int64]*types.Chunk)
for _, chunk := range chunks {
chunkBySeqID[chunk.SeqID] = chunk
}
// Build tag seq_id set for validation
tagSeqIDSet := make(map[int64]bool)
for _, tagSeqID := range updates {
if tagSeqID != nil && *tagSeqID > 0 {
tagSeqIDSet[*tagSeqID] = true
}
}
// Validate all tags in batch by seq_id
tagMap := make(map[int64]*types.KnowledgeTag)
if len(tagSeqIDSet) > 0 {
tagSeqIDs := make([]int64, 0, len(tagSeqIDSet))
for tagSeqID := range tagSeqIDSet {
tagSeqIDs = append(tagSeqIDs, tagSeqID)
}
tags, err := s.tagRepo.GetBySeqIDs(ctx, tenantID, tagSeqIDs)
if err != nil {
return err
}
for _, tag := range tags {
if tag.KnowledgeBaseID != kb.ID {
return werrors.NewBadRequestError(fmt.Sprintf("标签 %d 不属于当前知识库", tag.SeqID))
}
tagMap[tag.SeqID] = tag
}
}
// Update chunks
chunksToUpdate := make([]*types.Chunk, 0)
for entrySeqID, tagSeqID := range updates {
chunk, exists := chunkBySeqID[entrySeqID]
if !exists {
continue
}
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
continue
}
var resolvedTagID string
if tagSeqID != nil && *tagSeqID > 0 {
tag, ok := tagMap[*tagSeqID]
if !ok {
return werrors.NewBadRequestError(fmt.Sprintf("标签 %d 不存在", *tagSeqID))
}
resolvedTagID = tag.ID
}
chunk.TagID = resolvedTagID
chunk.UpdatedAt = time.Now()
chunksToUpdate = append(chunksToUpdate, chunk)
}
if len(chunksToUpdate) > 0 {
if err := s.chunkRepo.UpdateChunks(ctx, chunksToUpdate); err != nil {
return err
}
// Sync tag updates to retriever engines
tagUpdates := make(map[string]string)
for _, chunk := range chunksToUpdate {
tagUpdates[chunk.ID] = chunk.TagID
}
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
ctx, s.retrieveEngine, s.ownership, tenantID, kb.VectorStoreID)
if err != nil {
return err
}
if err := retrieveEngine.BatchUpdateChunkTagID(ctx, tagUpdates); err != nil {
return err
}
}
return nil
return s.UpdateFAQEntryFieldsBatch(ctx, kbID, req)
}
// SearchFAQEntries searches FAQ entries using hybrid search.
@ -1389,41 +1258,47 @@ func (s *knowledgeService) DeleteFAQEntries(ctx context.Context,
if len(entrySeqIDs) == 0 {
return werrors.NewBadRequestError("请选择需要删除的 FAQ 条目")
}
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
var faqKnowledge *types.Knowledge
chunksToRemove := make([]*types.Chunk, 0, len(entrySeqIDs))
for _, seqID := range entrySeqIDs {
if seqID <= 0 {
continue
}
chunk, err := s.chunkRepo.GetChunkBySeqID(ctx, tenantID, seqID)
if err != nil {
return werrors.NewNotFoundError("FAQ条目不存在")
}
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
return werrors.NewBadRequestError("包含无效的 FAQ 条目")
}
if err := s.chunkService.DeleteChunk(ctx, chunk.ID); err != nil {
return err
}
if faqKnowledge == nil {
faqKnowledge, err = s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
selected, err := s.loadFAQWriteChunks(ctx, kb, entrySeqIDs)
if err != nil {
return err
}
chunksToRemove := make([]*types.Chunk, 0, len(selected))
knowledges := make(map[string]*types.Knowledge)
groups := make(map[string][]*types.Chunk)
for _, id := range sortedFAQIDs(selected) {
chunk := selected[id]
if knowledges[chunk.KnowledgeID] == nil {
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
if err != nil {
return err
}
if knowledge == nil || knowledge.TenantID != tenantID || knowledge.KnowledgeBaseID != kb.ID ||
knowledge.Type != types.KnowledgeTypeFAQ {
return werrors.NewForbiddenError("FAQ 文档不属于当前知识库")
}
knowledges[chunk.KnowledgeID] = knowledge
}
groups[chunk.KnowledgeID] = append(groups[chunk.KnowledgeID], chunk)
chunksToRemove = append(chunksToRemove, chunk)
}
if len(chunksToRemove) > 0 && faqKnowledge != nil {
if err := s.deleteFAQChunkVectors(ctx, kb, faqKnowledge, chunksToRemove); err != nil {
// All entries and parent documents are authorized before any deletion.
for _, chunk := range chunksToRemove {
if err := s.chunkService.DeleteChunk(ctx, chunk.ID); err != nil {
return err
}
}
for id, chunks := range groups {
if err := s.deleteFAQChunkVectors(ctx, kb, knowledges[id], chunks); err != nil {
return err
}
}
details := map[string]any{"count": len(chunksToRemove), "source_type": "faq"}
titles := make([]string, 0, len(chunksToRemove))
for _, chunk := range chunksToRemove {
@ -1630,6 +1505,9 @@ func (s *knowledgeService) validateFAQKnowledgeBase(ctx context.Context, kbID st
if err != nil {
return nil, err
}
if kb == nil || kb.ID != kbID {
return nil, werrors.NewNotFoundError("知识库不存在")
}
kb.EnsureDefaults()
if kb.Type != types.KnowledgeBaseTypeFAQ {
return nil, werrors.NewBadRequestError("仅 FAQ 知识库支持该操作")
@ -1906,7 +1784,9 @@ func (s *knowledgeService) buildFAQTagResolver(
if len(seqIDs) > 0 {
if tags, err := s.tagRepo.GetBySeqIDs(ctx, tenantID, seqIDs); err == nil {
for _, t := range tags {
seqIDToUUID[t.SeqID] = t.ID
if validateFAQTagScope(t, tenantID, kbID) == nil {
seqIDToUUID[t.SeqID] = t.ID
}
}
} else {
logger.Warnf(ctx, "buildFAQTagResolver: batch GetBySeqIDs failed (%d ids), fallback per-entry: %v",
@ -1953,6 +1833,9 @@ func (s *knowledgeService) resolveTagID(ctx context.Context, kbID string, payloa
if err != nil {
return "", fmt.Errorf("failed to find tag by seq_id %d: %w", payload.TagID, err)
}
if err := validateFAQTagScope(tag, tenantID, kbID); err != nil {
return "", err
}
return tag.ID, nil
}

View File

@ -0,0 +1,149 @@
package service
import (
"context"
"fmt"
"sort"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
)
func sortedFAQIDs[V any](values map[int64]V) []int64 {
ids := make([]int64, 0, len(values))
for id := range values {
ids = append(ids, id)
}
sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] })
return ids
}
// loadFAQWriteChunks validates the entire selection before the first mutation.
// Copies keep planning from mutating repository-owned snapshots on rejection.
func (s *knowledgeService) loadFAQWriteChunks(
ctx context.Context,
kb *types.KnowledgeBase,
ids []int64,
) (map[int64]*types.Chunk, error) {
wanted := make(map[int64]bool, len(ids))
for _, id := range ids {
if id <= 0 {
return nil, apperrors.NewBadRequestError("FAQ 条目 ID 必须为正整数")
}
wanted[id] = true
}
result := make(map[int64]*types.Chunk, len(wanted))
if len(wanted) == 0 {
return result, nil
}
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, kb.TenantID, sortedFAQIDs(wanted))
if err != nil {
return nil, err
}
for _, chunk := range chunks {
if chunk == nil || !wanted[chunk.SeqID] {
continue
}
if chunk.TenantID != kb.TenantID || chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
return nil, apperrors.NewForbiddenError("FAQ 条目不属于当前知识库")
}
snapshot := *chunk
result[chunk.SeqID] = &snapshot
}
if len(result) != len(wanted) {
return nil, apperrors.NewNotFoundError("FAQ 条目不存在")
}
return result, nil
}
type faqFieldPlan struct {
chunks map[int64]*types.Chunk
chunksByID map[string]*types.Chunk
tags map[int64]*types.KnowledgeTag
excludeIDs []string
}
func (s *knowledgeService) planFAQFields(
ctx context.Context,
kb *types.KnowledgeBase,
req *types.FAQEntryFieldsBatchUpdate,
) (*faqFieldPlan, error) {
ids := append(sortedFAQIDs(req.ByID), req.ExcludeIDs...)
chunks, err := s.loadFAQWriteChunks(ctx, kb, ids)
if err != nil {
return nil, err
}
plan := &faqFieldPlan{
chunks: chunks,
chunksByID: make(map[string]*types.Chunk),
tags: make(map[int64]*types.KnowledgeTag),
}
for _, chunk := range chunks {
plan.chunksByID[chunk.ID] = chunk
}
for _, id := range req.ExcludeIDs {
plan.excludeIDs = append(plan.excludeIDs, chunks[id].ID)
}
tagIDs := make(map[int64]bool)
for id := range req.ByTag {
if id <= 0 {
return nil, apperrors.NewBadRequestError("标签 ID 必须为正整数")
}
tagIDs[id] = true
}
for _, updates := range []map[int64]types.FAQEntryFieldsUpdate{req.ByTag, req.ByID} {
for _, update := range updates {
if update.TagID != nil && *update.TagID > 0 {
tagIDs[*update.TagID] = true
}
}
}
if len(tagIDs) > 0 {
tags, err := s.tagRepo.GetBySeqIDs(ctx, kb.TenantID, sortedFAQIDs(tagIDs))
if err != nil {
return nil, err
}
for _, tag := range tags {
if tag == nil || !tagIDs[tag.SeqID] {
continue
}
if err := validateFAQTagScope(tag, kb.TenantID, kb.ID); err != nil {
return nil, err
}
plan.tags[tag.SeqID] = tag
}
for _, id := range sortedFAQIDs(tagIDs) {
if plan.tags[id] == nil {
return nil, apperrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", id))
}
}
}
return plan, nil
}
func validateFAQTagScope(tag *types.KnowledgeTag, tenantID uint64, kbID string) error {
if tag == nil {
return apperrors.NewNotFoundError("标签不存在")
}
if tag.TenantID != tenantID || tag.KnowledgeBaseID != kbID {
return apperrors.NewForbiddenError("标签不属于当前知识库")
}
return nil
}
func (s *knowledgeService) validateFAQImportTags(
ctx context.Context,
kb *types.KnowledgeBase,
entries []types.FAQEntryPayload,
) error {
// Import still reports per-entry content validation failures. Referenced
// resources must all be in scope before admission or execution, however.
req := &types.FAQEntryFieldsBatchUpdate{ByTag: make(map[int64]types.FAQEntryFieldsUpdate)}
for _, entry := range entries {
if entry.TagID != 0 {
req.ByTag[entry.TagID] = types.FAQEntryFieldsUpdate{}
}
}
_, err := s.planFAQFields(ctx, kb, req)
return err
}

View File

@ -11,6 +11,8 @@ import (
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
werrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
@ -39,10 +41,13 @@ func (s *knowledgeService) UpsertFAQEntries(ctx context.Context,
}
// 验证知识库是否存在且有效
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return "", err
}
if err := s.validateFAQImportTags(ctx, kb, payload.Entries); err != nil {
return "", err
}
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
@ -1386,7 +1391,7 @@ func (s *knowledgeService) executeFAQImport(ctx context.Context, taskID string,
}
}()
kb, err = s.validateFAQKnowledgeBase(ctx, kbID)
kb, ctx, err = s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
@ -2245,7 +2250,29 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
ctx = logger.WithRequestID(ctx, uuid.New().String())
ctx = logger.WithField(ctx, "faq_import", payload.TaskID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
kb, err := s.validateFAQKnowledgeBase(ctx, payload.KBID)
if err != nil {
if errors.Is(err, repository.ErrKnowledgeBaseNotFound) {
return fmt.Errorf("%w: FAQ task KB no longer exists", asynq.SkipRetry)
}
return err
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
if err != nil {
return fmt.Errorf("%w: FAQ task KB does not belong to its tenant", asynq.SkipRetry)
}
knowledge, err := s.repo.GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
if err != nil {
if errors.Is(err, repository.ErrKnowledgeNotFound) {
return fmt.Errorf("%w: FAQ task document no longer exists", asynq.SkipRetry)
}
return err
}
if knowledge == nil || knowledge.TenantID != payload.TenantID || knowledge.KnowledgeBaseID != payload.KBID ||
knowledge.Type != types.KnowledgeTypeFAQ {
return fmt.Errorf("%w: FAQ task document does not belong to its KB", asynq.SkipRetry)
}
// 获取任务重试信息,用于判断是否是最后一次重试
retryCount, _ := asynq.GetRetryCount(ctx)
@ -2287,6 +2314,9 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
logger.Infof(ctx, "Processing FAQ import task: task_id=%s, kb_id=%s, total_entries=%d, dry_run=%v, retry=%d/%d",
payload.TaskID, payload.KBID, len(payload.Entries), payload.DryRun, retryCount, maxRetry)
if err := s.validateFAQImportTags(ctx, kb, payload.Entries); err != nil {
return err
}
// 保存原始总数量
originalTotalEntries := len(payload.Entries)
@ -2361,31 +2391,6 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
logger.Warnf(ctx, "Failed to update FAQ import progress: %v", err)
}
// 幂等性检查:获取knowledge记录(FAQ任务使用knowledge ID作为taskID)
knowledge, err := s.repo.GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
if err != nil {
logger.Errorf(ctx, "failed to get FAQ knowledge: %v", err)
return nil
}
if knowledge == nil {
return nil
}
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, payload.KBID)
if err != nil {
logger.Errorf(ctx, "Failed to get knowledge base: %v", err)
// 如果是最后一次重试,更新状态为失败
if isLastRetry {
if updateErr := s.updateFAQImportProgressStatus(ctx, payload.TaskID, payload.InstanceID, payload.EnqueuedAt, types.FAQImportStatusFailed, 0, originalTotalEntries, 0, "获取知识库失败", err.Error()); updateErr != nil {
logger.Errorf(ctx, "Failed to update task status to failed: %v", updateErr)
}
s.recordFAQImportKBActivity(ctx, &payload, progress, originalTotalEntries, types.AuditActionFAQImportFailed, types.AuditOutcomeFailed)
}
s.cleanupFAQEntriesFileOnFinalFailure(ctx, payload.EntriesURL, retryCount, maxRetry)
return fmt.Errorf("failed to get knowledge base: %w", err)
}
// 检查任务状态 - 幂等性处理(复用之前获取的 existingProgress)
var processedCount int
if existingProgress != nil {
@ -2825,8 +2830,11 @@ func (s *knowledgeService) UpdateLastFAQImportResultDisplayStatus(ctx context.Co
return werrors.NewBadRequestError("invalid display status, must be 'open' or 'close'")
}
// 获取当前空间ID
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
if err != nil {
return err
}
tenantID := kb.TenantID
// 查找FAQ类型的knowledge
knowledgeList, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, tenantID, kbID)

View File

@ -4,6 +4,7 @@ import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/assert"
@ -40,12 +41,23 @@ func (r *folderMoveRepoStub) RenameKnowledgeFolderPath(
}
func folderMoveContext() context.Context {
return context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
return (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 1,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 1,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
}
func TestMoveKnowledgeToFolderNormalizesDestination(t *testing.T) {
repo := &folderMoveRepoStub{}
svc := &knowledgeService{repo: repo}
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
affected, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1", "k2"}, "/docs//spec/")
require.NoError(t, err)
@ -56,7 +68,7 @@ func TestMoveKnowledgeToFolderNormalizesDestination(t *testing.T) {
func TestMoveKnowledgeToFolderAcceptsRootAndRejectsEmptyBatch(t *testing.T) {
repo := &folderMoveRepoStub{}
svc := &knowledgeService{repo: repo}
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
// The empty destination is meaningful: it files documents back at the top level.
_, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1"}, "")
@ -70,7 +82,7 @@ func TestMoveKnowledgeToFolderAcceptsRootAndRejectsEmptyBatch(t *testing.T) {
func TestMoveKnowledgeToFolderRejectsTraversalAndUnsafeInput(t *testing.T) {
repo := &folderMoveRepoStub{}
svc := &knowledgeService{repo: repo}
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
_, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1"}, "../../etc/docs")
require.NoError(t, err)
@ -82,7 +94,7 @@ func TestMoveKnowledgeToFolderRejectsTraversalAndUnsafeInput(t *testing.T) {
func TestRenameKnowledgeFolderValidatesPaths(t *testing.T) {
repo := &folderMoveRepoStub{}
svc := &knowledgeService{repo: repo}
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
ctx := folderMoveContext()
affected, err := svc.RenameKnowledgeFolder(ctx, "kb-1", "docs", "handbook")
@ -112,3 +124,15 @@ func TestRenameKnowledgeFolderValidatesPaths(t *testing.T) {
assert.Equal(t, int64(0), affected)
assert.Equal(t, before, repo.renameCalls)
}
func (r *folderMoveRepoStub) GetKnowledgeBatch(
_ context.Context,
tenant uint64,
ids []string,
) ([]*types.Knowledge, error) {
rows := make([]*types.Knowledge, 0, len(ids))
for _, id := range ids {
rows = append(rows, &types.Knowledge{ID: id, TenantID: tenant, KnowledgeBaseID: "kb-1"})
}
return rows, nil
}

View File

@ -6,6 +6,7 @@ import (
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
@ -74,7 +75,7 @@ type metadataUpdateKBService struct {
func (metadataUpdateKBService) GetKnowledgeBaseByID(
_ context.Context, id string,
) (*types.KnowledgeBase, error) {
return &types.KnowledgeBase{ID: id, SummaryModelID: "summary-model"}, nil
return &types.KnowledgeBase{ID: id, TenantID: 7, SummaryModelID: "summary-model"}, nil
}
type metadataUpdateTaskEnqueuer struct {
@ -114,6 +115,17 @@ func TestUpdateKnowledgeMetadataDoesNotReindexChunks(t *testing.T) {
repo: repo, chunkRepo: chunkRepo, kbService: metadataUpdateKBService{}, task: taskEnqueuer,
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
err := service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",
@ -143,6 +155,17 @@ func TestUpdateKnowledgeMetadataDoesNotLeaveTasklessPendingStatus(t *testing.T)
task: &metadataUpdateTaskEnqueuer{err: errors.New("queue unavailable")},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
require.NoError(t, service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1", CustomMetadata: types.JSON(`{"region":"Shanghai"}`),
@ -162,6 +185,17 @@ func TestUpdateKnowledgeUnchangedMetadataDoesNotRefreshSummary(t *testing.T) {
repo: repo, chunkRepo: &metadataUpdateChunkRepo{}, kbService: metadataUpdateKBService{}, task: taskEnqueuer,
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
require.NoError(t, service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",

View File

@ -2,8 +2,11 @@ package service
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/assert"
@ -19,6 +22,7 @@ import (
type moveWikiKnowledgeRepo struct {
interfaces.KnowledgeRepository
knowledge *types.Knowledge
commitErr error
}
func (r *moveWikiKnowledgeRepo) GetKnowledgeByID(
@ -40,22 +44,33 @@ func (r *moveWikiKnowledgeRepo) DeleteKnowledgeTagRelations(_ context.Context, _
type moveWikiPageRepo struct {
interfaces.WikiPageRepository
listedKBs []string
listedKBs []string
listErr error
beforeList func()
pages []*types.WikiPage
}
func (r *moveWikiPageRepo) ListBySourceRef(
_ context.Context, kbID, _ string,
) ([]*types.WikiPage, error) {
r.listedKBs = append(r.listedKBs, kbID)
return nil, nil
if r.beforeList != nil {
r.beforeList()
}
return r.pages, r.listErr
}
type moveWikiPendingRepo struct {
interfaces.TaskPendingOpsRepository
ops []*types.TaskPendingOp
ops []*types.TaskPendingOp
err error
failKB string
}
func (r *moveWikiPendingRepo) Enqueue(_ context.Context, op *types.TaskPendingOp) error {
if r.err != nil && (r.failKB == "" || r.failKB == op.ScopeID) {
return r.err
}
r.ops = append(r.ops, op)
return nil
}
@ -67,6 +82,8 @@ func (r *moveWikiPendingRepo) DeleteByDedupKey(_ context.Context, _, _, _, _, _
type moveWikiChunkRepo struct {
interfaces.ChunkRepository
movedToKB string
moves int
moveErr error
}
func (r *moveWikiChunkRepo) ListChunksByKnowledgeID(
@ -78,6 +95,10 @@ func (r *moveWikiChunkRepo) ListChunksByKnowledgeID(
func (r *moveWikiChunkRepo) MoveChunksByKnowledgeID(
_ context.Context, _ uint64, _ string, targetKBID string,
) error {
r.moves++
if r.moveErr != nil {
return r.moveErr
}
r.movedToKB = targetKBID
return nil
}
@ -85,6 +106,7 @@ func (r *moveWikiChunkRepo) MoveChunksByKnowledgeID(
func wikiEnabledKB(id string) *types.KnowledgeBase {
return &types.KnowledgeBase{
ID: id,
TenantID: 1,
IndexingStrategy: types.IndexingStrategy{WikiEnabled: true},
}
}
@ -124,25 +146,25 @@ func newMoveWikiService(t *testing.T) (
}
func moveWikiCtx() context.Context {
return context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
ctx, _ := access.WithKBTransferTask(
context.Background(),
wikiEnabledKB("kb-src"),
wikiEnabledKB("kb-dst"),
1,
access.KBTransferMove,
"move-task",
false,
)
return ctx
}
func TestMoveOneKnowledgeRetractsWikiFromSourceKB(t *testing.T) {
// An unknown mode makes the move itself a no-op, which pins the cleanup as
// unconditional: it runs before the mode dispatch, while the knowledge still
// belongs to the source KB.
func TestMoveOneKnowledgeRejectsUnknownModeBeforeWikiCleanup(t *testing.T) {
svc, wikiRepo, pendingRepo, _ := newMoveWikiService(t)
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1",
wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "bogus")
require.Error(t, err)
assert.Equal(t, []string{"kb-src"}, wikiRepo.listedKBs)
srcOps := opsFor(pendingRepo.ops, "kb-src")
require.Len(t, srcOps, 1)
assert.Equal(t, WikiOpRetract, srcOps[0].Op)
assert.Equal(t, "kn-1", srcOps[0].DedupKey)
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1", wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "bogus")
require.ErrorContains(t, err, "unknown move mode")
require.Empty(t, wikiRepo.listedKBs)
require.Empty(t, pendingRepo.ops)
require.Equal(t, types.ParseStatusCompleted, svc.repo.(*moveWikiKnowledgeRepo).knowledge.ParseStatus)
}
func TestMoveOneKnowledgeReuseVectorsIngestsIntoTargetKB(t *testing.T) {
@ -165,10 +187,187 @@ func TestMoveOneKnowledgeReuseVectorsIngestsIntoTargetKB(t *testing.T) {
func TestMoveOneKnowledgeSkipsWikiWorkForNonWikiKBs(t *testing.T) {
svc, wikiRepo, pendingRepo, _ := newMoveWikiService(t)
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1",
&types.KnowledgeBase{ID: "kb-src"}, &types.KnowledgeBase{ID: "kb-dst"}, "reuse_vectors")
err := svc.moveOneKnowledge(
moveWikiCtx(),
"kn-1",
&types.KnowledgeBase{
ID: "kb-src",
TenantID: 1,
},
&types.KnowledgeBase{ID: "kb-dst", TenantID: 1},
"reuse_vectors",
)
require.NoError(t, err)
assert.Empty(t, wikiRepo.listedKBs)
assert.Empty(t, pendingRepo.ops)
}
func (r *moveWikiKnowledgeRepo) UpdateKnowledgeForTransfer(
ctx context.Context,
_ *types.Knowledge,
after *types.Knowledge,
) error {
if after.KnowledgeBaseID == "kb-dst" && r.commitErr != nil {
return r.commitErr
}
return r.UpdateKnowledge(ctx, after)
}
func (r *moveWikiChunkRepo) ListAllChunksByKnowledgeID(context.Context, uint64, string) ([]*types.Chunk, error) {
return nil, nil
}
type moveWikiMetaService struct {
interfaces.WikiPageService
updated []*types.WikiPage
}
func (s *moveWikiMetaService) UpdatePageMeta(_ context.Context, page *types.WikiPage) error {
s.updated = append(s.updated, page)
return nil
}
func TestMoveWikiCleanupRunsOnlyAfterOwnershipCommit(t *testing.T) {
for _, stage := range []string{"chunks", "CAS"} {
t.Run(stage, func(t *testing.T) {
svc, wiki, pending, chunks := newMoveWikiService(t)
repo := svc.repo.(*moveWikiKnowledgeRepo)
failure := errors.New("transfer failed")
if stage == "chunks" {
chunks.moveErr = failure
} else {
repo.commitErr = failure
}
err := svc.moveOneKnowledge(
moveWikiCtx(),
"kn-1",
wikiEnabledKB("kb-src"),
wikiEnabledKB("kb-dst"),
"reuse_vectors",
)
require.ErrorIs(t, err, failure)
require.Equal(t, "kb-src", repo.knowledge.KnowledgeBaseID)
require.Empty(t, wiki.listedKBs)
require.Empty(t, pending.ops)
})
}
}
func TestMoveWikiCleanupFailureRetriesWithoutMovingAgain(t *testing.T) {
for _, stage := range []string{"lookup", "enqueue", "target enqueue"} {
t.Run(stage, func(t *testing.T) {
svc, wiki, pending, chunks := newMoveWikiService(t)
repo := svc.repo.(*moveWikiKnowledgeRepo)
failure := errors.New("wiki unavailable")
if stage == "lookup" {
wiki.listErr = failure
} else {
pending.err = failure
if stage == "target enqueue" {
pending.failKB = "kb-dst"
}
}
wiki.beforeList = func() { require.Equal(t, "kb-dst", repo.knowledge.KnowledgeBaseID) }
err := svc.moveOneKnowledge(
moveWikiCtx(),
"kn-1",
wikiEnabledKB("kb-src"),
wikiEnabledKB("kb-dst"),
"reuse_vectors",
)
require.ErrorIs(t, err, failure)
require.Equal(t, "kb-dst", repo.knowledge.KnowledgeBaseID)
require.Equal(t, []string{"kb-src"}, wiki.listedKBs)
require.Empty(t, opsFor(pending.ops, "kb-dst"))
wiki.listErr = nil
pending.err = nil
require.NoError(
t,
svc.moveOneKnowledge(
moveWikiCtx(),
"kn-1",
wikiEnabledKB("kb-src"),
wikiEnabledKB("kb-dst"),
"reuse_vectors",
),
)
require.Equal(t, 1, chunks.moves)
require.Equal(t, []string{"kb-src", "kb-src"}, wiki.listedKBs)
require.NotEmpty(t, opsFor(pending.ops, "kb-dst"))
})
}
}
func TestMoveReparseWikiUsesDurableSourceRefsAfterChunksDeleted(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
f.kbs.values["kb"].IndexingStrategy.WikiEnabled = true
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
row.SetManualMetadata(types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1)),
)
row.Description = "source summary"
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
page := &types.WikiPage{
ID: "page",
KnowledgeBaseID: "kb",
SourceRefs: types.StringArray{"doc", "other-doc"},
ChunkRefs: types.StringArray{"chunk", "other-chunk"},
}
wiki := &moveWikiPageRepo{pages: []*types.WikiPage{page}}
meta := &moveWikiMetaService{}
pending := &moveWikiPendingRepo{}
wiki.beforeList = func() {
saved, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "other", saved.KnowledgeBaseID)
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Empty(t, chunks)
}
f.svc.wikiRepo = wiki
f.svc.wikiService = meta
f.svc.taskPendingRepo = pending
f.svc.task = &transferQueue{}
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, "reparse")))
require.Equal(t, []string{"kb"}, wiki.listedKBs)
require.Len(t, meta.updated, 1)
require.Equal(t, types.StringArray{"other-doc"}, meta.updated[0].SourceRefs)
require.Equal(t, types.StringArray{"other-chunk"}, meta.updated[0].ChunkRefs)
ops := opsFor(pending.ops, "kb")
require.Len(t, ops, 1)
var payload WikiPendingOp
require.NoError(t, json.Unmarshal(ops[0].Payload, &payload))
require.Equal(t, "source summary", payload.DocSummary)
}
func TestMoveWikiRetractionIsDurableBeforeRemovingPageSources(t *testing.T) {
svc, wiki, pending, chunks := newMoveWikiService(t)
page := &types.WikiPage{ID: "page", Slug: "concept/source", SourceRefs: types.StringArray{"kn-1", "other"}}
wiki.pages = []*types.WikiPage{page}
meta := &moveWikiMetaService{}
svc.wikiService = meta
pending.err = errors.New("enqueue unavailable")
err := svc.moveOneKnowledge(
moveWikiCtx(),
"kn-1",
wikiEnabledKB("kb-src"),
wikiEnabledKB("kb-dst"),
"reuse_vectors",
)
require.ErrorIs(t, err, pending.err)
require.Empty(t, meta.updated)
require.Contains(t, page.SourceRefs, "kn-1")
pending.err = nil
require.NoError(
t,
svc.moveOneKnowledge(moveWikiCtx(), "kn-1", wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "reuse_vectors"),
)
require.Equal(t, 1, chunks.moves)
require.Len(t, meta.updated, 1)
var op WikiPendingOp
require.NoError(t, json.Unmarshal(opsFor(pending.ops, "kb-src")[0].Payload, &op))
require.Equal(t, []string{"concept/source"}, op.PageSlugs)
}

View File

@ -8,6 +8,7 @@ import (
"sort"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
"github.com/Tencent/WeKnora/internal/types"
@ -83,11 +84,39 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
logger.Infof(ctx, "[KnowledgePostProcess] Orchestrating post processing for knowledge: %s", payload.KnowledgeID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
// 1. Fetch Knowledge and KB
knowledge, err := s.knowledgeRepo.GetKnowledgeByIDOnly(ctx, payload.KnowledgeID)
if err != nil {
return fmt.Errorf("get knowledge %s: %w", payload.KnowledgeID, err)
}
if knowledge == nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Knowledge %s not found, aborting.", payload.KnowledgeID)
return nil
}
if err := validateProcessingKnowledge(knowledge,
payload.TenantID,
payload.KnowledgeBaseID,
payload.KnowledgeID); err != nil {
return err
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil || kb == nil {
return fmt.Errorf("get knowledge base %s: %w", payload.KnowledgeBaseID, err)
}
if kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
return fmt.Errorf("postprocess task KB owner changed: %w", asynq.SkipRetry)
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
if err != nil {
return fmt.Errorf("invalid postprocess scope: %v: %w", err, asynq.SkipRetry)
}
// Resolve attempt: payload carries it from the upstream stage, but
// fall back to the latest known attempt for compatibility with
// in-flight tasks queued before this code shipped.
@ -108,16 +137,6 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
postSpan := s.tracker().BeginStage(ctx, payload.KnowledgeID, attempt, types.StagePostProcess, nil)
// 1. Fetch Knowledge and KB
knowledge, err := s.knowledgeRepo.GetKnowledgeByIDOnly(ctx, payload.KnowledgeID)
if err != nil {
return fmt.Errorf("get knowledge %s: %w", payload.KnowledgeID, err)
}
if knowledge == nil {
logger.Warnf(ctx, "[KnowledgePostProcess] Knowledge %s not found, aborting.", payload.KnowledgeID)
return nil
}
// Skip post-processing entirely when the knowledge has been cancelled
// by the user or marked for deletion. We must NOT enqueue summary /
// question / graph / wiki child tasks for an aborted knowledge. We
@ -136,11 +155,6 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
return nil
}
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
if err != nil || kb == nil {
return fmt.Errorf("get knowledge base %s: %w", payload.KnowledgeBaseID, err)
}
processOverrides, _ := knowledge.ProcessOverrides()
eff := ResolveProcessConfig(kb, processOverrides)
@ -150,6 +164,13 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
return fmt.Errorf("list chunks for knowledge %s: %w", payload.KnowledgeID, err)
}
for _, chunk := range chunks {
if chunk == nil || chunk.TenantID != payload.TenantID || chunk.KnowledgeID != payload.KnowledgeID ||
chunk.KnowledgeBaseID != payload.KnowledgeBaseID {
return fmt.Errorf("postprocess chunk binding changed: %w", asynq.SkipRetry)
}
}
// Gather all text-like chunks (including newly added OCR and Caption from multimodal tasks)
var textChunks []*types.Chunk
for _, c := range chunks {

View File

@ -117,21 +117,30 @@ func newWikiEnqueueTestService(
) (*KnowledgePostProcessService, *wikiEnqueueFailureKnowledgeRepo) {
repo := &wikiEnqueueFailureKnowledgeRepo{
knowledge: &types.Knowledge{
ID: knowledgeID,
ParseStatus: types.ParseStatusProcessing,
ID: knowledgeID,
TenantID: 7,
KnowledgeBaseID: "kb-wiki",
ParseStatus: types.ParseStatusProcessing,
},
}
pendingRepo.knowledgeRepo = repo
return &KnowledgePostProcessService{
knowledgeRepo: repo,
kbService: &wikiEnqueueFailureKBService{kb: &types.KnowledgeBase{
ID: "kb-wiki",
ID: "kb-wiki",
TenantID: 7,
IndexingStrategy: types.IndexingStrategy{
WikiEnabled: true,
},
}},
chunkService: &wikiEnqueueFailureChunkService{chunks: []*types.Chunk{
{ID: "chunk-1", ChunkType: types.ChunkTypeText},
{
ID: "chunk-1",
TenantID: 7,
KnowledgeID: knowledgeID,
KnowledgeBaseID: "kb-wiki",
ChunkType: types.ChunkTypeText,
},
}},
taskEnqueuer: queue,
pendingRepo: pendingRepo,
@ -219,3 +228,13 @@ func TestKnowledgePostProcessRetriesWikiTriggerWithoutDoubleAccounting(t *testin
queue.taskTypes,
)
}
func TestPostProcessRejectsMovedKnowledgeBeforeWikiOrSpanWrites(t *testing.T) {
pending := &wikiEnqueueFailurePendingRepo{}
queue := &wikiEnqueueFailureTaskQueue{}
svc, repo := newWikiEnqueueTestService("moved-doc", pending, queue)
repo.knowledge.KnowledgeBaseID = "other-kb"
require.ErrorIs(t, svc.Handle(context.Background(), newWikiEnqueuePostProcessTask(t, "moved-doc")), asynq.SkipRetry)
require.Zero(t, repo.expectedSubtasks)
require.Equal(t, types.ParseStatusProcessing, repo.knowledge.ParseStatus)
}

View File

@ -10,6 +10,8 @@ import (
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
"github.com/Tencent/WeKnora/internal/common"
werrors "github.com/Tencent/WeKnora/internal/errors"
@ -32,13 +34,21 @@ func (s *knowledgeService) cloneKnowledge(
src *types.Knowledge,
targetKB *types.KnowledgeBase,
) (err error) {
if src.ParseStatus != "completed" {
logger.GetLogger(ctx).WithField("knowledge_id", src.ID).Errorf("MoveKnowledge parse status is not completed")
return nil
sourceKB, err := knowledgeWriteKB(ctx, s.kbService, src)
if err != nil {
return err
}
if err := access.RequireKBTransfer(ctx, sourceKB, targetKB, access.KBTransferClone); err != nil {
return err
}
if src.ParseStatus != types.ParseStatusCompleted {
return fmt.Errorf("source knowledge %s is not completed", src.ID)
}
if _, err := s.transferChunks(ctx, src, sourceKB.ID); err != nil {
return err
}
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
dst := &types.Knowledge{
ID: uuid.New().String(),
ID: uuid.NewString(),
TenantID: targetKB.TenantID,
KnowledgeBaseID: targetKB.ID,
Type: src.Type,
@ -55,8 +65,21 @@ func (s *knowledgeService) cloneKnowledge(
FileSize: src.FileSize,
FileHash: src.FileHash,
FilePath: src.FilePath,
StorageSize: src.StorageSize,
StorageSize: 0,
Metadata: src.Metadata,
CustomMetadata: src.CustomMetadata,
}
state := knowledgeTransferState{
TaskID: access.TransferTaskID(ctx),
Operation: access.KBTransferClone,
SourceKB: sourceKB.ID,
TargetKB: targetKB.ID,
SourceID: src.ID,
Phase: "cloning",
}
if err := setTransferState(dst, state); err != nil {
return err
}
// Deep-copy the source document file into an object owned by the destination
@ -80,37 +103,52 @@ func (s *knowledgeService) cloneKnowledge(
}
defer func() {
if err != nil {
if len(copiedFilePaths) > 0 {
cleanupCopiedObjects(ctx, s.resolveFileService(ctx, targetKB), copiedFilePaths)
}
dst.ParseStatus = "failed"
dst.ErrorMessage = err.Error()
_ = s.repo.UpdateKnowledge(ctx, dst)
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge failed to move knowledge")
} else {
dst.ParseStatus = "completed"
dst.EnableStatus = "enabled"
_ = s.repo.UpdateKnowledge(ctx, dst)
logger.GetLogger(ctx).WithField("knowledge_id", dst.ID).Infof("MoveKnowledge move knowledge successfully")
if err == nil {
return
}
stored, loadErr := s.repo.GetKnowledgeByID(ctx, dst.TenantID, dst.ID)
if errors.Is(loadErr, repository.ErrKnowledgeNotFound) {
cleanupCopiedObjects(ctx, s.resolveFileService(ctx, targetKB), copiedFilePaths)
return
}
// An uncertain save/read must not delete objects that may be referenced.
if loadErr != nil || stored == nil || validateTransferKnowledge(stored, targetKB) != nil {
return
}
marker, markerErr := transferState(stored)
if markerErr != nil || !matchesTransfer(ctx, marker, sourceKB, targetKB, access.KBTransferClone, src.ID, "") ||
stored.ParseStatus == types.ParseStatusCompleted {
return
}
before, after := *stored, *stored
after.ParseStatus = types.ParseStatusFailed
after.ErrorMessage = err.Error()
_ = s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after)
}()
if err = s.repo.CreateKnowledge(ctx, dst); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge create knowledge failed")
return
return err
}
tenantInfo.StorageUsed += dst.StorageSize
if err = s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, dst.StorageSize); err != nil {
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge update tenant storage used failed")
return
// Create timestamps can be rounded by the database before the first CAS.
persisted, err := s.repo.GetKnowledgeByID(ctx, dst.TenantID, dst.ID)
if err != nil {
return err
}
if err := validateTransferKnowledge(persisted, targetKB); err != nil {
return err
}
dst.UpdatedAt = persisted.UpdatedAt
if err = s.CloneChunk(ctx, src, dst); err != nil {
logger.GetLogger(ctx).WithField("knowledge_id", dst.ID).
WithField("error", err).Errorf("MoveKnowledge move chunks failed")
return
return err
}
return
before := *dst
state.Phase = "done"
if err = setTransferState(dst, state); err != nil {
return err
}
dst.ParseStatus = types.ParseStatusCompleted
dst.EnableStatus = "enabled"
dst.StorageSize = src.StorageSize
return s.repo.UpdateKnowledgeForTransfer(ctx, &before, dst)
}
// processDocumentFromPassage handles asynchronous processing of text passages
@ -294,7 +332,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
logger.Infof(ctx, "Cleaning up existing chunks and index data for knowledge: %s", knowledge.ID)
// 删除旧的chunks
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
logger.Warnf(ctx, "Failed to delete existing chunks (may not exist): %v", err)
// 不返回错误,继续处理(可能没有旧数据)
}
@ -494,7 +532,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
s.beginStage(ctx, knowledge.ID, types.StageChunking, types.JSONMap{
"chunks_planned": len(insertChunks),
})
if err := s.chunkService.CreateChunks(ctx, insertChunks); err != nil {
if err := s.chunkRepo.CreateChunks(ctx, insertChunks); err != nil {
knowledge.ParseStatus = types.ParseStatusFailed
knowledge.ErrorMessage = err.Error()
knowledge.UpdatedAt = time.Now()
@ -574,7 +612,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
if aborted, status := s.isKnowledgeAborted(ctx, knowledge.TenantID, knowledge.ID); aborted {
logger.Infof(ctx, "Knowledge aborted (%s) before indexing: %s", status, knowledge.ID)
if status == types.ParseStatusDeleting {
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
logger.Warnf(ctx, "Failed to cleanup chunks after deletion detected: %v", err)
}
}
@ -589,7 +627,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
s.repo.UpdateKnowledge(ctx, knowledge)
// delete failed chunks
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
logger.Errorf(ctx, "Delete chunks failed: %v", err)
}
@ -622,7 +660,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
if aborted, status := s.isKnowledgeAborted(ctx, knowledge.TenantID, knowledge.ID); aborted {
logger.Infof(ctx, "Knowledge aborted (%s) after indexing: %s", status, knowledge.ID)
if status == types.ParseStatusDeleting {
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
logger.Warnf(ctx, "Failed to cleanup chunks after deletion detected: %v", err)
}
if err := retrieveEngine.DeleteByKnowledgeIDList(ctx, []string{knowledge.ID}, embeddingModel.GetDimensions(), kb.Type); err != nil {
@ -1319,7 +1357,7 @@ func (s *knowledgeService) ProcessSummaryGeneration(ctx context.Context, t *asyn
}
// Save summary chunk
if err := s.chunkService.CreateChunks(ctx, []*types.Chunk{summaryChunk}); err != nil {
if err := s.chunkRepo.CreateChunks(ctx, []*types.Chunk{summaryChunk}); err != nil {
logger.Errorf(ctx, "Failed to create summary chunk: %v", err)
summaryErr = err
return fmt.Errorf("failed to create summary chunk: %w", err)
@ -1726,7 +1764,7 @@ func (s *knowledgeService) processQuestionGenerationForKnowledge(ctx context.Con
}
// Update chunk in database
if err := s.chunkService.UpdateChunk(ctx, chunk); err != nil {
if err := s.chunkRepo.UpdateChunk(ctx, chunk); err != nil {
chunkUpdateFailed++
logger.Warnf(ctx, "Failed to update chunk %s: %v", chunk.ID, err)
continue
@ -2059,7 +2097,7 @@ func (s *knowledgeService) processQuestionGenerationForChunks(ctx context.Contex
logger.Warnf(ctx, "Failed to set document metadata for chunk %s: %v", chunk.ID, err)
continue
}
if err := s.chunkService.UpdateChunk(ctx, chunk); err != nil {
if err := s.chunkRepo.UpdateChunk(ctx, chunk); err != nil {
logger.Warnf(ctx, "Failed to update chunk %s: %v", chunk.ID, err)
continue
}
@ -2180,13 +2218,12 @@ func (s *knowledgeService) RegenerateChunkQuestions(
return nil, fmt.Errorf("questions can only be generated for text chunks")
}
generationRevision := chunk.ContentRevision
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
knowledge, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, chunk.KnowledgeID)
if err != nil {
return nil, err
}
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, chunk.KnowledgeBaseID)
if err != nil {
return nil, err
if knowledge.KnowledgeBaseID != chunk.KnowledgeBaseID || chunk.TenantID != knowledge.TenantID {
return nil, werrors.NewForbiddenError("chunk does not belong to its knowledge document")
}
if kb.SummaryModelID == "" {
return nil, fmt.Errorf("summary model is required for question generation")
@ -2200,7 +2237,7 @@ func (s *knowledgeService) RegenerateChunkQuestions(
return ""
}
neighbor, getErr := s.chunkRepo.GetChunkByID(ctx, tenantID, id)
if getErr != nil {
if getErr != nil || !sameChunkDocument(chunk, neighbor) {
return ""
}
return neighbor.Content
@ -2414,13 +2451,14 @@ func (s *knowledgeService) ReparseKnowledge(
) (*types.Knowledge, error) {
logger.Info(ctx, "Start re-parsing knowledge")
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
existing, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
existing, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
if err != nil {
logger.Errorf(ctx, "Failed to load knowledge: %v", err)
return nil, err
}
tenantID := existing.TenantID
// Allocate a fresh span tree attempt up front. Doing this BEFORE
// the cleanup + enqueue means: (a) the UI immediately sees a new
// attempt with all five stages back to "pending" instead of the
@ -2434,13 +2472,6 @@ func (s *knowledgeService) ReparseKnowledge(
logger.Warnf(ctx, "[Reparse] OpenAttempt failed for %s: %v (will fall back in worker)", existing.ID, err)
}
// Get knowledge base configuration
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, existing.KnowledgeBaseID)
if err != nil {
logger.Errorf(ctx, "Failed to get knowledge base for reparse: %v", err)
return nil, err
}
// When the caller supplies new overrides (e.g. via the reparse confirm
// dialog), validate them against this knowledge's file type, then persist
// to metadata so both this call's enqueue and the worker re-read the same
@ -2914,6 +2945,10 @@ func (s *knowledgeService) UpdateImageInfo(
chunkID string,
imageInfo string,
) error {
knowledge, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
if err != nil {
return err
}
imageInfo = common.CleanInvalidUTF8(imageInfo)
var images []*types.ImageInfo
if err := json.Unmarshal([]byte(imageInfo), &images); err != nil {
@ -2932,6 +2967,11 @@ func (s *knowledgeService) UpdateImageInfo(
logger.Errorf(ctx, "Failed to get chunk: %v", err)
return err
}
if chunk == nil || chunk.ID != chunkID || chunk.KnowledgeID != knowledge.ID ||
chunk.TenantID != knowledge.TenantID ||
chunk.KnowledgeBaseID != knowledge.KnowledgeBaseID {
return werrors.NewForbiddenError("chunk does not belong to its knowledge document")
}
chunk.ImageInfo = imageInfo
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
chunkChildren, err := s.chunkService.ListChunkByParentID(ctx, tenantID, chunkID)
@ -2942,6 +2982,11 @@ func (s *knowledgeService) UpdateImageInfo(
})
return err
}
for _, child := range chunkChildren {
if !sameChunkDocument(chunk, child) {
return werrors.NewForbiddenError("image child does not belong to its document")
}
}
logger.Infof(ctx, "Found %d chunks with parent chunk ID: %s", len(chunkChildren), chunkID)
// Iterate through each chunk and update its content based on the image information
@ -3029,7 +3074,7 @@ func (s *knowledgeService) UpdateImageInfo(
logger.Infof(ctx, "Updated %d chunks out of %d total chunks", len(updateChunk), len(chunkChildren)+1)
if len(addChunk) > 0 {
err := s.chunkService.CreateChunks(ctx, addChunk)
err := s.chunkRepo.CreateChunks(ctx, addChunk)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
"add_chunk_size": len(addChunk),
@ -3040,7 +3085,7 @@ func (s *knowledgeService) UpdateImageInfo(
// Update the chunks
for _, c := range updateChunk {
err := s.chunkService.UpdateChunk(ctx, c)
err := s.chunkRepo.UpdateChunk(ctx, c)
if err != nil {
logger.ErrorWithFields(ctx, err, map[string]interface{}{
"chunk_id": c.ID,
@ -3061,7 +3106,7 @@ func (s *knowledgeService) UpdateImageInfo(
}
// Update the knowledge file hash
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
knowledge, err = s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
if err != nil {
logger.Errorf(ctx, "Failed to get knowledge: %v", err)
return err
@ -3088,7 +3133,7 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
ctx = logger.WithRequestID(ctx, payload.RequestId)
ctx = logger.WithField(ctx, "manual_process", payload.KnowledgeID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
tenantInfo, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
if err != nil {
@ -3107,6 +3152,13 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
return nil
}
if err := validateProcessingKnowledge(knowledge,
payload.TenantID,
payload.KnowledgeBaseID,
payload.KnowledgeID); err != nil {
return err
}
// Skip if already completed or being deleted
if knowledge.ParseStatus == types.ParseStatusCompleted {
logger.Infof(ctx, "ProcessManualUpdate: already completed, skipping: %s", payload.KnowledgeID)
@ -3130,6 +3182,13 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
s.repo.UpdateKnowledge(ctx, knowledge)
return nil
}
if kb == nil || kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
return fmt.Errorf("processing task KB owner changed: %w", asynq.SkipRetry)
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
if err != nil {
return fmt.Errorf("invalid processing scope: %v: %w", err, asynq.SkipRetry)
}
// Re-check abort status right before marking processing — see the same
// note in ProcessDocument for the cancel race this guards.
@ -3185,7 +3244,7 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
ctx = logger.WithRequestID(ctx, payload.RequestId)
ctx = logger.WithField(ctx, "document_process", payload.KnowledgeID)
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
if payload.Language != "" {
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
}
@ -3216,6 +3275,13 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
return nil
}
if err := validateProcessingKnowledge(knowledge,
payload.TenantID,
payload.KnowledgeBaseID,
payload.KnowledgeID); err != nil {
return err
}
// 检查是否正在删除 / 已被用户取消 - 如果是则直接退出
if knowledge.ParseStatus == types.ParseStatusDeleting {
logger.Infof(ctx, "Knowledge is being deleted, aborting processing: %s", payload.KnowledgeID)
@ -3261,6 +3327,13 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
s.repo.UpdateKnowledge(ctx, knowledge)
return nil
}
if kb == nil || kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
return fmt.Errorf("processing task KB owner changed: %w", asynq.SkipRetry)
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
if err != nil {
return fmt.Errorf("invalid processing scope: %v: %w", err, asynq.SkipRetry)
}
processOverrides, _ := knowledge.ProcessOverrides()
eff := ResolveProcessConfig(kb, processOverrides)
@ -3926,6 +3999,11 @@ func (s *knowledgeService) ProcessKnowledgeListReparse(ctx context.Context, t *a
logger.Infof(ctx, "Processing knowledge list reparse task for %d knowledge items", len(payload.KnowledgeIDs))
ctx, ids, err := s.reparseTaskScope(ctx, payload)
if err != nil || len(ids) == 0 {
return err
}
tenant, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
if err != nil {
logger.Errorf(ctx, "Failed to get tenant %d: %v", payload.TenantID, err)
@ -3935,7 +4013,7 @@ func (s *knowledgeService) ProcessKnowledgeListReparse(ctx context.Context, t *a
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
ctx = context.WithValue(ctx, types.TenantInfoContextKey, tenant)
outcome, err := runKnowledgeListReparseSubmissions(payload.KnowledgeIDs, func(id string) error {
outcome, err := runKnowledgeListReparseSubmissions(ids, func(id string) error {
_, err := s.ReparseKnowledge(ctx, id, payload.ProcessConfig)
return err
})

View File

@ -29,11 +29,11 @@ func (r *parentChildKnowledgeRepo) UpdateKnowledge(
}
type parentChildChunkService struct {
interfaces.ChunkService
interfaces.ChunkRepository
created []*types.Chunk
}
func (s *parentChildChunkService) DeleteChunksByKnowledgeID(context.Context, string) error {
func (s *parentChildChunkService) DeleteChunksByKnowledgeID(context.Context, uint64, string) error {
return nil
}
@ -160,7 +160,7 @@ func TestProcessChunksIndexesEveryTextChild(t *testing.T) {
ctx := context.WithValue(context.Background(), types.TenantInfoContextKey, tenant)
svc := &knowledgeService{
repo: &parentChildKnowledgeRepo{knowledge: knowledge},
chunkService: chunkService,
chunkRepo: chunkService,
modelService: parentChildModelService{embedder: parentChildEmbedder{}},
retrieveEngine: parentChildRetrieveRegistry{engine: retrieveEngine},
graphEngine: parentChildGraphRepo{},

View File

@ -0,0 +1,60 @@
package service
import (
"context"
"fmt"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/hibiken/asynq"
)
// Validate the whole batch before any destructive submission. The exact KB
// grant is consumed again when ReparseKnowledge reloads each document.
func (s *knowledgeService) reparseTaskScope(
ctx context.Context, payload types.KnowledgeListReparsePayload,
) (context.Context, []string, error) {
if payload.TenantID == 0 {
return ctx, nil, fmt.Errorf("invalid reparse task tenant: %w", asynq.SkipRetry)
}
ids, err := writeResourceIDs(payload.KnowledgeIDs)
if err != nil {
return ctx, nil, fmt.Errorf("invalid reparse task IDs: %w", asynq.SkipRetry)
}
if len(ids) == 0 {
return ctx, ids, nil
}
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
rows, err := s.repo.GetKnowledgeBatch(ctx, payload.TenantID, ids)
if err != nil {
return ctx, nil, err
}
kbID := payload.KnowledgeBaseID
seen := make(map[string]bool, len(rows))
for _, row := range rows {
if row == nil || row.TenantID != payload.TenantID || row.KnowledgeBaseID == "" ||
(kbID != "" && row.KnowledgeBaseID != kbID) {
return ctx, nil, fmt.Errorf("reparse task binding changed: %w", asynq.SkipRetry)
}
if err := access.RejectMovingKnowledge(row); err != nil {
return ctx, nil, fmt.Errorf("reparse task document unavailable: %v: %w", err, asynq.SkipRetry)
}
// Legacy payloads can reconstruct only their current unambiguous KB.
kbID = row.KnowledgeBaseID
seen[row.ID] = true
}
for _, id := range ids {
if !seen[id] {
return ctx, nil, fmt.Errorf("reparse task document missing: %w", asynq.SkipRetry)
}
}
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, kbID)
if err != nil {
return ctx, nil, err
}
if kb == nil || kb.ID != kbID || kb.TenantID != payload.TenantID {
return ctx, nil, fmt.Errorf("reparse task KB binding changed: %w", asynq.SkipRetry)
}
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
return ctx, ids, err
}

View File

@ -21,45 +21,64 @@ type fakeKBShareService struct {
func (f *fakeKBShareService) ShareKnowledgeBase(context.Context, string, string, string, uint64, types.OrgMemberRole) (*types.KnowledgeBaseShare, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) UpdateSharePermission(context.Context, string, types.OrgMemberRole, string, uint64) error {
return errors.New("not implemented")
}
func (f *fakeKBShareService) RemoveShare(context.Context, string, string, uint64) error {
return errors.New("not implemented")
}
func (f *fakeKBShareService) ListSharesByKnowledgeBase(context.Context, string, uint64) ([]*types.KnowledgeBaseShare, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) ListSharesByOrganization(context.Context, string) ([]*types.KnowledgeBaseShare, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) ListSharedKnowledgeBases(context.Context, uint64, types.TenantRole) ([]*types.SharedKnowledgeBaseInfo, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) ListSharedKnowledgeBasesInOrganization(context.Context, string, uint64, types.TenantRole) ([]*types.OrganizationSharedKnowledgeBaseItem, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) ListSharedKnowledgeBaseIDsByOrganizations(context.Context, []string, uint64) (map[string][]string, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) GetShare(context.Context, string) (*types.KnowledgeBaseShare, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) GetShareByKBAndOrg(context.Context, string, string) (*types.KnowledgeBaseShare, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) CheckTenantKBPermission(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
return "", false, errors.New("not implemented")
func (f *fakeKBShareService) CheckTenantKBPermission(
_ context.Context,
kbID string,
_ uint64,
_ types.TenantRole,
) (types.OrgMemberRole, bool, error) {
return types.OrgRoleViewer, f.allowedKBs[kbID], nil
}
func (f *fakeKBShareService) HasTenantKBPermission(ctx context.Context, kbID string, callerTenantID uint64, callerTenantRole types.TenantRole, requiredRole types.OrgMemberRole) (bool, error) {
return f.allowedKBs[kbID], nil
}
func (f *fakeKBShareService) GetKBSourceTenant(context.Context, string) (uint64, error) {
return 0, errors.New("not implemented")
}
func (f *fakeKBShareService) CountSharesByKnowledgeBaseIDs(context.Context, []string) (map[string]int64, error) {
return nil, errors.New("not implemented")
}
func (f *fakeKBShareService) CountByOrganizations(context.Context, []string) (map[string]int64, error) {
return nil, errors.New("not implemented")
}
@ -153,3 +172,57 @@ func TestGetKnowledgeBatchWithSharedAccess_ExcludesSharedKnowledgeWithoutPermiss
}
var _ interfaces.KBShareService = (*fakeKBShareService)(nil)
type countingKBShareService struct {
fakeKBShareService
calls int
}
func (s *countingKBShareService) CheckTenantKBPermission(
ctx context.Context,
id string,
tenantID uint64,
role types.TenantRole,
) (types.OrgMemberRole, bool, error) {
s.calls++
return s.fakeKBShareService.CheckTenantKBPermission(ctx, id, tenantID, role)
}
func TestGetKnowledgeBatchWithSharedAccessChecksEachKBOnceAndRechecksNextRequest(t *testing.T) {
shares := &countingKBShareService{
fakeKBShareService: fakeKBShareService{allowedKBs: map[string]bool{"kb-shared": true}},
}
svc, db := newKnowledgeSharedAccessService(t, shares)
for _, id := range []string{"first", "second"} {
seedKnowledge(t, db, &types.Knowledge{ID: id, TenantID: 2, KnowledgeBaseID: "kb-shared", Type: "file"})
}
got, err := svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"first", "second"})
require.NoError(t, err)
require.Len(t, got, 2)
require.Equal(t, 1, shares.calls)
shares.allowedKBs["kb-shared"] = false
got, err = svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"first", "second"})
require.NoError(t, err)
require.Empty(t, got)
require.Equal(t, 2, shares.calls)
}
func TestResolveKBReadTenantPreservesServiceBoundary(t *testing.T) {
kb := &types.KnowledgeBase{ID: "kb", TenantID: 2}
shares := &fakeKBShareService{allowedKBs: map[string]bool{"kb": true}}
tenant, err := resolveKBReadTenant(newSharedAccessContext(), kb, shares)
require.NoError(t, err)
require.Equal(t, uint64(2), tenant)
_, err = resolveKBReadTenant(
context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1)),
kb,
shares,
)
require.Error(t, err, "direct cross-tenant calls still require a user identity")
_, err = resolveKBReadTenant(newSharedAccessContext(), kb, nil)
require.Error(t, err, "a missing share service cannot grant access")
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(2))
tenant, err = resolveKBReadTenant(ctx, kb, nil)
require.NoError(t, err, "a service execution context already scoped to the KB remains usable")
require.Equal(t, uint64(2), tenant)
}

View File

@ -0,0 +1,46 @@
package service
import (
"context"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/require"
)
type failingSharedKnowledgeRepo struct {
interfaces.KnowledgeRepository
err error
}
func (r *failingSharedKnowledgeRepo) GetKnowledgeBatch(context.Context, uint64, []string) ([]*types.Knowledge, error) {
return []*types.Knowledge{{ID: "own", TenantID: 1, KnowledgeBaseID: "own-kb"}}, nil
}
func (r *failingSharedKnowledgeRepo) GetKnowledgeByIDOnly(context.Context, string) (*types.Knowledge, error) {
return nil, r.err
}
func TestSharedDocumentReadsPropagateStorageFailure(t *testing.T) {
for _, failure := range []error{errors.New("database unavailable"), repository.ErrKnowledgeNotFound} {
repo := &failingSharedKnowledgeRepo{err: failure}
svc := &knowledgeService{repo: repo}
search := &knowledgeBaseService{kgRepo: repo}
rows, err := svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"own", "shared"})
found, searchErr := search.fetchKnowledgeDataWithShared(newSharedAccessContext(), 1, []string{"own", "shared"})
if errors.Is(failure, repository.ErrKnowledgeNotFound) {
require.NoError(t, err)
require.NoError(t, searchErr)
require.Len(t, rows, 1)
require.Len(t, found, 1)
} else {
require.ErrorIs(t, err, failure)
require.ErrorIs(t, searchErr, failure)
require.Nil(t, rows)
require.Nil(t, found)
}
}
}

View File

@ -0,0 +1,525 @@
package service
import (
"context"
"encoding/json"
"fmt"
"sort"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
"github.com/hibiken/asynq"
)
// Transfer state lives with the document, so queue retries do not depend on
// an expiring progress cache. It is server metadata, never custom metadata.
type knowledgeTransferState struct {
TaskID string `json:"task_id"`
Operation access.KBTransferOperation `json:"operation"`
SourceKB string `json:"source_kb"`
TargetKB string `json:"target_kb"`
SourceID string `json:"source_id"`
Mode string `json:"mode,omitempty"`
WikiChunkIDs []string `json:"wiki_chunk_ids,omitempty"`
WikiSummary string `json:"wiki_summary,omitempty"`
Phase string `json:"phase"`
}
func transferState(k *types.Knowledge) (*knowledgeTransferState, error) {
fields := map[string]json.RawMessage{}
if len(k.Metadata) != 0 {
if err := json.Unmarshal(k.Metadata, &fields); err != nil {
return nil, err
}
}
raw := fields[types.KnowledgeTransferMetadataKey]
if len(raw) == 0 {
return nil, nil
}
var state knowledgeTransferState
if err := json.Unmarshal(raw, &state); err != nil {
return nil, err
}
return &state, nil
}
func setTransferState(k *types.Knowledge, state knowledgeTransferState) error {
fields := map[string]json.RawMessage{}
if len(k.Metadata) != 0 {
if err := json.Unmarshal(k.Metadata, &fields); err != nil {
return err
}
}
if fields == nil {
fields = map[string]json.RawMessage{}
}
raw, err := json.Marshal(state)
if err != nil {
return err
}
fields[types.KnowledgeTransferMetadataKey] = raw
k.Metadata, err = json.Marshal(fields)
return err
}
func matchesTransfer(
ctx context.Context,
state *knowledgeTransferState,
source, target *types.KnowledgeBase,
operation access.KBTransferOperation,
sourceID, mode string,
) bool {
return state != nil && state.TaskID != "" && state.TaskID == access.TransferTaskID(ctx) &&
state.Operation == operation &&
state.SourceKB == source.ID &&
state.TargetKB == target.ID &&
state.SourceID == sourceID &&
state.Mode == mode
}
func validateTransferKnowledge(k *types.Knowledge, kb *types.KnowledgeBase) error {
if k == nil || k.ID == "" || k.TenantID != kb.TenantID || k.KnowledgeBaseID != kb.ID {
return fmt.Errorf("knowledge binding does not match the transfer scope")
}
return nil
}
// Read every chunk type and validate persisted document/KB/tag relationships
// before a clone or move can delete, index, create tags, or copy files.
func (s *knowledgeService) transferChunks(
ctx context.Context,
k *types.Knowledge,
allowedKBs ...string,
) ([]*types.Chunk, error) {
allowed := map[string]bool{}
for _, id := range allowedKBs {
allowed[id] = true
}
var all []*types.Chunk
seen := map[string]bool{}
tags := map[string]string{}
chunks, err := s.chunkRepo.ListAllChunksByKnowledgeID(ctx, k.TenantID, k.ID)
if err != nil {
return nil, err
}
for _, c := range chunks {
if c == nil || c.ID == "" || seen[c.ID] || c.TenantID != k.TenantID || c.KnowledgeID != k.ID ||
!allowed[c.KnowledgeBaseID] {
return nil, fmt.Errorf("chunk binding does not match the transfer scope")
}
seen[c.ID] = true
if c.TagID != "" && tags[c.TagID] != c.KnowledgeBaseID {
tag, err := s.tagRepo.GetByID(ctx, k.TenantID, c.TagID)
if err != nil {
return nil, err
}
if tag == nil || tag.ID != c.TagID || tag.TenantID != k.TenantID ||
tag.KnowledgeBaseID != c.KnowledgeBaseID {
return nil, fmt.Errorf("chunk tag belongs to another knowledge base")
}
tags[c.TagID] = c.KnowledgeBaseID
}
all = append(all, c)
}
for _, c := range all {
if c.ParentChunkID != "" && !seen[c.ParentChunkID] {
return nil, fmt.Errorf("chunk parent belongs to another document")
}
}
return all, nil
}
func (s *knowledgeService) planKnowledgeMove(
ctx context.Context,
source, target *types.KnowledgeBase,
ids []string,
mode string,
) ([]*types.Knowledge, error) {
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferMove); err != nil {
return nil, err
}
tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
if err := access.ValidateKBTransferCompatibility(source, target, access.KBTransferMove, mode, tenant); err != nil {
return nil, err
}
ids, err := writeResourceIDs(ids)
if err != nil {
return nil, err
}
if len(ids) == 0 {
return nil, fmt.Errorf("knowledge IDs cannot be empty")
}
rows, err := s.repo.GetKnowledgeBatch(ctx, source.TenantID, ids)
if err != nil {
return nil, err
}
byID := map[string]*types.Knowledge{}
for _, row := range rows {
if row != nil {
byID[row.ID] = row
}
}
engineChecked := false
result := make([]*types.Knowledge, 0, len(ids))
for _, id := range ids {
row := byID[id]
if row == nil || row.TenantID != source.TenantID {
return nil, fmt.Errorf("knowledge %s not found in transfer tenant", id)
}
if err := validateMoveItem(ctx, row, source, target, mode); err != nil {
return nil, err
}
allowed := []string{row.KnowledgeBaseID}
state, _ := transferState(row)
if matchesTransfer(ctx, state, source, target, access.KBTransferMove, row.ID, mode) && state.Phase == "moving" {
allowed = []string{source.ID, target.ID}
}
_, err = s.transferChunks(ctx, row, allowed...)
if err != nil {
return nil, err
}
if mode == "reuse_vectors" && row.EmbeddingModelID != "" {
if row.EmbeddingModelID != source.EmbeddingModelID {
return nil, fmt.Errorf("knowledge %s uses a different embedding model", row.ID)
}
if !engineChecked {
engine, err := retriever.CreateRetrieveEngineForKB(
ctx,
s.retrieveEngine,
s.ownership,
source.TenantID,
source.VectorStoreID,
)
if err != nil {
return nil, err
}
if err := engine.ValidateKnowledgeIndexMove(ctx); err != nil {
return nil, err
}
engineChecked = true
}
}
snapshot := *row
result = append(result, &snapshot)
}
return result, nil
}
func validateMoveItem(ctx context.Context, k *types.Knowledge, source, target *types.KnowledgeBase, mode string) error {
if k == nil || k.TenantID != source.TenantID {
return access.ErrForbidden
}
state, err := transferState(k)
if err != nil {
return err
}
if matchesTransfer(ctx, state, source, target, access.KBTransferMove, k.ID, mode) {
switch state.Phase {
case "moving":
if k.KnowledgeBaseID == source.ID &&
(k.ParseStatus == types.ParseStatusProcessing || k.ParseStatus == types.ParseStatusFailed) {
return nil
}
case "reparse_pending", "done":
if k.KnowledgeBaseID == target.ID {
return nil
}
}
return fmt.Errorf("knowledge %s changed during move", k.ID)
}
if k.KnowledgeBaseID != source.ID || k.ParseStatus != types.ParseStatusCompleted {
return fmt.Errorf("knowledge %s must be completed in the source knowledge base", k.ID)
}
if state != nil && state.Phase != "done" {
return fmt.Errorf("knowledge %s has an unfinished transfer", k.ID)
}
if mode == "reparse" {
if k.IsManual() {
meta, err := k.ManualMetadata()
if err != nil || meta == nil || meta.Content == "" {
return fmt.Errorf("knowledge %s has no manual content to reparse", k.ID)
}
} else if k.FilePath == "" {
return fmt.Errorf("knowledge %s has no stored file to reparse", k.ID)
}
}
return nil
}
type knowledgeClonePlan struct {
add []*types.Knowledge
remove []string
}
func (s *knowledgeService) planKnowledgeClone(
ctx context.Context,
source, target *types.KnowledgeBase,
) (*knowledgeClonePlan, error) {
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferClone); err != nil {
return nil, err
}
src, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, source.TenantID, source.ID)
if err != nil {
return nil, err
}
dst, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, target.TenantID, target.ID)
if err != nil {
return nil, err
}
for _, pair := range []struct {
rows []*types.Knowledge
kb *types.KnowledgeBase
}{{src, source}, {dst, target}} {
for _, k := range pair.rows {
if err := validateTransferKnowledge(k, pair.kb); err != nil {
return nil, err
}
if _, err := s.transferChunks(ctx, k, pair.kb.ID); err != nil {
return nil, err
}
}
}
sort.Slice(src, func(i, j int) bool { return src[i].ID < src[j].ID })
sort.Slice(dst, func(i, j int) bool { return dst[i].ID < dst[j].ID })
plan := &knowledgeClonePlan{}
used := map[string]bool{}
for _, k := range src {
if k.ParseStatus != types.ParseStatusCompleted {
return nil, fmt.Errorf("source knowledge %s is not completed", k.ID)
}
matched := false
// Prefer exact source bindings created by this task; unlike file hashes,
// these also identify manual/empty-hash documents across retries.
for _, other := range dst {
state, err := transferState(other)
if err != nil {
return nil, err
}
if !used[other.ID] && matchesTransfer(ctx, state, source, target, access.KBTransferClone, k.ID, "") {
used[other.ID] = true
if state.Phase == "done" && other.ParseStatus == types.ParseStatusCompleted {
matched = true
} else {
plan.remove = append(plan.remove, other.ID)
}
break
}
}
if !matched && k.FileHash != "" {
for _, other := range dst {
if !used[other.ID] && other.FileHash == k.FileHash && other.ParseStatus == types.ParseStatusCompleted {
used[other.ID] = true
matched = true
break
}
}
}
if !matched {
plan.add = append(plan.add, k)
}
}
for _, k := range dst {
if used[k.ID] {
continue
}
if k.ParseStatus == types.ParseStatusProcessing || k.ParseStatus == types.ParseStatusPending ||
k.ParseStatus == types.ParseStatusDeleting {
return nil, fmt.Errorf("target knowledge %s is busy", k.ID)
}
plan.remove = append(plan.remove, k.ID)
}
return plan, nil
}
func (s *knowledgeService) executeKnowledgeClone(
ctx context.Context,
source, target *types.KnowledgeBase,
progress func(int, int),
) error {
plan, err := s.planKnowledgeClone(ctx, source, target)
if err != nil {
return err
}
total, done := len(plan.add)+len(plan.remove), 0
if progress != nil {
progress(done, total)
}
if len(plan.remove) > 0 {
// The complete binding/preflight plan above precedes the first deletion.
if err := deleteReferencedKnowledge(ctx, s, target.ID, plan.remove); err != nil {
return err
}
done += len(plan.remove)
if progress != nil {
progress(done, total)
}
}
// Serial execution keeps storage accounting, tag creation and progress
// coherent; each completed document is independently resumable.
for _, k := range plan.add {
if err := s.cloneKnowledge(ctx, k, target); err != nil {
return err
}
done++
if progress != nil {
progress(done, total)
}
}
return nil
}
// acknowledgeMovedReparse completes transfer admission, not document parsing.
// The target processing pipeline owns ParseStatus after its task is enqueued.
func (s *knowledgeService) acknowledgeMovedReparse(
ctx context.Context,
knowledge *types.Knowledge,
source, target *types.KnowledgeBase,
) error {
// The processing worker can already have updated the row after enqueue.
// Reload before checkpointing; never overwrite its parse status or metadata.
current, err := s.repo.GetKnowledgeByID(ctx, knowledge.TenantID, knowledge.ID)
if err != nil {
return err
}
if err := validateTransferKnowledge(current, target); err != nil {
return err
}
state, err := transferState(current)
if err != nil {
return err
}
if !matchesTransfer(ctx, state, source, target, access.KBTransferMove, knowledge.ID, "reparse") {
return access.ErrForbidden
}
if state.Phase == "done" {
return nil
}
before := *current
after := *current
state.Phase = "done"
if err := setTransferState(&after, *state); err != nil {
return err
}
return s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after)
}
func (s *knowledgeService) preflightFAQClone(
ctx context.Context,
source, target *types.KnowledgeBase,
) (map[string]*types.Chunk, map[string]*types.Chunk, error) {
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferClone); err != nil {
return nil, nil, err
}
maps := []map[string]*types.Chunk{{}, {}}
for i, kb := range []*types.KnowledgeBase{source, target} {
rows, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, kb.TenantID, kb.ID)
if err != nil {
return nil, nil, err
}
if len(rows) > 1 {
return nil, nil, fmt.Errorf("FAQ knowledge base has multiple document containers")
}
for _, row := range rows {
if err := validateTransferKnowledge(row, kb); err != nil {
return nil, nil, err
}
if row.Type != types.KnowledgeTypeFAQ {
return nil, nil, fmt.Errorf("invalid FAQ container")
}
chunks, err := s.transferChunks(ctx, row, kb.ID)
if err != nil {
return nil, nil, err
}
for _, chunk := range chunks {
if chunk.ChunkType != types.ChunkTypeFAQ {
return nil, nil, fmt.Errorf("invalid FAQ chunk type")
}
maps[i][chunk.ID] = chunk
}
}
}
return maps[0], maps[1], nil
}
func (s *knowledgeService) markMoveItemFailed(
ctx context.Context,
id string,
source, target *types.KnowledgeBase,
mode string,
taskErr error,
) {
if access.RequireKBTransfer(ctx, source, target, access.KBTransferMove) != nil {
return
}
row, err := s.repo.GetKnowledgeByID(ctx, source.TenantID, id)
if err != nil || row == nil || row.ID != id || row.TenantID != source.TenantID {
return
}
state, err := transferState(row)
if err != nil || !matchesTransfer(ctx, state, source, target, access.KBTransferMove, id, mode) {
return
}
sourceFailed := row.KnowledgeBaseID == source.ID && state.Phase == "moving" &&
row.ParseStatus == types.ParseStatusProcessing
enqueueFailed := row.KnowledgeBaseID == target.ID && state.Phase == "reparse_pending" &&
row.ParseStatus == types.ParseStatusPending
if !sourceFailed && !enqueueFailed {
return
}
before, after := *row, *row
after.ParseStatus = types.ParseStatusFailed
after.ErrorMessage = taskErr.Error()
if err := s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after); err != nil {
logger.Errorf(ctx, "Failed to record move failure for knowledge %s, task %s: %v (move error: %v)",
row.ID, state.TaskID, err, taskErr)
}
}
// A stale parser must not start after a move has claimed a completed document,
// including the interval before its knowledge_base_id changes. Reparse tasks
// emitted by the move are admitted only once the destination checkpoint exists.
func validateProcessingKnowledge(knowledge *types.Knowledge, tenant uint64, kbID, knowledgeID string) error {
if knowledge == nil || tenant == 0 || kbID == "" || knowledge.ID != knowledgeID || knowledge.TenantID != tenant ||
knowledge.KnowledgeBaseID != kbID {
return fmt.Errorf("processing task binding changed: %w", asynq.SkipRetry)
}
state, err := transferState(knowledge)
if err != nil {
return fmt.Errorf("invalid processing metadata: %v: %w", err, asynq.SkipRetry)
}
if state != nil && state.Operation == access.KBTransferMove && state.Phase == "moving" {
return fmt.Errorf("knowledge is being moved: %w", asynq.SkipRetry)
}
return nil
}
// Cleanup consumes the durable source references only after target ownership
// is committed. A done transfer is retried here when wiki reconciliation fails.
func (s *knowledgeService) cleanupMovedSourceWiki(ctx context.Context, knowledge *types.Knowledge,
source, target *types.KnowledgeBase,
) error {
if !source.IsWikiEnabled() {
return nil
}
if knowledge.KnowledgeBaseID != target.ID {
return access.ErrForbidden
}
state, err := transferState(knowledge)
if err != nil {
return err
}
if state == nil || state.SourceKB != source.ID || state.TargetKB != target.ID {
return access.ErrForbidden
}
old := *knowledge
old.KnowledgeBaseID = source.ID
old.Description = state.WikiSummary
refs := make(map[string]bool, len(state.WikiChunkIDs))
for _, id := range state.WikiChunkIDs {
refs[id] = true
}
return s.cleanupWikiReferences(ctx, &old, refs)
}

View File

@ -0,0 +1,457 @@
package service
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/alicebob/miniredis/v2"
"github.com/hibiken/asynq"
"github.com/redis/go-redis/v9"
"github.com/stretchr/testify/require"
)
func transferFixture(t *testing.T, operation access.KBTransferOperation) *documentWriteFixture {
t.Helper()
f := newDocumentWriteFixture(t)
require.NoError(t, f.db.AutoMigrate(&types.Tenant{}))
require.NoError(t, f.db.Create(&types.Tenant{ID: 7, Name: "tenant", StorageUsed: 5}).Error)
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("parse_status", types.ParseStatusCompleted).Error,
)
ctx, err := access.WithKBTransferTask(
context.Background(),
f.kbs.values["kb"],
f.kbs.values["other"],
7,
operation,
"transfer-task",
false,
)
require.NoError(t, err)
f.ctx = context.WithValue(ctx, types.TenantInfoContextKey, &types.Tenant{ID: 7, StorageUsed: 5})
mr := miniredis.RunT(t)
f.svc.redisClient = redis.NewClient(&redis.Options{Addr: mr.Addr()})
t.Cleanup(func() { require.NoError(t, f.svc.redisClient.Close()) })
return f
}
func moveTask(t *testing.T, ids []string, mode string) *asynq.Task {
t.Helper()
raw, err := json.Marshal(
types.KnowledgeMovePayload{
TenantID: 7,
TaskID: "transfer-task",
SourceKBID: "kb",
TargetKBID: "other",
KnowledgeIDs: ids,
Mode: mode,
},
)
require.NoError(t, err)
return asynq.NewTask(types.TypeKnowledgeMove, raw)
}
func TestMoveBatchPreflightHasNoPartialMutations(t *testing.T) {
for _, ids := range [][]string{{"doc", "missing"}, {"doc", "other-doc"}, {"doc", ""}} {
t.Run(ids[1], func(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
err := f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, ids, "reuse_vectors"))
require.Error(t, err)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, types.ParseStatusCompleted, row.ParseStatus)
require.Equal(t, "kb", row.KnowledgeBaseID)
require.Empty(t, row.Metadata)
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
})
}
}
func TestMoveRejectsInvalidPairAndModeBeforeWrites(t *testing.T) {
for _, change := range []string{"tenant", "same KB", "mode", "chunk", "tag"} {
t.Run(change, func(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
mode := "reuse_vectors"
switch change {
case "tenant":
f.kbs.values["other"].TenantID = 8
case "same KB":
f.kbs.values["other"].ID = "kb"
case "mode":
mode = "bogus"
case "chunk":
require.NoError(
t,
f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("knowledge_base_id", "third").Error,
)
case "tag":
require.NoError(t, f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("tag_id", "tag").Error)
require.NoError(
t,
f.db.Model(&types.KnowledgeTag{}).Where("id = ?", "tag").Update("knowledge_base_id", "other").Error,
)
}
require.Error(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, mode)))
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, types.ParseStatusCompleted, row.ParseStatus)
require.Empty(t, row.Metadata)
})
}
}
type transferFaultRepo struct {
interfaces.KnowledgeRepository
failID string
claims int
}
func (r *transferFaultRepo) DeleteKnowledgeTagRelations(ctx context.Context, id string) error {
if id == r.failID {
return errors.New("temporary tag cleanup failure")
}
return r.KnowledgeRepository.DeleteKnowledgeTagRelations(ctx, id)
}
func (r *transferFaultRepo) UpdateKnowledgeForTransfer(ctx context.Context, before, after *types.Knowledge) error {
r.claims++
return r.KnowledgeRepository.UpdateKnowledgeForTransfer(ctx, before, after)
}
func TestMovePartialFailureResumesAndSkipsCompletedDocuments(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
require.NoError(
t,
f.db.Create(
&types.Knowledge{ID: "second", TenantID: 7, KnowledgeBaseID: "kb", ParseStatus: types.ParseStatusCompleted},
).Error,
)
require.NoError(
t,
f.db.Create(
&types.Chunk{
ID: "second-chunk",
TenantID: 7,
KnowledgeID: "second",
KnowledgeBaseID: "kb",
ChunkType: types.ChunkTypeImageOCR,
},
).Error,
)
fault := &transferFaultRepo{KnowledgeRepository: f.repo, failID: "second"}
f.svc.repo = fault
task := moveTask(t, []string{"doc", "doc", "second"}, "reuse_vectors")
require.ErrorContains(t, f.svc.ProcessKnowledgeMove(context.Background(), task), "temporary tag cleanup failure")
first, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "other", first.KnowledgeBaseID)
firstUpdated := first.UpdatedAt
second, err := f.repo.GetKnowledgeByID(f.ctx, 7, "second")
require.NoError(t, err)
require.Equal(t, "kb", second.KnowledgeBaseID)
state, err := transferState(second)
require.NoError(t, err)
require.Equal(t, "moving", state.Phase)
fault.failID = ""
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), task))
first, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, firstUpdated, first.UpdatedAt, "completed item must not be moved again")
second, err = f.repo.GetKnowledgeByID(f.ctx, 7, "second")
require.NoError(t, err)
require.Equal(t, "other", second.KnowledgeBaseID)
require.Equal(t, types.ParseStatusCompleted, second.ParseStatus)
progress, err := f.svc.GetKnowledgeMoveProgress(f.ctx, "transfer-task")
require.NoError(t, err)
require.Equal(t, 2, progress.Total)
require.Zero(t, progress.Failed)
require.Equal(t, types.KBCloneStatusCompleted, progress.Status)
}
type transferQueue struct {
fail bool
ids []string
}
func (q *transferQueue) Enqueue(_ *asynq.Task, options ...asynq.Option) (*asynq.TaskInfo, error) {
for _, option := range options {
if option.Type() == asynq.TaskIDOpt {
q.ids = append(q.ids, option.Value().(string))
}
}
if q.fail {
return nil, errors.New("queue unavailable")
}
return &asynq.TaskInfo{ID: "processing-task"}, nil
}
func TestMoveReparseRetryOnlyReenqueuesAndAccountsOnce(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
row.SetManualMetadata(types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1)),
)
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
queue := &transferQueue{fail: true}
f.svc.task = queue
task := moveTask(t, []string{"doc"}, "reparse")
require.ErrorContains(t, f.svc.ProcessKnowledgeMove(context.Background(), task), "queue unavailable")
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "other", row.KnowledgeBaseID)
require.Zero(t, row.StorageSize)
cleanupCalls := f.graph.calls
var tenant types.Tenant
require.NoError(t, f.db.First(&tenant, 7).Error)
require.Zero(t, tenant.StorageUsed)
// A parser may already have written manual metadata before the move's ack.
meta, err := row.ManualMetadata()
require.NoError(t, err)
require.NoError(t, row.SetManualMetadata(meta))
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
queue.fail = false
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), task))
require.Equal(t, cleanupCalls, f.graph.calls)
require.Len(t, queue.ids, 2)
require.Equal(t, queue.ids[0], queue.ids[1])
require.NoError(t, f.db.First(&tenant, 7).Error)
require.Zero(t, tenant.StorageUsed)
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
state, err := transferState(row)
require.NoError(t, err)
require.Equal(t, "done", state.Phase)
}
type cloneFaultChunks struct {
interfaces.ChunkRepository
fail bool
}
func (r *cloneFaultChunks) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
if err := r.ChunkRepository.CreateChunks(ctx, chunks); err != nil {
return err
}
if r.fail {
return errors.New("lost clone acknowledgement")
}
return nil
}
func TestCloneRetryReplacesIncompleteCopyWithoutDuplicatingEmptyHashes(t *testing.T) {
f := transferFixture(t, access.KBTransferClone)
fault := &cloneFaultChunks{ChunkRepository: f.chunkRepo, fail: true}
f.svc.chunkRepo = fault
require.ErrorContains(
t,
f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil),
"lost clone acknowledgement",
)
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
require.NoError(t, err)
require.Len(t, rows, 1)
require.Equal(t, types.ParseStatusFailed, rows[0].ParseStatus)
failedChunks, chunkErr := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, rows[0].ID)
require.NoError(t, chunkErr)
require.Empty(t, failedChunks, "a lost create acknowledgement must also be rolled back")
fault.fail = false
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
rows, err = f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
require.NoError(t, err)
require.Len(t, rows, 1)
require.Equal(t, types.ParseStatusCompleted, rows[0].ParseStatus)
id := rows[0].ID
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
rows, err = f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
require.NoError(t, err)
require.Len(t, rows, 1)
require.Equal(t, id, rows[0].ID)
var tenant types.Tenant
require.NoError(t, f.db.First(&tenant, 7).Error)
require.EqualValues(t, 10, tenant.StorageUsed)
}
func TestClonePreflightDoesNotDeleteTargetForUnreadySource(t *testing.T) {
f := transferFixture(t, access.KBTransferClone)
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("parse_status", types.ParseStatusProcessing).Error,
)
require.Error(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
_, err := f.repo.GetKnowledgeByID(f.ctx, 7, "other-doc")
require.NoError(t, err)
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
}
func TestTransferCheckpointCannotResurrectOrOverwriteChangedDocument(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
before, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
after := *before
after.StorageSize = 0
after.KnowledgeBaseID = "other"
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "third").Error)
require.Error(t, f.repo.UpdateKnowledgeForTransfer(f.ctx, before, &after))
var tenant types.Tenant
require.NoError(t, f.db.First(&tenant, 7).Error)
require.EqualValues(t, 5, tenant.StorageUsed)
require.NoError(t, f.db.Where("id = ?", "doc").Delete(&types.Knowledge{}).Error)
require.Error(t, f.repo.UpdateKnowledgeForTransfer(f.ctx, before, &after))
_, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.Error(t, err)
}
func TestProcessingTasksRejectOldKnowledgeBaseAfterMove(t *testing.T) {
for _, taskType := range []string{types.TypeDocumentProcess, types.TypeManualProcess} {
t.Run(taskType, func(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
require.NoError(
t,
f.db.Model(&types.Knowledge{}).
Where("id = ?", "doc").
Updates(map[string]any{"knowledge_base_id": "other", "parse_status": types.ParseStatusPending}).
Error,
)
payload, err := json.Marshal(
types.DocumentProcessPayload{TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb"},
)
require.NoError(t, err)
task := asynq.NewTask(taskType, payload)
if taskType == types.TypeDocumentProcess {
err = f.svc.ProcessDocument(context.Background(), task)
} else {
err = f.svc.ProcessManualUpdate(context.Background(), task)
}
require.ErrorIs(t, err, asynq.SkipRetry)
require.Zero(t, f.repo.writes)
require.Zero(t, f.graph.calls)
})
}
}
func TestCopyServiceResumesReservedTargetAndRequiresTransferGrant(t *testing.T) {
repo := newFakeKBRepo()
repo.rows["src"] = &types.KnowledgeBase{ID: "src", TenantID: 1}
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
require.ErrorIs(t, err, access.ErrForbidden)
require.Len(t, repo.rows, 1)
ctx, err := access.WithKBTransferTask(
context.Background(),
repo.rows["src"],
&types.KnowledgeBase{ID: "reserved", TenantID: 1, CreatorID: "creator"},
1,
access.KBTransferClone,
"task",
true,
)
require.NoError(t, err)
for range 2 {
_, target, err := svc.CopyKnowledgeBase(ctx, "src", "reserved")
require.NoError(t, err)
require.Equal(t, "reserved", target.ID)
require.Equal(t, "creator", target.CreatorID)
}
require.Len(t, repo.rows, 2)
_, _, err = svc.CopyKnowledgeBase(ctx, "src", "other")
require.ErrorIs(t, err, access.ErrForbidden)
}
func TestProcessingCannotEnterClaimedMoveBeforeKBBindingChanges(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
setTransferState(
row,
knowledgeTransferState{
TaskID: "transfer-task",
Operation: access.KBTransferMove,
SourceKB: "kb",
TargetKB: "other",
SourceID: "doc",
Mode: "reuse_vectors",
Phase: "moving",
},
),
)
row.ParseStatus = types.ParseStatusProcessing
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
f.repo.writes = 0
payload, err := json.Marshal(types.DocumentProcessPayload{TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb"})
require.NoError(t, err)
require.ErrorIs(
t,
f.svc.ProcessDocument(context.Background(), asynq.NewTask(types.TypeDocumentProcess, payload)),
asynq.SkipRetry,
)
require.Zero(t, f.repo.writes)
require.Zero(t, f.graph.calls)
}
func TestAdmissionProgressCannotOverwriteFinishedWorker(t *testing.T) {
f := transferFixture(t, access.KBTransferMove)
require.NoError(
t,
f.svc.saveKnowledgeMoveProgress(
f.ctx,
&types.KnowledgeMoveProgress{TaskID: "task", Status: types.KBCloneStatusCompleted, Progress: 100},
),
)
require.NoError(
t,
f.svc.SaveKnowledgeMoveProgress(
f.ctx,
&types.KnowledgeMoveProgress{TaskID: "task", Status: types.KBCloneStatusPending},
),
)
progress, err := f.svc.GetKnowledgeMoveProgress(f.ctx, "task")
require.NoError(t, err)
require.Equal(t, types.KBCloneStatusCompleted, progress.Status)
require.Equal(t, 100, progress.Progress)
}
func TestClonePreservesFailedChunkIndexStatus(t *testing.T) {
f := transferFixture(t, access.KBTransferClone)
require.NoError(t, f.db.Exec(`CREATE TRIGGER truncate_clone_created_time AFTER INSERT ON knowledges
BEGIN UPDATE knowledges SET updated_at = strftime('%Y-%m-%d %H:%M:%S', NEW.updated_at) || '+00:00'
WHERE id = NEW.id; END`).Error)
require.NoError(t, f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("index_status", "failed").Error)
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
require.NoError(t, err)
require.Len(t, rows, 1)
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, rows[0].ID)
require.NoError(t, err)
require.Len(t, chunks, 1)
require.Equal(t, "failed", chunks[0].IndexStatus)
}
func TestTableSummaryCleanupDoesNotFollowMovedDocument(t *testing.T) {
f := newDocumentWriteFixture(t)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
chunk, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
require.NoError(t, err)
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error)
worker := &DataTableSummaryService{knowledgeService: f.svc, chunkService: f.chunks}
worker.cleanupOnFailure(
f.ctx, &extractionResources{knowledge: row}, []*types.Chunk{chunk}, errors.New("index failed"),
)
f.requireNoWrites(t)
current, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "other", current.KnowledgeBaseID)
require.Equal(t, row.ParseStatus, current.ParseStatus)
}

View File

@ -4,6 +4,7 @@ import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
@ -21,6 +22,17 @@ func TestUpdateKnowledgeExplicitEmptyDescriptionClearsSummary(t *testing.T) {
repo: repo, kbService: metadataUpdateKBService{},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
err := service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",
@ -45,6 +57,17 @@ func TestUpdateKnowledgeOmittedDescriptionPreservesExisting(t *testing.T) {
repo: repo, kbService: metadataUpdateKBService{},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
err := service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",
@ -68,6 +91,17 @@ func TestUpdateKnowledgeManualDescriptionSetsCompletedStatus(t *testing.T) {
repo: repo, kbService: metadataUpdateKBService{},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
err := service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",
@ -93,6 +127,17 @@ func TestUpdateKnowledgeDescriptionOnlyDoesNotTouchMetadata(t *testing.T) {
repo: repo, kbService: metadataUpdateKBService{},
}
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
ctx = (&access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{
ID: "kb-1",
TenantID: 7,
},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}).Context(
ctx,
)
err := service.UpdateKnowledge(ctx, &types.Knowledge{
ID: "knowledge-1",

View File

@ -0,0 +1,141 @@
package service
import (
"context"
"strings"
"github.com/Tencent/WeKnora/internal/application/access"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
type knowledgeBaseWriteLookup interface {
GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error)
}
func writeResourceIDs(ids []string) ([]string, error) {
result := make([]string, 0, len(ids))
seen := make(map[string]bool, len(ids))
for _, id := range ids {
if strings.TrimSpace(id) == "" {
return nil, apperrors.NewBadRequestError("resource ID cannot be empty")
}
if !seen[id] {
seen[id] = true
result = append(result, id)
}
}
return result, nil
}
func writeExecutionTenant(ctx context.Context) (uint64, error) {
tenant, ok := types.TenantIDFromContext(ctx)
if !ok || tenant == 0 {
return 0, apperrors.NewUnauthorizedError("workspace context unavailable")
}
return tenant, nil
}
// Resolve persisted bindings before consuming grants. An input object's KB or
// tenant must never select the scope used to authorize an existing resource.
func knowledgeWriteKB(
ctx context.Context,
lookup knowledgeBaseWriteLookup,
knowledge *types.Knowledge,
) (*types.KnowledgeBase, error) {
if knowledge == nil || knowledge.ID == "" || knowledge.KnowledgeBaseID == "" || knowledge.TenantID == 0 {
return nil, apperrors.NewNotFoundError("knowledge not found")
}
kb, err := lookup.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
if err != nil {
return nil, err
}
if kb == nil || kb.ID != knowledge.KnowledgeBaseID || kb.TenantID != knowledge.TenantID {
return nil, apperrors.NewForbiddenError("knowledge does not belong to its knowledge base")
}
return kb, nil
}
func loadKnowledgeWrite(
ctx context.Context,
repo interfaces.KnowledgeRepository,
lookup knowledgeBaseWriteLookup,
id string,
) (*types.Knowledge, *types.KnowledgeBase, error) {
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return nil, nil, err
}
knowledge, err := repo.GetKnowledgeByID(ctx, tenant, id)
if err != nil {
return nil, nil, err
}
if knowledge == nil || knowledge.ID != id || knowledge.TenantID != tenant {
return nil, nil, apperrors.NewNotFoundError("knowledge not found")
}
if err := access.RejectMovingKnowledge(knowledge); err != nil {
return nil, nil, err
}
kb, err := knowledgeWriteKB(ctx, lookup, knowledge)
if err != nil {
return nil, nil, err
}
if _, err := requireKBWrite(ctx, kb); err != nil {
return nil, nil, err
}
copyOfKnowledge := *knowledge
return &copyOfKnowledge, kb, nil
}
// loadKnowledgeWriteBatch validates every requested ID, including missing
// entries, before any caller can apply changes. Each KB needs its own grant.
func loadKnowledgeWriteBatch(
ctx context.Context,
repo interfaces.KnowledgeRepository,
lookup knowledgeBaseWriteLookup,
ids []string,
) ([]*types.Knowledge, error) {
ids, err := writeResourceIDs(ids)
if err != nil || len(ids) == 0 {
return nil, err
}
tenant, err := writeExecutionTenant(ctx)
if err != nil {
return nil, err
}
rows, err := repo.GetKnowledgeBatch(ctx, tenant, ids)
if err != nil {
return nil, err
}
byID := make(map[string]*types.Knowledge, len(rows))
for _, row := range rows {
if row != nil && row.TenantID == tenant {
byID[row.ID] = row
}
}
checked := make(map[string]bool)
result := make([]*types.Knowledge, 0, len(ids))
for _, id := range ids {
row := byID[id]
if row == nil {
return nil, apperrors.NewNotFoundError("knowledge not found")
}
if err := access.RejectMovingKnowledge(row); err != nil {
return nil, err
}
if !checked[row.KnowledgeBaseID] {
kb, err := knowledgeWriteKB(ctx, lookup, row)
if err != nil {
return nil, err
}
if _, err := requireKBWrite(ctx, kb); err != nil {
return nil, err
}
checked[row.KnowledgeBaseID] = true
}
copyOfRow := *row
result = append(result, &copyOfRow)
}
return result, nil
}

View File

@ -0,0 +1,442 @@
package service
import (
"context"
"encoding/json"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
"github.com/stretchr/testify/require"
"gorm.io/driver/sqlite"
"gorm.io/gorm"
)
type writeKBLookup struct {
interfaces.KnowledgeBaseService
kb *types.KnowledgeBase
}
func (s *writeKBLookup) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
return s.kb, nil
}
type writeChunkSpy struct {
interfaces.ChunkRepository
writes int
}
func (s *writeChunkSpy) UpdateChunkFieldsByTagID(
ctx context.Context,
tenant uint64,
kb, tag string,
enabled *bool,
set, clearFlags types.ChunkFlags,
next *string,
exclude []string,
) ([]string, error) {
s.writes++
return s.ChunkRepository.UpdateChunkFieldsByTagID(ctx, tenant, kb, tag, enabled, set, clearFlags, next, exclude)
}
func (s *writeChunkSpy) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
s.writes++
return s.ChunkRepository.UpdateChunks(ctx, chunks)
}
func (s *writeChunkSpy) UpdateChunkFlagsBatch(
ctx context.Context,
tenant uint64,
kb string,
set, clearFlags map[string]types.ChunkFlags,
) error {
s.writes++
return s.ChunkRepository.UpdateChunkFlagsBatch(ctx, tenant, kb, set, clearFlags)
}
type writeChunkDeleter struct {
interfaces.ChunkService
deleted []string
}
func (s *writeChunkDeleter) DeleteChunk(_ context.Context, id string) error {
s.deleted = append(s.deleted, id)
return nil
}
type writeIndexSpy struct {
interfaces.RetrieveEngineService
enabled map[string]bool
tags map[string]string
}
func (*writeIndexSpy) EngineType() types.RetrieverEngineType {
return types.PostgresRetrieverEngineType
}
func (*writeIndexSpy) Support() []types.RetrieverType {
return []types.RetrieverType{types.VectorRetrieverType}
}
func (s *writeIndexSpy) BatchUpdateChunkEnabledStatus(_ context.Context, values map[string]bool) error {
s.enabled = values
return nil
}
func (s *writeIndexSpy) BatchUpdateChunkTagID(_ context.Context, values map[string]string) error {
s.tags = values
return nil
}
type faqWriteFixture struct {
svc *knowledgeService
tags *knowledgeTagService
db *gorm.DB
ctx context.Context
chunks *writeChunkSpy
deletes *writeChunkDeleter
index *writeIndexSpy
kb *types.KnowledgeBase
}
func newFAQWriteFixture(t *testing.T) *faqWriteFixture {
t.Helper()
db, err := gorm.Open(sqlite.Open("file:"+t.Name()+"?mode=memory&cache=shared"), &gorm.Config{})
require.NoError(t, err)
require.NoError(t, db.AutoMigrate(&types.Knowledge{}, &types.Chunk{}, &types.KnowledgeTag{}))
store := "write-store"
kb := &types.KnowledgeBase{ID: "kb", TenantID: 7, Type: types.KnowledgeBaseTypeFAQ, VectorStoreID: &store}
lookup := &writeKBLookup{kb: kb}
chunks := &writeChunkSpy{ChunkRepository: repository.NewChunkRepository(db)}
deletes := &writeChunkDeleter{}
index := &writeIndexSpy{}
svc := &knowledgeService{
kbService: lookup, chunkRepo: chunks, chunkService: deletes,
tagRepo: repository.NewKnowledgeTagRepository(db), repo: repository.NewKnowledgeRepository(db),
retrieveEngine: &fakeFanoutRegistry{byStore: map[string]interfaces.RetrieveEngineService{store: index}},
ownership: &fakeOwnership{owned: map[string]uint64{store: 7}},
}
base := types.WithCaller(
context.Background(),
types.Caller{TenantID: 1, UserID: "caller", Role: types.TenantRoleAdmin},
)
grant := &access.KBAccess{
KnowledgeBase: kb,
Caller: types.CallerFromContext(base),
EffectiveTenantID: 7,
Permission: types.OrgRoleEditor,
}
ctx := context.WithValue(grant.Context(base), types.TenantInfoContextKey, &types.Tenant{ID: 7})
for _, tag := range []*types.KnowledgeTag{
{ID: "tag", SeqID: 11, TenantID: 7, KnowledgeBaseID: "kb", Name: "tag"},
{ID: "next", SeqID: 12, TenantID: 7, KnowledgeBaseID: "kb", Name: "next"},
{ID: "foreign", SeqID: 22, TenantID: 7, KnowledgeBaseID: "other", Name: "foreign"},
} {
require.NoError(t, db.Create(tag).Error)
}
for _, chunk := range []*types.Chunk{
{
ID: "one",
SeqID: 1,
TenantID: 7,
KnowledgeBaseID: "kb",
KnowledgeID: "faq",
ChunkType: types.ChunkTypeFAQ,
TagID: "tag",
IsEnabled: true,
},
{
ID: "two",
SeqID: 2,
TenantID: 7,
KnowledgeBaseID: "other",
KnowledgeID: "foreign-faq",
ChunkType: types.ChunkTypeFAQ,
TagID: "foreign",
IsEnabled: true,
},
} {
require.NoError(t, db.Create(chunk).Error)
}
require.NoError(
t,
db.Create(&types.Knowledge{ID: "faq", TenantID: 7, KnowledgeBaseID: "kb", Type: types.KnowledgeTypeFAQ}).Error,
)
tags := &knowledgeTagService{kbService: lookup, repo: svc.tagRepo}
tags.chunkRepo = chunks
svc.tagService = tags
return &faqWriteFixture{
svc: svc,
tags: tags,
db: db,
ctx: ctx,
chunks: chunks,
deletes: deletes,
index: index,
kb: kb,
}
}
func TestFAQAndTagWritesRejectUnscopedServiceCalls(t *testing.T) {
f := newFAQWriteFixture(t)
ctx := types.WithExecutionTenant(types.WithCaller(context.Background(), types.Caller{TenantID: 1}), 7)
operations := map[string]func(context.Context) error{
"create FAQ": func(ctx context.Context) error {
_, err := f.svc.CreateFAQEntry(ctx, "kb", &types.FAQEntryPayload{})
return err
},
"update FAQ": func(ctx context.Context) error {
_, err := f.svc.UpdateFAQEntry(ctx, "kb", 1, &types.FAQEntryPayload{})
return err
},
"similar questions": func(ctx context.Context) error {
_, err := f.svc.AddSimilarQuestions(ctx, "kb", 1, []string{"question"})
return err
},
"status": func(ctx context.Context) error { return f.svc.UpdateFAQEntryStatus(ctx, "kb", "one", false) },
"fields": func(ctx context.Context) error {
return f.svc.UpdateFAQEntryFieldsBatch(
ctx,
"kb",
&types.FAQEntryFieldsBatchUpdate{ByID: map[int64]types.FAQEntryFieldsUpdate{1: {}}},
)
},
"single tag": func(ctx context.Context) error { return f.svc.UpdateFAQEntryTag(ctx, "kb", "one", nil) },
"batch tags": func(ctx context.Context) error {
return f.svc.UpdateFAQEntryTagBatch(ctx, "kb", map[int64]*int64{1: nil})
},
"delete": func(ctx context.Context) error { return f.svc.DeleteFAQEntries(ctx, "kb", []int64{1}) },
"import": func(ctx context.Context) error {
_, err := f.svc.UpsertFAQEntries(
ctx,
"kb",
&types.FAQBatchUpsertPayload{Entries: []types.FAQEntryPayload{{}}},
)
return err
},
"import display": func(ctx context.Context) error {
return f.svc.UpdateLastFAQImportResultDisplayStatus(ctx, "kb", "close")
},
"create tag": func(ctx context.Context) error {
_,
err := f.tags.CreateTag(ctx,
"kb",
"new",
"",
0)
return err
},
"find/create tag": func(ctx context.Context) error {
_,
err := f.tags.FindOrCreateTagByName(ctx,
"kb",
"tag")
return err
},
"update tag": func(ctx context.Context) error {
_,
err := f.tags.UpdateTag(ctx,
"tag",
nil,
nil,
nil)
return err
},
"delete tag": func(ctx context.Context) error { return f.tags.DeleteTag(ctx, "tag", true, false, nil) },
}
for name, operation := range operations {
t.Run(name, func(t *testing.T) { require.Error(t, operation(ctx)) })
}
require.Zero(t, f.chunks.writes)
require.Empty(t, f.deletes.deleted)
// The same service accepts a correctly scoped shared Editor operation.
tag, err := f.tags.CreateTag(f.ctx, "kb", "created", "", 0)
require.NoError(t, err)
require.Equal(t, uint64(7), tag.TenantID)
}
func TestFAQBatchPreflightRejectsBeforeAnyWrite(t *testing.T) {
no := false
foreign := int64(22)
for name, req := range map[string]*types.FAQEntryFieldsBatchUpdate{
"foreign entry after group": {
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
ByID: map[int64]types.FAQEntryFieldsUpdate{2: {IsEnabled: &no}},
},
"missing entry after group": {
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
ByID: map[int64]types.FAQEntryFieldsUpdate{999: {}},
},
"foreign source tag": {ByTag: map[int64]types.FAQEntryFieldsUpdate{
11: {IsEnabled: &no},
22: {IsEnabled: &no},
}},
"foreign destination tag": {ByID: map[int64]types.FAQEntryFieldsUpdate{1: {TagID: &foreign}}},
"foreign exclusion": {
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
ExcludeIDs: []int64{2},
},
"missing exclusion": {
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
ExcludeIDs: []int64{999},
},
} {
t.Run(name, func(t *testing.T) {
f := newFAQWriteFixture(t)
require.Error(t, f.svc.UpdateFAQEntryFieldsBatch(f.ctx, "kb", req))
require.Zero(t, f.chunks.writes)
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
require.NoError(t, err)
require.True(t, chunk.IsEnabled)
require.Equal(t, "tag", chunk.TagID)
})
}
}
func TestFAQDeleteValidatesEntireSelectionAndParents(t *testing.T) {
for _, ids := range [][]int64{{1, 2}, {1, 999}, {1, 0}} {
t.Run(string(mustWriteJSON(t, ids)), func(t *testing.T) {
f := newFAQWriteFixture(t)
require.Error(t, f.svc.DeleteFAQEntries(f.ctx, "kb", ids))
require.Empty(t, f.deletes.deleted)
})
}
t.Run("foreign parent", func(t *testing.T) {
f := newFAQWriteFixture(t)
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "faq").Update("knowledge_base_id", "other").Error,
)
require.Error(t, f.svc.DeleteFAQEntries(f.ctx, "kb", []int64{1}))
require.Empty(t, f.deletes.deleted)
})
}
func TestFAQFieldGroupsAndExplicitUpdatesPreservePrecedence(t *testing.T) {
f := newFAQWriteFixture(t)
yes, no := true, false
next := int64(12)
req := &types.FAQEntryFieldsBatchUpdate{
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {TagID: &next, IsRecommended: &yes}},
ByID: map[int64]types.FAQEntryFieldsUpdate{1: {IsEnabled: &no, IsRecommended: &no}},
}
require.NoError(t, f.svc.UpdateFAQEntryFieldsBatch(f.ctx, "kb", req))
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
require.NoError(t, err)
require.False(t, chunk.IsEnabled)
require.Equal(t, "next", chunk.TagID, "an explicit enabled patch must preserve the group's tag change")
require.False(t, chunk.Flags.HasFlag(types.ChunkFlagRecommended))
require.Equal(t, map[string]string{"one": "next"}, f.index.tags)
require.Equal(t, map[string]bool{"one": false}, f.index.enabled)
}
func TestFAQTagOnlyBatchUsesValidatedPlanAndSynchronizesIndex(t *testing.T) {
f := newFAQWriteFixture(t)
foreign := int64(22)
require.Error(t, f.svc.UpdateFAQEntryTagBatch(f.ctx, "kb", map[int64]*int64{1: &foreign}))
require.Zero(t, f.chunks.writes)
require.NoError(t, f.svc.UpdateFAQEntryTagBatch(f.ctx, "kb", map[int64]*int64{1: nil}))
require.Equal(t, map[string]string{"one": ""}, f.index.tags)
}
func TestFAQImportRejectsForeignTagsBeforeEnqueue(t *testing.T) {
f := newFAQWriteFixture(t)
_, err := f.svc.UpsertFAQEntries(
f.ctx,
"kb",
&types.FAQBatchUpsertPayload{Entries: []types.FAQEntryPayload{{TagID: 22}}},
)
require.Error(t, err) // Redis, file storage and the queue are intentionally unwired.
require.Zero(t, f.chunks.writes)
_, err = f.svc.resolveTagID(f.ctx, "kb", &types.FAQEntryPayload{TagID: 22})
require.Error(t, err)
resolver := f.svc.buildFAQTagResolver(f.ctx, "kb", []types.FAQEntryPayload{{TagID: 22}})
_, err = resolver(&types.FAQEntryPayload{TagID: 22})
require.Error(t, err)
}
func TestFAQImportWorkerRejectsMismatchedScopeBeforeSideEffects(t *testing.T) {
for _, tenant := range []uint64{7, 8} {
t.Run(string(mustWriteJSON(t, tenant)), func(t *testing.T) {
f := newFAQWriteFixture(t)
require.NoError(
t,
f.db.Create(
&types.Knowledge{
ID: "foreign-faq",
TenantID: 7,
KnowledgeBaseID: "other",
Type: types.KnowledgeTypeFAQ,
},
).Error,
)
payload := types.FAQImportPayload{TenantID: tenant, KBID: "kb", KnowledgeID: "foreign-faq", TaskID: "task"}
err := f.svc.ProcessFAQImport(
context.Background(),
asynq.NewTask(types.TypeFAQImport, mustWriteJSON(t, payload)),
)
require.ErrorIs(t, err, asynq.SkipRetry)
require.Zero(t, f.chunks.writes)
})
}
}
func TestTagDeleteRejectsInvalidExclusionsBeforeDeletion(t *testing.T) {
for _, id := range []string{"two", "missing"} {
t.Run(id, func(t *testing.T) {
f := newFAQWriteFixture(t)
require.Error(t, f.tags.DeleteTag(f.ctx, "tag", true, false, []string{id}))
tag, err := f.svc.tagRepo.GetByID(f.ctx, 7, "tag")
require.NoError(t, err)
require.NotNil(t, tag)
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
require.NoError(t, err)
require.NotNil(t, chunk)
})
}
}
func mustWriteJSON(t *testing.T, value interface{}) []byte {
t.Helper()
data, err := json.Marshal(value)
require.NoError(t, err)
return data
}
func TestSharedFAQWriteLoadsOwnerTenantInfoWithoutReplacingCaller(t *testing.T) {
f := newFAQWriteFixture(t)
f.svc.tenantRepo = &processSyncTenantRepo{tenant: &types.Tenant{ID: 7}}
ctx := context.WithValue(f.ctx, types.TenantInfoContextKey, &types.Tenant{ID: 1})
_, scoped, err := f.svc.writableFAQKnowledgeBase(ctx, "kb")
require.NoError(t, err)
owner, _ := types.TenantInfoFromContext(scoped)
require.Equal(t, uint64(7), owner.ID)
require.Equal(t, uint64(1), types.CallerFromContext(scoped).TenantID)
original, _ := types.TenantInfoFromContext(ctx)
require.Equal(t, uint64(1), original.ID)
}
func TestDataSourceTagCreationReceivesOnlyItsTaskKBGrant(t *testing.T) {
h := newSyncDeletionHarness(t, false, "ds-tag-scope", "log-tag-scope", nil, nil)
tags := h.svc.tagService.(*processSyncTagService)
_, err := h.run(t)
require.NoError(t, err)
require.NotNil(t, tags.ctx)
require.Zero(t, types.CallerFromContext(tags.ctx).TenantID)
require.NoError(
t,
access.RequireKBWrite(tags.ctx, &types.KnowledgeBase{ID: h.ds.KnowledgeBaseID, TenantID: h.ds.TenantID}),
)
require.Error(t, access.RequireKBWrite(tags.ctx, &types.KnowledgeBase{ID: "other", TenantID: h.ds.TenantID}))
}

View File

@ -9,6 +9,8 @@ import (
"strings"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
"github.com/Tencent/WeKnora/internal/datasource"
apperrors "github.com/Tencent/WeKnora/internal/errors"
@ -1099,72 +1101,57 @@ func (s *knowledgeBaseService) SetEmbeddingModel(ctx context.Context, id string,
// CopyKnowledgeBase copies a knowledge base to a new knowledge base (shallow copy).
// Source and target must belong to the tenant in context; cross-tenant access is rejected.
//
// Defensive checks:
//
// - When dstKB != "" (clone into an existing target), the source's
// EmbeddingModelID and VectorStoreID must match the target's. Mismatched
// embedding models would silently mix incompatible vector spaces;
// mismatched vector stores would require copying physical vector data
// between stores, which is not yet supported.
// - When dstKB == "" (create a new target), VectorStoreID is copied from
// the source so the new KB shares the same physical vector index. GORM
// `<-:create` allows INSERT, so the new row is well-formed.
//
// The handler's CopyKnowledgeBase endpoint runs the same checks synchronously
// before enqueueing the async clone task, so the 400 errors here are
// defense-in-depth for the worker entry point.
// The transfer grant fixes both resource IDs and whether a new destination
// may be created. Existing targets must be compatible with the source. A
// worker retry reloads the reserved target instead of creating another KB.
func (s *knowledgeBaseService) CopyKnowledgeBase(ctx context.Context,
srcKB string, dstKB string,
) (*types.KnowledgeBase, *types.KnowledgeBase, error) {
tenantID := types.MustTenantIDFromContext(ctx)
// Load source KB with tenant scope to prevent cross-tenant cloning
sourceKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, srcKB, tenantID)
destinationID, create, creatorID, err := access.CloneDestination(ctx, srcKB)
if err != nil {
logger.Errorf(ctx, "Get source knowledge base failed: %v", err)
return nil, nil, err
}
if dstKB != "" && dstKB != destinationID {
return nil, nil, access.ErrForbidden
}
tenantID := types.MustTenantIDFromContext(ctx)
sourceKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, srcKB, tenantID)
if err != nil {
return nil, nil, err
}
if sourceKB == nil || sourceKB.ID != srcKB || sourceKB.TenantID != tenantID {
return nil, nil, access.ErrForbidden
}
sourceCopy := *sourceKB
sourceKB = &sourceCopy
sourceKB.EnsureDefaults()
var targetKB *types.KnowledgeBase
if dstKB != "" {
// Load target KB with tenant scope so we only clone into the caller's tenant
targetKB, err = s.repo.GetKnowledgeBaseByIDAndTenant(ctx, dstKB, tenantID)
if err != nil {
targetKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, destinationID, tenantID)
if err != nil && (!create || !errors.Is(err, repository.ErrKnowledgeBaseNotFound)) {
return nil, nil, err
}
if targetKB != nil {
targetCopy := *targetKB
targetKB = &targetCopy
targetKB.EnsureDefaults()
if err := access.RequireKBTransfer(ctx, sourceKB, targetKB, access.KBTransferClone); err != nil {
return nil, nil, err
}
// Defense 1: embedding model must match. Mixing incompatible
// vector spaces would produce semantically broken search results.
if sourceKB.EmbeddingModelID != targetKB.EmbeddingModelID {
return nil, nil, apperrors.NewBadRequestError(
"source and target knowledge bases use different embedding models; " +
"clone into a target with the same embedding model")
}
// Defense 2: vector store binding must match. Cross-store cloning
// would require copying physical vector data between stores.
// (both nil → equal; both same UUID → equal; otherwise → rejected)
if !sourceKB.SharesStoreWith(targetKB) {
return nil, nil, apperrors.NewBadRequestError(
"source and target knowledge bases are bound to different vector stores; " +
"cross-store cloning is not yet supported")
}
// Defense 3: the concrete storage instance must match. Comparing only
// provider names would incorrectly allow COS-A -> COS-B clones.
if tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant); tenant != nil {
defaultID, defaultProvider := "", ""
if tenant.DefaultStorageBackendID != nil {
defaultID = *tenant.DefaultStorageBackendID
}
if tenant.StorageEngineConfig != nil {
defaultProvider = tenant.StorageEngineConfig.DefaultProvider
}
if !sourceKB.SharesStorageBackendWith(targetKB, defaultID, defaultProvider) {
return nil, nil, apperrors.NewBadRequestError(
"source and target knowledge bases use different storage instances; cross-storage-backend cloning is not supported")
}
tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
if err := access.ValidateKBTransferCompatibility(sourceKB,
targetKB,
access.KBTransferClone,
"",
tenant); err != nil {
return nil, nil, apperrors.NewBadRequestError(err.Error())
}
} else {
reserved := &types.KnowledgeBase{ID: destinationID, TenantID: tenantID}
if !create {
return nil, nil, access.ErrNotFound
}
if err := access.RequireKBTransfer(ctx, sourceKB, reserved, access.KBTransferClone); err != nil {
return nil, nil, err
}
var faqConfig *types.FAQConfig
if sourceKB.FAQConfig != nil {
cfg := *sourceKB.FAQConfig
@ -1173,7 +1160,8 @@ func (s *knowledgeBaseService) CopyKnowledgeBase(ctx context.Context,
// Preserve VectorStoreID so the cloned KB lands on the same
// physical index. GORM `<-:create` permits the value at INSERT.
targetKB = &types.KnowledgeBase{
ID: uuid.New().String(),
ID: destinationID,
CreatorID: creatorID,
Name: sourceKB.Name,
Type: sourceKB.Type,
Description: sourceKB.Description,

View File

@ -0,0 +1,74 @@
package service
import (
"context"
"github.com/Tencent/WeKnora/internal/application/access"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
)
// Service reads accept exact upstream grants. Otherwise cross-tenant reads
// require a user and organization permission resolved for the original caller.
func kbReadPermissions(ctx context.Context, shares access.KBShareLookup) *access.KBPermissions {
if types.CallerFromContext(ctx).UserID == "" {
shares = nil
}
return access.NewKBPermissions(ctx, shares)
}
func resolveKBReadTenant(ctx context.Context, kb *types.KnowledgeBase, shares access.KBShareLookup) (uint64, error) {
if kb != nil {
allowed, err := kbReadPermissions(ctx, shares).Check(kb.ID, kb.TenantID, types.OrgRoleViewer)
if err == nil && allowed {
return kb.TenantID, nil
}
}
return 0, apperrors.NewForbiddenError("无权访问该知识库")
}
func requireKBWrite(ctx context.Context, kb *types.KnowledgeBase) (context.Context, error) {
if err := access.RequireKBWrite(ctx, kb); err != nil {
return ctx, apperrors.NewForbiddenError("无权修改该知识库")
}
return types.WithExecutionTenant(ctx, kb.TenantID), nil
}
func (s *knowledgeService) writableFAQKnowledgeBase(
ctx context.Context,
kbID string,
) (*types.KnowledgeBase, context.Context, error) {
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
if err != nil {
return nil, ctx, err
}
ctx, err = requireKBWrite(ctx, kb)
if err == nil {
ctx, err = withKBWriteTenantInfo(ctx, kb, s.tenantRepo)
}
return kb, ctx, err
}
// A shared KB mutation must also resolve models/index backends against its
// owner. The authenticated caller remains unchanged when TenantInfo changes.
func withKBWriteTenantInfo(
ctx context.Context,
kb *types.KnowledgeBase,
tenants interfaces.TenantRepository,
) (context.Context, error) {
if tenant, ok := types.TenantInfoFromContext(ctx); ok && tenant != nil && tenant.ID == kb.TenantID {
return ctx, nil
}
if tenants == nil {
return ctx, apperrors.NewServiceUnavailableError("无法获取知识库所属空间")
}
tenant, err := tenants.GetTenantByID(ctx, kb.TenantID)
if err != nil {
return ctx, err
}
if tenant == nil || tenant.ID != kb.TenantID {
return ctx, apperrors.NewNotFoundError("知识库所属空间不存在")
}
return context.WithValue(ctx, types.TenantInfoContextKey, tenant), nil
}

View File

@ -6,6 +6,8 @@ import (
"testing"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/storageallowlist"
@ -87,22 +89,27 @@ func (r *fakeKBRepo) CreateKnowledgeBase(_ context.Context, kb *types.KnowledgeB
r.rows[kb.ID] = kb
return nil
}
func (r *fakeKBRepo) GetKnowledgeBaseByID(_ context.Context, id string) (*types.KnowledgeBase, error) {
return r.rows[id], nil
}
func (r *fakeKBRepo) GetKnowledgeBaseByIDAndTenant(_ context.Context, id string, tenantID uint64) (*types.KnowledgeBase, error) {
kb := r.rows[id]
if kb == nil || kb.TenantID != tenantID {
return nil, stderrors.New("not found")
return nil, repository.ErrKnowledgeBaseNotFound
}
return kb, nil
}
func (r *fakeKBRepo) GetKnowledgeBaseByIDs(_ context.Context, _ []string) ([]*types.KnowledgeBase, error) {
return nil, nil
}
func (r *fakeKBRepo) ListKnowledgeBases(_ context.Context) ([]*types.KnowledgeBase, error) {
return nil, nil
}
func (r *fakeKBRepo) ListKnowledgeBasesByTenantID(_ context.Context, tenantID uint64) ([]*types.KnowledgeBase, error) {
rows := make([]*types.KnowledgeBase, 0, len(r.rows))
for _, kb := range r.rows {
@ -112,6 +119,7 @@ func (r *fakeKBRepo) ListKnowledgeBasesByTenantID(_ context.Context, tenantID ui
}
return rows, nil
}
func (r *fakeKBRepo) UpdateKnowledgeBase(_ context.Context, _ *types.KnowledgeBase) error {
return nil
}
@ -119,9 +127,11 @@ func (r *fakeKBRepo) DeleteKnowledgeBase(_ context.Context, _ string) error { re
func (r *fakeKBRepo) TogglePinKnowledgeBase(_ context.Context, _ string, _ uint64) (*types.KnowledgeBase, error) {
return nil, nil
}
func (r *fakeKBRepo) CountByVectorStoreID(_ context.Context, _ *gorm.DB, _ uint64, _ string) (int64, error) {
return 0, nil
}
func (r *fakeKBRepo) CountByModelID(_ context.Context, _ uint64, _ string) (int64, error) {
return 0, nil
}
@ -322,7 +332,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["src"] = mkKB("src", 1, "embed-1", &storeA)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
src, tgt, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
src, tgt, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, ""), "src", "")
require.NoError(t, err)
require.NotNil(t, src.VectorStoreID)
require.NotNil(t, tgt.VectorStoreID)
@ -340,7 +350,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["src"] = mkKB("src", 1, "embed-1", nil)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, tgt, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
_, tgt, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, ""), "src", "")
require.NoError(t, err)
assert.Nil(t, tgt.VectorStoreID)
})
@ -351,7 +361,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["dst"] = mkKB("dst", 1, "embed-1", nil)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
require.NoError(t, err)
})
@ -361,7 +371,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeA)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
require.NoError(t, err)
})
@ -371,7 +381,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["dst"] = mkKB("dst", 1, "embed-2", &storeA)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
require.Error(t, err)
assert.Contains(t, err.Error(), "different embedding models")
})
@ -382,7 +392,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeB)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
require.Error(t, err)
assert.Contains(t, err.Error(), "different vector stores")
assert.NotContains(t, err.Error(), "Phase 4", "internal roadmap labels must not leak to end-user error messages")
@ -394,7 +404,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeA)
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
require.Error(t, err)
assert.Contains(t, err.Error(), "different vector stores")
})
@ -525,3 +535,22 @@ func TestDuplicateKnowledgeBase_UsesLocalizedEnglishSuffix(t *testing.T) {
assert.Equal(t, "Source KB Copy", target.Name)
}
func copyTaskTestContext(t *testing.T, repo *fakeKBRepo, dst string) context.Context {
t.Helper()
target := repo.rows[dst]
if target == nil {
target = &types.KnowledgeBase{ID: "reserved-copy", TenantID: 1}
}
ctx, err := access.WithKBTransferTask(
context.Background(),
repo.rows["src"],
target,
1,
access.KBTransferClone,
"clone-task",
dst == "",
)
require.NoError(t, err)
return ctx
}

View File

@ -97,7 +97,7 @@ func (s *knowledgeBaseService) ResolveEmbeddingModelKeys(ctx context.Context, kb
// Resolve each unique (modelID, tenantID) to a model identity key
resolvedKeys := make(map[modelRef]string, len(uniqueRefs))
for ref := range uniqueRefs {
tenantCtx := context.WithValue(ctx, types.TenantIDContextKey, ref.TenantID)
tenantCtx := types.WithExecutionTenant(ctx, ref.TenantID)
model, err := s.modelService.GetModelByID(tenantCtx, ref.ModelID)
if err != nil || model == nil {
logger.Warnf(ctx, "ResolveEmbeddingModelKeys: cannot resolve model %s for tenant %d: %v", ref.ModelID, ref.TenantID, err)
@ -143,7 +143,6 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
searchKBIDs, secutils.SanitizeForLog(params.QueryText))
tenantInfo, _ := types.TenantInfoFromContext(ctx)
requestTenantID := types.MustTenantIDFromContext(ctx)
// Batch-load every KB in scope. Required for store grouping,
// embedding-model consistency validation, and FAQ type detection.
@ -162,13 +161,13 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
return nil, apperrors.NewNotFoundError("knowledge base not found")
}
// Authorize every KB the caller asked for. Same-tenant KBs are
// always accessible; foreign-tenant KBs (Organization-shared) must
// pass an explicit per-KB permission check. Without this guard, a
// Authorize every KB for the original caller, using exact upstream
// grants or organization permissions. Execution in a shared tenant
// does not grant access to its other KBs. Without this guard, a
// caller could pass arbitrary KB UUIDs in params.KnowledgeBaseIDs
// and reach foreign tenants' bound vector stores via the per-group
// engine resolution downstream.
if err := s.authorizeKBAccess(ctx, kbs, requestTenantID); err != nil {
if err := s.authorizeKBAccess(ctx, kbs); err != nil {
return nil, err
}

View File

@ -415,27 +415,35 @@ func (f *fakeRetrieveEngineService) Retrieve(ctx context.Context, p types.Retrie
func (f *fakeRetrieveEngineService) Index(context.Context, embedding.Embedder, *types.IndexInfo, []types.RetrieverType) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) BatchIndex(context.Context, embedding.Embedder, []*types.IndexInfo, []types.RetrieverType) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) EstimateStorageSize(context.Context, embedding.Embedder, []*types.IndexInfo, []types.RetrieverType) int64 {
panic("unused")
}
func (f *fakeRetrieveEngineService) CopyIndices(context.Context, string, map[string]string, map[string]string, string, int, string) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) DeleteByChunkIDList(context.Context, []string, int, string) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) DeleteBySourceIDList(context.Context, []string, int, string) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) DeleteByKnowledgeIDList(context.Context, []string, int, string) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) BatchUpdateChunkEnabledStatus(context.Context, map[string]bool) error {
panic("unused")
}
func (f *fakeRetrieveEngineService) BatchUpdateChunkTagID(context.Context, map[string]string) error {
panic("unused")
}
@ -455,9 +463,11 @@ func (r *fakeFanoutRegistry) Register(interfaces.RetrieveEngineService) error {
func (r *fakeFanoutRegistry) GetRetrieveEngineService(types.RetrieverEngineType) (interfaces.RetrieveEngineService, error) {
return nil, stderrors.New("not used in fan-out tests")
}
func (r *fakeFanoutRegistry) GetAllRetrieveEngineServices() []interfaces.RetrieveEngineService {
return nil
}
func (r *fakeFanoutRegistry) GetByStoreID(id string) (interfaces.RetrieveEngineService, error) {
if svc, ok := r.byStore[id]; ok {
return svc, nil
@ -775,7 +785,7 @@ func TestRetrieveFromStores_PerGroupTimeout(t *testing.T) {
// ---------------------------------------------------------------------------
// fakeKBShareForAuth implements just enough of KBShareService for the
// authorizeKBAccess test matrix. Only HasTenantKBPermission is exercised;
// authorizeKBAccess test matrix. Only CheckTenantKBPermission is exercised;
// the embedded interface keeps the type assignable.
type fakeKBShareForAuth struct {
// allowed maps kbID → tenantID → allowed. Mirrors the Plan 3 (#1303)
@ -785,17 +795,10 @@ type fakeKBShareForAuth struct {
interfaces.KBShareService
}
func (f *fakeKBShareForAuth) HasTenantKBPermission(
_ context.Context, kbID string, callerTenantID uint64,
_ types.TenantRole, _ types.OrgMemberRole,
) (bool, error) {
if f.err != nil {
return false, f.err
}
if perTenant, ok := f.allowed[kbID]; ok {
return perTenant[callerTenantID], nil
}
return false, nil
func (f *fakeKBShareForAuth) CheckTenantKBPermission(
_ context.Context, kbID string, callerTenantID uint64, _ types.TenantRole,
) (types.OrgMemberRole, bool, error) {
return types.OrgRoleViewer, f.allowed[kbID][callerTenantID], f.err
}
func ctxWithTenantForAuth(tenantID uint64) context.Context {
@ -809,7 +812,7 @@ func TestAuthorizeKBAccess_SameTenantAllPass(t *testing.T) {
{ID: "kb-1", TenantID: 7},
{ID: "kb-2", TenantID: 7},
}
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
require.NoError(t, err)
}
@ -825,7 +828,7 @@ func TestAuthorizeKBAccess_ForeignTenantWithShare_OK(t *testing.T) {
{ID: "kb-own", TenantID: 7},
{ID: "kb-foreign", TenantID: 99},
}
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
require.NoError(t, err)
}
@ -840,7 +843,7 @@ func TestAuthorizeKBAccess_ForeignTenantNoShare_NotFound(t *testing.T) {
kbs := []*types.KnowledgeBase{
{ID: "kb-foreign", TenantID: 99},
}
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
require.Error(t, err)
app, ok := apperrors.IsAppError(err)
require.True(t, ok, "expected typed AppError, got %T", err)
@ -853,7 +856,7 @@ func TestAuthorizeKBAccess_PermissionLookupError_500(t *testing.T) {
share := &fakeKBShareForAuth{err: stderrors.New("share infra down")}
s := &knowledgeBaseService{kbShareService: share}
kbs := []*types.KnowledgeBase{{ID: "kb-foreign", TenantID: 99}}
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
require.Error(t, err)
app, ok := apperrors.IsAppError(err)
require.True(t, ok)
@ -863,7 +866,7 @@ func TestAuthorizeKBAccess_PermissionLookupError_500(t *testing.T) {
func TestAuthorizeKBAccess_EmptyKBs_OK(t *testing.T) {
t.Parallel()
s := &knowledgeBaseService{kbShareService: &fakeKBShareForAuth{}}
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), nil, 7)
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), nil)
require.NoError(t, err)
}

View File

@ -2,7 +2,9 @@ package service
import (
"context"
"errors"
"github.com/Tencent/WeKnora/internal/application/repository"
"github.com/Tencent/WeKnora/internal/logger"
"github.com/Tencent/WeKnora/internal/types"
)
@ -30,52 +32,42 @@ func (s *knowledgeBaseService) fetchKnowledgeData(ctx context.Context,
}
// fetchKnowledgeDataWithShared gets knowledge data in batch, including knowledge
// from shared KBs the user has access to.
// from KBs authorized for the original caller. Initial tenant-scoped rows
// require the same permission check as cross-tenant expansion.
func (s *knowledgeBaseService) fetchKnowledgeDataWithShared(ctx context.Context,
tenantID uint64,
knowledgeIDs []string,
) (map[string]*types.Knowledge, error) {
knowledgeMap, err := s.fetchKnowledgeData(ctx, tenantID, knowledgeIDs)
rows, err := s.fetchKnowledgeData(ctx, tenantID, knowledgeIDs)
if err != nil {
return nil, err
}
missingIDs := s.findMissingIDs(knowledgeIDs, func(id string) bool {
return knowledgeMap[id] != nil
})
if len(missingIDs) == 0 {
return knowledgeMap, nil
permissions := kbReadPermissions(ctx, s.kbShareService)
knowledgeMap := make(map[string]*types.Knowledge, len(rows))
appendAllowed := func(k *types.Knowledge) {
if k == nil {
return
}
allowed, err := permissions.Check(k.KnowledgeBaseID, k.TenantID, types.OrgRoleViewer)
if err == nil && allowed {
knowledgeMap[k.ID] = k
}
}
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] %d knowledge IDs not found in current tenant, attempting shared KB lookup", len(missingIDs))
userID, ok := s.extractUserID(ctx)
if !ok {
logger.Warnf(ctx, "[fetchKnowledgeDataWithShared] userID not found or empty in context, skipping shared KB lookup")
return knowledgeMap, nil
for _, k := range rows {
appendAllowed(k)
}
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] Looking up %d missing knowledge IDs with userID=%s", len(missingIDs), userID)
callerTenantRole := types.TenantRoleFromContext(ctx)
for _, id := range missingIDs {
for _, id := range knowledgeIDs {
if rows[id] != nil {
continue
}
k, err := s.kgRepo.GetKnowledgeByIDOnly(ctx, id)
if err != nil || k == nil || k.KnowledgeBaseID == "" {
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Knowledge %s not found or has no KB", id)
continue
if err != nil && !errors.Is(err, repository.ErrKnowledgeNotFound) {
return nil, err
}
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, k.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
if err != nil {
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Permission check error for KB %s: %v", k.KnowledgeBaseID, err)
continue
if err == nil {
appendAllowed(k)
}
if !hasPermission {
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] No permission for KB %s", k.KnowledgeBaseID)
continue
}
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Found shared knowledge %s in KB %s", id, k.KnowledgeBaseID)
knowledgeMap[k.ID] = k
}
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] After shared lookup, total knowledge found: %d", len(knowledgeMap))
return knowledgeMap, nil
}
@ -84,58 +76,41 @@ func (s *knowledgeBaseService) listChunksByIDWithShared(ctx context.Context,
tenantID uint64,
chunkIDs []string,
) ([]*types.Chunk, error) {
chunks, err := s.chunkRepo.ListChunksByID(ctx, tenantID, chunkIDs)
rows, err := s.chunkRepo.ListChunksByID(ctx, tenantID, chunkIDs)
if err != nil {
return nil, err
}
foundSet := make(map[string]bool, len(chunks))
for _, c := range chunks {
permissions := kbReadPermissions(ctx, s.kbShareService)
chunks := make([]*types.Chunk, 0, len(rows))
foundSet := make(map[string]bool)
appendAllowed := func(c *types.Chunk) {
if c == nil || foundSet[c.ID] {
return
}
allowed, err := permissions.Check(c.KnowledgeBaseID, c.TenantID, types.OrgRoleViewer)
if err == nil && allowed {
chunks = append(chunks, c)
foundSet[c.ID] = true
}
}
for _, c := range rows {
appendAllowed(c)
if c != nil {
foundSet[c.ID] = true
}
}
missing := s.findMissingIDs(chunkIDs, func(id string) bool {
return foundSet[id]
})
missing := s.findMissingIDs(chunkIDs, func(id string) bool { return foundSet[id] })
if len(missing) == 0 {
return chunks, nil
}
logger.Infof(ctx, "[listChunksByIDWithShared] %d chunks not found in current tenant, attempting shared KB lookup", len(missing))
userID, ok := s.extractUserID(ctx)
if !ok {
logger.Warnf(ctx, "[listChunksByIDWithShared] userID not found or empty in context, skipping shared KB lookup")
return chunks, nil
}
logger.Infof(ctx, "[listChunksByIDWithShared] Looking up %d missing chunks with userID=%s", len(missing), userID)
callerTenantRole := types.TenantRoleFromContext(ctx)
crossChunks, err := s.chunkRepo.ListChunksByIDOnly(ctx, missing)
if err != nil {
logger.Warnf(ctx, "[listChunksByIDWithShared] Failed to fetch chunks by ID only: %v", err)
return chunks, nil
return nil, err
}
logger.Infof(ctx, "[listChunksByIDWithShared] Found %d chunks without tenant filter", len(crossChunks))
for _, c := range crossChunks {
if c == nil || c.KnowledgeBaseID == "" {
continue
}
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, c.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
if err != nil {
logger.Debugf(ctx, "[listChunksByIDWithShared] Permission check error for KB %s: %v", c.KnowledgeBaseID, err)
continue
}
if !hasPermission {
logger.Debugf(ctx, "[listChunksByIDWithShared] No permission for KB %s", c.KnowledgeBaseID)
continue
}
chunks = append(chunks, c)
appendAllowed(c)
}
logger.Infof(ctx, "[listChunksByIDWithShared] After shared lookup, total chunks: %d", len(chunks))
return chunks, nil
}
@ -149,16 +124,3 @@ func (s *knowledgeBaseService) findMissingIDs(ids []string, exists func(string)
}
return missing
}
// extractUserID extracts the user ID from context, returning ("", false) if not found.
func (s *knowledgeBaseService) extractUserID(ctx context.Context) (string, bool) {
userIDVal := ctx.Value(types.UserIDContextKey)
if userIDVal == nil {
return "", false
}
userID, ok := userIDVal.(string)
if !ok || userID == "" {
return "", false
}
return userID, true
}

View File

@ -6,6 +6,7 @@ import (
"fmt"
"time"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/application/service/retriever"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/logger"
@ -184,9 +185,9 @@ func classifyFactoryError(
}
// authorizeKBAccess rejects multi-KB searches whose scope includes a KB
// that the caller is not entitled to read. Same-tenant KBs always pass.
// Foreign-tenant KBs (Organization-shared) must pass an explicit
// tenant-scoped permission check via kbShareService.HasTenantKBPermission,
// that the original caller is not entitled to read. Exact upstream grants
// support shared KB/agent execution; other foreign KBs need an organization
// permission check for the caller via access.KBPermissions,
// applying the 3-D cap (share role + caller's tenant-org role + tenant
// Viewer cap) introduced in Plan 3 of #1303.
//
@ -198,20 +199,24 @@ func classifyFactoryError(
func (s *knowledgeBaseService) authorizeKBAccess(
ctx context.Context,
kbs []*types.KnowledgeBase,
requestTenantID uint64,
) error {
if len(kbs) == 0 {
return nil
}
callerTenantRole := types.TenantRoleFromContext(ctx)
kbIDs := make([]string, 0, len(kbs))
for _, kb := range kbs {
kbIDs = append(kbIDs, kb.ID)
}
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbIDs...); err != nil {
return err
}
requestTenantID := types.CallerFromContext(ctx).TenantID
permissions := access.NewKBPermissions(ctx, s.kbShareService)
for _, kb := range kbs {
if kb.TenantID == requestTenantID {
continue
}
hasPermission, permErr := s.kbShareService.HasTenantKBPermission(
ctx, kb.ID, requestTenantID, callerTenantRole, types.OrgRoleViewer)
hasPermission, permErr := permissions.Check(kb.ID, kb.TenantID, types.OrgRoleViewer)
if permErr != nil {
logger.ErrorWithFields(ctx, permErr, map[string]interface{}{
"caller_tenant_id": requestTenantID,

View File

@ -446,7 +446,7 @@ func (s *messageService) DeleteMessageKnowledge(ctx context.Context, knowledgeID
return
}
logger.Infof(ctx, "Deleting chat history knowledge entry: %s", knowledgeID)
if err := s.knowService.DeleteKnowledge(ctx, knowledgeID); err != nil {
if err := deleteReferencedKnowledge(ctx, s.knowService, "", []string{knowledgeID}); err != nil {
logger.Warnf(ctx, "Failed to delete chat history knowledge %s: %v", knowledgeID, err)
}
}
@ -466,7 +466,7 @@ func (s *messageService) DeleteSessionKnowledge(ctx context.Context, sessionID s
}
logger.Infof(ctx, "Deleting %d chat history knowledge entries for session %s", len(knowledgeIDs), sessionID)
if err := s.knowService.DeleteKnowledgeList(ctx, knowledgeIDs); err != nil {
if err := deleteReferencedKnowledge(ctx, s.knowService, "", knowledgeIDs); err != nil {
logger.Warnf(ctx, "Failed to batch delete chat history knowledge for session %s: %v", sessionID, err)
}
}

View File

@ -155,6 +155,47 @@ func (s *resourceCatalog) Bind(ctx context.Context, reference, ownerType, ownerI
})
}
func (s *resourceCatalog) IsReferencedByKnowledgeBase(
ctx context.Context,
tenantID uint64,
kbID, reference string,
) (bool, error) {
physical, resource, err := s.ResolvePath(ctx, reference)
if err != nil {
return false, err
}
if resource == nil {
resource, err = s.repo.GetByTenantLocation(ctx, tenantID, resourceLocationHash(physical))
if err != nil {
return false, err
}
}
if resource == nil || resource.TenantID != tenantID {
return false, nil
}
return s.repo.IsReferencedByKnowledgeBase(ctx, tenantID, kbID, resource.ID)
}
// GetMessageFileBindings resolves registered aliases before reading authoritative origins.
func (s *resourceCatalog) GetMessageFileBindings(
ctx context.Context, tenantID uint64, reference, messageID string,
) (*types.MessageFileBindings, error) {
physical, resource, err := s.ResolvePath(ctx, reference)
if err != nil {
return nil, err
}
if resource == nil {
resource, err = s.repo.GetByTenantLocation(ctx, tenantID, resourceLocationHash(physical))
if err != nil {
return nil, err
}
}
if resource == nil || resource.TenantID != tenantID {
return &types.MessageFileBindings{}, nil
}
return s.repo.GetMessageFileBindings(ctx, tenantID, resource.ID, messageID)
}
// Release implements interfaces.ResourceCatalog.
//
// Unbinding and counting are deliberately not a single transaction. A racing

View File

@ -0,0 +1,169 @@
package service
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/require"
)
type catalogFileShares struct{}
func (catalogFileShares) CheckTenantKBPermission(
_ context.Context,
kb string,
_ uint64,
_ types.TenantRole,
) (types.OrgMemberRole, bool, error) {
return types.OrgRoleViewer, kb == "shared", nil
}
func (catalogFileShares) GetKnowledgeBasesByIDsOnly(context.Context, []string) ([]*types.KnowledgeBase, error) {
return []*types.KnowledgeBase{{ID: "shared", TenantID: 7}}, nil
}
type catalogFileAgent struct{ agent *types.CustomAgent }
func (a *catalogFileAgent) GetSharedAgentForTenant(
context.Context,
uint64,
types.TenantRole,
string,
...uint64,
) (*types.CustomAgent, error) {
return a.agent, nil
}
func TestCatalogFileAuthorizationRejectsPrivateHandlesInSharedText(t *testing.T) {
catalog, db := newResourceCatalogForTest(t)
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
ctx := newSharedAccessContext()
const physical = "local://7/exports/private.png"
ref, err := catalog.Register(ctx, 7, physical, interfaces.ResourceRegistration{})
require.NoError(t, err)
for _, kb := range []string{"private", "shared"} {
require.NoError(t, db.Create(&types.KnowledgeBase{ID: kb, TenantID: 7}).Error)
require.NoError(
t,
db.Create(&types.Knowledge{ID: kb + "-doc", TenantID: 7, KnowledgeBaseID: kb, Type: "file"}).Error,
)
}
require.NoError(
t,
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "private-doc", types.ResourceRelationSourceFile),
)
text := "![private](" + ref + ") ![legacy](" + physical + ")"
require.NoError(
t,
db.Create(
&types.Chunk{ID: "chunk", TenantID: 7, KnowledgeBaseID: "shared", KnowledgeID: "shared-doc", Content: text},
).Error,
)
require.NoError(
t,
db.Create(&types.WikiPage{ID: "wiki", TenantID: 7, KnowledgeBaseID: "shared", Content: text}).Error,
)
grant := &access.KBAccess{
KnowledgeBase: &types.KnowledgeBase{ID: "shared", TenantID: 7},
Caller: types.CallerFromContext(ctx),
EffectiveTenantID: 7,
Permission: types.OrgRoleViewer,
}
lookup := catalog.(interfaces.KBResourceLookup)
message := &types.Message{
ID: "message",
AgentTenantID: 1,
Content: text,
KnowledgeReferences: []*types.SearchResult{{KnowledgeBaseID: "shared", Content: text}},
}
shares := access.MessageKBShareAuthorizer{ShareGuard: catalogFileShares{}, KBs: catalogFileShares{}}
check := func(allowed bool) {
t.Helper()
for _, path := range []string{ref, physical} {
_, err := access.ResolveKBFile(ctx, grant, "shared", path, catalog, lookup)
if allowed {
require.NoError(t, err)
} else {
require.ErrorIs(t, err, access.ErrForbidden)
}
_, err = access.AuthorizeMessageFile(ctx, message, path, nil, catalog, shares)
if allowed && path == ref {
require.NoError(t, err)
} else {
require.ErrorIs(t, err, access.ErrForbidden)
}
}
}
check(false)
require.NoError(
t,
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "shared-doc", types.ResourceRelationSourceFile),
)
check(true)
require.NoError(t, db.Where("id = ?", "shared").Delete(&types.KnowledgeBase{}).Error)
check(false)
}
func TestCatalogSharedAgentFileRequiresCurrentSelectionOrExactArtifactBinding(t *testing.T) {
catalog, db := newResourceCatalogForTest(t)
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}))
ctx := newSharedAccessContext()
const physical = "local://7/exports/source.pdf"
ref, err := catalog.Register(ctx, 7, physical, interfaces.ResourceRegistration{})
require.NoError(t, err)
require.NoError(t, db.Create(&types.KnowledgeBase{ID: "private", TenantID: 7}).Error)
doc := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "private", Type: "file"}
require.NoError(t, db.Create(doc).Error)
require.NoError(t, catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "doc", types.ResourceRelationSourceFile))
agents := &catalogFileAgent{agent: &types.CustomAgent{ID: "agent", TenantID: 7}}
message := &types.Message{
ID: "message",
AgentID: "agent",
AgentTenantID: 7,
Content: ref + " " + physical,
Artifacts: types.MessageArtifacts{{URL: ref}},
}
check := func(allowed bool) {
t.Helper()
for _, path := range []string{ref, physical} {
_, err := access.AuthorizeMessageFile(
ctx,
message,
path,
agents,
catalog,
access.MessageKBShareAuthorizer{},
)
if allowed {
require.NoError(t, err)
} else {
require.ErrorIs(t, err, access.ErrForbidden)
}
}
}
for _, mode := range []string{"none", "", "unknown", "selected", "all"} {
agents.agent.Config = types.CustomAgentConfig{KBSelectionMode: mode, KnowledgeBases: []string{"private"}}
check(mode == "selected" || mode == "all")
}
agents.agent.Config = types.CustomAgentConfig{KBSelectionMode: "selected", KnowledgeBases: []string{"other"}}
check(false)
agents.agent.Config.KnowledgeBases = []string{"private"}
scoped := types.WithTenantAPIKeyScope(ctx, types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"other"}})
_, err = access.AuthorizeMessageFile(scoped, message, ref, agents, catalog, access.MessageKBShareAuthorizer{})
require.ErrorIs(t, err, access.ErrForbidden)
require.NoError(t, db.Delete(doc).Error)
check(false)
agents.agent.Config.KBSelectionMode = "none"
require.NoError(
t,
catalog.Bind(ctx, ref, types.ResourceOwnerMessage, "another-message", types.ResourceRelationArtifact),
)
check(false)
require.NoError(t, catalog.Bind(ctx, ref, types.ResourceOwnerMessage, "message", types.ResourceRelationArtifact))
check(true)
agents.agent = nil
check(false)
}

View File

@ -0,0 +1,71 @@
package service
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/stretchr/testify/require"
)
func TestKBFileBindingRequiresLiveKnowledgeInExactKB(t *testing.T) {
catalog, db := newResourceCatalogForTest(t)
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
ctx := context.Background()
ref, err := catalog.Register(ctx, 7, "local://7/exports/image.png", interfaces.ResourceRegistration{})
require.NoError(t, err)
require.NoError(t, db.Create(&types.KnowledgeBase{ID: "kb", TenantID: 7}).Error)
knowledge := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "kb", Type: "file"}
require.NoError(t, db.Create(knowledge).Error)
require.NoError(
t,
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "doc", types.ResourceRelationExtractedImage),
)
lookup := catalog.(interfaces.KBResourceLookup)
for _, reference := range []string{ref, "local://7/exports/image.png"} {
ok, err := lookup.IsReferencedByKnowledgeBase(ctx, 7, "kb", reference)
require.NoError(t, err)
require.True(t, ok)
ok, err = lookup.IsReferencedByKnowledgeBase(ctx, 7, "other-kb", reference)
require.NoError(t, err)
require.False(t, ok)
}
require.NoError(t, db.Delete(knowledge).Error)
ok, err := lookup.IsReferencedByKnowledgeBase(ctx, 7, "kb", ref)
require.NoError(t, err)
require.False(t, ok, "a stale binding cannot authorize a deleted document")
}
func TestKBFileLegacyTextReferencesDoNotAuthorizeFiles(t *testing.T) {
catalog, db := newResourceCatalogForTest(t)
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
lookup := catalog.(interfaces.KBResourceLookup)
const reference = "local://7/exports/image.png"
chunk := &types.Chunk{
ID: "chunk",
TenantID: 7,
KnowledgeBaseID: "kb",
Content: "![image](" + reference + ".private)",
}
require.NoError(t, db.Create(chunk).Error)
ok, err := lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
require.NoError(t, err)
require.False(t, ok)
require.NoError(t, db.Model(chunk).Update("image_info", `[{"url":"`+reference+`"}]`).Error)
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
require.NoError(t, err)
require.False(t, ok, "text is not an ownership binding")
require.NoError(t, db.Delete(chunk).Error)
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
require.NoError(t, err)
require.False(t, ok)
page := &types.WikiPage{ID: "page", TenantID: 7, KnowledgeBaseID: "kb", Content: "![image](" + reference + ")"}
require.NoError(t, db.Create(page).Error)
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
require.NoError(t, err)
require.False(t, ok, "text is not an ownership binding")
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 8, "kb", reference)
require.NoError(t, err)
require.False(t, ok)
}

View File

@ -0,0 +1,284 @@
package service
import (
"context"
"encoding/json"
"errors"
"testing"
"github.com/Tencent/WeKnora/internal/application/access"
apperrors "github.com/Tencent/WeKnora/internal/errors"
"github.com/Tencent/WeKnora/internal/types"
"github.com/Tencent/WeKnora/internal/types/interfaces"
"github.com/hibiken/asynq"
"github.com/stretchr/testify/require"
)
func TestReparseRejectsMovingDocumentBeforeSideEffects(t *testing.T) {
f := newDocumentWriteFixture(t)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
f.repo.writes = 0
_, err = f.svc.ReparseKnowledge(f.ctx, "doc", nil)
var appErr *apperrors.AppError
require.ErrorAs(t, err, &appErr)
require.Equal(t, 409, appErr.HTTPCode)
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
require.Zero(t, f.graph.calls)
}
func TestReparseRequiresExplicitWriteGrant(t *testing.T) {
f := newDocumentWriteFixture(t)
ctx := types.WithExecutionTenant(context.Background(), 7)
_, err := f.svc.ReparseKnowledge(ctx, "doc", nil)
require.Error(t, err)
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
}
func TestReparseTaskPinsOriginalKBBeforeAnySubmission(t *testing.T) {
for _, scenario := range []string{"moved", "missing", "mixed legacy", "blank ID", "moving"} {
t.Run(scenario, func(t *testing.T) {
f := newDocumentWriteFixture(t)
payload := types.KnowledgeListReparsePayload{
TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc"},
}
switch scenario {
case "moved":
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error,
)
case "missing":
payload.KnowledgeIDs = append(payload.KnowledgeIDs, "missing")
case "mixed legacy":
payload.KnowledgeBaseID = ""
payload.KnowledgeIDs = append(payload.KnowledgeIDs, "other-doc")
case "blank ID":
payload.KnowledgeIDs = append(payload.KnowledgeIDs, " ")
case "moving":
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}),
)
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
}
f.repo.writes = 0
raw, err := json.Marshal(payload)
require.NoError(t, err)
err = f.svc.ProcessKnowledgeListReparse(
context.Background(), asynq.NewTask(types.TypeKnowledgeListReparse, raw),
)
require.ErrorIs(t, err, asynq.SkipRetry)
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
})
}
}
func TestReparseTaskGrantCannotFollowMoveAfterPreflight(t *testing.T) {
f := newDocumentWriteFixture(t)
ctx, ids, err := f.svc.reparseTaskScope(context.Background(), types.KnowledgeListReparsePayload{
TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc", "doc"},
})
require.NoError(t, err)
require.Equal(t, []string{"doc"}, ids)
require.NoError(t, access.RequireKBWrite(ctx, f.kbs.values["kb"]))
require.Error(t, access.RequireKBWrite(ctx, f.kbs.values["other"]))
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error)
_, err = f.svc.ReparseKnowledge(ctx, "doc", nil)
require.Error(t, err)
require.Zero(t, f.repo.writes)
require.Zero(t, f.chunkRepo.writes)
}
type sharedChunkFailureRepo struct {
interfaces.ChunkRepository
failure error
}
func (r sharedChunkFailureRepo) ListChunksByID(context.Context, uint64, []string) ([]*types.Chunk, error) {
return []*types.Chunk{{ID: "own", TenantID: 1, KnowledgeBaseID: "own"}}, nil
}
func (r sharedChunkFailureRepo) ListChunksByIDOnly(context.Context, []string) ([]*types.Chunk, error) {
return nil, r.failure
}
func TestSharedChunkReadPropagatesStorageFailure(t *testing.T) {
failure := errors.New("shared chunk storage unavailable")
svc := &knowledgeBaseService{chunkRepo: sharedChunkFailureRepo{failure: failure}}
rows, err := svc.listChunksByIDWithShared(newSharedAccessContext(), 1, []string{"own", "shared"})
require.ErrorIs(t, err, failure)
require.Nil(t, rows)
}
func TestFAQCloneDoesNotInheritTransferState(t *testing.T) {
f := transferFixture(t, access.KBTransferClone)
kb := &types.KnowledgeBase{ID: "new-faq", TenantID: 7}
src := &types.Knowledge{
Metadata: types.JSON(`{"custom":{"value":3},"_knowledge_transfer":{"operation":"move","phase":"moving"}}`),
}
before := string(src.Metadata)
dst, err := f.svc.getOrCreateFAQKnowledge(f.ctx, kb, src)
require.NoError(t, err)
require.JSONEq(t, `{"custom":{"value":3}}`, string(dst.Metadata))
require.Equal(t, before, string(src.Metadata))
require.NoError(t, access.RejectMovingKnowledge(dst))
}
type partialCloneEngine struct {
parentChildRetrieveEngine
vectors map[string]int
cleanupFailure bool
}
func (e *partialCloneEngine) Support() []types.RetrieverType {
return []types.RetrieverType{types.VectorRetrieverType, types.KeywordsRetrieverType}
}
func (e *partialCloneEngine) CopyIndices(
_ context.Context, _ string, documents, _ map[string]string, _ string, _ int, _ string,
) error {
for _, id := range documents {
e.vectors[id] = 1
}
return errors.New("partial vector copy")
}
func (e *partialCloneEngine) DeleteByKnowledgeIDList(_ context.Context, ids []string, _ int, _ string) error {
if e.cleanupFailure {
return errors.New("vector cleanup unavailable")
}
for _, id := range ids {
delete(e.vectors, id)
}
return nil
}
func TestClonePartialIndexFailureCleansOnlyItsDestination(t *testing.T) {
for _, cleanupFailure := range []bool{false, true} {
t.Run(map[bool]string{false: "cleaned", true: "cleanup fails visibly"}[cleanupFailure], func(t *testing.T) {
t.Setenv("RETRIEVE_DRIVER", "postgres")
f := transferFixture(t, access.KBTransferClone)
engine := &partialCloneEngine{vectors: map[string]int{"doc": 1}, cleanupFailure: cleanupFailure}
f.svc.retrieveEngine = parentChildRetrieveRegistry{engine: engine}
f.svc.modelService = parentChildModelService{embedder: parentChildEmbedder{}}
f.kbs.values["other"].EmbeddingModelID = "model"
src, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
err = f.svc.cloneKnowledge(f.ctx, src, f.kbs.values["other"])
require.ErrorContains(t, err, "partial vector copy")
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
require.NoError(t, err)
var dst *types.Knowledge
for _, row := range rows {
if row.ID != "other-doc" {
dst = row
}
}
require.NotNil(t, dst)
require.Equal(t, types.ParseStatusFailed, dst.ParseStatus)
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, dst.ID)
require.NoError(t, err)
if cleanupFailure {
require.Len(t, chunks, 1, "retain image references when index cleanup is uncertain")
require.Equal(t, 1, engine.vectors[dst.ID])
require.Contains(t, dst.ErrorMessage, "vector cleanup unavailable")
} else {
require.Empty(t, chunks)
require.Zero(t, engine.vectors[dst.ID])
}
require.Equal(t, 1, engine.vectors["doc"])
var tenant types.Tenant
require.NoError(t, f.db.First(&tenant, 7).Error)
require.EqualValues(t, 5, tenant.StorageUsed, "failed clones must not change storage accounting")
})
}
}
func TestReparseWorkerSubmitsPinnedAndLegacySingleKBTasks(t *testing.T) {
for _, kbID := range []string{"kb", ""} {
t.Run("scope="+kbID, func(t *testing.T) {
f := newDocumentWriteFixture(t)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.NoError(
t,
row.SetManualMetadata(
types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1),
),
)
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
f.svc.task = &transferQueue{}
raw, err := json.Marshal(types.KnowledgeListReparsePayload{
TenantID: 7, KnowledgeBaseID: kbID, KnowledgeIDs: []string{"doc"},
})
require.NoError(t, err)
require.NoError(
t,
f.svc.ProcessKnowledgeListReparse(
context.Background(), asynq.NewTask(types.TypeKnowledgeListReparse, raw),
),
)
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, types.ParseStatusPending, row.ParseStatus)
})
}
}
func TestMovedReparseTransferPhaseDoesNotBlockItsParser(t *testing.T) {
row := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "target", ParseStatus: types.ParseStatusPending}
for _, phase := range []string{"reparse_pending", "done"} {
require.NoError(
t,
setTransferState(row, knowledgeTransferState{
Operation: access.KBTransferMove, Mode: "reparse", Phase: phase,
}),
)
require.NoError(t, validateProcessingKnowledge(row, 7, "target", "doc"))
require.ErrorIs(t, validateProcessingKnowledge(row, 7, "source", "doc"), asynq.SkipRetry)
require.Equal(t, types.ParseStatusPending, row.ParseStatus, "transfer state is independent of parse completion")
}
}
type orphanMoveEngine struct {
partialCloneEngine
calls int
chunks []string
}
func (e *orphanMoveEngine) MoveKnowledgeIndices(
_ context.Context, _, _, _ string, chunks []string, _ int, _ string,
) error {
e.calls++
e.chunks = append([]string(nil), chunks...)
return nil
}
func TestMoveStillVisitsVectorBackendWithoutDatabaseChunks(t *testing.T) {
t.Setenv("RETRIEVE_DRIVER", "postgres")
f := transferFixture(t, access.KBTransferMove)
engine := &orphanMoveEngine{}
f.svc.retrieveEngine = parentChildRetrieveRegistry{engine: engine}
f.svc.modelService = parentChildModelService{embedder: parentChildEmbedder{}}
f.kbs.values["kb"].EmbeddingModelID = "model"
f.kbs.values["other"].EmbeddingModelID = "model"
require.NoError(
t,
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("embedding_model_id", "model").Error,
)
require.NoError(t, f.db.Where("knowledge_id = ?", "doc").Delete(&types.Chunk{}).Error)
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, "reuse_vectors")))
require.Equal(t, 1, engine.calls)
require.Empty(t, engine.chunks)
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
require.NoError(t, err)
require.Equal(t, "other", row.KnowledgeBaseID)
}

View File

@ -312,3 +312,42 @@ func (c *CompositeRetrieveEngine) EstimateStorageSize(ctx context.Context,
}
return sum.Load()
}
// ValidateKnowledgeIndexMove checks all stores before the first mutation.
func (c *CompositeRetrieveEngine) ValidateKnowledgeIndexMove(ctx context.Context) error {
for _, info := range c.engineInfos {
if _, ok := info.retrieveEngine.(interfaces.KnowledgeIndexMover); !ok {
return fmt.Errorf("retriever %s does not support moving indices", info.retrieveEngine.EngineType())
}
if validator, ok := info.retrieveEngine.(interface{ ValidateKnowledgeIndexMove(context.Context) error }); ok {
if err := validator.ValidateKnowledgeIndexMove(ctx); err != nil {
return err
}
}
}
return nil
}
// MoveKnowledgeIndices updates each validated store with retry-safe operations.
func (c *CompositeRetrieveEngine) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
chunkIDs []string,
dimension int,
knowledgeType string,
) error {
if err := c.ValidateKnowledgeIndexMove(ctx); err != nil {
return err
}
return c.concurrentExecWithError(ctx, func(ctx context.Context, info *engineInfo) error {
return info.retrieveEngine.(interfaces.KnowledgeIndexMover).MoveKnowledgeIndices(
ctx,
sourceKB,
targetKB,
knowledgeID,
chunkIDs,
dimension,
knowledgeType,
)
})
}

View File

@ -2,6 +2,7 @@ package retriever
import (
"context"
"fmt"
"regexp"
"slices"
"strings"
@ -346,3 +347,37 @@ func (v *KeywordsVectorHybridRetrieveEngineService) BatchUpdateChunkTagID(
) error {
return v.indexRepository.BatchUpdateChunkTagID(ctx, chunkTagMap)
}
// ValidateKnowledgeIndexMove checks backend support before any mutation.
func (v *KeywordsVectorHybridRetrieveEngineService) ValidateKnowledgeIndexMove(ctx context.Context) error {
if _, ok := v.indexRepository.(interfaces.KnowledgeIndexMover); !ok {
return fmt.Errorf("retriever %s does not support moving indices", v.EngineType())
}
if validator, ok := v.indexRepository.(interface{ ValidateKnowledgeIndexMove(context.Context) error }); ok {
return validator.ValidateKnowledgeIndexMove(ctx)
}
return nil
}
// MoveKnowledgeIndices delegates metadata relocation to the supported backend.
func (v *KeywordsVectorHybridRetrieveEngineService) MoveKnowledgeIndices(
ctx context.Context,
sourceKB, targetKB, knowledgeID string,
chunkIDs []string,
dimension int,
knowledgeType string,
) error {
if err := v.ValidateKnowledgeIndexMove(ctx); err != nil {
return err
}
return v.indexRepository.(interfaces.KnowledgeIndexMover).MoveKnowledgeIndices(
ctx,
sourceKB,
targetKB,
knowledgeID,
chunkIDs,
dimension,
knowledgeType,
)
}

View File

@ -0,0 +1,38 @@
package retriever
import (
"context"
"testing"
"github.com/Tencent/WeKnora/internal/types"
"github.com/stretchr/testify/require"
)
type moveTestEngine struct {
fakeEngine
calls int
}
func (e *moveTestEngine) MoveKnowledgeIndices(context.Context, string, string, string, []string, int, string) error {
e.calls++
return nil
}
func TestCompositeMoveChecksAllEnginesBeforeMutation(t *testing.T) {
supported := &moveTestEngine{fakeEngine: fakeEngine{engineType: types.PostgresRetrieverEngineType}}
unsupported := &fakeEngine{engineType: types.ElasticsearchRetrieverEngineType}
c := &CompositeRetrieveEngine{
engineInfos: []*engineInfo{{retrieveEngine: supported}, {retrieveEngine: unsupported}},
}
require.Error(
t,
c.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
)
require.Zero(t, supported.calls)
c.engineInfos = c.engineInfos[:1]
require.NoError(
t,
c.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
)
require.Equal(t, 1, supported.calls)
}

Some files were not shown because too many files have changed in this diff Show More