refactor(resource-access): unify scoped authorization and transfer recovery (#3097)
* refactor(resource-access): unify scoped authorization and transfer recovery * fix(resource-access): close reparse and transfer review gaps * fix: enforce explicit file bindings and recover move cleanup * fix: align mutation admission and cleanup scope checksmain
parent
e18cf138a3
commit
3d81b977f1
|
|
@ -0,0 +1,294 @@
|
|||
# Resource access boundaries
|
||||
|
||||
`access` contains shared permission resolution without depending on Gin or the
|
||||
application service implementations. HTTP adapters keep parameter parsing, status
|
||||
codes, RBAC rollout behavior, and response projection at their existing boundaries.
|
||||
|
||||
| Entry point | Shared rule | Additional boundary |
|
||||
| --- | --- | --- |
|
||||
| Resource mutations with URL/body KB selectors | `CheckOwnershipOrRole`: creator or sufficient tenant role | API-key capabilities, KB scope and tenant visibility remain separate checks |
|
||||
| KB/document detail and content routes | `ResolveKB`: own tenant, organization share, then read-only shared agent | Tenant role/ownership guards and API-key capabilities |
|
||||
| Shared documents, chunks, search targets and suggestions | `KBSharePermissions`: organization grants only | Existing user requirements and caller-specific filtering/error behavior |
|
||||
| FAQ and tag service reads | `resolveKBReadTenant`: execution tenant or organization share | Direct cross-tenant calls require a user; no implicit agent fallback |
|
||||
| KB list, document search and batch restoration with `agent_id` | Shared-agent lookup, then `SharedAgentKBScope` | API-key intersection; dynamic list/search selections also apply capability filtering |
|
||||
| Wiki fixer source tenant | Organization Editor permission | Built-in fixer only; exactly one KB |
|
||||
| Message file shared-KB fallback | Organization Viewer permission | Persisted message reference, exact resource handle, KB/resource owner match |
|
||||
| KB file proxy | `ResolveKBFile`: exact Viewer grant and current explicit KB binding | Owner tenant and exports namespace; original uploads retain their separate download permission |
|
||||
| Message file proxy | `ResolveMessageFile`: session ownership and exact persisted output reference | Current shared-agent or organization-KB permission for source-owned resources |
|
||||
| Message artifact download | `ResolveMessageArtifact`: session ownership and persisted artifact index | Session-owned output remains accessible; source-owned output rechecks current sharing |
|
||||
| FAQ and tag mutations | `RequireKBWrite`: exact Editor operation grant | API-key ingest capability and KB scope; tenant-role/creator checks remain at the admission boundary |
|
||||
|
||||
## Ownership and write roles
|
||||
|
||||
URL middleware and body-based handlers use the same lazy ownership decision.
|
||||
The middleware adapter resolves identity, RBAC rollout configuration and the
|
||||
configuration-gated superuser flag. API-key principals, sufficient roles,
|
||||
cross-tenant superusers and disabled enforcement skip the creator lookup.
|
||||
API keys still require their route capabilities and allowed KB scope.
|
||||
|
||||
When a lookup is needed, it must check tenant visibility before returning the
|
||||
creator. An empty creator never matches an empty user. Middleware passes missing
|
||||
resources to the handler, reports lookup failures as 503 and audits only actual
|
||||
policy denials. Body-based handlers retain their existing 404/403/500 mapping.
|
||||
KB, initialization, Wiki and body-based KB lookups share the same tenant check.
|
||||
|
||||
## Caller and resource tenant
|
||||
|
||||
`types.Caller` holds the authenticated tenant, user and tenant role. Auth records
|
||||
it once; `KBRequest.Caller` and `KBAccess.Caller` use that same identity. The
|
||||
legacy `TenantIDContextKey` continues to scope repository/model execution, so
|
||||
existing storage APIs do not need a coordinated signature migration.
|
||||
|
||||
Use `types.WithExecutionTenant` to switch execution tenants. It captures the
|
||||
caller first, even when identity is missing, and grants no access by itself.
|
||||
Use `KBAccess.Context` after authorization to carry both the exact KB grant and
|
||||
its execution tenant, or `WithGrant` to carry the grant without switching yet.
|
||||
Grants bind the KB ID, owner tenant, caller and resolved permission; changing
|
||||
execution tenant cannot turn a Viewer grant into Editor or expose another KB.
|
||||
HTTP adapters retain Gin's caller keys for existing handlers.
|
||||
|
||||
Service reads use `KBPermissions` to check caller ownership, exact upstream
|
||||
grants, then organization shares for the original caller. This check also applies
|
||||
to documents/chunks returned by the initial tenant-scoped batch query, not just
|
||||
rows found during cross-tenant expansion. Search preflight, result enrichment,
|
||||
FAQ/tag reads, search targets and suggestion scopes follow the same boundary.
|
||||
General document search enumerates the caller's own/shared KBs; its execution
|
||||
tenant is not a new identity.
|
||||
|
||||
After authorizing a shared agent, `WithSharedAgent` carries its configured,
|
||||
source-tenant-bound read scope into the pipeline. An arbitrary execution switch
|
||||
cannot substitute for that grant. `logger.CloneContext` preserves the caller and
|
||||
resource grants for detached work within the operation. Context grants contain
|
||||
immutable snapshots; deriving a context does not mutate a sibling search branch.
|
||||
They are not persisted as authority for subsequent requests. API-key scope is
|
||||
reapplied when consuming any grant.
|
||||
|
||||
## Organization shares
|
||||
|
||||
`KBSharePermissions` fixes the caller tenant and role for one operation and caches
|
||||
the granular decision per KB. Reading many documents/chunks from one KB therefore
|
||||
queries its membership once. Both denied decisions and errors are cached within
|
||||
that operation. The next request creates a new resolver and observes revocations.
|
||||
The resolver is not a process-wide cache and is not shared between goroutines.
|
||||
|
||||
The share service remains responsible for the role cap across share permission,
|
||||
organization membership and caller tenant role. The resolver returns lookup errors
|
||||
so strict search preflight can report an infrastructure failure while best-effort
|
||||
result expansion can omit inaccessible rows. `ResolveKB` retains its independent
|
||||
read-only agent fallback when organization resolution cannot grant access.
|
||||
|
||||
## Shared-agent KB scope
|
||||
|
||||
A scope is created only after looking up an accessible shared agent. `all` means
|
||||
all KBs in that agent's tenant; `selected` means its explicit nonempty ID set.
|
||||
`none`, unknown modes and empty selections authorize no KBs. A nil/empty ID slice
|
||||
never implicitly grants an entire tenant. Specifying an agent does not permit
|
||||
falling back to a different shared agent.
|
||||
|
||||
These rules do not replace message/session ownership, persisted file-reference
|
||||
validation, resource binding, Agent tool search-target restrictions, or embed/IM
|
||||
capability checks. Raw tenant files and message files have different authorization
|
||||
contracts and must not be admitted by a generic KB grant alone.
|
||||
|
||||
## FAQ and tag writes
|
||||
|
||||
FAQ/tag service mutations consume an explicit Editor grant; setting an execution
|
||||
tenant or supplying a KB ID is insufficient. `ResolveKB` keeps the effective
|
||||
permission used for response projection, but caps its context grant to the
|
||||
requested operation. Thus an owner read does not mint a reusable write grant.
|
||||
HTTP routes retain their existing role/creator admission checks. API-key ingest
|
||||
capability and KB scope are reapplied when consuming a write grant.
|
||||
|
||||
Trusted FAQ-import, data-source sync and automatic-tagging workers use
|
||||
`WithKBTaskWrite` after loading and validating their target KB and child-resource
|
||||
bindings. This represents continuation of an admitted job, not renewed user
|
||||
authorization on every retry. It carries exactly one KB, preserves the caller
|
||||
or its absence, survives in-operation context cloning, and grants no tenant-wide
|
||||
role. Services use the KB owner's TenantInfo for model/index configuration.
|
||||
|
||||
Batch FAQ updates resolve all source/destination tags, entries and exclusions
|
||||
before mutation. Batch deletion also resolves all parent documents first.
|
||||
Tag-only batch updates use the same plan. Tag groups execute in ascending seq_id
|
||||
order, then explicit entry updates override only their specified fields.
|
||||
Import retains per-entry content-validation results, but out-of-scope resource
|
||||
references are rejected before admission and checked again in the worker.
|
||||
This preflight prevents partial writes caused by invalid resource selections;
|
||||
it does not add a transaction across the database, task queue and search index.
|
||||
|
||||
## File authorization and transport
|
||||
|
||||
File resolvers return request-local `FileAccess` locators before any storage
|
||||
reader is opened. The KB catalog accepts only explicit resource bindings to live
|
||||
documents in the exact live KB. Registered physical aliases resolve to the same
|
||||
resource ID; chunk/wiki text is never authorization evidence. Historical files
|
||||
without explicit bindings fail closed and need a trusted migration. A same-tenant
|
||||
path or handle prefix cannot replace a KB binding. Message evidence comes from
|
||||
persisted content, references, images, artifacts and tool results; tool arguments
|
||||
do not prove that the message returned a file. The organization-shared KB
|
||||
fallback also checks the catalog for an independent live KB/file binding;
|
||||
retrieval text and a currently shared KB ID alone are insufficient.
|
||||
|
||||
Source-owned shared-agent files also require the current live agent share.
|
||||
KB sources need an explicit live KB binding allowed by the agent's current
|
||||
`all`/`selected` selection and the caller's API-key KB scope; `none`, unknown
|
||||
modes, removed KB selections and deleted sources fail closed. Independently
|
||||
generated artifacts remain accessible when explicitly bound to the exact
|
||||
authorized message as an artifact. Mentioning a file in message text or the
|
||||
artifact list does not create that binding. Caller-owned artifacts retain their
|
||||
session ownership contract.
|
||||
|
||||
The storage adapter uses the authorized owner and backend. The shared
|
||||
`filetransport` package owns response headers, safe inline/download disposition,
|
||||
streaming and reader closure. Seekable readers use standard HTTP Range and HEAD
|
||||
handling; streaming-only backends return full content with `Accept-Ranges: none`
|
||||
without buffering the object (RFC 9110 sections 14.2 and 14.3). Permission-controlled responses use `private, no-store` so a later
|
||||
request rechecks current access. Tenant-wide and signed-token file routes keep
|
||||
their own authorization and caching contracts.
|
||||
|
||||
## Document and chunk writes
|
||||
|
||||
Document metadata, manual content, reparse, document tags/folders, image edits, chunk mutations
|
||||
and generated-question edits consume explicit Editor grants after loading the
|
||||
persisted document/KB binding. Full chunk objects cannot reparent existing rows.
|
||||
Batch inputs are deduplicated and fully validated, including missing IDs and
|
||||
each KB's grant, before writes. Text edits also check the parent/image-child
|
||||
relationships that they will modify. Body-based HTTP routes explicitly resolve
|
||||
an Editor operation; a previously cached owner read is not a write admission.
|
||||
|
||||
Document deletion shares a preflight plan and executor for single and batch
|
||||
calls. The plan validates every selection and KB, loads owner TenantInfo and
|
||||
routing metadata, and captures image references before marking rows as deleting.
|
||||
Index/graph failures leave the document and original file available for retry.
|
||||
Wiki deletion collects all chunk types before deleting chunks, so parent, image
|
||||
and summary citations are removed along with text citations.
|
||||
Physical file cleanup remains best effort after database deletion; this does not
|
||||
introduce a transaction across database, indexes and storage or durable blob GC.
|
||||
|
||||
New queued deletion payloads pin `knowledge_base_id` as well as tenant and exact
|
||||
document IDs. Workers consume a private cleanup scope bound to those references,
|
||||
not general KB write permission; already deleted IDs are successful no-ops.
|
||||
A moved document rejects the batch before side effects. Pre-upgrade payloads that
|
||||
omit the KB are accepted only when remaining rows resolve to one same-tenant KB;
|
||||
they cannot reconstruct the KB that existed at enqueue time. Failure callbacks
|
||||
skip rejected scopes and condition status updates on tenant, KB and deleting
|
||||
state. Scope/identity errors are not retried; infrastructure failures are.
|
||||
|
||||
Single/batch delete, clear-content and batch-reparse admission reject already
|
||||
moving documents with HTTP 409 before enqueue. Workers recheck current state;
|
||||
a deletion conflict is terminal for that admitted task, so retries cannot
|
||||
follow a document into a different KB. This does not serialize admission with
|
||||
a concurrent move. Synchronous metadata/image/chunk/summary handlers preserve
|
||||
application 403/409 errors, including wrapped errors.
|
||||
|
||||
Batch reparse and batch document tags resolve an Editor operation and apply the
|
||||
same creator-or-Admin admission policy as single-document writes, respecting
|
||||
RBAC rollout/API-key behavior. Batch reparse payloads likewise carry the admitted
|
||||
KB ID. Workers validate the whole batch before any submission and use exactly one `WithKBTaskWrite` grant;
|
||||
each reparse reload consumes that grant and rejects a document still moving.
|
||||
Missing/mixed/moved selections fail without submitting earlier batch members.
|
||||
Legacy reparse payloads can reconstruct only a present, single-KB binding.
|
||||
|
||||
Session-history, temporary-KB and clone cleanup use exact server-derived
|
||||
references from their already admitted operations. These cleanup contexts cannot
|
||||
be used for document/chunk updates or additional IDs. Parsing and enrichment
|
||||
workers use repositories for their internal persistence and rollback; they must
|
||||
keep their existing task ownership checks, abort checks and resource bindings.
|
||||
Repository access is not an incoming-request authorization API.
|
||||
|
||||
## Clone and move operations
|
||||
|
||||
`WithKBTransfer` admits the exact source/target pair after HTTP role/creator
|
||||
checks. Clone consumes source Viewer, target Editor and the API-key
|
||||
`manage_kbs` capability; move consumes Editor on both sides and `ingest`.
|
||||
Both operations remain within one tenant. Creating a clone reserves its target
|
||||
ID before enqueue and carries its creator into the worker. The ID of a new
|
||||
resource is not checked against an existing-resource API-key allow-list.
|
||||
|
||||
Workers reload both KBs, validate their owners and consume a private pair grant.
|
||||
Document/FAQ plans validate all selected resources, chunks, parents and tags
|
||||
before destructive work. Lifecycle reads include every chunk type and indexing
|
||||
state. A document's `_knowledge_transfer` metadata records task identity and
|
||||
progress; retries reuse the reserved target, replace incomplete clones, skip
|
||||
completed vector moves and retry outstanding wiki cleanup/parser enqueue.
|
||||
This field is internal operation state, not caller-supplied custom metadata.
|
||||
For `reparse`, transfer `done` means target binding and parser admission are
|
||||
complete, not that parsing has finished. `ParseStatus` remains the parser's
|
||||
state. `reparse_pending` already admits the target parser; extending the
|
||||
`moving` write guard to that phase would also block legitimate worker writes.
|
||||
Locking all writes until parsing finishes is a separate processing/concurrency
|
||||
contract, including ordinary reparses, and is not provided by this transfer marker.
|
||||
|
||||
Failed clone chunk/index writes attempt rollback after conditionally claiming
|
||||
the original destination. Rollback failures are returned, references retained
|
||||
when cleanup is uncertain, and later retries replace the failed copy. Accounting
|
||||
is committed only after the clone succeeds. FAQ containers do not inherit the
|
||||
source document's transfer marker.
|
||||
|
||||
Source wiki cleanup starts only after the document ownership CAS succeeds.
|
||||
The transfer checkpoint preserves source chunk IDs and summary before reparse
|
||||
removes them. Retract page lists are persisted before page source refs are removed, so queue
|
||||
failures retain retry evidence. Cleanup errors propagate; redelivery resumes cleanup after a
|
||||
committed move without repeating vector/chunk relocation. These retries do not
|
||||
provide a distributed transaction or recovery after retries are exhausted.
|
||||
|
||||
Vector reuse relocates KB metadata while preserving physical vector/chunk IDs.
|
||||
Weaviate and Milvus drain the first source-filtered page until a follow-up query
|
||||
is empty, avoiding offset limits/skipped rows. Repeated IDs or partial failures
|
||||
return errors and leave the document checkpoint pending for retry.
|
||||
OpenSearch selects the entire source KB/document pair, even without DB chunks,
|
||||
and rejects version conflicts, timeouts and partial failures before DB relocation.
|
||||
OpenSearch and Elasticsearch 7/8 require present non-negative `total`/`updated`
|
||||
counts with `updated == total`; their move scripts do not skip/delete documents.
|
||||
A zero/zero result allows a retry after metadata already reached the target.
|
||||
Reparse cleanup loads and validates the source KB before touching any backend,
|
||||
reusing its storage binding throughout; a failed/missing lookup cannot fall
|
||||
back to the tenant default store.
|
||||
Vector reuse never copies vectors and then deletes by the same document ID. Source and
|
||||
target must use the same vector store for reuse, and the same concrete file
|
||||
storage instance for either mode. Doris ANN tables cannot safely replace rows
|
||||
without a delete/insert gap, so their moves require `reparse`. Clone still
|
||||
requires compatible embedding models and vector stores; a move also retains
|
||||
the existing same-model requirement. This is not a cross-storage migration API.
|
||||
|
||||
Conditional document checkpoints bind tenant, KB, status and the stored update
|
||||
time. They keep storage accounting in the same database transaction and read
|
||||
back timestamp precision. Delete stops when its checkpoint no longer matches.
|
||||
Ordinary document/chunk writes reject a document in the `moving` phase. Parser
|
||||
admission and failure callbacks verify the task's persisted tenant/KB binding;
|
||||
table-summary failure cleanup also stops if its document changed.
|
||||
|
||||
## Follow-up work outside this refactor
|
||||
|
||||
These are separate follow-ups, not capabilities provided by this package:
|
||||
|
||||
1. **Complete worker contracts and concurrency control.** Audit enrichment,
|
||||
table-summary, multimodal, wiki and indexing workers end to end. Standardize
|
||||
payload resource bindings, processing-attempt IDs and conditional writes,
|
||||
including operations admitted before a concurrent move. Do not replace
|
||||
these checks with an execution-tenant switch or a blanket task write grant.
|
||||
2. **Durable recovery across systems.** Add a transactional outbox and explicit
|
||||
recovery/cancellation for exhausted transfer tasks. Current retries use the
|
||||
same admitted task identity; a new move request is not a resume API. Database,
|
||||
queue, vector, graph and blob operations are not one atomic transaction.
|
||||
Physical file cleanup remains best effort and needs durable garbage collection.
|
||||
3. **External-backend integration coverage.** Exercise clone/move, partial
|
||||
backend failures, retries, pagination and search/edit behavior against live
|
||||
PostgreSQL, Elasticsearch 7/8, OpenSearch, Qdrant, Milvus, Weaviate,
|
||||
TencentVectorDB and Doris. Local SQLite vector tests and backend compilation
|
||||
do not establish this coverage. Doris ANN vector reuse remains unsupported.
|
||||
4. **Finish service-layer migration.** Apply explicit operation grants to the
|
||||
remaining KB configuration/duplicate/delete, wiki and resource-administration
|
||||
service methods, preserving their existing route admission policies. Keep
|
||||
user/organization membership, embed, IM and signed-resource contracts separate.
|
||||
5. **Trusted legacy file migration.** Backfill historical file bindings from
|
||||
verified provenance, not arbitrary chunk/wiki mentions. Unbound historical
|
||||
files now fail closed; request-path text fallback has been removed. Migrate
|
||||
old queued payloads and transfer records without captured source wiki refs.
|
||||
A legacy delete task without a KB ID can reconstruct only its present
|
||||
unambiguous binding, not its original enqueue-time binding.
|
||||
|
||||
No public request schema or database column migration is required by this
|
||||
refactor. Internal service interfaces require operation contexts, queued payloads
|
||||
add optional scope fields, and malformed/out-of-scope batch selections now fail
|
||||
before mutation instead of being silently skipped. Callers must also handle a
|
||||
409 for document edits/deletes while a move is in progress.
|
||||
|
|
@ -0,0 +1,94 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// Context grants contain values, not mutable KB objects. Deriving a context
|
||||
// copies the grant slice so parallel search branches cannot widen each other.
|
||||
type kbGrant struct {
|
||||
caller types.Caller
|
||||
kbID string
|
||||
tenantID uint64
|
||||
permission types.OrgMemberRole
|
||||
task bool
|
||||
}
|
||||
|
||||
type agentGrant struct {
|
||||
caller types.Caller
|
||||
scope types.SharedAgentKBScope
|
||||
}
|
||||
|
||||
// WithSharedAgent scopes execution after the caller has authorized this exact
|
||||
// agent. Only its configured KBs receive read access; other source-tenant KBs
|
||||
// and the source tenant's organization memberships are not inherited.
|
||||
func WithSharedAgent(ctx context.Context, agent *types.CustomAgent) context.Context {
|
||||
grant := agentGrant{caller: types.CallerFromContext(ctx), scope: types.NewSharedAgentKBScope(agent)}
|
||||
ctx = context.WithValue(ctx, types.SharedAgentGrantContextKey, grant)
|
||||
if agent == nil {
|
||||
return ctx
|
||||
}
|
||||
return types.WithExecutionTenant(ctx, agent.TenantID)
|
||||
}
|
||||
|
||||
// HasKBGrant checks only previously resolved resource access. It never infers
|
||||
// ownership from the execution tenant, and always reapplies API-key scope.
|
||||
func HasKBGrant(ctx context.Context, kbID string, tenantID uint64, required types.OrgMemberRole) bool {
|
||||
if !required.IsValid() || types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID) != nil {
|
||||
return false
|
||||
}
|
||||
caller := types.CallerFromContext(ctx)
|
||||
if kbID == "" || tenantID == 0 {
|
||||
return false
|
||||
}
|
||||
grants, _ := ctx.Value(types.KBGrantsContextKey).([]kbGrant)
|
||||
for _, grant := range grants {
|
||||
if (caller.TenantID != 0 || grant.task) && grant.caller == caller && grant.kbID == kbID &&
|
||||
grant.tenantID == tenantID &&
|
||||
grant.permission.HasPermission(required) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
grant, ok := ctx.Value(types.SharedAgentGrantContextKey).(agentGrant)
|
||||
return ok && caller.TenantID != 0 && required == types.OrgRoleViewer && grant.caller == caller &&
|
||||
grant.scope.Allows(kbID, tenantID)
|
||||
}
|
||||
|
||||
// KBPermissions combines caller ownership, exact context grants and organization
|
||||
// shares for one service operation. Pass nil shares when that entry point does
|
||||
// not permit organization expansion (for example, a caller without a user).
|
||||
type KBPermissions struct {
|
||||
ctx context.Context
|
||||
caller types.Caller
|
||||
shares *KBSharePermissions
|
||||
}
|
||||
|
||||
// NewKBPermissions resolves reads for one caller and operation.
|
||||
func NewKBPermissions(ctx context.Context, shares KBShareLookup) *KBPermissions {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
return &KBPermissions{
|
||||
ctx: ctx,
|
||||
caller: caller,
|
||||
shares: NewKBSharePermissions(ctx, shares, caller.TenantID, caller.Role),
|
||||
}
|
||||
}
|
||||
|
||||
// Check combines caller ownership, exact grants and organization sharing.
|
||||
func (p *KBPermissions) Check(kbID string, ownerTenantID uint64, required types.OrgMemberRole) (bool, error) {
|
||||
if kbID == "" || ownerTenantID == 0 || !required.IsValid() {
|
||||
return false, nil
|
||||
}
|
||||
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(p.ctx, kbID); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if (p.caller.TenantID == ownerTenantID && required == types.OrgRoleViewer) ||
|
||||
HasKBGrant(p.ctx, kbID, ownerTenantID, required) {
|
||||
return true, nil
|
||||
}
|
||||
if p.caller.TenantID == 0 || p.caller.TenantID == ownerTenantID {
|
||||
return false, nil
|
||||
}
|
||||
return p.shares.Check(kbID, required)
|
||||
}
|
||||
|
|
@ -0,0 +1,140 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func callerContext() context.Context {
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
|
||||
ctx = context.WithValue(ctx, types.UserIDContextKey, "user")
|
||||
return context.WithValue(ctx, types.TenantRoleContextKey, types.TenantRoleContributor)
|
||||
}
|
||||
|
||||
func TestKBGrantSurvivesExecutionSwitchAndDetachWithoutWidening(t *testing.T) {
|
||||
ctx := callerContext()
|
||||
kb := &types.KnowledgeBase{ID: "shared", TenantID: 2}
|
||||
grant, err := ResolveKB(
|
||||
ctx,
|
||||
KBRequest{Caller: types.CallerFromContext(ctx)},
|
||||
kb,
|
||||
types.OrgRoleViewer,
|
||||
&shareLookup{permission: types.OrgRoleViewer},
|
||||
nil,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
ctx = grant.Context(ctx)
|
||||
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx))
|
||||
ctx = logger.CloneContext(types.WithExecutionTenant(ctx, 3))
|
||||
require.Equal(
|
||||
t,
|
||||
types.Caller{TenantID: 1, UserID: "user", Role: types.TenantRoleContributor},
|
||||
types.CallerFromContext(ctx),
|
||||
)
|
||||
require.True(t, HasKBGrant(ctx, "shared", 2, types.OrgRoleViewer))
|
||||
// A later mutation of the service model cannot mutate the captured grant.
|
||||
kb.ID = "private"
|
||||
require.False(t, HasKBGrant(ctx, "private", 2, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(ctx, "shared", 3, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(ctx, "shared", 2, types.OrgRoleEditor))
|
||||
for _, caller := range []types.Caller{
|
||||
{TenantID: 9, UserID: "user", Role: types.TenantRoleContributor},
|
||||
{TenantID: 1, UserID: "other", Role: types.TenantRoleContributor},
|
||||
{TenantID: 1, UserID: "user", Role: types.TenantRoleViewer},
|
||||
} {
|
||||
otherCtx := types.WithCaller(ctx, caller)
|
||||
require.False(t, HasKBGrant(otherCtx, "shared", 2, types.OrgRoleViewer))
|
||||
require.Equal(
|
||||
t,
|
||||
caller,
|
||||
types.CallerFromContext(grant.Context(otherCtx)),
|
||||
"reusing a grant cannot impersonate its caller",
|
||||
)
|
||||
require.Equal(
|
||||
t,
|
||||
uint64(3),
|
||||
types.MustTenantIDFromContext(grant.Context(otherCtx)),
|
||||
"a caller mismatch must not change execution scope",
|
||||
)
|
||||
}
|
||||
narrowed := types.WithTenantAPIKeyScope(
|
||||
ctx,
|
||||
types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"another"}},
|
||||
)
|
||||
require.False(t, HasKBGrant(narrowed, "shared", 2, types.OrgRoleViewer))
|
||||
}
|
||||
|
||||
func TestExecutionTenantDoesNotGrantOwnershipOrChangeShareIdentity(t *testing.T) {
|
||||
ctx := types.WithExecutionTenant(callerContext(), 2)
|
||||
shares := &shareLookup{}
|
||||
allowed, err := NewKBPermissions(ctx, shares).Check("private", 2, types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed)
|
||||
require.Equal(t, uint64(1), shares.caller)
|
||||
require.Equal(t, types.TenantRoleContributor, shares.role)
|
||||
ctx = types.WithExecutionTenant(context.Background(), 2)
|
||||
allowed, err = NewKBPermissions(ctx, nil).Check("private", 2, types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed, "an absent caller must not become the execution tenant")
|
||||
}
|
||||
|
||||
func TestSharedAgentGrantPreservesConfiguredScopeAcrossDetach(t *testing.T) {
|
||||
for _, mode := range []string{"all", "selected", "none", "unknown"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
agent := &types.CustomAgent{
|
||||
TenantID: 2,
|
||||
Config: types.CustomAgentConfig{KBSelectionMode: mode, KnowledgeBases: []string{"selected"}},
|
||||
}
|
||||
ctx := logger.CloneContext(WithSharedAgent(callerContext(), agent))
|
||||
require.Equal(t, uint64(1), types.CallerFromContext(ctx).TenantID)
|
||||
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx))
|
||||
require.Equal(t, mode == "all" || mode == "selected", HasKBGrant(ctx, "selected", 2, types.OrgRoleViewer))
|
||||
require.Equal(t, mode == "all", HasKBGrant(ctx, "other", 2, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(ctx, "selected", 3, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(ctx, "selected", 2, types.OrgRoleEditor))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestKBGrantBranchesRemainIndependent(t *testing.T) {
|
||||
base := callerContext()
|
||||
grant := &KBAccess{Caller: types.CallerFromContext(base), EffectiveTenantID: 2, Permission: types.OrgRoleViewer}
|
||||
grant.KnowledgeBase = &types.KnowledgeBase{ID: "first", TenantID: 2}
|
||||
first := grant.WithGrant(base)
|
||||
grant.KnowledgeBase = &types.KnowledgeBase{ID: "second", TenantID: 2}
|
||||
second := grant.WithGrant(base)
|
||||
require.True(t, HasKBGrant(first, "first", 2, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(first, "second", 2, types.OrgRoleViewer))
|
||||
require.True(t, HasKBGrant(second, "second", 2, types.OrgRoleViewer))
|
||||
require.False(t, HasKBGrant(second, "first", 2, types.OrgRoleViewer))
|
||||
}
|
||||
|
||||
func TestKBPermissionsOwnerShortcutOnlyGrantsRead(t *testing.T) {
|
||||
roles := []types.TenantRole{types.TenantRoleViewer, types.TenantRoleContributor, types.TenantRoleAdmin}
|
||||
for _, role := range roles {
|
||||
ctx := context.WithValue(callerContext(), types.TenantRoleContextKey, role)
|
||||
permissions := NewKBPermissions(ctx, nil)
|
||||
for _, required := range []types.OrgMemberRole{types.OrgRoleViewer, types.OrgRoleEditor, types.OrgRoleAdmin} {
|
||||
allowed, err := permissions.Check("kb", 1, required)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, required == types.OrgRoleViewer, allowed)
|
||||
}
|
||||
grant := &KBAccess{
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
KnowledgeBase: &types.KnowledgeBase{ID: "kb", TenantID: 1},
|
||||
EffectiveTenantID: 1,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}
|
||||
permissions = NewKBPermissions(grant.Context(ctx), nil)
|
||||
allowed, err := permissions.Check("kb", 1, types.OrgRoleEditor)
|
||||
require.NoError(t, err)
|
||||
require.True(t, allowed)
|
||||
allowed, err = permissions.Check("kb", 1, types.OrgRoleAdmin)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,256 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
secutils "github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
// FileAccess is a request-local result, produced before storage is opened.
|
||||
// Physical locators are not a substitute for KB/message authorization.
|
||||
type FileAccess struct {
|
||||
OwnerTenantID uint64
|
||||
Path string
|
||||
Filename string
|
||||
StorageBackendID string
|
||||
}
|
||||
|
||||
// FileCatalog resolves logical references to their stored owner and locator.
|
||||
type FileCatalog interface {
|
||||
ResolvePath(context.Context, string) (string, *types.StoredResource, error)
|
||||
}
|
||||
|
||||
func resolveFile(
|
||||
ctx context.Context,
|
||||
catalog FileCatalog,
|
||||
reference string,
|
||||
) (FileAccess, *types.StoredResource, error) {
|
||||
file := FileAccess{Path: reference, Filename: reference}
|
||||
if catalog == nil {
|
||||
if _, resource := types.ParseResourcePath(reference); resource {
|
||||
return file, nil, ErrNotFound
|
||||
}
|
||||
return file, nil, nil
|
||||
}
|
||||
path, resource, err := catalog.ResolvePath(ctx, reference)
|
||||
if err != nil {
|
||||
return file, nil, ErrNotFound
|
||||
}
|
||||
file.Path = path
|
||||
file.Filename = path
|
||||
if resource != nil {
|
||||
file.OwnerTenantID = resource.TenantID
|
||||
file.StorageBackendID = resource.StorageBackendID
|
||||
if strings.TrimSpace(resource.OriginalName) != "" {
|
||||
file.Filename = resource.OriginalName
|
||||
}
|
||||
}
|
||||
return file, resource, nil
|
||||
}
|
||||
|
||||
// ResolveKBFile requires the route's exact KB grant and an independent live
|
||||
// explicit binding check. Rewriting the execution tenant is insufficient.
|
||||
func ResolveKBFile(
|
||||
ctx context.Context,
|
||||
grant *KBAccess,
|
||||
kbID, reference string,
|
||||
catalog FileCatalog,
|
||||
bindings interfaces.KBResourceLookup,
|
||||
) (FileAccess, error) {
|
||||
var empty FileAccess
|
||||
if grant == nil || grant.KnowledgeBase == nil || grant.KnowledgeBase.ID != kbID ||
|
||||
grant.Caller != types.CallerFromContext(ctx) || !grant.Permission.HasPermission(types.OrgRoleViewer) {
|
||||
return empty, ErrForbidden
|
||||
}
|
||||
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID); err != nil {
|
||||
return empty, err
|
||||
}
|
||||
file, resource, err := resolveFile(ctx, catalog, reference)
|
||||
if err != nil {
|
||||
return empty, err
|
||||
}
|
||||
owner := grant.EffectiveTenantID
|
||||
if owner == 0 || (resource != nil && resource.TenantID != owner) {
|
||||
return empty, ErrForbidden
|
||||
}
|
||||
if err := secutils.ValidateKBScopedStoragePath(file.Path, owner); err != nil {
|
||||
return empty, ErrForbidden
|
||||
}
|
||||
if bindings == nil {
|
||||
return empty, ErrForbidden
|
||||
}
|
||||
bound, err := bindings.IsReferencedByKnowledgeBase(ctx, owner, kbID, reference)
|
||||
if err != nil {
|
||||
return empty, fmt.Errorf("check KB file binding: %w", err)
|
||||
}
|
||||
if !bound {
|
||||
return empty, ErrForbidden
|
||||
}
|
||||
file.OwnerTenantID = owner
|
||||
return file, nil
|
||||
}
|
||||
|
||||
// MessageReferencesFile examines only persisted rendering/output fields. Tool
|
||||
// arguments and request metadata are not evidence of a returned file.
|
||||
func MessageReferencesFile(message *types.Message, reference string) bool {
|
||||
if message == nil {
|
||||
return false
|
||||
}
|
||||
if types.ContainsStorageReference(message.Content, reference) {
|
||||
return true
|
||||
}
|
||||
for _, artifact := range message.Artifacts {
|
||||
if artifact.URL == reference {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, value := range []interface{}{message.KnowledgeReferences, message.Images} {
|
||||
data, _ := json.Marshal(value)
|
||||
if types.ContainsStorageReference(string(data), reference) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, step := range message.AgentSteps {
|
||||
for _, call := range step.ToolCalls {
|
||||
if call.Result == nil {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(call.Result)
|
||||
if types.ContainsStorageReference(string(data), reference) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ResolveMessageFile loads the session-authorized message using the original
|
||||
// caller and rechecks its current sharing relationships on every request.
|
||||
func ResolveMessageFile(ctx context.Context, sessionID, messageID, reference string, messages MessageFileLookup,
|
||||
agents SharedAgentFileLookup, catalog FileCatalog, kbShares MessageKBShareAuthorizer,
|
||||
) (FileAccess, error) {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
if caller.TenantID == 0 {
|
||||
return FileAccess{}, ErrUnauthorized
|
||||
}
|
||||
if messages == nil {
|
||||
return FileAccess{}, ErrNotFound
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, caller.TenantID)
|
||||
message, err := messages.GetMessage(ctx, sessionID, messageID)
|
||||
if err != nil || message == nil {
|
||||
return FileAccess{}, ErrNotFound
|
||||
}
|
||||
return AuthorizeMessageFile(ctx, message, reference, agents, catalog, kbShares)
|
||||
}
|
||||
|
||||
// AuthorizeMessageFile also serves index-based artifact downloads after their
|
||||
// session/message lookup. It never accepts an unchecked client message.
|
||||
func AuthorizeMessageFile(ctx context.Context, message *types.Message, reference string, agents SharedAgentFileLookup,
|
||||
catalog FileCatalog, kbShares MessageKBShareAuthorizer,
|
||||
) (FileAccess, error) {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
if caller.TenantID == 0 {
|
||||
return FileAccess{}, ErrUnauthorized
|
||||
}
|
||||
if !MessageReferencesFile(message, reference) {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
file, resource, err := resolveFile(ctx, catalog, reference)
|
||||
if err != nil {
|
||||
return FileAccess{}, err
|
||||
}
|
||||
owner := message.AgentTenantID
|
||||
if resource != nil {
|
||||
owner = resource.TenantID
|
||||
}
|
||||
if owner == 0 {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
if message.Role == "user" && owner != caller.TenantID {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
if kbShares.Bindings == nil {
|
||||
kbShares.Bindings, _ = catalog.(interfaces.KBResourceLookup)
|
||||
}
|
||||
kbAuthorized := false
|
||||
if resource != nil && message.AgentTenantID != 0 && message.AgentTenantID != owner {
|
||||
kbAuthorized = kbShares.resourceAccessibleViaSharedKB(ctx, message, resource, caller.TenantID, caller.Role)
|
||||
if !kbAuthorized {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
}
|
||||
if owner != caller.TenantID && !kbAuthorized {
|
||||
if message.AgentTenantID == 0 {
|
||||
kbAuthorized = kbShares.resourceAccessibleViaSharedKB(ctx, message, resource, caller.TenantID, caller.Role)
|
||||
}
|
||||
if !kbAuthorized {
|
||||
if message.AgentID == "" || agents == nil {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
agent, err := agents.GetSharedAgentForTenant(ctx, caller.TenantID, caller.Role, message.AgentID, owner)
|
||||
if err != nil || agent == nil || agent.TenantID != owner {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
bindings, ok := catalog.(interfaces.MessageFileBindingLookup)
|
||||
if !ok {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
origins, err := bindings.GetMessageFileBindings(ctx, owner, reference, message.ID)
|
||||
if err != nil || origins == nil {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
allowed := origins.MessageArtifact
|
||||
scope := types.NewSharedAgentKBScope(agent)
|
||||
for _, kbID := range origins.KnowledgeBaseIDs {
|
||||
if scope.Allows(kbID, owner) && types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbID) == nil {
|
||||
allowed = true
|
||||
}
|
||||
}
|
||||
if !allowed {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
}
|
||||
}
|
||||
if resource == nil {
|
||||
if err := secutils.ValidateStoragePathTenant(file.Path, owner); err != nil {
|
||||
return FileAccess{}, ErrForbidden
|
||||
}
|
||||
}
|
||||
file.OwnerTenantID = owner
|
||||
return file, nil
|
||||
}
|
||||
|
||||
// ResolveMessageArtifact selects a persisted artifact after session ownership
|
||||
// was checked. Session-owned output stays downloadable independently of the
|
||||
// agent; source-owned output still requires its current sharing permission.
|
||||
func ResolveMessageArtifact(ctx context.Context, message *types.Message, index int, agents SharedAgentFileLookup,
|
||||
catalog FileCatalog, kbShares MessageKBShareAuthorizer,
|
||||
) (FileAccess, error) {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
if caller.TenantID == 0 {
|
||||
return FileAccess{}, ErrUnauthorized
|
||||
}
|
||||
if message == nil || index < 0 || index >= len(message.Artifacts) {
|
||||
return FileAccess{}, ErrNotFound
|
||||
}
|
||||
artifact := message.Artifacts[index]
|
||||
file, resource, err := resolveFile(ctx, catalog, artifact.URL)
|
||||
if err != nil {
|
||||
return FileAccess{}, err
|
||||
}
|
||||
if (resource != nil && resource.TenantID == caller.TenantID) ||
|
||||
(resource == nil && secutils.ValidateStoragePathTenant(file.Path, caller.TenantID) == nil) {
|
||||
file.OwnerTenantID = caller.TenantID
|
||||
file.Filename = artifact.FileName
|
||||
return file, nil
|
||||
}
|
||||
file, err = AuthorizeMessageFile(ctx, message, artifact.URL, agents, catalog, kbShares)
|
||||
file.Filename = artifact.FileName
|
||||
return file, err
|
||||
}
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type fileCatalog struct{ resource *types.StoredResource }
|
||||
|
||||
func (c fileCatalog) ResolvePath(_ context.Context, ref string) (string, *types.StoredResource, error) {
|
||||
if c.resource == nil {
|
||||
return ref, nil, nil
|
||||
}
|
||||
return c.resource.PhysicalPath, c.resource, nil
|
||||
}
|
||||
|
||||
type fileBinding struct {
|
||||
allowed bool
|
||||
err error
|
||||
kb string
|
||||
tenant uint64
|
||||
}
|
||||
|
||||
func (b *fileBinding) IsReferencedByKnowledgeBase(_ context.Context, tenant uint64, kb, _ string) (bool, error) {
|
||||
b.kb, b.tenant = kb, tenant
|
||||
return b.allowed, b.err
|
||||
}
|
||||
|
||||
type fileMessages struct {
|
||||
message *types.Message
|
||||
tenant uint64
|
||||
}
|
||||
|
||||
func (m *fileMessages) GetMessage(ctx context.Context, _, _ string) (*types.Message, error) {
|
||||
m.tenant = types.MustTenantIDFromContext(ctx)
|
||||
return m.message, nil
|
||||
}
|
||||
|
||||
func TestKBFilesRequireExactGrantAndBinding(t *testing.T) {
|
||||
base := callerContext()
|
||||
grant := &KBAccess{
|
||||
Caller: types.CallerFromContext(base),
|
||||
KnowledgeBase: &types.KnowledgeBase{ID: "kb", TenantID: 2},
|
||||
EffectiveTenantID: 2,
|
||||
Permission: types.OrgRoleViewer,
|
||||
}
|
||||
ctx := grant.Context(base)
|
||||
bindings := &fileBinding{allowed: true}
|
||||
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
|
||||
catalog := fileCatalog{
|
||||
&types.StoredResource{TenantID: 2, PhysicalPath: "local://2/exports/a.png", OriginalName: "a.png"},
|
||||
}
|
||||
file, err := ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(2), file.OwnerTenantID)
|
||||
require.Equal(t, "kb", bindings.kb)
|
||||
require.Equal(t, uint64(2), bindings.tenant)
|
||||
bindings.allowed = false
|
||||
_, err = ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
|
||||
require.ErrorIs(t, err, ErrForbidden, "a same-tenant file still needs an exact KB binding")
|
||||
bindings.allowed = true
|
||||
_, err = ResolveKBFile(ctx, grant, "another", ref, catalog, bindings)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
_, err = ResolveKBFile(ctx, nil, "kb", ref, catalog, bindings)
|
||||
require.ErrorIs(t, err, ErrForbidden, "execution tenant alone is not a grant")
|
||||
bindings.err = errors.New("database unavailable")
|
||||
_, err = ResolveKBFile(ctx, grant, "kb", ref, catalog, bindings)
|
||||
require.ErrorIs(t, err, bindings.err)
|
||||
}
|
||||
|
||||
func TestMessageFilesRequireReferenceAndRecheckRevocation(t *testing.T) {
|
||||
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
|
||||
messages := &fileMessages{
|
||||
message: &types.Message{ID: "message", AgentID: "agent", AgentTenantID: 2, Role: "assistant"},
|
||||
}
|
||||
agents := &agentLookup{agent: &types.CustomAgent{ID: "agent", TenantID: 2}}
|
||||
catalog := artifactFileCatalog{
|
||||
fileCatalog{&types.StoredResource{TenantID: 2, PhysicalPath: "local://2/exports/a.png"}},
|
||||
}
|
||||
ctx := types.WithExecutionTenant(callerContext(), 2)
|
||||
_, err := ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
messages.message.Content = ""
|
||||
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
|
||||
require.ErrorIs(t, err, ErrForbidden, "a longer handle is not the requested handle")
|
||||
messages.message.Content = ""
|
||||
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(1), messages.tenant, "session lookup must use the caller")
|
||||
agents.agent = nil
|
||||
_, err = ResolveMessageFile(ctx, "session", "message", ref, messages, agents, catalog, MessageKBShareAuthorizer{})
|
||||
require.ErrorIs(t, err, ErrForbidden, "historical references cannot bypass share revocation")
|
||||
}
|
||||
|
||||
func TestMessageArtifactsKeepSessionOwnershipSeparateFromAgentOutput(t *testing.T) {
|
||||
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
|
||||
message := &types.Message{
|
||||
AgentID: "agent",
|
||||
AgentTenantID: 2,
|
||||
Artifacts: types.MessageArtifacts{{URL: ref, FileName: "report.pdf"}},
|
||||
}
|
||||
catalog := fileCatalog{&types.StoredResource{TenantID: 1, PhysicalPath: "local://1/exports/report.pdf"}}
|
||||
file, err := ResolveMessageArtifact(callerContext(), message, 0, nil, catalog, MessageKBShareAuthorizer{})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(1), file.OwnerTenantID)
|
||||
catalog.resource.TenantID = 2
|
||||
catalog.resource.PhysicalPath = "local://2/exports/report.pdf"
|
||||
_, err = ResolveMessageArtifact(callerContext(), message, 0, nil, catalog, MessageKBShareAuthorizer{})
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
}
|
||||
|
||||
type messageFileKBs struct{ kb *types.KnowledgeBase }
|
||||
|
||||
func (k messageFileKBs) GetKnowledgeBasesByIDsOnly(context.Context, []string) ([]*types.KnowledgeBase, error) {
|
||||
return []*types.KnowledgeBase{k.kb}, nil
|
||||
}
|
||||
|
||||
func TestMessageSharedKBFilesRequireLiveBinding(t *testing.T) {
|
||||
const ref = "resource://AbCdEfGhIjKlMnOpQrStUv"
|
||||
message := &types.Message{
|
||||
AgentTenantID: 1, Role: "assistant",
|
||||
KnowledgeReferences: types.References{{KnowledgeBaseID: "shared", Content: ref}},
|
||||
}
|
||||
catalog := fileCatalog{&types.StoredResource{
|
||||
Handle: "AbCdEfGhIjKlMnOpQrStUv", TenantID: 2, PhysicalPath: "local://2/exports/a.png",
|
||||
}}
|
||||
binding := &fileBinding{allowed: true}
|
||||
shares := &shareLookup{permission: types.OrgRoleViewer}
|
||||
authorizer := MessageKBShareAuthorizer{
|
||||
ShareGuard: shares, KBs: messageFileKBs{kb: &types.KnowledgeBase{ID: "shared", TenantID: 2}}, Bindings: binding,
|
||||
}
|
||||
_, err := AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "shared", binding.kb)
|
||||
require.Equal(t, uint64(2), binding.tenant)
|
||||
binding.allowed = false
|
||||
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
|
||||
require.ErrorIs(t, err, ErrForbidden, "retrieval text alone must not authorize an unrelated same-tenant resource")
|
||||
binding.allowed = true
|
||||
binding.err = errors.New("binding lookup unavailable")
|
||||
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
binding.err = nil
|
||||
authorizer.Bindings = nil
|
||||
_, err = AuthorizeMessageFile(callerContext(), message, ref, nil, catalog, authorizer)
|
||||
require.ErrorIs(t, err, ErrForbidden, "missing live lookup must fail closed")
|
||||
}
|
||||
|
||||
// This fixture represents a catalog claim made by the artifact collector.
|
||||
type artifactFileCatalog struct{ fileCatalog }
|
||||
|
||||
func (artifactFileCatalog) GetMessageFileBindings(
|
||||
_ context.Context,
|
||||
_ uint64,
|
||||
_, messageID string,
|
||||
) (*types.MessageFileBindings, error) {
|
||||
return &types.MessageFileBindings{MessageArtifact: messageID == "message"}, nil
|
||||
}
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// KBSharePermissions resolves organization grants for one caller during one
|
||||
// operation. It deliberately does not grant tenant ownership or shared-agent
|
||||
// access. Callers retain their own policy for lookup failures (abort or filter).
|
||||
// Create a new instance for each operation; it is not a cross-request cache and
|
||||
// must not be shared between goroutines.
|
||||
type KBSharePermissions struct {
|
||||
ctx context.Context
|
||||
lookup KBShareLookup
|
||||
tenantID uint64
|
||||
role types.TenantRole
|
||||
results map[string]kbSharePermission
|
||||
}
|
||||
|
||||
type kbSharePermission struct {
|
||||
role types.OrgMemberRole
|
||||
shared bool
|
||||
err error
|
||||
}
|
||||
|
||||
// NewKBSharePermissions creates a request-local permission cache.
|
||||
func NewKBSharePermissions(
|
||||
ctx context.Context,
|
||||
lookup KBShareLookup,
|
||||
tenantID uint64,
|
||||
role types.TenantRole,
|
||||
) *KBSharePermissions {
|
||||
return &KBSharePermissions{
|
||||
ctx: ctx,
|
||||
lookup: lookup,
|
||||
tenantID: tenantID,
|
||||
role: role,
|
||||
results: make(map[string]kbSharePermission),
|
||||
}
|
||||
}
|
||||
|
||||
// Permission caches both grants and failures so multiple documents/chunks in a
|
||||
// KB observe the same decision, without repeating the membership queries.
|
||||
func (p *KBSharePermissions) Permission(kbID string) (types.OrgMemberRole, bool, error) {
|
||||
if kbID == "" || p.tenantID == 0 || p.lookup == nil {
|
||||
return "", false, nil
|
||||
}
|
||||
result, ok := p.results[kbID]
|
||||
if !ok {
|
||||
result.role, result.shared, result.err = p.lookup.CheckTenantKBPermission(p.ctx, kbID, p.tenantID, p.role)
|
||||
p.results[kbID] = result
|
||||
}
|
||||
return result.role, result.shared, result.err
|
||||
}
|
||||
|
||||
// Check tests the required role against the cached organization permission.
|
||||
func (p *KBSharePermissions) Check(kbID string, required types.OrgMemberRole) (bool, error) {
|
||||
role, shared, err := p.Permission(kbID)
|
||||
return err == nil && shared && role.IsValid() && required.IsValid() && role.HasPermission(required), err
|
||||
}
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type permissionLookupFunc func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error)
|
||||
|
||||
func (f permissionLookupFunc) CheckTenantKBPermission(
|
||||
ctx context.Context,
|
||||
id string,
|
||||
tenant uint64,
|
||||
role types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
return f(ctx, id, tenant, role)
|
||||
}
|
||||
|
||||
func TestKBSharePermissionsCachePerOperation(t *testing.T) {
|
||||
calls := map[string]int{}
|
||||
granted := true
|
||||
lookup := permissionLookupFunc(
|
||||
func(_ context.Context, kb string, tenant uint64, role types.TenantRole) (types.OrgMemberRole, bool, error) {
|
||||
require.Equal(t, uint64(1), tenant)
|
||||
require.Equal(t, types.TenantRoleContributor, role)
|
||||
calls[kb]++
|
||||
return types.OrgRoleEditor, granted, nil
|
||||
},
|
||||
)
|
||||
permissions := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleContributor)
|
||||
for _, required := range []types.OrgMemberRole{types.OrgRoleViewer, types.OrgRoleEditor, types.OrgRoleAdmin} {
|
||||
allowed, err := permissions.Check("kb", required)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, required != types.OrgRoleAdmin, allowed)
|
||||
}
|
||||
require.Equal(t, 1, calls["kb"], "different minimum roles share the same membership lookup")
|
||||
_, err := permissions.Check("other", types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 1, calls["other"])
|
||||
granted = false
|
||||
fresh := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleContributor)
|
||||
allowed, err := fresh.Check("kb", types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed, "a new operation must observe share revocation")
|
||||
require.Equal(t, 2, calls["kb"])
|
||||
}
|
||||
|
||||
func TestKBSharePermissionsCachesFailuresWithoutGranting(t *testing.T) {
|
||||
failure := errors.New("database unavailable")
|
||||
calls := 0
|
||||
lookup := permissionLookupFunc(
|
||||
func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
|
||||
calls++
|
||||
return types.OrgRoleAdmin, true, failure
|
||||
},
|
||||
)
|
||||
permissions := NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleAdmin)
|
||||
for i := 0; i < 3; i++ {
|
||||
allowed, err := permissions.Check("kb", types.OrgRoleViewer)
|
||||
require.False(t, allowed)
|
||||
require.ErrorIs(t, err, failure, "callers must retain the infrastructure error")
|
||||
}
|
||||
require.Equal(t, 1, calls)
|
||||
}
|
||||
|
||||
func TestKBSharePermissionsMissingLookupAndInvalidRolesDeny(t *testing.T) {
|
||||
permissions := NewKBSharePermissions(context.Background(), nil, 1, types.TenantRoleAdmin)
|
||||
allowed, err := permissions.Check("kb", types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed)
|
||||
lookup := permissionLookupFunc(
|
||||
func(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
|
||||
return types.OrgRoleEditor, true, nil
|
||||
},
|
||||
)
|
||||
permissions = NewKBSharePermissions(context.Background(), lookup, 1, types.TenantRoleAdmin)
|
||||
allowed, err = permissions.Check("kb", "unknown")
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed)
|
||||
}
|
||||
|
|
@ -0,0 +1,227 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// KBTransferOperation selects the source and destination permission contract.
|
||||
type KBTransferOperation string
|
||||
|
||||
// Supported knowledge-base transfer operations.
|
||||
const (
|
||||
KBTransferClone KBTransferOperation = "clone"
|
||||
KBTransferMove KBTransferOperation = "move"
|
||||
)
|
||||
|
||||
type kbTransferGrant struct {
|
||||
caller types.Caller
|
||||
tenant uint64
|
||||
source, target, taskID, creatorID string
|
||||
operation KBTransferOperation
|
||||
create bool
|
||||
}
|
||||
|
||||
// WithKBTransfer admits a pair of resources after the entry point has resolved
|
||||
// the source and destination grants and applied its creator/role policy. A new
|
||||
// clone destination must be a server-reserved ID; it is not an existing KB.
|
||||
func WithKBTransfer(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
operation KBTransferOperation,
|
||||
taskID string,
|
||||
create bool,
|
||||
) (context.Context, error) {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
if taskID == "" {
|
||||
return ctx, ErrForbidden
|
||||
}
|
||||
if err := validateTransferPair(source, target, caller.TenantID, operation, create); err != nil {
|
||||
return ctx, err
|
||||
}
|
||||
if err := transferAPIScope(ctx, source.ID, target.ID, operation, create); err != nil {
|
||||
return ctx, err
|
||||
}
|
||||
required := types.OrgRoleViewer
|
||||
if operation == KBTransferMove {
|
||||
required = types.OrgRoleEditor
|
||||
}
|
||||
if !HasKBGrant(ctx, source.ID, source.TenantID, required) ||
|
||||
(!create && !HasKBGrant(ctx, target.ID, target.TenantID, types.OrgRoleEditor)) {
|
||||
return ctx, ErrForbidden
|
||||
}
|
||||
return withTransferGrant(ctx, source, target, operation, taskID, create, false), nil
|
||||
}
|
||||
|
||||
// WithKBTransferTask continues an admitted task. Both KBs must first be loaded
|
||||
// with their persisted owners (or a reserved new clone destination). This scope
|
||||
// cannot authorize a different pair or upgrade the worker into a tenant user.
|
||||
func WithKBTransferTask(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
expectedTenant uint64,
|
||||
operation KBTransferOperation,
|
||||
taskID string,
|
||||
create bool,
|
||||
) (context.Context, error) {
|
||||
if taskID == "" {
|
||||
return ctx, ErrForbidden
|
||||
}
|
||||
if err := validateTransferPair(source, target, expectedTenant, operation, create); err != nil {
|
||||
return ctx, err
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, expectedTenant)
|
||||
return withTransferGrant(ctx, source, target, operation, taskID, create, true), nil
|
||||
}
|
||||
|
||||
func withTransferGrant(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
operation KBTransferOperation,
|
||||
taskID string,
|
||||
create, task bool,
|
||||
) context.Context {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
grant := kbTransferGrant{
|
||||
caller: caller,
|
||||
tenant: source.TenantID,
|
||||
source: source.ID,
|
||||
target: target.ID,
|
||||
taskID: taskID,
|
||||
creatorID: target.CreatorID,
|
||||
operation: operation,
|
||||
create: create,
|
||||
}
|
||||
required := types.OrgRoleViewer
|
||||
if operation == KBTransferMove {
|
||||
required = types.OrgRoleEditor
|
||||
}
|
||||
ctx = context.WithValue(ctx, types.KBGrantsContextKey, []kbGrant{
|
||||
{caller: caller, kbID: source.ID, tenantID: source.TenantID, permission: required, task: task},
|
||||
{caller: caller, kbID: target.ID, tenantID: target.TenantID, permission: types.OrgRoleEditor, task: task},
|
||||
})
|
||||
return context.WithValue(ctx, types.KBTransferContextKey, grant)
|
||||
}
|
||||
|
||||
// RequireKBTransfer consumes the exact caller, operation and resource-pair grant.
|
||||
func RequireKBTransfer(ctx context.Context, source, target *types.KnowledgeBase, operation KBTransferOperation) error {
|
||||
grant, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
|
||||
if !ok || grant.caller != types.CallerFromContext(ctx) || grant.operation != operation {
|
||||
return ErrForbidden
|
||||
}
|
||||
if err := validateTransferPair(source, target, grant.tenant, operation, grant.create); err != nil {
|
||||
return err
|
||||
}
|
||||
if grant.source != source.ID || grant.target != target.ID {
|
||||
return ErrForbidden
|
||||
}
|
||||
return transferAPIScope(ctx, source.ID, target.ID, operation, grant.create)
|
||||
}
|
||||
|
||||
// CloneDestination exposes only the destination already admitted by the
|
||||
// request/worker; CopyKnowledgeBase cannot turn an empty ID into ambient write.
|
||||
func CloneDestination(ctx context.Context, sourceID string) (id string, create bool, creatorID string, err error) {
|
||||
g, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
|
||||
if !ok || g.caller != types.CallerFromContext(ctx) || g.operation != KBTransferClone || g.source != sourceID {
|
||||
return "", false, "", ErrForbidden
|
||||
}
|
||||
return g.target, g.create, g.creatorID, nil
|
||||
}
|
||||
|
||||
// TransferTaskID returns the admitted operation identity for retry checkpoints.
|
||||
func TransferTaskID(ctx context.Context) string {
|
||||
g, ok := ctx.Value(types.KBTransferContextKey).(kbTransferGrant)
|
||||
if !ok || g.caller != types.CallerFromContext(ctx) {
|
||||
return ""
|
||||
}
|
||||
return g.taskID
|
||||
}
|
||||
|
||||
func validateTransferPair(
|
||||
source, target *types.KnowledgeBase,
|
||||
tenant uint64,
|
||||
operation KBTransferOperation,
|
||||
create bool,
|
||||
) error {
|
||||
if source == nil || target == nil || tenant == 0 || source.ID == "" || target.ID == "" ||
|
||||
source.TenantID != tenant ||
|
||||
target.TenantID != tenant {
|
||||
return ErrForbidden
|
||||
}
|
||||
if source.ID == target.ID {
|
||||
return fmt.Errorf("source and target knowledge bases must differ")
|
||||
}
|
||||
if operation != KBTransferClone && operation != KBTransferMove || create && operation != KBTransferClone {
|
||||
return ErrForbidden
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func transferAPIScope(
|
||||
ctx context.Context,
|
||||
sourceID, targetID string,
|
||||
operation KBTransferOperation,
|
||||
create bool,
|
||||
) error {
|
||||
ids := []string{sourceID}
|
||||
if !create {
|
||||
ids = append(ids, targetID)
|
||||
}
|
||||
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, ids...); err != nil {
|
||||
return err
|
||||
}
|
||||
capability := types.APIKeyCapabilityManageKnowledgeBases
|
||||
if operation == KBTransferMove {
|
||||
capability = types.APIKeyCapabilityIngest
|
||||
}
|
||||
if scope, ok := types.TenantAPIKeyScopeFromContext(ctx); ok && !scope.FullAccess &&
|
||||
!scope.HasCapability(capability) {
|
||||
return ErrForbidden
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ValidateKBTransferCompatibility is shared by synchronous admission and task
|
||||
// execution, so invalid modes/configurations fail before resource mutations.
|
||||
func ValidateKBTransferCompatibility(
|
||||
source, target *types.KnowledgeBase,
|
||||
operation KBTransferOperation,
|
||||
mode string,
|
||||
tenant *types.Tenant,
|
||||
) error {
|
||||
if source == nil || target == nil {
|
||||
return ErrNotFound
|
||||
}
|
||||
if source.ID == target.ID {
|
||||
return fmt.Errorf("source and target knowledge bases must differ")
|
||||
}
|
||||
if source.Type != target.Type {
|
||||
return fmt.Errorf("source and target knowledge bases must have the same type")
|
||||
}
|
||||
if source.EmbeddingModelID != target.EmbeddingModelID {
|
||||
return fmt.Errorf("source and target knowledge bases use different embedding models")
|
||||
}
|
||||
if operation == KBTransferMove && mode != "reuse_vectors" && mode != "reparse" {
|
||||
return fmt.Errorf("unknown move mode: %s", mode)
|
||||
}
|
||||
if (operation == KBTransferClone || mode == "reuse_vectors") && !source.SharesStoreWith(target) {
|
||||
return fmt.Errorf("source and target knowledge bases use different vector stores; use reparse mode for moves")
|
||||
}
|
||||
// A move retains file paths too, so resolving them through another storage
|
||||
// instance would make the document unreadable, even when vectors are reparsed.
|
||||
defaultID, defaultProvider := "", ""
|
||||
if tenant != nil {
|
||||
if tenant.DefaultStorageBackendID != nil {
|
||||
defaultID = *tenant.DefaultStorageBackendID
|
||||
}
|
||||
if tenant.StorageEngineConfig != nil {
|
||||
defaultProvider = tenant.StorageEngineConfig.DefaultProvider
|
||||
}
|
||||
}
|
||||
if !source.SharesStorageBackendWith(target, defaultID, defaultProvider) {
|
||||
return fmt.Errorf("source and target knowledge bases use different storage instances")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -0,0 +1,96 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestTransferAdmissionRequiresIndependentGrants(t *testing.T) {
|
||||
source, target := &types.KnowledgeBase{ID: "source", TenantID: 1}, &types.KnowledgeBase{ID: "target", TenantID: 1}
|
||||
ctx := types.WithCaller(
|
||||
context.Background(),
|
||||
types.Caller{TenantID: 1, UserID: "user", Role: types.TenantRoleAdmin},
|
||||
)
|
||||
_, err := WithKBTransfer(ctx, source, target, KBTransferMove, "task", false)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
request := KBRequest{Caller: types.CallerFromContext(ctx)}
|
||||
sourceRead, err := ResolveKB(ctx, request, source, types.OrgRoleViewer, nil, nil)
|
||||
require.NoError(t, err)
|
||||
targetRead, err := ResolveKB(ctx, request, target, types.OrgRoleViewer, nil, nil)
|
||||
require.NoError(t, err)
|
||||
reads := targetRead.WithGrant(sourceRead.WithGrant(ctx))
|
||||
_, err = WithKBTransfer(reads, source, target, KBTransferClone, "task", false)
|
||||
require.ErrorIs(t, err, ErrForbidden, "owner read projection must not authorize target replacement")
|
||||
targetWrite, err := ResolveKB(ctx, request, target, types.OrgRoleEditor, nil, nil)
|
||||
require.NoError(t, err)
|
||||
cloneCtx, err := WithKBTransfer(targetWrite.WithGrant(reads), source, target, KBTransferClone, "task", false)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, RequireKBTransfer(logger.CloneContext(cloneCtx), source, target, KBTransferClone))
|
||||
require.False(t, HasKBGrant(cloneCtx, source.ID, 1, types.OrgRoleEditor))
|
||||
_, err = WithKBTransfer(targetWrite.WithGrant(reads), source, target, KBTransferMove, "task", false)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
}
|
||||
|
||||
func TestTransferTaskScopeIsExactAndDoesNotCreateCaller(t *testing.T) {
|
||||
source, target := &types.KnowledgeBase{ID: "source", TenantID: 7}, &types.KnowledgeBase{ID: "target", TenantID: 7}
|
||||
ctx, err := WithKBTransferTask(context.Background(), source, target, 7, KBTransferMove, "task", false)
|
||||
require.NoError(t, err)
|
||||
ctx = logger.CloneContext(ctx)
|
||||
require.Zero(t, types.CallerFromContext(ctx).TenantID)
|
||||
require.NoError(t, RequireKBTransfer(ctx, source, target, KBTransferMove))
|
||||
require.ErrorIs(t, RequireKBTransfer(ctx, source, target, KBTransferClone), ErrForbidden)
|
||||
require.ErrorIs(
|
||||
t,
|
||||
RequireKBTransfer(ctx, source, &types.KnowledgeBase{ID: "third", TenantID: 7}, KBTransferMove),
|
||||
ErrForbidden,
|
||||
)
|
||||
require.False(t, HasKBGrant(ctx, "third", 7, types.OrgRoleViewer))
|
||||
_, err = WithKBTransferTask(
|
||||
ctx,
|
||||
source,
|
||||
&types.KnowledgeBase{ID: "target", TenantID: 8},
|
||||
7,
|
||||
KBTransferMove,
|
||||
"task",
|
||||
false,
|
||||
)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
changed := types.WithCaller(ctx, types.Caller{TenantID: 7, UserID: "different"})
|
||||
require.ErrorIs(t, RequireKBTransfer(changed, source, target, KBTransferMove), ErrForbidden)
|
||||
}
|
||||
|
||||
func TestTransferAPICapabilitiesRemainSeparate(t *testing.T) {
|
||||
source, target := &types.KnowledgeBase{ID: "source", TenantID: 1}, &types.KnowledgeBase{ID: "target", TenantID: 1}
|
||||
for _, tc := range []struct {
|
||||
operation KBTransferOperation
|
||||
capability string
|
||||
create bool
|
||||
ids []string
|
||||
allowed bool
|
||||
}{
|
||||
{KBTransferClone, "manage_kbs", false, []string{"source", "target"}, true},
|
||||
{KBTransferClone, "ingest", false, []string{"source", "target"}, false},
|
||||
{KBTransferMove, "ingest", false, []string{"source", "target"}, true},
|
||||
{KBTransferMove, "manage_kbs", false, []string{"source", "target"}, false},
|
||||
{KBTransferMove, "ingest", false, []string{"source"}, false},
|
||||
{KBTransferClone, "manage_kbs", true, []string{"source"}, true},
|
||||
} {
|
||||
t.Run(
|
||||
fmt.Sprintf("%s/%s/create=%v/ids=%d", tc.operation, tc.capability, tc.create, len(tc.ids)),
|
||||
func(t *testing.T) {
|
||||
ctx, err := WithKBTransferTask(context.Background(), source, target, 1, tc.operation, "task", tc.create)
|
||||
require.NoError(t, err)
|
||||
ctx = types.WithTenantAPIKeyScope(
|
||||
ctx,
|
||||
types.TenantAPIKeyScope{Capabilities: types.StringArray{tc.capability}, KnowledgeBaseIDs: tc.ids},
|
||||
)
|
||||
require.Equal(t, tc.allowed, RequireKBTransfer(ctx, source, target, tc.operation) == nil)
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,45 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// RequireKBWrite consumes an explicit operation grant. Tenant ownership and
|
||||
// role checks belong to the entry point that resolved the grant; an execution
|
||||
// tenant alone, or a shared-agent/read grant, never authorizes a mutation.
|
||||
func RequireKBWrite(ctx context.Context, kb *types.KnowledgeBase) error {
|
||||
if kb == nil || kb.ID == "" || kb.TenantID == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
if scope, ok := types.TenantAPIKeyScopeFromContext(ctx); ok && !scope.FullAccess &&
|
||||
!scope.HasCapability(types.APIKeyCapabilityIngest) {
|
||||
return ErrForbidden
|
||||
}
|
||||
if !HasKBGrant(ctx, kb.ID, kb.TenantID, types.OrgRoleEditor) {
|
||||
return ErrForbidden
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// WithKBTaskWrite is for trusted workers continuing an admitted operation.
|
||||
// The worker supplies a server-loaded KB and verifies its child-resource
|
||||
// bindings before any side effects. It grants only this KB, preserves the
|
||||
// original caller (including an absent caller), and never creates an Admin.
|
||||
// This is a task execution grant, not a way to authorize an incoming request.
|
||||
func WithKBTaskWrite(ctx context.Context, kb *types.KnowledgeBase, expectedTenant uint64) (context.Context, error) {
|
||||
if kb == nil || kb.ID == "" || expectedTenant == 0 || kb.TenantID != expectedTenant {
|
||||
return ctx, ErrForbidden
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, expectedTenant)
|
||||
grant := kbGrant{
|
||||
caller: types.CallerFromContext(ctx),
|
||||
kbID: kb.ID,
|
||||
tenantID: expectedTenant,
|
||||
permission: types.OrgRoleEditor,
|
||||
task: true,
|
||||
}
|
||||
// A worker gets exactly one scope, without inheriting another task's KBs.
|
||||
return context.WithValue(ctx, types.KBGrantsContextKey, []kbGrant{grant}), nil
|
||||
}
|
||||
|
|
@ -0,0 +1,85 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestKBWriteConsumesOperationGrant(t *testing.T) {
|
||||
ctx := callerContext()
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: types.CallerFromContext(ctx).TenantID}
|
||||
read, err := ResolveKB(ctx, KBRequest{Caller: types.CallerFromContext(ctx)}, kb, types.OrgRoleViewer, nil, nil)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.OrgRoleAdmin, read.Permission, "owner permission projection remains unchanged")
|
||||
require.ErrorIs(t, RequireKBWrite(read.Context(ctx), kb), ErrForbidden, "an owner read does not grant mutation")
|
||||
require.ErrorIs(t, RequireKBWrite(types.WithExecutionTenant(ctx, kb.TenantID), kb), ErrForbidden)
|
||||
write, err := ResolveKB(ctx, KBRequest{Caller: types.CallerFromContext(ctx)}, kb, types.OrgRoleEditor, nil, nil)
|
||||
require.NoError(t, err)
|
||||
ctx = logger.CloneContext(write.Context(ctx))
|
||||
require.NoError(t, RequireKBWrite(ctx, kb))
|
||||
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: "other", TenantID: kb.TenantID}), ErrForbidden)
|
||||
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: kb.ID, TenantID: kb.TenantID + 1}), ErrForbidden)
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
scope types.TenantAPIKeyScope
|
||||
allowed bool
|
||||
}{
|
||||
{"retrieve only", types.TenantAPIKeyScope{Capabilities: types.StringArray{"retrieve"}}, false},
|
||||
{
|
||||
"ingest",
|
||||
types.TenantAPIKeyScope{
|
||||
Capabilities: types.StringArray{"ingest"},
|
||||
KnowledgeBaseIDs: types.StringArray{"kb"},
|
||||
},
|
||||
true,
|
||||
},
|
||||
|
||||
{
|
||||
"wrong KB",
|
||||
types.TenantAPIKeyScope{
|
||||
Capabilities: types.StringArray{"ingest"},
|
||||
KnowledgeBaseIDs: types.StringArray{"other"},
|
||||
},
|
||||
false,
|
||||
},
|
||||
|
||||
{"full access", types.TenantAPIKeyScope{FullAccess: true}, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
err := RequireKBWrite(types.WithTenantAPIKeyScope(ctx, tt.scope), kb)
|
||||
require.Equal(t, tt.allowed, err == nil)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestKBTaskWriteIsExplicitAndBounded(t *testing.T) {
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: 7}
|
||||
base := context.Background()
|
||||
ctx, err := WithKBTaskWrite(base, kb, 7)
|
||||
require.NoError(t, err)
|
||||
ctx = logger.CloneContext(ctx)
|
||||
require.Zero(t, types.CallerFromContext(ctx).TenantID, "a worker must not impersonate the tenant owner")
|
||||
require.NoError(t, RequireKBWrite(ctx, kb))
|
||||
allowed, err := NewKBPermissions(ctx, nil).Check("kb", 7, types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.True(t, allowed, "a scoped write task can read its KB")
|
||||
allowed, err = NewKBPermissions(ctx, nil).Check("other", 7, types.OrgRoleViewer)
|
||||
require.NoError(t, err)
|
||||
require.False(t, allowed)
|
||||
require.ErrorIs(t, RequireKBWrite(ctx, &types.KnowledgeBase{ID: "other", TenantID: 7}), ErrForbidden)
|
||||
_, err = WithKBTaskWrite(base, kb, 8)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
_, err = WithKBTaskWrite(base, kb, 0)
|
||||
require.ErrorIs(t, err, ErrForbidden)
|
||||
require.False(t, HasKBGrant(base, "kb", 7, types.OrgRoleEditor), "the parent context stays unmodified")
|
||||
request := callerContext()
|
||||
task, err := WithKBTaskWrite(request, kb, 7)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.CallerFromContext(request), types.CallerFromContext(task))
|
||||
changed := types.WithCaller(task, types.Caller{TenantID: 99, UserID: "another"})
|
||||
require.ErrorIs(t, RequireKBWrite(changed, kb), ErrForbidden)
|
||||
}
|
||||
|
|
@ -0,0 +1,38 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// RejectMovingKnowledge is shared by admission and persisted service checks.
|
||||
// Admission prevents misleading task acceptance; workers must recheck after
|
||||
// enqueue because a move can claim the document in the intervening window.
|
||||
func RejectMovingKnowledge(knowledge *types.Knowledge) error {
|
||||
if knowledge == nil {
|
||||
return apperrors.NewNotFoundError("knowledge not found")
|
||||
}
|
||||
fields := map[string]json.RawMessage{}
|
||||
if len(knowledge.Metadata) != 0 {
|
||||
if err := json.Unmarshal(knowledge.Metadata, &fields); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
raw := fields[types.KnowledgeTransferMetadataKey]
|
||||
if len(raw) == 0 {
|
||||
return nil
|
||||
}
|
||||
var state struct {
|
||||
Operation KBTransferOperation `json:"operation"`
|
||||
Phase string `json:"phase"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &state); err != nil {
|
||||
return err
|
||||
}
|
||||
if state.Operation == KBTransferMove && state.Phase == "moving" {
|
||||
return apperrors.NewConflictError("knowledge has an unfinished move; retry the move first")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
// Package access resolves resource permissions independently of HTTP routing.
|
||||
// Tenant roles, API-key capabilities, and resource ownership guards remain
|
||||
// separate checks; a KB grant does not replace them.
|
||||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// Resource access errors retain distinct adapter response mappings.
|
||||
var (
|
||||
ErrUnauthorized = errors.New("kb_access: unauthorized")
|
||||
ErrNotFound = errors.New("kb_access: not found")
|
||||
ErrForbidden = errors.New("kb_access: forbidden")
|
||||
ErrInvalidAgentSource = errors.New("kb_access: invalid agent source")
|
||||
)
|
||||
|
||||
// KBRequest keeps the authenticated caller separate from the resource tenant.
|
||||
// AgentSourceTenantID is parsed only when the explicit-agent fallback is used.
|
||||
type KBRequest struct {
|
||||
Caller types.Caller
|
||||
AgentID string
|
||||
AgentSourceTenantID string
|
||||
}
|
||||
|
||||
// KBAccess binds a resolved resource permission to its original caller.
|
||||
type KBAccess struct {
|
||||
KnowledgeBase *types.KnowledgeBase
|
||||
Caller types.Caller
|
||||
EffectiveTenantID uint64
|
||||
Permission types.OrgMemberRole
|
||||
// A read resolution must not mint a write grant even for the KB owner.
|
||||
operationPermission types.OrgMemberRole
|
||||
}
|
||||
|
||||
// Context scopes resource operations without changing the authenticated caller.
|
||||
func (a *KBAccess) Context(ctx context.Context) context.Context {
|
||||
if a == nil || a.KnowledgeBase == nil {
|
||||
return ctx
|
||||
}
|
||||
scoped := a.WithGrant(ctx)
|
||||
if types.CallerFromContext(scoped) != a.Caller.Normalize() {
|
||||
return ctx
|
||||
}
|
||||
return types.WithExecutionTenant(scoped, a.EffectiveTenantID)
|
||||
}
|
||||
|
||||
// WithGrant carries this authorization without changing execution scope.
|
||||
func (a *KBAccess) WithGrant(ctx context.Context) context.Context {
|
||||
if a == nil || a.KnowledgeBase == nil {
|
||||
return ctx
|
||||
}
|
||||
caller := a.Caller.Normalize()
|
||||
if existing, ok := ctx.Value(types.CallerContextKey).(types.Caller); ok && existing != caller {
|
||||
return ctx // A grant cannot replace an already authenticated caller.
|
||||
}
|
||||
grants, _ := ctx.Value(types.KBGrantsContextKey).([]kbGrant)
|
||||
permission := a.Permission
|
||||
if a.operationPermission.IsValid() && permission.HasPermission(a.operationPermission) {
|
||||
permission = a.operationPermission
|
||||
}
|
||||
grants = append(
|
||||
append([]kbGrant(nil), grants...),
|
||||
kbGrant{caller: caller, kbID: a.KnowledgeBase.ID, tenantID: a.EffectiveTenantID, permission: permission},
|
||||
)
|
||||
return context.WithValue(types.WithCaller(ctx, caller), types.KBGrantsContextKey, grants)
|
||||
}
|
||||
|
||||
// KBShareLookup resolves organization sharing with the caller role cap.
|
||||
type KBShareLookup interface {
|
||||
CheckTenantKBPermission(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error)
|
||||
}
|
||||
|
||||
// SharedAgentLookup verifies access to a particular shared agent.
|
||||
type SharedAgentLookup interface {
|
||||
GetSharedAgentForTenant(context.Context, uint64, types.TenantRole, string, ...uint64) (*types.CustomAgent, error)
|
||||
}
|
||||
|
||||
// AgentShareLookup also supports read access through any reachable agent.
|
||||
type AgentShareLookup interface {
|
||||
SharedAgentLookup
|
||||
TenantCanAccessKBViaSomeSharedAgent(context.Context, uint64, types.TenantRole, *types.KnowledgeBase) (bool, error)
|
||||
}
|
||||
|
||||
// ResolveKB authorizes a server-loaded KB (or a document's persisted KB/tenant
|
||||
// reference). Its tenant is authoritative; resolving the owner from a share
|
||||
// record again would add a lookup and a second source of truth. It never treats
|
||||
// an effective resource tenant as the caller.
|
||||
// Share lookup errors do not grant access; an independent agent grant may still
|
||||
// allow reading. An explicit agent selector never falls back to another agent.
|
||||
func ResolveKB(ctx context.Context, request KBRequest, kb *types.KnowledgeBase, required types.OrgMemberRole,
|
||||
shares KBShareLookup, agents AgentShareLookup,
|
||||
) (*KBAccess, error) {
|
||||
if request.Caller.TenantID == 0 {
|
||||
return nil, ErrUnauthorized
|
||||
}
|
||||
if kb == nil || kb.ID == "" {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kb.ID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
request.Caller = request.Caller.Normalize()
|
||||
if caller, ok := ctx.Value(types.CallerContextKey).(types.Caller); ok && caller != request.Caller {
|
||||
return nil, ErrForbidden
|
||||
}
|
||||
grant := func(permission types.OrgMemberRole) (*KBAccess, error) {
|
||||
return &KBAccess{
|
||||
KnowledgeBase: kb, Caller: request.Caller,
|
||||
EffectiveTenantID: kb.TenantID, Permission: permission, operationPermission: required,
|
||||
}, nil
|
||||
}
|
||||
if kb.TenantID == request.Caller.TenantID {
|
||||
return grant(types.OrgRoleAdmin)
|
||||
}
|
||||
if shares != nil {
|
||||
permissions := NewKBSharePermissions(ctx, shares, request.Caller.TenantID, request.Caller.Role)
|
||||
permission, shared, err := permissions.Permission(kb.ID)
|
||||
if err == nil && shared && permission.HasPermission(required) {
|
||||
return grant(permission)
|
||||
}
|
||||
}
|
||||
if required != types.OrgRoleViewer || agents == nil {
|
||||
return nil, ErrForbidden
|
||||
}
|
||||
if request.AgentID != "" {
|
||||
source, err := types.ParseAgentSourceTenantID(request.AgentSourceTenantID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: %s", ErrInvalidAgentSource, err)
|
||||
}
|
||||
agent, err := agents.GetSharedAgentForTenant(
|
||||
ctx,
|
||||
request.Caller.TenantID,
|
||||
request.Caller.Role,
|
||||
request.AgentID,
|
||||
source,
|
||||
)
|
||||
if err == nil && types.SharedAgentIncludesKB(agent, kb) {
|
||||
return grant(types.OrgRoleViewer)
|
||||
}
|
||||
} else {
|
||||
allowed,
|
||||
err := agents.TenantCanAccessKBViaSomeSharedAgent(ctx,
|
||||
request.Caller.TenantID,
|
||||
request.Caller.Role,
|
||||
kb)
|
||||
if err == nil && allowed {
|
||||
return grant(types.OrgRoleViewer)
|
||||
}
|
||||
}
|
||||
return nil, ErrForbidden
|
||||
}
|
||||
|
|
@ -0,0 +1,282 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type shareLookup struct {
|
||||
permission types.OrgMemberRole
|
||||
err error
|
||||
caller uint64
|
||||
role types.TenantRole
|
||||
}
|
||||
|
||||
func (s *shareLookup) CheckTenantKBPermission(
|
||||
_ context.Context,
|
||||
_ string,
|
||||
caller uint64,
|
||||
role types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
s.caller, s.role = caller, role
|
||||
return s.permission, s.permission != "", s.err
|
||||
}
|
||||
|
||||
type agentLookup struct {
|
||||
agent *types.CustomAgent
|
||||
any bool
|
||||
err error
|
||||
anyCalls int
|
||||
source uint64
|
||||
}
|
||||
|
||||
func (s *agentLookup) GetSharedAgentForTenant(
|
||||
_ context.Context,
|
||||
_ uint64,
|
||||
_ types.TenantRole,
|
||||
_ string,
|
||||
source ...uint64,
|
||||
) (*types.CustomAgent, error) {
|
||||
s.source = source[0]
|
||||
return s.agent, s.err
|
||||
}
|
||||
|
||||
func (s *agentLookup) TenantCanAccessKBViaSomeSharedAgent(
|
||||
context.Context,
|
||||
uint64,
|
||||
types.TenantRole,
|
||||
*types.KnowledgeBase,
|
||||
) (bool, error) {
|
||||
s.anyCalls++
|
||||
return s.any, s.err
|
||||
}
|
||||
|
||||
func TestResolveKBPermissionMatrix(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
own bool
|
||||
permission types.OrgMemberRole
|
||||
required types.OrgMemberRole
|
||||
shareError bool
|
||||
agentID, source, mode string
|
||||
selected []string
|
||||
wrongTenant, missingAgent, agentError, anyAgent bool
|
||||
want types.OrgMemberRole
|
||||
wantErr error
|
||||
}{
|
||||
{name: "own KB", own: true, required: types.OrgRoleEditor, want: types.OrgRoleAdmin},
|
||||
{
|
||||
name: "organization reader",
|
||||
permission: types.OrgRoleViewer,
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleViewer,
|
||||
},
|
||||
|
||||
{
|
||||
name: "organization editor",
|
||||
permission: types.OrgRoleEditor,
|
||||
required: types.OrgRoleEditor,
|
||||
want: types.OrgRoleEditor,
|
||||
},
|
||||
|
||||
{
|
||||
name: "reader cannot write",
|
||||
permission: types.OrgRoleViewer,
|
||||
required: types.OrgRoleEditor,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "share lookup fails closed",
|
||||
permission: types.OrgRoleEditor,
|
||||
shareError: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "independent agent grant after share error",
|
||||
shareError: true,
|
||||
anyAgent: true,
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleViewer,
|
||||
},
|
||||
|
||||
{name: "any shared agent read", anyAgent: true, required: types.OrgRoleViewer, want: types.OrgRoleViewer},
|
||||
{name: "any agent cannot write", anyAgent: true, required: types.OrgRoleEditor, wantErr: ErrForbidden},
|
||||
{
|
||||
name: "agent lookup error",
|
||||
anyAgent: true,
|
||||
agentError: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "explicit all",
|
||||
agentID: "agent",
|
||||
source: "2",
|
||||
mode: "all",
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleViewer,
|
||||
},
|
||||
|
||||
{
|
||||
name: "explicit selected",
|
||||
agentID: "agent",
|
||||
mode: "selected",
|
||||
selected: []string{"kb"},
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleViewer,
|
||||
},
|
||||
|
||||
{
|
||||
name: "selection miss does not fall back",
|
||||
agentID: "agent",
|
||||
mode: "selected",
|
||||
selected: []string{"other"},
|
||||
anyAgent: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "none does not fall back",
|
||||
agentID: "agent",
|
||||
mode: "none",
|
||||
anyAgent: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "unknown mode denied",
|
||||
agentID: "agent",
|
||||
mode: "invalid",
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "tenant mismatch denied",
|
||||
agentID: "agent",
|
||||
mode: "all",
|
||||
wrongTenant: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "missing explicit agent does not fall back",
|
||||
agentID: "agent",
|
||||
missingAgent: true,
|
||||
anyAgent: true,
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrForbidden,
|
||||
},
|
||||
|
||||
{
|
||||
name: "invalid source rejected",
|
||||
agentID: "agent",
|
||||
source: "bad",
|
||||
mode: "all",
|
||||
required: types.OrgRoleViewer,
|
||||
wantErr: ErrInvalidAgentSource,
|
||||
},
|
||||
|
||||
{
|
||||
name: "own grant precedes agent parsing",
|
||||
own: true,
|
||||
agentID: "agent",
|
||||
source: "bad",
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleAdmin,
|
||||
},
|
||||
|
||||
{
|
||||
name: "share grant precedes agent parsing",
|
||||
permission: types.OrgRoleViewer,
|
||||
agentID: "agent",
|
||||
source: "bad",
|
||||
required: types.OrgRoleViewer,
|
||||
want: types.OrgRoleViewer,
|
||||
},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: 2}
|
||||
if tt.own {
|
||||
kb.TenantID = 1
|
||||
}
|
||||
shares := &shareLookup{permission: tt.permission}
|
||||
if tt.shareError {
|
||||
shares.err = errors.New("unavailable")
|
||||
}
|
||||
agents := &agentLookup{
|
||||
any: tt.anyAgent,
|
||||
agent: &types.CustomAgent{
|
||||
TenantID: 2,
|
||||
Config: types.CustomAgentConfig{KBSelectionMode: tt.mode, KnowledgeBases: tt.selected},
|
||||
},
|
||||
}
|
||||
if tt.wrongTenant {
|
||||
agents.agent.TenantID = 3
|
||||
}
|
||||
if tt.missingAgent {
|
||||
agents.agent = nil
|
||||
}
|
||||
if tt.agentError {
|
||||
agents.err = errors.New("unavailable")
|
||||
}
|
||||
request := KBRequest{
|
||||
Caller: types.Caller{TenantID: 1, Role: types.TenantRoleViewer},
|
||||
AgentID: tt.agentID,
|
||||
AgentSourceTenantID: tt.source,
|
||||
}
|
||||
grant, err := ResolveKB(context.Background(), request, kb, tt.required, shares, agents)
|
||||
if tt.wantErr != nil {
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
require.Nil(t, grant)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, tt.want, grant.Permission)
|
||||
require.Equal(t, uint64(1), grant.Caller.TenantID)
|
||||
require.Equal(t, kb.TenantID, grant.EffectiveTenantID)
|
||||
}
|
||||
if !tt.own {
|
||||
require.Equal(t, uint64(1), shares.caller)
|
||||
require.Equal(t, types.TenantRoleViewer, shares.role)
|
||||
}
|
||||
if tt.agentID != "" {
|
||||
require.Zero(t, agents.anyCalls)
|
||||
}
|
||||
if tt.source == "2" {
|
||||
require.Equal(t, uint64(2), agents.source)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveKBMissingIdentityResourceAndAPIKeyScope(t *testing.T) {
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: 1}
|
||||
_, err := ResolveKB(context.Background(), KBRequest{}, kb, types.OrgRoleViewer, nil, nil)
|
||||
require.ErrorIs(t, err, ErrUnauthorized)
|
||||
_, err = ResolveKB(
|
||||
context.Background(),
|
||||
KBRequest{Caller: types.Caller{TenantID: 1}},
|
||||
nil,
|
||||
types.OrgRoleViewer,
|
||||
nil,
|
||||
nil,
|
||||
)
|
||||
require.ErrorIs(t, err, ErrNotFound)
|
||||
ctx := types.WithTenantAPIKeyScope(
|
||||
context.Background(),
|
||||
types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"other"}},
|
||||
)
|
||||
_, err = ResolveKB(ctx, KBRequest{Caller: types.Caller{TenantID: 1}}, kb, types.OrgRoleViewer, nil, nil)
|
||||
require.Error(t, err, "API key scope must apply even to owned KBs")
|
||||
}
|
||||
|
|
@ -0,0 +1,249 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
||||
// MessageFileLookup is the narrow message-service surface needed by the
|
||||
// message-scoped file proxy. Keeping it small makes the authorization boundary
|
||||
// independently testable.
|
||||
type MessageFileLookup interface {
|
||||
GetMessage(ctx context.Context, sessionID, messageID string) (*types.Message, error)
|
||||
}
|
||||
|
||||
// SharedAgentFileLookup verifies that a source workspace's agent is still
|
||||
// shared to the caller. Revoking the share therefore also revokes historical
|
||||
// message-file access.
|
||||
type SharedAgentFileLookup interface {
|
||||
GetSharedAgentForTenant(
|
||||
ctx context.Context,
|
||||
tenantID uint64,
|
||||
callerTenantRole types.TenantRole,
|
||||
agentID string,
|
||||
sourceTenantID ...uint64,
|
||||
) (*types.CustomAgent, error)
|
||||
}
|
||||
|
||||
// KBSharePermissionGuard is the org-share surface the message-scoped proxy
|
||||
// needs for its shared-KB fallback.
|
||||
type KBSharePermissionGuard = KBShareLookup
|
||||
|
||||
// KBTenantLookup resolves knowledge bases without a tenant filter, so a
|
||||
// message's persisted retrieval evidence can be mapped back to the knowledge
|
||||
// bases the chunks were retrieved from.
|
||||
type KBTenantLookup interface {
|
||||
GetKnowledgeBasesByIDsOnly(ctx context.Context, ids []string) ([]*types.KnowledgeBase, error)
|
||||
}
|
||||
|
||||
// KnowledgeOwnerLookup resolves a knowledge entry to its knowledge base for
|
||||
// message references that predate the denormalized knowledge_base_id field.
|
||||
type KnowledgeOwnerLookup interface {
|
||||
GetKnowledgeByIDOnly(ctx context.Context, id string) (*types.Knowledge, error)
|
||||
}
|
||||
|
||||
// MessageKBShareAuthorizer bundles the read-only lookups behind the
|
||||
// message proxy's org-shared-KB fallback. A nil ShareGuard or KBs lookup
|
||||
// disables the fallback. A live Bindings lookup is also required and can
|
||||
// be supplied by the resource catalog. A nil Knowledges lookup only disables the
|
||||
// pre-denormalization KnowledgeID path.
|
||||
type MessageKBShareAuthorizer struct {
|
||||
ShareGuard KBSharePermissionGuard
|
||||
KBs KBTenantLookup
|
||||
Knowledges KnowledgeOwnerLookup
|
||||
Bindings interfaces.KBResourceLookup
|
||||
}
|
||||
|
||||
// resourceAccessibleViaSharedKB reports whether the message's persisted
|
||||
// retrieval evidence proves the resource came from an org-shared KB the
|
||||
// caller may read. This is the fallback for replies whose agent belongs to
|
||||
// the caller's own workspace while the retrieved resources belong to the
|
||||
// workspace that shared the knowledge base (#3022).
|
||||
//
|
||||
// Evidence required from one retrieval record: a canonical resource://
|
||||
// handle (not a prefix of a longer token) appears in the chunk text or
|
||||
// image_info, its knowledge base belongs to the resource's tenant, and
|
||||
// that KB is org-shared to the caller with at least viewer permission, and
|
||||
// an independent live resource binding confirms the file still belongs to it.
|
||||
// Smart-reasoning turns persist that evidence on AgentSteps when
|
||||
// KnowledgeReferences was never filled. Any lookup failure fails closed.
|
||||
func (a MessageKBShareAuthorizer) resourceAccessibleViaSharedKB(
|
||||
ctx context.Context,
|
||||
message *types.Message,
|
||||
resource *types.StoredResource,
|
||||
callerTenantID uint64,
|
||||
callerTenantRole types.TenantRole,
|
||||
) bool {
|
||||
if a.ShareGuard == nil || a.KBs == nil || a.Bindings == nil || message == nil || resource == nil {
|
||||
return false
|
||||
}
|
||||
handle, ok := types.ParseResourcePath(types.BuildResourcePath(resource.Handle))
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
|
||||
kbIDs := a.collectSharedKBEvidenceIDs(ctx, message, handle)
|
||||
if len(kbIDs) == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
kbs, err := a.KBs.GetKnowledgeBasesByIDsOnly(ctx, kbIDs)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
permissions := NewKBSharePermissions(ctx, a.ShareGuard, callerTenantID, callerTenantRole)
|
||||
for _, kb := range kbs {
|
||||
if kb == nil || kb.TenantID != resource.TenantID {
|
||||
continue
|
||||
}
|
||||
shared, err := permissions.Check(kb.ID, types.OrgRoleViewer)
|
||||
if err == nil && shared {
|
||||
bound, bindingErr := a.Bindings.IsReferencedByKnowledgeBase(
|
||||
ctx, resource.TenantID, kb.ID, types.BuildResourcePath(handle))
|
||||
if bindingErr == nil && bound {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (a MessageKBShareAuthorizer) collectSharedKBEvidenceIDs(
|
||||
ctx context.Context,
|
||||
message *types.Message,
|
||||
handle string,
|
||||
) []string {
|
||||
seenKB := make(map[string]bool)
|
||||
seenKnowledge := make(map[string]bool)
|
||||
var kbIDs, knowledgeIDs []string
|
||||
addKB := func(id string) {
|
||||
if id == "" || seenKB[id] {
|
||||
return
|
||||
}
|
||||
seenKB[id] = true
|
||||
kbIDs = append(kbIDs, id)
|
||||
}
|
||||
addKnowledge := func(id string) {
|
||||
if id == "" || seenKnowledge[id] {
|
||||
return
|
||||
}
|
||||
seenKnowledge[id] = true
|
||||
knowledgeIDs = append(knowledgeIDs, id)
|
||||
}
|
||||
|
||||
for _, ref := range message.KnowledgeReferences {
|
||||
if !searchResultHasResourceHandle(ref, handle) {
|
||||
continue
|
||||
}
|
||||
if ref.KnowledgeBaseID != "" {
|
||||
addKB(ref.KnowledgeBaseID)
|
||||
continue
|
||||
}
|
||||
addKnowledge(ref.KnowledgeID)
|
||||
}
|
||||
collectKBEvidenceFromValue(message.AgentSteps, handle, "", "", addKB, addKnowledge)
|
||||
|
||||
if a.Knowledges != nil {
|
||||
for _, knowledgeID := range knowledgeIDs {
|
||||
knowledge, err := a.Knowledges.GetKnowledgeByIDOnly(ctx, knowledgeID)
|
||||
if err != nil || knowledge == nil {
|
||||
continue
|
||||
}
|
||||
addKB(knowledge.KnowledgeBaseID)
|
||||
}
|
||||
}
|
||||
return kbIDs
|
||||
}
|
||||
|
||||
func searchResultHasResourceHandle(ref *types.SearchResult, handle string) bool {
|
||||
if ref == nil {
|
||||
return false
|
||||
}
|
||||
return textHasResourceHandle(ref.Content, handle) ||
|
||||
textHasResourceHandle(ref.MatchedContent, handle) ||
|
||||
textHasResourceHandle(ref.ImageInfo, handle)
|
||||
}
|
||||
|
||||
func textHasResourceHandle(text, handle string) bool {
|
||||
if text == "" || handle == "" {
|
||||
return false
|
||||
}
|
||||
want := types.BuildResourcePath(handle)
|
||||
for _, ref := range types.ScanResourceReferences(text) {
|
||||
if ref == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func collectKBEvidenceFromValue(
|
||||
v interface{},
|
||||
handle, kbID, knowledgeID string,
|
||||
addKB, addKnowledge func(string),
|
||||
) {
|
||||
switch val := v.(type) {
|
||||
case string:
|
||||
if !textHasResourceHandle(val, handle) {
|
||||
return
|
||||
}
|
||||
if kbID != "" {
|
||||
addKB(kbID)
|
||||
return
|
||||
}
|
||||
addKnowledge(knowledgeID)
|
||||
case types.AgentSteps:
|
||||
for _, step := range val {
|
||||
collectKBEvidenceFromValue(step, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
}
|
||||
case types.AgentStep:
|
||||
collectKBEvidenceFromValue(val.ToolCalls, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
case []types.ToolCall:
|
||||
for _, call := range val {
|
||||
collectKBEvidenceFromValue(call, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
}
|
||||
case types.ToolCall:
|
||||
if val.Result != nil {
|
||||
collectKBEvidenceFromValue(val.Result.Output, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
collectKBEvidenceFromValue(val.Result.Data, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
}
|
||||
case map[string]interface{}:
|
||||
nextKB := firstNonEmptyString(
|
||||
stringFromAnyMap(val, "knowledge_base_id"),
|
||||
stringFromAnyMap(val, "knowledge_base"),
|
||||
kbID,
|
||||
)
|
||||
nextKnowledge := firstNonEmptyString(stringFromAnyMap(val, "knowledge_id"), knowledgeID)
|
||||
for _, nested := range val {
|
||||
collectKBEvidenceFromValue(nested, handle, nextKB, nextKnowledge, addKB, addKnowledge)
|
||||
}
|
||||
case []interface{}:
|
||||
for _, item := range val {
|
||||
collectKBEvidenceFromValue(item, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
}
|
||||
case []map[string]interface{}:
|
||||
for _, item := range val {
|
||||
collectKBEvidenceFromValue(item, handle, kbID, knowledgeID, addKB, addKnowledge)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func stringFromAnyMap(m map[string]interface{}, key string) string {
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
s, _ := m[key].(string)
|
||||
return strings.TrimSpace(s)
|
||||
}
|
||||
|
||||
func firstNonEmptyString(values ...string) string {
|
||||
for _, value := range values {
|
||||
if strings.TrimSpace(value) != "" {
|
||||
return strings.TrimSpace(value)
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"errors"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
var (
|
||||
// ErrResourceNotFound means the ownership lookup found no resource visible
|
||||
// in the caller's tenant. HTTP adapters decide how to produce the 404.
|
||||
ErrResourceNotFound = errors.New("rbac: resource not found")
|
||||
// ErrOwnershipForbidden indicates neither ownership nor the required role.
|
||||
ErrOwnershipForbidden = errors.New("rbac: ownership or role insufficient")
|
||||
)
|
||||
|
||||
// OwnershipRequest carries server-resolved identity and rollout policy. The
|
||||
// cross-tenant bypass must already account for the cluster configuration.
|
||||
// This gate does not replace API-key capabilities or resource-scope checks.
|
||||
type OwnershipRequest struct {
|
||||
UserID string
|
||||
Role types.TenantRole
|
||||
APIKey bool
|
||||
CrossTenantSuperuser bool
|
||||
Enforce bool
|
||||
}
|
||||
|
||||
// OwnershipDecision contains the details adapters need for logs and audits.
|
||||
// The error returned with it determines whether access was allowed.
|
||||
type OwnershipDecision struct {
|
||||
CreatorID string
|
||||
EnforcementSkipped bool
|
||||
LookupFailed bool
|
||||
}
|
||||
|
||||
// CheckOwnershipOrRole performs no resource lookup when API-key authorization,
|
||||
// role, superuser status, or the RBAC rollout policy already settles the gate.
|
||||
// The lookup must enforce tenant visibility before returning a creator ID.
|
||||
func CheckOwnershipOrRole(
|
||||
request OwnershipRequest,
|
||||
required types.TenantRole,
|
||||
lookup func() (string, error),
|
||||
) (OwnershipDecision, error) {
|
||||
var decision OwnershipDecision
|
||||
if request.APIKey || request.Role.HasPermission(required) || request.CrossTenantSuperuser {
|
||||
return decision, nil
|
||||
}
|
||||
if !request.Enforce {
|
||||
decision.EnforcementSkipped = true
|
||||
return decision, nil
|
||||
}
|
||||
if lookup == nil {
|
||||
decision.LookupFailed = true
|
||||
return decision, errors.New("rbac: creator lookup unavailable")
|
||||
}
|
||||
creator, err := lookup()
|
||||
decision.CreatorID = creator
|
||||
if errors.Is(err, ErrResourceNotFound) {
|
||||
return decision, ErrResourceNotFound
|
||||
}
|
||||
if err != nil {
|
||||
decision.LookupFailed = true
|
||||
return decision, err
|
||||
}
|
||||
if creator != "" && creator == request.UserID {
|
||||
return decision, nil
|
||||
}
|
||||
return decision, ErrOwnershipForbidden
|
||||
}
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
package access
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestCheckOwnershipOrRoleSkipsLookupWhenGateAlreadyResolved(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
request OwnershipRequest
|
||||
skipped bool
|
||||
}{
|
||||
{name: "API key", request: OwnershipRequest{APIKey: true, Role: types.TenantRoleViewer, Enforce: true}},
|
||||
{name: "admin", request: OwnershipRequest{Role: types.TenantRoleAdmin, Enforce: true}},
|
||||
{name: "owner role", request: OwnershipRequest{Role: types.TenantRoleOwner, Enforce: true}},
|
||||
{
|
||||
name: "cross tenant superuser",
|
||||
request: OwnershipRequest{
|
||||
Role: types.TenantRoleViewer,
|
||||
CrossTenantSuperuser: true,
|
||||
Enforce: true,
|
||||
},
|
||||
},
|
||||
|
||||
{name: "enforcement disabled", request: OwnershipRequest{Role: types.TenantRoleViewer}, skipped: true},
|
||||
{name: "role before rollout", request: OwnershipRequest{Role: types.TenantRoleAdmin}},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
decision, err := CheckOwnershipOrRole(tt.request, types.TenantRoleAdmin, func() (string, error) {
|
||||
t.Fatal("ownership lookup must not run on a fast path")
|
||||
return "", errors.New("unavailable")
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, tt.skipped, decision.EnforcementSkipped)
|
||||
require.False(t, decision.LookupFailed)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckOwnershipOrRoleResolvesCreator(t *testing.T) {
|
||||
failure := errors.New("database unavailable")
|
||||
for _, tt := range []struct {
|
||||
name, user, creator string
|
||||
lookupErr, wantErr error
|
||||
lookupFailed bool
|
||||
}{
|
||||
{name: "creator", user: "user", creator: "user"},
|
||||
{name: "other creator", user: "user", creator: "other", wantErr: ErrOwnershipForbidden},
|
||||
{name: "tenant owned", user: "user", wantErr: ErrOwnershipForbidden},
|
||||
{name: "missing identity cannot match empty creator", wantErr: ErrOwnershipForbidden},
|
||||
{name: "missing resource", user: "user", lookupErr: ErrResourceNotFound, wantErr: ErrResourceNotFound},
|
||||
{
|
||||
name: "wrapped missing resource",
|
||||
user: "user",
|
||||
lookupErr: fmt.Errorf("load: %w",
|
||||
ErrResourceNotFound),
|
||||
wantErr: ErrResourceNotFound,
|
||||
},
|
||||
|
||||
{
|
||||
name: "lookup failure wins over creator match",
|
||||
user: "user",
|
||||
creator: "user",
|
||||
lookupErr: failure,
|
||||
wantErr: failure,
|
||||
lookupFailed: true,
|
||||
},
|
||||
|
||||
{
|
||||
name: "lookup denial remains a lookup failure",
|
||||
lookupErr: ErrOwnershipForbidden,
|
||||
wantErr: ErrOwnershipForbidden,
|
||||
lookupFailed: true,
|
||||
},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
calls := 0
|
||||
decision, err := CheckOwnershipOrRole(
|
||||
OwnershipRequest{UserID: tt.user, Role: types.TenantRoleContributor, Enforce: true},
|
||||
types.TenantRoleAdmin,
|
||||
func() (string, error) {
|
||||
calls++
|
||||
return tt.creator, tt.lookupErr
|
||||
},
|
||||
)
|
||||
require.Equal(t, 1, calls)
|
||||
require.ErrorIs(t, err, tt.wantErr)
|
||||
require.Equal(t, tt.creator, decision.CreatorID)
|
||||
require.Equal(t, tt.lookupFailed, decision.LookupFailed)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckOwnershipOrRoleMissingLookupFailsClosed(t *testing.T) {
|
||||
decision, err := CheckOwnershipOrRole(OwnershipRequest{Enforce: true}, types.TenantRoleAdmin, nil)
|
||||
require.Error(t, err)
|
||||
require.True(t, decision.LookupFailed)
|
||||
}
|
||||
|
|
@ -575,7 +575,7 @@ func (r *chunkRepository) DeleteByKnowledgeList(ctx context.Context, tenantID ui
|
|||
func (r *chunkRepository) MoveChunksByKnowledgeID(ctx context.Context, tenantID uint64, knowledgeID string, targetKBID string) error {
|
||||
return r.db.WithContext(ctx).Model(&types.Chunk{}).
|
||||
Where("tenant_id = ? AND knowledge_id = ?", tenantID, knowledgeID).
|
||||
Update("knowledge_base_id", targetKBID).Error
|
||||
Updates(map[string]any{"knowledge_base_id": targetKBID, "tag_id": ""}).Error
|
||||
}
|
||||
|
||||
// DeleteChunksByTagID deletes all chunks with the specified tag ID
|
||||
|
|
@ -953,30 +953,22 @@ func (r *chunkRepository) UpdateChunkFieldsByTagID(
|
|||
newTagID *string,
|
||||
excludeIDs []string,
|
||||
) ([]string, error) {
|
||||
// First, get the IDs of chunks that will be affected (for is_enabled sync)
|
||||
if isEnabled == nil && setFlags == 0 && clearFlags == 0 && newTagID == nil {
|
||||
return nil, nil
|
||||
}
|
||||
// Return every affected entry, including tag-only and flag-only changes.
|
||||
// Callers use these IDs for index synchronization and subsequent patches.
|
||||
var affectedIDs []string
|
||||
if isEnabled != nil {
|
||||
var chunks []*types.Chunk
|
||||
query := r.db.WithContext(ctx).
|
||||
Select("id").
|
||||
Where("tenant_id = ? AND knowledge_base_id = ? AND chunk_type = ?",
|
||||
tenantID, kbID, types.ChunkTypeFAQ)
|
||||
if tagID != "" {
|
||||
query = query.Where("tag_id = ?", tagID)
|
||||
}
|
||||
|
||||
if len(excludeIDs) > 0 {
|
||||
query = query.Where("id NOT IN ?", excludeIDs)
|
||||
}
|
||||
|
||||
// Only get chunks that need to change
|
||||
query = query.Where("is_enabled != ?", *isEnabled)
|
||||
if err := query.Find(&chunks).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, c := range chunks {
|
||||
affectedIDs = append(affectedIDs, c.ID)
|
||||
}
|
||||
selection := r.db.WithContext(ctx).Model(&types.Chunk{}).
|
||||
Where("tenant_id = ? AND knowledge_base_id = ? AND chunk_type = ?", tenantID, kbID, types.ChunkTypeFAQ)
|
||||
if tagID != "" {
|
||||
selection = selection.Where("tag_id = ?", tagID)
|
||||
}
|
||||
if len(excludeIDs) > 0 {
|
||||
selection = selection.Where("id NOT IN ?", excludeIDs)
|
||||
}
|
||||
if err := selection.Pluck("id", &affectedIDs).Error; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Build update query
|
||||
|
|
@ -1309,3 +1301,17 @@ func (r *chunkRepository) ListRecentDocumentChunksWithQuestions(
|
|||
|
||||
return chunks, nil
|
||||
}
|
||||
|
||||
func (r *chunkRepository) ListAllChunksByKnowledgeID(
|
||||
ctx context.Context,
|
||||
tenantID uint64,
|
||||
knowledgeID string,
|
||||
) ([]*types.Chunk, error) {
|
||||
var chunks []*types.Chunk
|
||||
err := r.db.WithContext(ctx).
|
||||
Where("tenant_id = ? AND knowledge_id = ?", tenantID, knowledgeID).
|
||||
Order("id ASC").
|
||||
Find(&chunks).
|
||||
Error
|
||||
return chunks, err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,60 @@
|
|||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestTagFieldUpdatesReturnAllAffectedChunks(t *testing.T) {
|
||||
for _, flagsOnly := range []bool{false, true} {
|
||||
name := "tag only"
|
||||
if flagsOnly {
|
||||
name = "flags only"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
db := setupChunkTestDB(t)
|
||||
repo := NewChunkRepository(db)
|
||||
for _, chunk := range []*types.Chunk{
|
||||
{ID: "selected", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
|
||||
{ID: "excluded", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
|
||||
{ID: "other-kb", TenantID: 7, KnowledgeBaseID: "other", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
|
||||
{ID: "other-tenant", TenantID: 8, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
|
||||
{ID: "deleted", TenantID: 7, KnowledgeBaseID: "kb", ChunkType: types.ChunkTypeFAQ, TagID: "tag"},
|
||||
} {
|
||||
require.NoError(t, db.Create(chunk).Error)
|
||||
}
|
||||
require.NoError(t, db.Delete(&types.Chunk{ID: "deleted"}).Error)
|
||||
next := "next"
|
||||
var tag *string
|
||||
var flags types.ChunkFlags
|
||||
if flagsOnly {
|
||||
flags = types.ChunkFlagRecommended
|
||||
} else {
|
||||
tag = &next
|
||||
}
|
||||
ids, err := repo.UpdateChunkFieldsByTagID(
|
||||
context.Background(),
|
||||
7,
|
||||
"kb",
|
||||
"tag",
|
||||
nil,
|
||||
flags,
|
||||
0,
|
||||
tag,
|
||||
[]string{"excluded"},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"selected"}, ids)
|
||||
selected, err := repo.GetChunkByID(context.Background(), 7, "selected")
|
||||
require.NoError(t, err)
|
||||
if flagsOnly {
|
||||
require.True(t, selected.Flags.HasFlag(types.ChunkFlagRecommended))
|
||||
} else {
|
||||
require.Equal(t, "next", selected.TagID)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -560,15 +560,20 @@ func (r *knowledgeRepository) UpdateKnowledgeColumns(
|
|||
// to normal queries and have not moved out of the transient deleting state.
|
||||
func (r *knowledgeRepository) UpdateActiveDeletingKnowledgeColumns(
|
||||
ctx context.Context,
|
||||
id string,
|
||||
tenantID uint64,
|
||||
kbID, id string,
|
||||
values map[string]interface{},
|
||||
) (bool, error) {
|
||||
if len(values) == 0 {
|
||||
if tenantID == 0 || kbID == "" || len(values) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
result := r.db.WithContext(ctx).
|
||||
Model(&types.Knowledge{}).
|
||||
Where("id = ? AND parse_status = ?", id, types.ParseStatusDeleting).
|
||||
Where("tenant_id = ? AND knowledge_base_id = ? AND id = ? AND parse_status = ?",
|
||||
tenantID,
|
||||
kbID,
|
||||
id,
|
||||
types.ParseStatusDeleting).
|
||||
Updates(values)
|
||||
if result.Error != nil {
|
||||
return false, result.Error
|
||||
|
|
|
|||
|
|
@ -353,22 +353,63 @@ func TestUpdateActiveDeletingKnowledgeColumns_GuardsStateAndSoftDelete(t *testin
|
|||
activeCompletedID := insertKnowledgeWithStatus(t, db, types.ParseStatusCompleted, false)
|
||||
deletedDeletingID := insertKnowledgeWithStatus(t, db, types.ParseStatusDeleting, true)
|
||||
|
||||
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, activeDeletingID, map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
"error_message": "delete task exhausted retries",
|
||||
})
|
||||
require.NoError(
|
||||
t,
|
||||
db.Exec(
|
||||
"UPDATE knowledges SET knowledge_base_id = ? WHERE id IN ?",
|
||||
"delete-kb",
|
||||
[]string{activeDeletingID, activeCompletedID, deletedDeletingID},
|
||||
).Error,
|
||||
)
|
||||
for _, scope := range []struct {
|
||||
tenant uint64
|
||||
kb string
|
||||
}{{2, "delete-kb"}, {1, "other-kb"}, {0, "delete-kb"}, {1, ""}} {
|
||||
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(
|
||||
ctx,
|
||||
scope.tenant,
|
||||
scope.kb,
|
||||
activeDeletingID,
|
||||
map[string]interface{}{"parse_status": types.ParseStatusFailed},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.False(t, updated)
|
||||
}
|
||||
|
||||
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(
|
||||
ctx,
|
||||
1,
|
||||
"delete-kb",
|
||||
activeDeletingID,
|
||||
map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
"error_message": "delete task exhausted retries",
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, updated)
|
||||
|
||||
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(ctx, activeCompletedID, map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
})
|
||||
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(
|
||||
ctx,
|
||||
1,
|
||||
"delete-kb",
|
||||
activeCompletedID,
|
||||
map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, updated)
|
||||
|
||||
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(ctx, deletedDeletingID, map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
})
|
||||
updated, err = repo.UpdateActiveDeletingKnowledgeColumns(
|
||||
ctx,
|
||||
1,
|
||||
"delete-kb",
|
||||
deletedDeletingID,
|
||||
map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
assert.False(t, updated)
|
||||
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ func TestGetKnowledgeBatch_TracksDeletionUntilSoftDelete(t *testing.T) {
|
|||
assert.Equal(t, id, rows[0].ID)
|
||||
assert.Equal(t, types.ParseStatusDeleting, rows[0].ParseStatus)
|
||||
|
||||
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, id, map[string]interface{}{
|
||||
updated, err := repo.UpdateActiveDeletingKnowledgeColumns(ctx, 1, kbID, id, map[string]interface{}{
|
||||
"parse_status": types.ParseStatusFailed,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
// UpdateKnowledgeForTransfer is a compare-and-swap checkpoint. Full-row Save
|
||||
// could resurrect a deleted document or overwrite a concurrently moved row.
|
||||
// Accounting changes are committed with the checkpoint, making retry cleanup
|
||||
// safe even if the worker crashes between the database and queue operations.
|
||||
func (r *knowledgeRepository) UpdateKnowledgeForTransfer(ctx context.Context, before, after *types.Knowledge) error {
|
||||
if before == nil || after == nil || before.ID == "" || before.ID != after.ID || before.TenantID == 0 ||
|
||||
before.TenantID != after.TenantID {
|
||||
return fmt.Errorf("invalid knowledge transfer checkpoint")
|
||||
}
|
||||
// PostgreSQL persists timestamps at microsecond precision. Keep the returned
|
||||
// checkpoint identical to storage so the next CAS in this task can match it.
|
||||
after.UpdatedAt = time.Now().UTC().Truncate(time.Microsecond)
|
||||
var storedUpdatedAt time.Time
|
||||
err := r.db.WithContext(ctx).Transaction(func(tx *gorm.DB) error {
|
||||
result := tx.Model(&types.Knowledge{}).
|
||||
Where("id = ? AND tenant_id = ? AND knowledge_base_id = ? AND parse_status = ? AND "+
|
||||
"updated_at = ?",
|
||||
before.ID,
|
||||
before.TenantID,
|
||||
before.KnowledgeBaseID,
|
||||
before.ParseStatus,
|
||||
before.UpdatedAt).
|
||||
Updates(map[string]any{
|
||||
"knowledge_base_id": after.KnowledgeBaseID, "parse_status": after.ParseStatus,
|
||||
"metadata": after.Metadata, "storage_size": after.StorageSize, "updated_at": after.UpdatedAt,
|
||||
"enable_status": after.EnableStatus, "embedding_model_id": after.EmbeddingModelID,
|
||||
"description": after.Description,
|
||||
"processed_at": after.ProcessedAt,
|
||||
"error_message": after.ErrorMessage,
|
||||
})
|
||||
if result.Error != nil {
|
||||
return result.Error
|
||||
}
|
||||
if result.RowsAffected != 1 {
|
||||
return fmt.Errorf("knowledge changed during transfer")
|
||||
}
|
||||
delta := after.StorageSize - before.StorageSize
|
||||
if delta != 0 {
|
||||
result := tx.Model(&types.Tenant{}).Where("id = ?", before.TenantID).
|
||||
UpdateColumn("storage_used",
|
||||
gorm.Expr("CASE WHEN storage_used + ? < 0 THEN 0 ELSE storage_used + ? END",
|
||||
delta,
|
||||
delta))
|
||||
if result.Error != nil {
|
||||
return result.Error
|
||||
}
|
||||
if result.RowsAffected != 1 {
|
||||
return fmt.Errorf("transfer tenant not found")
|
||||
}
|
||||
}
|
||||
var stored struct{ UpdatedAt time.Time }
|
||||
if err := tx.Model(&types.Knowledge{}).Select("updated_at").
|
||||
Where("id = ? AND tenant_id = ?", after.ID, after.TenantID).Take(&stored).Error; err != nil {
|
||||
return err
|
||||
}
|
||||
storedUpdatedAt = stored.UpdatedAt
|
||||
return nil
|
||||
})
|
||||
if err == nil {
|
||||
after.UpdatedAt = storedUpdatedAt
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
|
@ -0,0 +1,30 @@
|
|||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestTransferCheckpointReturnsStoredTimestampPrecision(t *testing.T) {
|
||||
db := setupKnowledgeTestDB(t)
|
||||
// Emulate a database that stores fewer fractional digits than time.Now.
|
||||
require.NoError(t, db.Exec(`CREATE TRIGGER truncate_checkpoint_time AFTER UPDATE ON knowledges
|
||||
BEGIN UPDATE knowledges SET updated_at = strftime('%Y-%m-%d %H:%M:%S', NEW.updated_at) || '+00:00'
|
||||
WHERE id = NEW.id; END`).Error)
|
||||
row := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "source", ParseStatus: types.ParseStatusCompleted}
|
||||
require.NoError(t, db.Omit("custom_metadata").Create(row).Error)
|
||||
repo := NewKnowledgeRepository(db)
|
||||
before, err := repo.GetKnowledgeByID(context.Background(), 7, "doc")
|
||||
require.NoError(t, err)
|
||||
after := *before
|
||||
after.ParseStatus = types.ParseStatusProcessing
|
||||
require.NoError(t, repo.UpdateKnowledgeForTransfer(context.Background(), before, &after))
|
||||
require.Zero(t, after.UpdatedAt.Nanosecond())
|
||||
final := after
|
||||
final.KnowledgeBaseID = "target"
|
||||
final.ParseStatus = types.ParseStatusCompleted
|
||||
require.NoError(t, repo.UpdateKnowledgeForTransfer(context.Background(), &after, &final))
|
||||
}
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
// IsReferencedByKnowledgeBase accepts only explicit bindings to live documents.
|
||||
// Text mentioning a handle or physical path is never ownership evidence.
|
||||
func (r *resourceRepository) IsReferencedByKnowledgeBase(
|
||||
ctx context.Context, tenantID uint64, kbID, resourceID string,
|
||||
) (bool, error) {
|
||||
if tenantID == 0 || kbID == "" || resourceID == "" {
|
||||
return false, nil
|
||||
}
|
||||
var count int64
|
||||
err := r.db.WithContext(ctx).Table("resource_bindings AS b").
|
||||
Joins("JOIN knowledges AS k ON k.id = b.owner_id AND k.tenant_id = b.tenant_id").
|
||||
Joins("JOIN knowledge_bases AS kb ON kb.id = k.knowledge_base_id "+
|
||||
"AND kb.tenant_id = k.tenant_id AND kb.deleted_at IS NULL").
|
||||
Where("b.resource_id = ? AND b.owner_type = ? AND b.tenant_id = ? "+
|
||||
"AND k.knowledge_base_id = ? AND k.deleted_at IS NULL",
|
||||
resourceID, types.ResourceOwnerKnowledge, tenantID, kbID).Count(&count).Error
|
||||
return count > 0, err
|
||||
}
|
||||
|
||||
func (r *resourceRepository) GetMessageFileBindings(
|
||||
ctx context.Context, tenantID uint64, resourceID, messageID string,
|
||||
) (*types.MessageFileBindings, error) {
|
||||
result := &types.MessageFileBindings{}
|
||||
if tenantID == 0 || resourceID == "" {
|
||||
return result, nil
|
||||
}
|
||||
err := r.db.WithContext(ctx).Table("resource_bindings AS b").
|
||||
Joins("JOIN knowledges AS k ON k.id = b.owner_id AND k.tenant_id = b.tenant_id").
|
||||
Joins("JOIN knowledge_bases AS kb ON kb.id = k.knowledge_base_id "+
|
||||
"AND kb.tenant_id = k.tenant_id AND kb.deleted_at IS NULL").
|
||||
Where("b.resource_id = ? AND b.owner_type = ? AND b.tenant_id = ? AND k.deleted_at IS NULL",
|
||||
resourceID, types.ResourceOwnerKnowledge, tenantID).
|
||||
Distinct("k.knowledge_base_id").Pluck("k.knowledge_base_id", &result.KnowledgeBaseIDs).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if messageID != "" {
|
||||
var count int64
|
||||
err = r.db.WithContext(ctx).Model(&types.ResourceBinding{}).
|
||||
Where("resource_id = ? AND tenant_id = ? AND owner_type = ? AND owner_id = ? AND relation = ?",
|
||||
resourceID, tenantID, types.ResourceOwnerMessage, messageID, types.ResourceRelationArtifact).
|
||||
Count(&count).Error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result.MessageArtifact = count > 0
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
package doris
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
func (r *dorisRepository) ValidateKnowledgeIndexMove(ctx context.Context) error {
|
||||
mode, err := r.resolveCompatMode(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if mode.usesRewriteChunkUpdates() {
|
||||
// ANN DUPLICATE KEY tables implement replacement as DELETE + INSERT.
|
||||
// A failed insert would lose the only vector copy; changing physical IDs
|
||||
// would break the stable source-ID identity used by subsequent edits.
|
||||
return fmt.Errorf("reuse_vectors move is not supported by Doris ANN tables; use reparse mode")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *dorisRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
dimension int,
|
||||
_ string,
|
||||
) error {
|
||||
if err := r.ValidateKnowledgeIndexMove(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
if dimension <= 0 {
|
||||
return fmt.Errorf("invalid embedding dimension")
|
||||
}
|
||||
query := fmt.Sprintf(
|
||||
"UPDATE `%s` SET knowledge_base_id = ?, tag_id = '' WHERE knowledge_base_id = ? AND knowledge_id = ?",
|
||||
r.getTableName(dimension),
|
||||
)
|
||||
_, err := r.db.ExecContext(ctx, query, targetKB, sourceKB, knowledgeID)
|
||||
return err
|
||||
}
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
package doris
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/DATA-DOG/go-sqlmock"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveIndicesRejectsNonAtomicANNReplacement(t *testing.T) {
|
||||
repo, mock, _, cleanup := newTestRepo(t)
|
||||
defer cleanup()
|
||||
primeCompatMode(repo, dorisCompatModeInnerProductDuplicate, nil)
|
||||
require.ErrorContains(
|
||||
t,
|
||||
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
|
||||
"use reparse mode",
|
||||
)
|
||||
require.NoError(t, mock.ExpectationsWereMet(), "rejection must issue no vector delete or insert")
|
||||
}
|
||||
|
||||
func TestMoveIndicesLegacyUpdatesOnlyExactSourceDocument(t *testing.T) {
|
||||
repo, mock, _, cleanup := newTestRepo(t)
|
||||
defer cleanup()
|
||||
primeCompatMode(repo, dorisCompatModeLegacy, nil)
|
||||
mock.ExpectExec("UPDATE `weknora_embeddings_2` SET knowledge_base_id = \\?, tag_id = '' WHERE "+
|
||||
"knowledge_base_id = \\? AND knowledge_id = \\?").
|
||||
WithArgs("target", "source", "doc").
|
||||
WillReturnResult(sqlmock.NewResult(0, 3))
|
||||
require.NoError(
|
||||
t,
|
||||
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
|
||||
)
|
||||
require.NoError(t, mock.ExpectationsWereMet())
|
||||
}
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
package v7
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
func (e *elasticsearchRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
_ int,
|
||||
_ string,
|
||||
) error {
|
||||
body, err := json.Marshal(map[string]any{
|
||||
"query": map[string]any{"bool": map[string]any{"filter": []any{
|
||||
map[string]any{"term": map[string]any{e.idField("knowledge_base_id"): sourceKB}},
|
||||
map[string]any{"term": map[string]any{e.idField("knowledge_id"): knowledgeID}},
|
||||
}}},
|
||||
"script": map[string]any{
|
||||
"lang": "painless",
|
||||
"source": "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = ''",
|
||||
"params": map[string]any{"target": targetKB},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp, err := e.client.UpdateByQuery(
|
||||
[]string{e.index},
|
||||
e.client.UpdateByQuery.WithContext(ctx),
|
||||
e.client.UpdateByQuery.WithBody(bytes.NewReader(body)),
|
||||
e.client.UpdateByQuery.WithRefresh(true),
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.IsError() {
|
||||
return fmt.Errorf("move indices: %s", resp.String())
|
||||
}
|
||||
var result struct {
|
||||
Total *int64 `json:"total"`
|
||||
Updated *int64 `json:"updated"`
|
||||
TimedOut bool `json:"timed_out"`
|
||||
Conflicts int `json:"version_conflicts"`
|
||||
Failures []json.RawMessage `json:"failures"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
|
||||
return err
|
||||
}
|
||||
if result.Total == nil || result.Updated == nil || *result.Total < 0 || *result.Total != *result.Updated ||
|
||||
result.TimedOut || result.Conflicts != 0 || len(result.Failures) != 0 {
|
||||
return fmt.Errorf("move indices was incomplete")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
package v7
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/elastic/go-elasticsearch/v7"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveIndicesRequiresCompleteCounts(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name, body string
|
||||
fails bool
|
||||
}{
|
||||
{"complete", `{"total":100,"updated":100}`, false},
|
||||
{"empty retry", `{"total":0,"updated":0}`, false},
|
||||
{"partial", `{"total":100,"updated":40}`, true},
|
||||
{"missing total", `{"updated":40}`, true},
|
||||
{"missing updated", `{"total":40}`, true},
|
||||
{"missing counts", `{}`, true},
|
||||
{"negative", `{"total":-1,"updated":-1}`, true},
|
||||
{"conflict", `{"total":1,"updated":1,"version_conflicts":1}`, true},
|
||||
{"timeout", `{"total":1,"updated":1,"timed_out":true}`, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("X-Elastic-Product", "Elasticsearch")
|
||||
if r.URL.Path == "/" {
|
||||
_, _ = w.Write(
|
||||
[]byte(
|
||||
`{"version":{"number":"7.17.0","build_flavor":"default"},"tagline":"You Know, for Search"}`,
|
||||
),
|
||||
)
|
||||
return
|
||||
}
|
||||
if r.URL.Path != "/vectors/_update_by_query" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
_, _ = w.Write([]byte(tt.body))
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := elasticsearch.NewClient(
|
||||
elasticsearch.Config{Addresses: []string{server.URL}, DisableRetry: true},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
repo := &elasticsearchRepository{client: client, index: "vectors"}
|
||||
err = repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, "file")
|
||||
if tt.fails {
|
||||
require.Error(t, err)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
// Package v8 implements retrieval against its configured vector backend.
|
||||
package v8
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/elastic/go-elasticsearch/v8/typedapi/types"
|
||||
)
|
||||
|
||||
func (e *elasticsearchRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
_ int,
|
||||
_ string,
|
||||
) error {
|
||||
query := &types.Query{Bool: &types.BoolQuery{Filter: []types.Query{
|
||||
{
|
||||
Terms: &types.TermsQuery{
|
||||
TermsQuery: map[string]types.TermsQueryField{e.idField("knowledge_base_id"): []string{sourceKB}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Terms: &types.TermsQuery{
|
||||
TermsQuery: map[string]types.TermsQueryField{e.idField("knowledge_id"): []string{knowledgeID}},
|
||||
},
|
||||
},
|
||||
}}}
|
||||
source := "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = ''"
|
||||
target, err := json.Marshal(targetKB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
script := &types.Script{Source: &source, Params: map[string]json.RawMessage{"target": target}}
|
||||
result, err := e.client.UpdateByQuery(e.index).Query(query).Script(script).Refresh(true).Do(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if result == nil || result.Total == nil || result.Updated == nil ||
|
||||
*result.Total < 0 || *result.Total != *result.Updated || (result.TimedOut != nil && *result.TimedOut) ||
|
||||
(result.VersionConflicts != nil && *result.VersionConflicts != 0) ||
|
||||
len(result.Failures) != 0 {
|
||||
return fmt.Errorf("move indices was incomplete")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
package v8
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/elastic/go-elasticsearch/v8"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveIndicesRequiresCompleteCounts(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name, body string
|
||||
fails bool
|
||||
}{
|
||||
{"complete", `{"total":100,"updated":100}`, false},
|
||||
{"empty retry", `{"total":0,"updated":0}`, false},
|
||||
{"partial", `{"total":100,"updated":40}`, true},
|
||||
{"missing total", `{"updated":40}`, true},
|
||||
{"missing updated", `{"total":40}`, true},
|
||||
{"missing counts", `{}`, true},
|
||||
{"negative", `{"total":-1,"updated":-1}`, true},
|
||||
{"conflict", `{"total":1,"updated":1,"version_conflicts":1}`, true},
|
||||
{"timeout", `{"total":1,"updated":1,"timed_out":true}`, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Header().Set("X-Elastic-Product", "Elasticsearch")
|
||||
if r.URL.Path == "/" {
|
||||
_, _ = w.Write(
|
||||
[]byte(
|
||||
`{"version":{"number":"7.17.0","build_flavor":"default"},"tagline":"You Know, for Search"}`,
|
||||
),
|
||||
)
|
||||
return
|
||||
}
|
||||
if r.URL.Path != "/vectors/_update_by_query" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
}
|
||||
_, _ = w.Write([]byte(tt.body))
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := elasticsearch.NewTypedClient(
|
||||
elasticsearch.Config{Addresses: []string{server.URL}, DisableRetry: true},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
repo := &elasticsearchRepository{client: client, index: "vectors"}
|
||||
err = repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, "file")
|
||||
if tt.fails {
|
||||
require.Error(t, err)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
package milvus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
)
|
||||
|
||||
func (m *milvusRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
dimension int,
|
||||
_ string,
|
||||
) error {
|
||||
collection := m.getCollectionName(dimension)
|
||||
filter := &universalFilterCondition{Operator: operatorAnd, Value: []*universalFilterCondition{
|
||||
{Field: fieldKnowledgeBaseID, Operator: operatorEqual, Value: sourceKB},
|
||||
{Field: fieldKnowledgeID, Operator: operatorEqual, Value: knowledgeID},
|
||||
}}
|
||||
size, offset := 100, 0
|
||||
return drainMoveRows(ctx, targetKB, func(ctx context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
|
||||
rows, _, err := m.searchByFilter(ctx, collection, filter, &size, &offset)
|
||||
return rows, err
|
||||
}, func(ctx context.Context, rows []*MilvusVectorEmbedding) error {
|
||||
_, err := m.client.Upsert(ctx, createUpsert(collection, rows))
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// Drain the source predicate without offset windows. A repeated ID means the
|
||||
// backend has not made the acknowledged update visible; fail for retry instead
|
||||
// of advancing the document checkpoint or silently omitting remaining vectors.
|
||||
func drainMoveRows(ctx context.Context, targetKB string,
|
||||
read func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error),
|
||||
write func(context.Context, []*MilvusVectorEmbedding) error,
|
||||
) error {
|
||||
seen := map[string]bool{}
|
||||
for {
|
||||
rows, err := read(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
batch := make([]*MilvusVectorEmbedding, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
if row == nil || row.ID == "" || seen[row.ID] {
|
||||
return fmt.Errorf("invalid or repeated move index")
|
||||
}
|
||||
seen[row.ID] = true
|
||||
copyOfRow := row.MilvusVectorEmbedding
|
||||
copyOfRow.KnowledgeBaseID = targetKB
|
||||
copyOfRow.TagID = ""
|
||||
batch = append(batch, ©OfRow)
|
||||
}
|
||||
if err := write(ctx, batch); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
package milvus
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveRowsDrainsPagesAndRetriesPartialUpsert(t *testing.T) {
|
||||
remaining := map[string]*MilvusVectorEmbeddingWithScore{}
|
||||
moved := map[string]*MilvusVectorEmbedding{}
|
||||
for i := range 205 {
|
||||
id := fmt.Sprint(i)
|
||||
remaining[id] = &MilvusVectorEmbeddingWithScore{MilvusVectorEmbedding: MilvusVectorEmbedding{
|
||||
ID: id,
|
||||
KnowledgeID: "doc",
|
||||
KnowledgeBaseID: "source",
|
||||
TagID: "old-tag",
|
||||
ChunkID: "chunk-" + id,
|
||||
Content: "text",
|
||||
Embedding: []float32{1, 2, 3},
|
||||
IsEnabled: i%2 == 0,
|
||||
}}
|
||||
}
|
||||
reads, writes := 0, 0
|
||||
failure := errors.New("partial upsert")
|
||||
read := func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
|
||||
reads++
|
||||
var rows []*MilvusVectorEmbeddingWithScore
|
||||
for _, row := range remaining {
|
||||
rows = append(rows, row)
|
||||
if len(rows) == 100 {
|
||||
break
|
||||
}
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
write := func(_ context.Context, rows []*MilvusVectorEmbedding) error {
|
||||
writes++
|
||||
for i, row := range rows {
|
||||
original := remaining[row.ID]
|
||||
require.NotNil(t, original)
|
||||
require.Equal(t, "source", original.KnowledgeBaseID, "read rows must not be mutated before acknowledgement")
|
||||
require.Equal(t, "target", row.KnowledgeBaseID)
|
||||
require.Empty(t, row.TagID)
|
||||
expected := original.MilvusVectorEmbedding
|
||||
expected.KnowledgeBaseID = "target"
|
||||
expected.TagID = ""
|
||||
require.Equal(t, expected, *row)
|
||||
moved[row.ID] = row
|
||||
delete(remaining, row.ID)
|
||||
if writes == 2 && i == 16 {
|
||||
return failure
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
require.ErrorIs(t, drainMoveRows(context.Background(), "target", read, write), failure)
|
||||
require.Len(t, remaining, 88)
|
||||
require.NoError(t, drainMoveRows(context.Background(), "target", read, write))
|
||||
require.Empty(t, remaining)
|
||||
require.Len(t, moved, 205)
|
||||
require.Equal(t, 4, reads, "retry must verify an empty source predicate")
|
||||
}
|
||||
|
||||
func TestMoveRowsRejectsUnacknowledgedProgress(t *testing.T) {
|
||||
row := &MilvusVectorEmbeddingWithScore{
|
||||
MilvusVectorEmbedding: MilvusVectorEmbedding{ID: "id", KnowledgeBaseID: "source"},
|
||||
}
|
||||
err := drainMoveRows(
|
||||
context.Background(),
|
||||
"target",
|
||||
func(context.Context) ([]*MilvusVectorEmbeddingWithScore, error) {
|
||||
return []*MilvusVectorEmbeddingWithScore{row}, nil
|
||||
},
|
||||
func(context.Context, []*MilvusVectorEmbedding) error { return nil },
|
||||
)
|
||||
require.ErrorContains(t, err, "repeated")
|
||||
}
|
||||
|
|
@ -17,8 +17,15 @@ import (
|
|||
// bounded id + type. A non-zero version_conflicts count with no hard failures
|
||||
// is logged as a warning but not treated as an error.
|
||||
func inspectByQueryResponse(body io.Reader) error {
|
||||
return inspectByQueryResult(body, false)
|
||||
}
|
||||
|
||||
func inspectByQueryResult(body io.Reader, requireComplete bool) error {
|
||||
var r struct {
|
||||
VersionConflicts int `json:"version_conflicts"`
|
||||
Total *int64 `json:"total"`
|
||||
Updated *int64 `json:"updated"`
|
||||
TimedOut bool `json:"timed_out"`
|
||||
VersionConflicts int `json:"version_conflicts"`
|
||||
Failures []struct {
|
||||
ID string `json:"id"`
|
||||
Cause struct {
|
||||
|
|
@ -30,6 +37,13 @@ func inspectByQueryResponse(body io.Reader) error {
|
|||
if err := json.NewDecoder(body).Decode(&r); err != nil {
|
||||
return fmt.Errorf("opensearch: parse by-query response: %w", ErrTransport)
|
||||
}
|
||||
if requireComplete && (r.TimedOut || r.VersionConflicts != 0) {
|
||||
return fmt.Errorf("opensearch: incomplete move (timed_out=%t, version_conflicts=%d): %w",
|
||||
r.TimedOut, r.VersionConflicts, ErrTransport)
|
||||
}
|
||||
if requireComplete && (r.Total == nil || r.Updated == nil || *r.Total < 0 || *r.Updated != *r.Total) {
|
||||
return fmt.Errorf("opensearch: incomplete move document counts: %w", ErrTransport)
|
||||
}
|
||||
log := logger.GetLogger(context.Background())
|
||||
if len(r.Failures) == 0 {
|
||||
if r.VersionConflicts > 0 {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,49 @@
|
|||
package opensearch
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
|
||||
osapi "github.com/opensearch-project/opensearch-go/v4/opensearchapi"
|
||||
)
|
||||
|
||||
// MoveKnowledgeIndices changes every vector for the source document, including
|
||||
// historical vectors whose chunks no longer exist, while preserving vector IDs.
|
||||
func (r *Repository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
_ int,
|
||||
_ string,
|
||||
) error {
|
||||
body, err := json.Marshal(map[string]any{
|
||||
"query": map[string]any{"bool": map[string]any{"filter": []any{
|
||||
map[string]any{"term": map[string]any{"knowledge_base_id": sourceKB}},
|
||||
map[string]any{"term": map[string]any{"knowledge_id": knowledgeID}},
|
||||
}}},
|
||||
"script": map[string]any{
|
||||
"lang": "painless",
|
||||
"source": "ctx._source.knowledge_base_id = params.target; ctx._source.tag_id = '';",
|
||||
"params": map[string]any{"target": targetKB},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("opensearch: marshal move: %w", err)
|
||||
}
|
||||
refresh := true
|
||||
resp, err := r.client.UpdateByQuery(ctx, osapi.UpdateByQueryReq{
|
||||
Indices: []string{r.baseIndex + "_*"}, Body: bytes.NewReader(body),
|
||||
Params: osapi.UpdateByQueryParams{Refresh: &refresh},
|
||||
})
|
||||
if err != nil {
|
||||
return wrapTransport(err)
|
||||
}
|
||||
if resp == nil {
|
||||
return fmt.Errorf("opensearch: missing move response: %w", ErrTransport)
|
||||
}
|
||||
defer drainAndClose(resp.Inspect().Response.Body)
|
||||
return inspectByQueryResult(io.LimitReader(resp.Inspect().Response.Body, 16<<20), true)
|
||||
}
|
||||
|
|
@ -0,0 +1,61 @@
|
|||
package opensearch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveIndicesRequiresCompleteResult(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name, response string
|
||||
fails bool
|
||||
}{
|
||||
{"complete", `{"total":2,"updated":2,"version_conflicts":0,"failures":[]}`, false},
|
||||
{"partial", `{"total":100,"updated":40,"failures":[]}`, true},
|
||||
{"empty", `{"total":0,"updated":0,"failures":[]}`, false},
|
||||
{"missing counts", `{"failures":[]}`, true},
|
||||
{"missing total", `{"updated":2,"failures":[]}`, true},
|
||||
{"missing updated", `{"total":2,"failures":[]}`, true},
|
||||
{"negative", `{"total":-1,"updated":-1,"failures":[]}`, true},
|
||||
{"conflict", `{"updated":1,"version_conflicts":1,"failures":[]}`, true},
|
||||
{"timeout", `{"timed_out":true,"version_conflicts":0,"failures":[]}`, true},
|
||||
{"failure", `{"failures":[{"id":"doc","cause":{"type":"unavailable"}}]}`, true},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
repo, server := newTestRepo(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(tt.response))
|
||||
})
|
||||
defer server.Close()
|
||||
err := repo.MoveKnowledgeIndices(
|
||||
context.Background(), "source", "target", "doc", []string{"chunk"}, 1, "file",
|
||||
)
|
||||
if tt.fails {
|
||||
require.ErrorIs(t, err, ErrTransport)
|
||||
} else {
|
||||
require.NoError(t, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoveIndicesIncludesOrphansEvenWithoutDatabaseChunks(t *testing.T) {
|
||||
var body map[string]any
|
||||
repo, server := newTestRepo(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
|
||||
t.Errorf("decode move request: %v", err)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"total":1,"updated":1,"failures":[]}`))
|
||||
})
|
||||
defer server.Close()
|
||||
require.NoError(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 1, "file"))
|
||||
want := `{"bool":{"filter":[{"term":{"knowledge_base_id":"source"}},{"term":{"knowledge_id":"doc"}}]}}`
|
||||
query, err := json.Marshal(body["query"])
|
||||
require.NoError(t, err)
|
||||
require.JSONEq(t, want, string(query), "all vectors for this exact source document must be selected")
|
||||
script := body["script"].(map[string]any)
|
||||
require.Equal(t, map[string]any{"target": "target"}, script["params"])
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
// Package postgres implements retrieval against its configured vector backend.
|
||||
package postgres
|
||||
|
||||
import "context"
|
||||
|
||||
func (g *pgRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
_ int,
|
||||
_ string,
|
||||
) error {
|
||||
return g.db.WithContext(ctx).Model(&pgVector{}).
|
||||
Where("knowledge_base_id = ? AND knowledge_id = ?", sourceKB, knowledgeID).
|
||||
Updates(map[string]any{"knowledge_base_id": targetKB, "tag_id": ""}).Error
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
// Package qdrant implements vector retrieval and metadata updates in Qdrant.
|
||||
package qdrant
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/qdrant/go-client/qdrant"
|
||||
)
|
||||
|
||||
func (q *qdrantRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
dimension int,
|
||||
_ string,
|
||||
) error {
|
||||
wait := true
|
||||
_, err := q.client.SetPayload(ctx, &qdrant.SetPayloadPoints{
|
||||
CollectionName: q.getCollectionName(dimension), Wait: &wait,
|
||||
Payload: newQdrantValueMap(map[string]any{fieldKnowledgeBaseID: targetKB, fieldTagID: ""}),
|
||||
PointsSelector: qdrant.NewPointsSelectorFilter(&qdrant.Filter{Must: []*qdrant.Condition{
|
||||
qdrant.NewMatch(fieldKnowledgeBaseID, sourceKB), qdrant.NewMatch(fieldKnowledgeID, knowledgeID),
|
||||
}}),
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
// Package sqlite implements local retrieval with SQLite FTS and vectors.
|
||||
package sqlite
|
||||
|
||||
import "context"
|
||||
|
||||
func (r *sqliteRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
_ int,
|
||||
_ string,
|
||||
) error {
|
||||
// FTS and vector rows keep their IDs. Queries join them to this metadata row.
|
||||
return r.db.WithContext(ctx).Model(&sqliteEmbedding{}).
|
||||
Where("knowledge_base_id = ? AND knowledge_id = ?", sourceKB, knowledgeID).
|
||||
Updates(map[string]any{"knowledge_base_id": targetKB, "tag_id": ""}).Error
|
||||
}
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
package sqlite
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestMoveIndicesPreservesVectorsAndExactScopeAcrossRetry(t *testing.T) {
|
||||
repo := newSQLiteRetrieverTestRepository(t)
|
||||
ctx := context.Background()
|
||||
for _, row := range []struct{ chunk, kb, doc string }{
|
||||
{
|
||||
"text",
|
||||
"source",
|
||||
"doc",
|
||||
},
|
||||
{
|
||||
"image",
|
||||
"source",
|
||||
"doc",
|
||||
},
|
||||
{
|
||||
"other",
|
||||
"source",
|
||||
"other-doc",
|
||||
},
|
||||
{
|
||||
"foreign",
|
||||
"third",
|
||||
"doc",
|
||||
},
|
||||
} {
|
||||
saveSQLiteTestVector(t, repo, sqliteTestIndex(row.chunk, row.kb, row.doc, "old-tag", true), []float32{1, 0})
|
||||
}
|
||||
var before []sqliteEmbedding
|
||||
require.NoError(t, repo.db.Order("id").Find(&before).Error)
|
||||
for range 2 {
|
||||
require.NoError(
|
||||
t,
|
||||
repo.MoveKnowledgeIndices(ctx, "source", "target", "doc", []string{"text", "image"}, 2, "document"),
|
||||
)
|
||||
}
|
||||
var after []sqliteEmbedding
|
||||
require.NoError(t, repo.db.Order("id").Find(&after).Error)
|
||||
require.Len(t, after, len(before))
|
||||
for i, row := range after {
|
||||
require.Equal(t, before[i].ID, row.ID)
|
||||
require.Equal(t, before[i].SourceID, row.SourceID)
|
||||
if row.KnowledgeID == "doc" && before[i].KnowledgeBaseID == "source" {
|
||||
require.Equal(t, "target", row.KnowledgeBaseID)
|
||||
require.Empty(t, row.TagID)
|
||||
} else {
|
||||
require.Equal(t, before[i].KnowledgeBaseID, row.KnowledgeBaseID)
|
||||
require.Equal(t, "old-tag", row.TagID)
|
||||
}
|
||||
}
|
||||
results, err := repo.vectorRetrieve(
|
||||
ctx,
|
||||
types.RetrieveParams{
|
||||
Embedding: []float32{1, 0},
|
||||
TopK: 10,
|
||||
KnowledgeBaseIDs: []string{"target"},
|
||||
RetrieverType: types.VectorRetrieverType,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, results, 1)
|
||||
require.Len(t, results[0].Results, 2, "target vectors remain searchable after an identity-preserving move")
|
||||
}
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
// Package tencentvectordb implements retrieval against its configured vector backend.
|
||||
package tencentvectordb
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/tencent/vectordatabase-sdk-go/tcvectordb"
|
||||
)
|
||||
|
||||
func (r *repository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
dimension int,
|
||||
_ string,
|
||||
) error {
|
||||
filter := tcvectordb.NewFilter(
|
||||
tcvectordb.In(
|
||||
fieldKnowledgeBaseID,
|
||||
[]string{sourceKB},
|
||||
) + " and " + tcvectordb.In(
|
||||
fieldKnowledgeID,
|
||||
[]string{knowledgeID},
|
||||
),
|
||||
)
|
||||
_, err := r.client.Database(r.databaseName).
|
||||
Collection(r.collectionName(dimension)).
|
||||
Update(ctx, tcvectordb.UpdateDocumentParams{
|
||||
QueryFilter: filter,
|
||||
UpdateFields: map[string]tcvectordb.Field{fieldKnowledgeBaseID: {Val: targetKB}, fieldTagID: {Val: ""}},
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
|
@ -0,0 +1,77 @@
|
|||
// Package weaviate implements retrieval and metadata updates in Weaviate.
|
||||
package weaviate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/weaviate/weaviate-go-client/v5/weaviate/filters"
|
||||
"github.com/weaviate/weaviate-go-client/v5/weaviate/graphql"
|
||||
)
|
||||
|
||||
func (w *weaviateRepository) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
_ []string,
|
||||
dimension int,
|
||||
_ string,
|
||||
) error {
|
||||
collection := w.getCollectionName(dimension)
|
||||
where := filters.Where().WithOperator(filters.And).WithOperands([]*filters.WhereBuilder{
|
||||
filters.Where().WithPath([]string{fieldKnowledgeBaseID}).WithOperator(filters.Equal).WithValueString(sourceKB),
|
||||
filters.Where().WithPath([]string{fieldKnowledgeID}).WithOperator(filters.Equal).WithValueString(knowledgeID),
|
||||
})
|
||||
seen := make(map[string]bool)
|
||||
for {
|
||||
result, err := w.client.GraphQL().Get().WithClassName(collection).WithWhere(where).WithLimit(100).
|
||||
WithFields(graphql.Field{Name: "_additional", Fields: []graphql.Field{{Name: "id"}}}).
|
||||
Do(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if result == nil || len(result.Errors) != 0 {
|
||||
return fmt.Errorf("failed to list move indices")
|
||||
}
|
||||
get, ok := result.Data["Get"].(map[string]interface{})
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid move index response")
|
||||
}
|
||||
rows, ok := get[collection].([]interface{})
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid move index collection")
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
var ids []string
|
||||
for _, row := range rows {
|
||||
data, ok := row.(map[string]interface{})
|
||||
if !ok {
|
||||
return fmt.Errorf("invalid move index row")
|
||||
}
|
||||
additional, ok := data["_additional"].(map[string]interface{})
|
||||
if !ok {
|
||||
return fmt.Errorf("missing move index ID")
|
||||
}
|
||||
id, ok := additional["id"].(string)
|
||||
if !ok || id == "" {
|
||||
return fmt.Errorf("invalid move index ID")
|
||||
}
|
||||
if seen[id] {
|
||||
return fmt.Errorf("move indices made no progress for %s", id)
|
||||
}
|
||||
seen[id] = true
|
||||
ids = append(ids, id)
|
||||
}
|
||||
// Drain the first filtered page; after+where is unsupported by Weaviate.
|
||||
// Successful updates remove these IDs from the predicate. Only an empty
|
||||
// follow-up query proves completion, including after partial-page updates.
|
||||
for _, id := range ids {
|
||||
if err := w.client.Data().Updater().WithClassName(collection).WithID(id).WithMerge().
|
||||
WithProperties(map[string]interface{}{fieldKnowledgeBaseID: targetKB, fieldTagID: ""}).
|
||||
Do(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
package weaviate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
sdk "github.com/weaviate/weaviate-go-client/v5/weaviate"
|
||||
)
|
||||
|
||||
func TestMoveKnowledgeIndicesDrainsBeyondOffsetLimitAndRetries(t *testing.T) {
|
||||
const count = 10005
|
||||
var mu sync.Mutex
|
||||
remaining := make(map[string]bool, count)
|
||||
for i := range count {
|
||||
remaining[fmt.Sprintf("00000000-0000-0000-0000-%012d", i)] = true
|
||||
}
|
||||
failed := false
|
||||
patches, queries := 0, 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case r.URL.Path == "/v1/graphql":
|
||||
queries++
|
||||
var body struct {
|
||||
Query string `json:"query"`
|
||||
}
|
||||
if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&body)) {
|
||||
w.WriteHeader(400)
|
||||
return
|
||||
}
|
||||
assert.Contains(t, body.Query, "knowledge_base_id")
|
||||
assert.Contains(t, body.Query, "knowledge_id")
|
||||
assert.Contains(t, body.Query, "source")
|
||||
assert.Contains(t, body.Query, "doc")
|
||||
if strings.Contains(body.Query, "offset") || strings.Contains(body.Query, "after") {
|
||||
_ = json.NewEncoder(w).
|
||||
Encode(map[string]any{"errors": []any{map[string]any{"message": "pagination unsupported"}}})
|
||||
return
|
||||
}
|
||||
rows := make([]any, 0, 100)
|
||||
for id := range remaining {
|
||||
rows = append(rows, map[string]any{"_additional": map[string]any{"id": id}})
|
||||
if len(rows) == 100 {
|
||||
break
|
||||
}
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{"data": map[string]any{"Get": map[string]any{"Move_3": rows}}})
|
||||
case r.Method == http.MethodPatch:
|
||||
id := path.Base(r.URL.Path)
|
||||
assert.True(t, remaining[id], "only source-filtered IDs should be moved")
|
||||
var body struct {
|
||||
Properties map[string]any `json:"properties"`
|
||||
Vector []float32 `json:"vector"`
|
||||
}
|
||||
if !assert.NoError(t, json.NewDecoder(r.Body).Decode(&body)) {
|
||||
w.WriteHeader(400)
|
||||
return
|
||||
}
|
||||
assert.Equal(t, map[string]any{fieldKnowledgeBaseID: "target", fieldTagID: ""}, body.Properties)
|
||||
assert.Empty(t, body.Vector, "metadata merge must preserve existing vectors")
|
||||
if patches == 137 && !failed {
|
||||
failed = true
|
||||
w.WriteHeader(http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
patches++
|
||||
delete(remaining, id)
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
default:
|
||||
// SDK probes the server version before object writes.
|
||||
if r.URL.Path == "/v1/meta" {
|
||||
_, _ = w.Write([]byte(`{"version":"1.30.0"}`))
|
||||
return
|
||||
}
|
||||
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := sdk.NewClient(sdk.Config{Scheme: "http", Host: strings.TrimPrefix(server.URL, "http://")})
|
||||
require.NoError(t, err)
|
||||
repo := &weaviateRepository{client: client, collectionBaseName: "Move"}
|
||||
require.Error(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""))
|
||||
mu.Lock()
|
||||
left := len(remaining)
|
||||
mu.Unlock()
|
||||
require.Equal(t, count-137, left)
|
||||
require.NoError(t, repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""))
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
require.Empty(t, remaining)
|
||||
require.Equal(t, count, patches)
|
||||
require.Greater(t, queries, 100)
|
||||
}
|
||||
|
||||
func TestMoveKnowledgeIndicesRejectsNoProgress(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.URL.Path {
|
||||
case "/v1/graphql":
|
||||
_, _ = w.Write(
|
||||
[]byte(`{"data":{"Get":{"Move_3":[{"_additional":{"id":"00000000-0000-0000-0000-000000000001"}}]}}}`),
|
||||
)
|
||||
case "/v1/meta":
|
||||
_, _ = w.Write([]byte(`{"version":"1.30.0"}`))
|
||||
default:
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := sdk.NewClient(sdk.Config{Scheme: "http", Host: strings.TrimPrefix(server.URL, "http://")})
|
||||
require.NoError(t, err)
|
||||
repo := &weaviateRepository{client: client, collectionBaseName: "Move"}
|
||||
require.ErrorContains(
|
||||
t,
|
||||
repo.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", nil, 3, ""),
|
||||
"no progress",
|
||||
)
|
||||
}
|
||||
|
|
@ -512,27 +512,9 @@ func (s *agentShareService) TenantCanAccessKBViaSomeSharedAgent(ctx context.Cont
|
|||
return false, err
|
||||
}
|
||||
for _, info := range list {
|
||||
if info.Agent == nil {
|
||||
continue
|
||||
}
|
||||
agent := info.Agent
|
||||
if agent.TenantID != kb.TenantID {
|
||||
continue
|
||||
}
|
||||
mode := agent.Config.KBSelectionMode
|
||||
if mode == "none" {
|
||||
continue
|
||||
}
|
||||
if mode == "all" {
|
||||
if types.SharedAgentIncludesKB(info.Agent, kb) {
|
||||
return true, nil
|
||||
}
|
||||
if mode == "selected" {
|
||||
for _, id := range agent.Config.KnowledgeBases {
|
||||
if id == kb.ID {
|
||||
return true, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -86,6 +86,9 @@ func (s *chunkService) GetRepository() interfaces.ChunkRepository {
|
|||
// Returns:
|
||||
// - error: Any error encountered during chunk creation
|
||||
func (s *chunkService) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
if err := s.validateChunkWrites(ctx, chunks, true); err != nil {
|
||||
return err
|
||||
}
|
||||
err := s.chunkRepository.CreateChunks(ctx, chunks)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
|
|
@ -217,6 +220,9 @@ func (s *chunkService) ListPagedChunksByKnowledgeID(ctx context.Context,
|
|||
//
|
||||
// This method handles the actual update logic for a chunk, including updating the vector database representation
|
||||
func (s *chunkService) UpdateChunk(ctx context.Context, chunk *types.Chunk) error {
|
||||
if err := s.validateChunkWrites(ctx, []*types.Chunk{chunk}, false); err != nil {
|
||||
return err
|
||||
}
|
||||
logger.Infof(ctx, "Updating chunk, ID: %s, knowledge ID: %s", chunk.ID, chunk.KnowledgeID)
|
||||
|
||||
// Update the chunk in the repository
|
||||
|
|
@ -235,6 +241,9 @@ func (s *chunkService) UpdateChunk(ctx context.Context, chunk *types.Chunk) erro
|
|||
|
||||
// UpdateChunks updates chunks in batch
|
||||
func (s *chunkService) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
if err := s.validateChunkWrites(ctx, chunks, false); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(chunks) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
|
@ -262,6 +271,9 @@ func (s *chunkService) UpdateChunks(ctx context.Context, chunks []*types.Chunk)
|
|||
// Returns:
|
||||
// - error: Any error encountered during deletion
|
||||
func (s *chunkService) DeleteChunk(ctx context.Context, id string) error {
|
||||
if _, err := s.writableChunk(ctx, id); err != nil {
|
||||
return err
|
||||
}
|
||||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
err := s.chunkRepository.DeleteChunk(ctx, tenantID, id)
|
||||
if err != nil {
|
||||
|
|
@ -289,10 +301,15 @@ func (s *chunkService) DeleteChunks(ctx context.Context, ids []string) error {
|
|||
logger.Info(ctx, "Start deleting chunks in batch")
|
||||
logger.Infof(ctx, "Deleting %d chunks", len(ids))
|
||||
|
||||
checkedIDs, err := s.writableChunkIDs(ctx, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ids = checkedIDs
|
||||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
logger.Infof(ctx, "Tenant ID: %d", tenantID)
|
||||
|
||||
err := s.chunkRepository.DeleteChunks(ctx, tenantID, ids)
|
||||
err = s.chunkRepository.DeleteChunks(ctx, tenantID, ids)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
"chunk_ids": ids,
|
||||
|
|
@ -314,6 +331,9 @@ func (s *chunkService) DeleteChunks(ctx context.Context, ids []string) error {
|
|||
// Returns:
|
||||
// - error: Any error encountered during bulk deletion
|
||||
func (s *chunkService) DeleteChunksByKnowledgeID(ctx context.Context, knowledgeID string) error {
|
||||
if _, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, []string{knowledgeID}); err != nil {
|
||||
return err
|
||||
}
|
||||
logger.Info(ctx, "Start deleting all chunks by knowledge ID")
|
||||
logger.Infof(ctx, "Knowledge ID: %s", knowledgeID)
|
||||
|
||||
|
|
@ -334,6 +354,9 @@ func (s *chunkService) DeleteChunksByKnowledgeID(ctx context.Context, knowledgeI
|
|||
}
|
||||
|
||||
func (s *chunkService) DeleteByKnowledgeList(ctx context.Context, ids []string) error {
|
||||
if _, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, ids); err != nil {
|
||||
return err
|
||||
}
|
||||
logger.Info(ctx, "Start deleting all chunks by knowledge IDs")
|
||||
logger.Infof(ctx, "Knowledge IDs: %v", ids)
|
||||
|
||||
|
|
@ -382,14 +405,16 @@ func (s *chunkService) ListChunkByParentID(
|
|||
func (s *chunkService) UpdateDocumentChunk(
|
||||
ctx context.Context, chunkID string, content *string, isEnabled *bool, expectedRevision *int,
|
||||
) (*types.Chunk, error) {
|
||||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenantID, chunkID)
|
||||
chunk, err := s.writableChunk(ctx, chunkID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if chunk.ChunkType != types.ChunkTypeText {
|
||||
return nil, fmt.Errorf("only text chunks can be edited")
|
||||
}
|
||||
if err := s.validateDocumentChunkRelations(ctx, chunk); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if expectedRevision != nil && *expectedRevision != chunk.ContentRevision {
|
||||
return nil, ErrChunkRevisionConflict
|
||||
}
|
||||
|
|
@ -481,7 +506,7 @@ func (s *chunkService) UpdateDocumentChunk(
|
|||
}
|
||||
}
|
||||
if bodyChanged || newEnabled != revision.IsEnabled {
|
||||
knowledge, getErr := s.knowledgeRepo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
|
||||
knowledge, getErr := s.knowledgeRepo.GetKnowledgeByID(ctx, chunk.TenantID, chunk.KnowledgeID)
|
||||
if getErr == nil {
|
||||
if err := enqueueSummaryRefresh(
|
||||
ctx, s.knowledgeRepo, s.task, s.kbRepository, s.spanTracker, knowledge,
|
||||
|
|
@ -700,7 +725,7 @@ func (s *chunkService) UpsertGeneratedQuestion(
|
|||
if question == "" {
|
||||
return nil, fmt.Errorf("question cannot be empty")
|
||||
}
|
||||
chunk, err := s.chunkRepository.GetChunkByID(ctx, types.MustTenantIDFromContext(ctx), chunkID)
|
||||
chunk, err := s.writableChunk(ctx, chunkID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -756,7 +781,7 @@ func (s *chunkService) DeleteGeneratedQuestion(ctx context.Context, chunkID stri
|
|||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
|
||||
// 1. Get the chunk
|
||||
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenantID, chunkID)
|
||||
chunk, err := s.writableChunk(ctx, chunkID)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
"chunk_id": chunkID,
|
||||
|
|
|
|||
|
|
@ -3,11 +3,11 @@ package service
|
|||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
|
@ -49,7 +49,7 @@ type editableChunkKBRepo struct {
|
|||
}
|
||||
|
||||
func (editableChunkKBRepo) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
|
||||
return &types.KnowledgeBase{}, nil
|
||||
return &types.KnowledgeBase{ID: "kb", TenantID: 1}, nil
|
||||
}
|
||||
|
||||
type editableChunkKnowledgeRepo struct {
|
||||
|
|
@ -57,7 +57,7 @@ type editableChunkKnowledgeRepo struct {
|
|||
}
|
||||
|
||||
func (editableChunkKnowledgeRepo) GetKnowledgeByID(context.Context, uint64, string) (*types.Knowledge, error) {
|
||||
return nil, errors.New("summary refresh not configured in unit test")
|
||||
return &types.Knowledge{ID: "knowledge", TenantID: 1, KnowledgeBaseID: "kb"}, nil
|
||||
}
|
||||
|
||||
type imageSyncChunkRepo struct {
|
||||
|
|
@ -165,6 +165,17 @@ func TestUpdateDocumentChunkPreservesGeneratedQuestionsAcrossRevision(t *testing
|
|||
kbRepository: editableChunkKBRepo{},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb",
|
||||
TenantID: 1,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 1,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
newContent := "new body"
|
||||
|
||||
updated, err := service.UpdateDocumentChunk(ctx, "chunk", &newContent, nil, nil)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,143 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
func (s *chunkService) writableChunk(ctx context.Context, id string) (*types.Chunk, error) {
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
chunk, err := s.chunkRepository.GetChunkByID(ctx, tenant, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if chunk == nil || chunk.ID != id || chunk.TenantID != tenant {
|
||||
return nil, apperrors.NewNotFoundError("chunk not found")
|
||||
}
|
||||
knowledge, _, err := loadKnowledgeWrite(ctx, s.knowledgeRepo, s.kbRepository, chunk.KnowledgeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if chunk.KnowledgeBaseID != knowledge.KnowledgeBaseID {
|
||||
return nil, apperrors.NewForbiddenError("chunk does not belong to its knowledge base")
|
||||
}
|
||||
copyOfChunk := *chunk
|
||||
return ©OfChunk, nil
|
||||
}
|
||||
|
||||
func sameChunkDocument(a, b *types.Chunk) bool {
|
||||
return a != nil && b != nil && a.TenantID == b.TenantID &&
|
||||
a.KnowledgeBaseID == b.KnowledgeBaseID && a.KnowledgeID == b.KnowledgeID
|
||||
}
|
||||
|
||||
// Batch validation is completed before calling the repository. Supplied full
|
||||
// rows cannot reparent an existing chunk to a different authorized document.
|
||||
func (s *chunkService) validateChunkWrites(ctx context.Context, chunks []*types.Chunk, create bool) error {
|
||||
ids := make([]string, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
if chunk == nil || chunk.ID == "" {
|
||||
return apperrors.NewBadRequestError("chunk ID cannot be empty")
|
||||
}
|
||||
ids = append(ids, chunk.KnowledgeID)
|
||||
}
|
||||
knowledge, err := loadKnowledgeWriteBatch(ctx, s.knowledgeRepo, s.kbRepository, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
parents := make(map[string]*types.Knowledge, len(knowledge))
|
||||
for _, row := range knowledge {
|
||||
parents[row.ID] = row
|
||||
}
|
||||
storedByID := make(map[string]*types.Chunk)
|
||||
if !create && len(chunks) > 0 {
|
||||
chunkIDs := make([]string, 0, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
chunkIDs = append(chunkIDs, chunk.ID)
|
||||
}
|
||||
stored, err := s.chunkRepository.ListChunksByID(ctx, knowledge[0].TenantID, chunkIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, chunk := range stored {
|
||||
if chunk != nil {
|
||||
storedByID[chunk.ID] = chunk
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, chunk := range chunks {
|
||||
parent := parents[chunk.KnowledgeID]
|
||||
if parent == nil || parent.TenantID != chunk.TenantID || parent.KnowledgeBaseID != chunk.KnowledgeBaseID {
|
||||
return apperrors.NewForbiddenError("chunk does not belong to its knowledge document")
|
||||
}
|
||||
if !create {
|
||||
stored := storedByID[chunk.ID]
|
||||
if stored == nil || stored.ID != chunk.ID || !sameChunkDocument(stored, chunk) {
|
||||
return apperrors.NewForbiddenError("chunk ownership cannot be changed")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *chunkService) writableChunkIDs(ctx context.Context, ids []string) ([]string, error) {
|
||||
ids, err := writeResourceIDs(ids)
|
||||
if err != nil || len(ids) == 0 {
|
||||
return ids, err
|
||||
}
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
chunks, err := s.chunkRepository.ListChunksByID(ctx, tenant, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byID := make(map[string]*types.Chunk, len(chunks))
|
||||
for _, chunk := range chunks {
|
||||
if chunk != nil {
|
||||
byID[chunk.ID] = chunk
|
||||
}
|
||||
}
|
||||
ordered := make([]*types.Chunk, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
if byID[id] == nil {
|
||||
return nil, apperrors.NewNotFoundError("chunk not found")
|
||||
}
|
||||
ordered = append(ordered, byID[id])
|
||||
}
|
||||
// Rows were loaded from storage already; only their parent grants remain.
|
||||
return ids, s.validateChunkWrites(ctx, ordered, true)
|
||||
}
|
||||
|
||||
// Editing a text chunk may also rewrite its parent and image children. Validate
|
||||
// those persisted relationships before saving the first revision.
|
||||
func (s *chunkService) validateDocumentChunkRelations(ctx context.Context, chunk *types.Chunk) error {
|
||||
parents := []*types.Chunk{chunk}
|
||||
if chunk.ParentChunkID != "" {
|
||||
parent, err := s.chunkRepository.GetChunkByID(ctx, chunk.TenantID, chunk.ParentChunkID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if parent == nil || parent.ID != chunk.ParentChunkID || !sameChunkDocument(chunk, parent) {
|
||||
return apperrors.NewForbiddenError("parent chunk does not belong to its document")
|
||||
}
|
||||
parents = append(parents, parent)
|
||||
}
|
||||
for _, parent := range parents {
|
||||
children, err := s.chunkRepository.ListChunkByParentID(ctx, chunk.TenantID, parent.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, child := range children {
|
||||
if !sameChunkDocument(chunk, child) || child.ParentChunkID != parent.ID {
|
||||
return apperrors.NewForbiddenError("child chunk does not belong to its document")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
|
@ -8,6 +8,7 @@ import (
|
|||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/agent/tools"
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
|
|
@ -579,7 +580,7 @@ func (s *customAgentService) getSuggestedQuestions(
|
|||
resolvedTags := resolvedSuggestionTagScopes{}
|
||||
if len(scopeTagIDs) > 0 {
|
||||
var err error
|
||||
resolvedTags, err = s.resolveSuggestionTagScopes(ctx, tenantID, tagScopes)
|
||||
resolvedTags, err = s.resolveSuggestionTagScopes(ctx, tagScopes)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
"agent_id": agentID,
|
||||
|
|
@ -673,7 +674,7 @@ func (s *customAgentService) getSuggestedQuestions(
|
|||
// querying a KB shared from tenant B would hit `tenant_id = A` and get zero
|
||||
// rows back — the symptom is "suggested questions never appear for shared KBs".
|
||||
scopeKBIDs := mergeUniqueStrings(queryKBIDs, resolvedTags.KnowledgeBaseIDs)
|
||||
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, tenantID, scopeKBIDs)
|
||||
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, scopeKBIDs)
|
||||
// Always keep the caller's tenant in the iteration so knowledge_ids-only
|
||||
// requests (no kbIDs) still execute one query under the caller's tenant.
|
||||
if len(scopeKBIDs) == 0 {
|
||||
|
|
@ -834,7 +835,6 @@ type resolvedSuggestionTagScopes struct {
|
|||
// source tenant that owns the tag and chunk rows.
|
||||
func (s *customAgentService) resolveSuggestionTagScopes(
|
||||
ctx context.Context,
|
||||
callerTenantID uint64,
|
||||
tagScopes []types.TagScope,
|
||||
) (resolvedSuggestionTagScopes, error) {
|
||||
result := resolvedSuggestionTagScopes{TagIDsByTenant: make(map[uint64][]string)}
|
||||
|
|
@ -862,7 +862,7 @@ func (s *customAgentService) resolveSuggestionTagScopes(
|
|||
for kbID := range byKB {
|
||||
kbIDs = append(kbIDs, kbID)
|
||||
}
|
||||
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, callerTenantID, kbIDs)
|
||||
kbGroups := s.groupKBIDsByEffectiveTenant(ctx, kbIDs)
|
||||
for tenantID, groupKBIDs := range kbGroups {
|
||||
for _, kbID := range groupKBIDs {
|
||||
requested := mergeUniqueStrings(nil, byKB[kbID])
|
||||
|
|
@ -1083,9 +1083,9 @@ func wikiSuggestionFromPage(page *types.WikiPage, locale string) string {
|
|||
// kbIDs is empty.
|
||||
func (s *customAgentService) groupKBIDsByEffectiveTenant(
|
||||
ctx context.Context,
|
||||
callerTenantID uint64,
|
||||
kbIDs []string,
|
||||
) map[uint64][]string {
|
||||
callerTenantID := types.CallerFromContext(ctx).TenantID
|
||||
out := make(map[uint64][]string)
|
||||
if len(kbIDs) == 0 {
|
||||
return out
|
||||
|
|
@ -1106,20 +1106,13 @@ func (s *customAgentService) groupKBIDsByEffectiveTenant(
|
|||
kbByID[kb.ID] = kb
|
||||
}
|
||||
}
|
||||
callerRole := types.TenantRoleFromContext(ctx)
|
||||
permissions := access.NewKBPermissions(ctx, s.kbShareService)
|
||||
for _, kbID := range kbIDs {
|
||||
kb := kbByID[kbID]
|
||||
if kb == nil {
|
||||
continue
|
||||
}
|
||||
if kb.TenantID == callerTenantID {
|
||||
out[callerTenantID] = append(out[callerTenantID], kbID)
|
||||
continue
|
||||
}
|
||||
if s.kbShareService == nil {
|
||||
continue
|
||||
}
|
||||
ok, err := s.kbShareService.HasTenantKBPermission(ctx, kbID, callerTenantID, callerRole, types.OrgRoleViewer)
|
||||
ok, err := permissions.Check(kbID, kb.TenantID, types.OrgRoleViewer)
|
||||
if err != nil || !ok {
|
||||
continue
|
||||
}
|
||||
|
|
|
|||
|
|
@ -66,14 +66,13 @@ type suggestionKBShareService struct {
|
|||
allowed map[string]bool
|
||||
}
|
||||
|
||||
func (s *suggestionKBShareService) HasTenantKBPermission(
|
||||
func (s *suggestionKBShareService) CheckTenantKBPermission(
|
||||
_ context.Context,
|
||||
kbID string,
|
||||
_ uint64,
|
||||
_ types.TenantRole,
|
||||
_ types.OrgMemberRole,
|
||||
) (bool, error) {
|
||||
return s.allowed[kbID], nil
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
return types.OrgRoleViewer, s.allowed[kbID], nil
|
||||
}
|
||||
|
||||
func TestResolveSuggestionTagScopes_UsesSourceTenantForSharedKB(t *testing.T) {
|
||||
|
|
@ -97,8 +96,7 @@ func TestResolveSuggestionTagScopes_UsesSourceTenantForSharedKB(t *testing.T) {
|
|||
}
|
||||
|
||||
resolved, err := svc.resolveSuggestionTagScopes(
|
||||
context.Background(),
|
||||
callerTenant,
|
||||
context.WithValue(context.Background(), types.TenantIDContextKey, callerTenant),
|
||||
[]types.TagScope{{KnowledgeBaseID: kbID, TagIDs: []string{tagID}}},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/datasource"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
|
||||
|
|
@ -513,8 +514,10 @@ func (s *DataSourceService) ManualSync(ctx context.Context, dsID string) (*types
|
|||
logger.Infof(ctx, "sync task enqueued: ds=%s syncLog=%s", dsID, syncLog.ID)
|
||||
recordKBActivity(ctx, s.audit, ds.TenantID, ds.KnowledgeBaseID, types.AuditActionDataSourceSyncStarted,
|
||||
"data_source", ds.ID, types.AuditOutcomeAccepted,
|
||||
map[string]any{"name": ds.Name, "type": ds.Type, "sync_log_id": syncLog.ID,
|
||||
"task_id": info.ID, "trigger": "manual", "processing_status": "pending"})
|
||||
map[string]any{
|
||||
"name": ds.Name, "type": ds.Type, "sync_log_id": syncLog.ID,
|
||||
"task_id": info.ID, "trigger": "manual", "processing_status": "pending",
|
||||
})
|
||||
return syncLog, nil
|
||||
}
|
||||
|
||||
|
|
@ -627,6 +630,10 @@ func (s *DataSourceService) ProcessSync(ctx context.Context, task *asynq.Task) e
|
|||
return nil
|
||||
}
|
||||
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, ds.TenantID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: data source KB does not belong to its tenant", asynq.SkipRetry)
|
||||
}
|
||||
wasPaused := ds.Status == types.DataSourceStatusPaused
|
||||
|
||||
// Get connector
|
||||
|
|
@ -719,7 +726,7 @@ func (s *DataSourceService) ProcessSync(ctx context.Context, task *asynq.Task) e
|
|||
}
|
||||
|
||||
// Process fetched items and write to knowledge base
|
||||
var result = &types.SyncResult{
|
||||
result := &types.SyncResult{
|
||||
Total: len(items),
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -254,9 +254,15 @@ func (r *processSyncTenantRepo) GetTenantByID(context.Context, uint64) (*types.T
|
|||
|
||||
type processSyncTagService struct {
|
||||
interfaces.KnowledgeTagService
|
||||
ctx context.Context
|
||||
}
|
||||
|
||||
func (*processSyncTagService) FindOrCreateTagByName(context.Context, string, string) (*types.KnowledgeTag, error) {
|
||||
func (s *processSyncTagService) FindOrCreateTagByName(
|
||||
ctx context.Context,
|
||||
_ string,
|
||||
_ string,
|
||||
) (*types.KnowledgeTag, error) {
|
||||
s.ctx = ctx
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,660 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type documentKBLookup struct {
|
||||
interfaces.KnowledgeBaseService
|
||||
values map[string]*types.KnowledgeBase
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *documentKBLookup) GetKnowledgeBaseByID(_ context.Context, id string) (*types.KnowledgeBase, error) {
|
||||
if r.err != nil {
|
||||
return nil, r.err
|
||||
}
|
||||
if r.values[id] == nil {
|
||||
return nil, repository.ErrKnowledgeBaseNotFound
|
||||
}
|
||||
return r.values[id], nil
|
||||
}
|
||||
|
||||
type documentKBRepo struct {
|
||||
interfaces.KnowledgeBaseRepository
|
||||
lookup *documentKBLookup
|
||||
}
|
||||
|
||||
func (r documentKBRepo) GetKnowledgeBaseByID(ctx context.Context, id string) (*types.KnowledgeBase, error) {
|
||||
return r.lookup.GetKnowledgeBaseByID(ctx, id)
|
||||
}
|
||||
|
||||
type documentKnowledgeSpy struct {
|
||||
interfaces.KnowledgeRepository
|
||||
writes int
|
||||
}
|
||||
|
||||
func (r *documentKnowledgeSpy) UpdateKnowledge(ctx context.Context, k *types.Knowledge) error {
|
||||
r.writes++
|
||||
return r.KnowledgeRepository.UpdateKnowledge(ctx, k)
|
||||
}
|
||||
|
||||
func (r *documentKnowledgeSpy) SetKnowledgeTags(ctx context.Context, id string, tags []string) error {
|
||||
r.writes++
|
||||
return r.KnowledgeRepository.SetKnowledgeTags(ctx, id, tags)
|
||||
}
|
||||
|
||||
func (r *documentKnowledgeSpy) DeleteKnowledgeList(ctx context.Context, tenant uint64, ids []string) error {
|
||||
r.writes++
|
||||
return r.KnowledgeRepository.DeleteKnowledgeList(ctx, tenant, ids)
|
||||
}
|
||||
|
||||
type documentChunkSpy struct {
|
||||
interfaces.ChunkRepository
|
||||
writes int
|
||||
imageErr error
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.CreateChunks(ctx, chunks)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.UpdateChunks(ctx, chunks)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) UpdateChunk(ctx context.Context, chunk *types.Chunk) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.UpdateChunk(ctx, chunk)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) DeleteChunks(ctx context.Context, tenant uint64, ids []string) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.DeleteChunks(ctx, tenant, ids)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) DeleteChunk(ctx context.Context, tenant uint64, id string) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.DeleteChunk(ctx, tenant, id)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) DeleteByKnowledgeList(ctx context.Context, tenant uint64, ids []string) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.DeleteByKnowledgeList(ctx, tenant, ids)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) DeleteChunksByKnowledgeID(ctx context.Context, tenant uint64, id string) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.DeleteChunksByKnowledgeID(ctx, tenant, id)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) SaveChunkRevision(
|
||||
ctx context.Context,
|
||||
chunk *types.Chunk,
|
||||
rev *types.ChunkRevision,
|
||||
expected int,
|
||||
) error {
|
||||
r.writes++
|
||||
return r.ChunkRepository.SaveChunkRevision(ctx, chunk, rev, expected)
|
||||
}
|
||||
|
||||
func (r *documentChunkSpy) ListImageInfoByKnowledgeIDs(
|
||||
ctx context.Context,
|
||||
tenant uint64,
|
||||
ids []string,
|
||||
) ([]interfaces.ChunkImageInfo, error) {
|
||||
if r.imageErr != nil {
|
||||
return nil, r.imageErr
|
||||
}
|
||||
return r.ChunkRepository.ListImageInfoByKnowledgeIDs(ctx, tenant, ids)
|
||||
}
|
||||
|
||||
type documentGraphSpy struct {
|
||||
interfaces.RetrieveGraphRepository
|
||||
err error
|
||||
calls int
|
||||
}
|
||||
|
||||
func (r *documentGraphSpy) DelGraph(context.Context, []types.NameSpace) error {
|
||||
r.calls++
|
||||
return r.err
|
||||
}
|
||||
|
||||
type documentTenantSpy struct {
|
||||
interfaces.TenantRepository
|
||||
adjustments []int64
|
||||
}
|
||||
|
||||
func (r *documentTenantSpy) GetTenantByID(_ context.Context, id uint64) (*types.Tenant, error) {
|
||||
return &types.Tenant{ID: id}, nil
|
||||
}
|
||||
|
||||
func (r *documentTenantSpy) AdjustStorageUsed(_ context.Context, _ uint64, delta int64) error {
|
||||
r.adjustments = append(r.adjustments, delta)
|
||||
return nil
|
||||
}
|
||||
|
||||
type documentFileSpy struct {
|
||||
interfaces.FileService
|
||||
deleted []string
|
||||
beforeDelete func()
|
||||
}
|
||||
|
||||
func (r *documentFileSpy) DeleteFile(_ context.Context, path string) error {
|
||||
if r.beforeDelete != nil {
|
||||
r.beforeDelete()
|
||||
}
|
||||
r.deleted = append(r.deleted, path)
|
||||
return nil
|
||||
}
|
||||
|
||||
type documentWriteFixture struct {
|
||||
svc *knowledgeService
|
||||
chunks *chunkService
|
||||
repo *documentKnowledgeSpy
|
||||
chunkRepo *documentChunkSpy
|
||||
kbs *documentKBLookup
|
||||
graph *documentGraphSpy
|
||||
files *documentFileSpy
|
||||
tenants *documentTenantSpy
|
||||
db *gorm.DB
|
||||
ctx context.Context
|
||||
}
|
||||
|
||||
func newDocumentWriteFixture(t *testing.T) *documentWriteFixture {
|
||||
t.Helper()
|
||||
db, err := gorm.Open(sqlite.Open(filepath.Join(t.TempDir(), "document.db")), &gorm.Config{})
|
||||
require.NoError(t, err)
|
||||
sqlDB, err := db.DB()
|
||||
require.NoError(t, err)
|
||||
sqlDB.SetMaxOpenConns(1)
|
||||
t.Cleanup(func() { require.NoError(t, sqlDB.Close()) })
|
||||
require.NoError(
|
||||
t,
|
||||
db.AutoMigrate(
|
||||
&types.Knowledge{},
|
||||
&types.Chunk{},
|
||||
&types.KnowledgeTag{},
|
||||
&types.KnowledgeTagRelation{},
|
||||
&types.ChunkRevision{},
|
||||
),
|
||||
)
|
||||
repo := &documentKnowledgeSpy{KnowledgeRepository: repository.NewKnowledgeRepository(db)}
|
||||
chunkRepo := &documentChunkSpy{ChunkRepository: repository.NewChunkRepository(db)}
|
||||
kbs := &documentKBLookup{values: map[string]*types.KnowledgeBase{
|
||||
"kb": {ID: "kb", TenantID: 7}, "other": {ID: "other", TenantID: 7},
|
||||
}}
|
||||
for _, row := range []*types.Knowledge{
|
||||
{
|
||||
ID: "doc",
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb",
|
||||
Title: "original",
|
||||
Type: types.KnowledgeTypeManual,
|
||||
ParseStatus: types.ManualKnowledgeStatusDraft,
|
||||
StorageSize: 5,
|
||||
},
|
||||
|
||||
{
|
||||
ID: "other-doc",
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "other",
|
||||
Title: "other",
|
||||
ParseStatus: types.ParseStatusCompleted,
|
||||
},
|
||||
} {
|
||||
require.NoError(t, db.Create(row).Error)
|
||||
}
|
||||
for _, row := range []*types.Chunk{
|
||||
{
|
||||
ID: "chunk",
|
||||
TenantID: 7,
|
||||
KnowledgeID: "doc",
|
||||
KnowledgeBaseID: "kb",
|
||||
ChunkType: types.ChunkTypeText,
|
||||
Content: "original",
|
||||
IndexStatus: "ready",
|
||||
},
|
||||
|
||||
{
|
||||
ID: "other-chunk",
|
||||
TenantID: 7,
|
||||
KnowledgeID: "other-doc",
|
||||
KnowledgeBaseID: "other",
|
||||
ChunkType: types.ChunkTypeText,
|
||||
},
|
||||
} {
|
||||
require.NoError(t, db.Create(row).Error)
|
||||
}
|
||||
require.NoError(t, db.Create(&types.KnowledgeTag{ID: "tag", TenantID: 7, KnowledgeBaseID: "kb", Name: "tag"}).Error)
|
||||
graph, files, tenants := &documentGraphSpy{}, &documentFileSpy{}, &documentTenantSpy{}
|
||||
chunks := &chunkService{chunkRepository: chunkRepo, knowledgeRepo: repo, kbRepository: documentKBRepo{lookup: kbs}}
|
||||
svc := &knowledgeService{
|
||||
repo: repo,
|
||||
chunkRepo: chunkRepo,
|
||||
chunkService: chunks,
|
||||
kbService: kbs,
|
||||
graphEngine: graph,
|
||||
fileSvc: files,
|
||||
tenantRepo: tenants,
|
||||
tagRepo: repository.NewKnowledgeTagRepository(db),
|
||||
}
|
||||
ctx := types.WithCaller(
|
||||
context.Background(),
|
||||
types.Caller{TenantID: 1, UserID: "editor", Role: types.TenantRoleAdmin},
|
||||
)
|
||||
grant := &access.KBAccess{
|
||||
KnowledgeBase: kbs.values["kb"],
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}
|
||||
ctx = grant.Context(ctx)
|
||||
ctx = context.WithValue(ctx, types.TenantInfoContextKey, &types.Tenant{ID: 7})
|
||||
return &documentWriteFixture{
|
||||
svc: svc,
|
||||
chunks: chunks,
|
||||
repo: repo,
|
||||
chunkRepo: chunkRepo,
|
||||
kbs: kbs,
|
||||
graph: graph,
|
||||
files: files,
|
||||
tenants: tenants,
|
||||
db: db,
|
||||
ctx: ctx,
|
||||
}
|
||||
}
|
||||
|
||||
func requireForbiddenWrite(t *testing.T, err error) {
|
||||
t.Helper()
|
||||
var appErr *apperrors.AppError
|
||||
require.ErrorAs(t, err, &appErr)
|
||||
require.Equal(t, 403, appErr.HTTPCode)
|
||||
}
|
||||
|
||||
func (f *documentWriteFixture) requireNoWrites(t *testing.T) {
|
||||
t.Helper()
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
require.Empty(t, f.files.deleted)
|
||||
}
|
||||
|
||||
func TestDocumentAndChunkWritesRequireOperationGrant(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
ctx := types.WithExecutionTenant(
|
||||
types.WithCaller(context.Background(), types.Caller{TenantID: 7, Role: types.TenantRoleAdmin}),
|
||||
7,
|
||||
)
|
||||
// Even an owner read resolution projects Admin without authorizing writes.
|
||||
grant, err := access.ResolveKB(
|
||||
ctx,
|
||||
access.KBRequest{Caller: types.CallerFromContext(ctx)},
|
||||
f.kbs.values["kb"],
|
||||
types.OrgRoleViewer,
|
||||
nil,
|
||||
nil,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
ctx = grant.Context(ctx)
|
||||
row, _ := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
ops := map[string]func() error{
|
||||
"set tags": func() error { return f.svc.SetKnowledgeTags(ctx, "doc", nil) },
|
||||
"move folder": func() error {
|
||||
_,
|
||||
err := f.svc.MoveKnowledgeToFolder(ctx,
|
||||
"kb",
|
||||
[]string{"doc"},
|
||||
"folder")
|
||||
return err
|
||||
},
|
||||
|
||||
"rename folder": func() error {
|
||||
_,
|
||||
err := f.svc.RenameKnowledgeFolder(ctx,
|
||||
"kb",
|
||||
"folder",
|
||||
"next")
|
||||
return err
|
||||
},
|
||||
|
||||
"metadata": func() error { return f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "new"}) },
|
||||
"manual": func() error {
|
||||
_, err := f.svc.UpdateManualKnowledge(
|
||||
ctx,
|
||||
"doc",
|
||||
&types.ManualKnowledgePayload{Content: "body", Status: "draft"},
|
||||
)
|
||||
return err
|
||||
},
|
||||
"tag": func() error { return f.svc.UpdateKnowledgeTag(ctx, "doc", []string{"tag"}) },
|
||||
"batch tags": func() error {
|
||||
return f.svc.UpdateKnowledgeTagBatch(ctx,
|
||||
"",
|
||||
map[string][]string{"doc": {"tag"}})
|
||||
},
|
||||
|
||||
"delete document": func() error { return f.svc.DeleteKnowledge(ctx, "doc") },
|
||||
"delete documents": func() error { return f.svc.DeleteKnowledgeList(ctx, []string{"doc"}) },
|
||||
"create chunks": func() error { return f.chunks.CreateChunks(ctx, []*types.Chunk{row}) },
|
||||
"update chunk": func() error { return f.chunks.UpdateChunk(ctx, row) },
|
||||
"update chunks": func() error { return f.chunks.UpdateChunks(ctx, []*types.Chunk{row}) },
|
||||
"edit chunk": func() error {
|
||||
body := "new"
|
||||
_, err := f.chunks.UpdateDocumentChunk(ctx, "chunk", &body, nil, nil)
|
||||
return err
|
||||
},
|
||||
"upsert question": func() error {
|
||||
_,
|
||||
err := f.chunks.UpsertGeneratedQuestion(ctx,
|
||||
"chunk",
|
||||
"",
|
||||
"new question")
|
||||
return err
|
||||
},
|
||||
|
||||
"delete question": func() error { return f.chunks.DeleteGeneratedQuestion(ctx, "chunk", "q") },
|
||||
"regenerate questions": func() error {
|
||||
_,
|
||||
err := f.svc.RegenerateChunkQuestions(ctx,
|
||||
"chunk")
|
||||
return err
|
||||
},
|
||||
|
||||
"update image": func() error { return f.svc.UpdateImageInfo(ctx, "doc", "chunk", "[]") },
|
||||
"delete chunk": func() error { return f.chunks.DeleteChunk(ctx, "chunk") },
|
||||
"delete chunks": func() error { return f.chunks.DeleteChunks(ctx, []string{"chunk"}) },
|
||||
"delete document chunks": func() error { return f.chunks.DeleteChunksByKnowledgeID(ctx, "doc") },
|
||||
"delete batch document chunks": func() error { return f.chunks.DeleteByKnowledgeList(ctx, []string{"doc"}) },
|
||||
}
|
||||
for name, op := range ops {
|
||||
t.Run(name, func(t *testing.T) { requireForbiddenWrite(t, op()); f.requireNoWrites(t) })
|
||||
}
|
||||
require.NoError(t, f.svc.UpdateKnowledge(f.ctx, &types.Knowledge{ID: "doc", Title: "shared edit"}))
|
||||
stored, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "shared edit", stored.Title)
|
||||
}
|
||||
|
||||
func TestDocumentBatchPreflightDoesNotPartiallyMutate(t *testing.T) {
|
||||
for _, invalid := range []string{"other-doc", "missing", ""} {
|
||||
t.Run(invalid, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.Error(t, f.svc.DeleteKnowledgeList(f.ctx, []string{"doc", invalid}))
|
||||
f.requireNoWrites(t)
|
||||
require.Error(
|
||||
t,
|
||||
f.svc.UpdateKnowledgeTagBatch(f.ctx, "", map[string][]string{"doc": {"tag"}, invalid: nil}),
|
||||
)
|
||||
f.requireNoWrites(t)
|
||||
_, err := f.svc.MoveKnowledgeToFolder(f.ctx, "kb", []string{"doc", invalid}, "next")
|
||||
require.Error(t, err)
|
||||
f.requireNoWrites(t)
|
||||
require.Error(t, f.chunks.DeleteByKnowledgeList(f.ctx, []string{"doc", invalid}))
|
||||
f.requireNoWrites(t)
|
||||
})
|
||||
}
|
||||
for _, invalid := range []string{"other-chunk", "missing", ""} {
|
||||
t.Run("chunks/"+invalid, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.Error(t, f.chunks.DeleteChunks(f.ctx, []string{"chunk", invalid}))
|
||||
f.requireNoWrites(t)
|
||||
good, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
require.NoError(t, err)
|
||||
forged := *good
|
||||
forged.ID = invalid
|
||||
require.Error(t, f.chunks.UpdateChunks(f.ctx, []*types.Chunk{good, &forged}))
|
||||
f.requireNoWrites(t)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeletePreflightResolvesKBAndImagesBeforeChangingStatus(t *testing.T) {
|
||||
for _, stage := range []string{"KB lookup", "KB owner", "image lookup"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
switch stage {
|
||||
case "KB lookup":
|
||||
f.kbs.err = errors.New("database unavailable")
|
||||
case "KB owner":
|
||||
f.kbs.values["kb"].TenantID = 9
|
||||
case "image lookup":
|
||||
f.chunkRepo.imageErr = errors.New("database unavailable")
|
||||
}
|
||||
require.Error(t, f.svc.DeleteKnowledge(f.ctx, "doc"))
|
||||
f.requireNoWrites(t)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.ManualKnowledgeStatusDraft, row.ParseStatus)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func deleteDocumentTask(t *testing.T, payload types.KnowledgeListDeletePayload) *asynq.Task {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(payload)
|
||||
require.NoError(t, err)
|
||||
return asynq.NewTask(types.TypeKnowledgeListDelete, data)
|
||||
}
|
||||
|
||||
func TestDeleteTaskScopeRejectsMovedDocumentsBeforeAnyMutation(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
task := deleteDocumentTask(
|
||||
t,
|
||||
types.KnowledgeListDeletePayload{
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb",
|
||||
KnowledgeIDs: []string{"doc", "other-doc"},
|
||||
},
|
||||
)
|
||||
require.ErrorIs(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task), asynq.SkipRetry)
|
||||
f.requireNoWrites(t)
|
||||
task = deleteDocumentTask(
|
||||
t,
|
||||
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeIDs: []string{"doc", "other-doc"}},
|
||||
)
|
||||
require.ErrorIs(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task), asynq.SkipRetry)
|
||||
f.requireNoWrites(t)
|
||||
}
|
||||
|
||||
func TestDeleteTaskRetriesAfterPartialCleanupAndAlreadyDeletedRows(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("file_path", "local://doc").Error)
|
||||
task := deleteDocumentTask(
|
||||
t,
|
||||
types.KnowledgeListDeletePayload{
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb",
|
||||
KnowledgeIDs: []string{"doc", "doc", "already-gone"},
|
||||
},
|
||||
)
|
||||
f.graph.err = errors.New("graph unavailable")
|
||||
err := f.svc.ProcessKnowledgeListDelete(context.Background(), task)
|
||||
require.ErrorContains(t, err, "graph unavailable")
|
||||
require.NotErrorIs(t, err, asynq.SkipRetry)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.ParseStatusDeleting, row.ParseStatus)
|
||||
require.Empty(t, f.files.deleted)
|
||||
require.Empty(t, f.tenants.adjustments)
|
||||
f.graph.err = nil
|
||||
f.files.beforeDelete = func() { _, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc"); require.Error(t, err) }
|
||||
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
|
||||
require.Equal(t, []string{"local://doc"}, f.files.deleted)
|
||||
require.Equal(t, []int64{-5}, f.tenants.adjustments)
|
||||
// A completed task does not need live KB metadata or current user grants.
|
||||
delete(f.kbs.values, "kb")
|
||||
writes, calls := f.repo.writes, f.graph.calls
|
||||
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
|
||||
require.Equal(t, writes, f.repo.writes)
|
||||
require.Equal(t, calls, f.graph.calls)
|
||||
}
|
||||
|
||||
func TestKnowledgeCleanupScopeCannotAuthorizeGeneralWritesOrExtraIDs(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
ctx := withKnowledgeCleanup(context.Background(), 7, map[string]string{"doc": "kb"})
|
||||
require.Zero(t, types.CallerFromContext(ctx).TenantID)
|
||||
requireForbiddenWrite(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "no"}))
|
||||
requireForbiddenWrite(t, f.svc.DeleteKnowledgeList(ctx, []string{"doc", "other-doc"}))
|
||||
requireForbiddenWrite(
|
||||
t,
|
||||
f.svc.DeleteKnowledgeList(types.WithCaller(ctx, types.Caller{TenantID: 7}), []string{"doc"}),
|
||||
)
|
||||
f.requireNoWrites(t)
|
||||
require.NoError(t, f.svc.DeleteKnowledge(ctx, "doc"))
|
||||
}
|
||||
|
||||
func TestReferencedKnowledgeCleanupPinsItsIndependentKB(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
requireForbiddenWrite(t, deleteReferencedKnowledge(f.ctx, f.svc, "kb", []string{"doc", "other-doc"}))
|
||||
f.requireNoWrites(t)
|
||||
// Session cleanup authority comes from exact persisted references and can
|
||||
// continue even when the caller has no permission to edit the history KB.
|
||||
ctx := types.WithExecutionTenant(types.WithCaller(context.Background(), types.Caller{TenantID: 7}), 7)
|
||||
require.NoError(t, deleteReferencedKnowledge(ctx, f.svc, "kb", []string{"doc", "already-gone"}))
|
||||
}
|
||||
|
||||
func TestChunkEditRejectsCrossDocumentParentBeforeSavingRevision(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("parent_chunk_id", "other-chunk").Error,
|
||||
)
|
||||
body := "edit"
|
||||
_, err := f.chunks.UpdateDocumentChunk(f.ctx, "chunk", &body, nil, nil)
|
||||
requireForbiddenWrite(t, err)
|
||||
f.requireNoWrites(t)
|
||||
}
|
||||
|
||||
func TestChunkEditRejectsCrossDocumentChildBeforeSavingRevision(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Chunk{}).Where("id = ?", "other-chunk").Update("parent_chunk_id", "chunk").Error,
|
||||
)
|
||||
body := "edit"
|
||||
_, err := f.chunks.UpdateDocumentChunk(f.ctx, "chunk", &body, nil, nil)
|
||||
requireForbiddenWrite(t, err)
|
||||
f.requireNoWrites(t)
|
||||
}
|
||||
|
||||
func TestChunkBatchWriteAllowsOnlyStoredOwnership(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
row, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
require.NoError(t, err)
|
||||
row.Content = "updated"
|
||||
require.NoError(t, f.chunks.UpdateChunks(f.ctx, []*types.Chunk{row}))
|
||||
stored, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "updated", stored.Content)
|
||||
require.NoError(t, f.chunks.DeleteChunks(f.ctx, []string{"chunk", "chunk"}))
|
||||
_, err = f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
require.ErrorIs(t, err, repository.ErrChunkNotFound)
|
||||
}
|
||||
|
||||
func TestDocumentWriteTaskAndAPIKeyScopesStayNarrow(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
ctx, err := access.WithKBTaskWrite(context.Background(), f.kbs.values["kb"], 7)
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "doc", Title: "worker edit"}))
|
||||
requireForbiddenWrite(t, f.svc.UpdateKnowledge(ctx, &types.Knowledge{ID: "other-doc", Title: "no"}))
|
||||
scope := types.TenantAPIKeyScope{
|
||||
KnowledgeBaseIDs: types.StringArray{"kb"},
|
||||
Capabilities: types.StringArray{string(types.APIKeyCapabilityRetrieve)},
|
||||
}
|
||||
requireForbiddenWrite(
|
||||
t,
|
||||
f.svc.UpdateKnowledge(types.WithTenantAPIKeyScope(f.ctx, scope), &types.Knowledge{ID: "doc", Title: "no"}),
|
||||
)
|
||||
require.Zero(t, types.CallerFromContext(ctx).TenantID)
|
||||
}
|
||||
|
||||
func TestLegacySingleKBDeleteTaskRemainsExecutable(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
task := deleteDocumentTask(
|
||||
t,
|
||||
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeIDs: []string{"doc", "already-gone"}},
|
||||
)
|
||||
require.NoError(t, f.svc.ProcessKnowledgeListDelete(context.Background(), task))
|
||||
_, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestMultiKBDeleteRequiresAndAcceptsEveryKBGrant(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
grant := &access.KBAccess{
|
||||
KnowledgeBase: f.kbs.values["other"],
|
||||
Caller: types.CallerFromContext(f.ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}
|
||||
require.NoError(t, f.svc.DeleteKnowledgeList(grant.Context(f.ctx), []string{"doc", "other-doc"}))
|
||||
rows, err := f.repo.GetKnowledgeBatch(f.ctx, 7, []string{"doc", "other-doc"})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, rows)
|
||||
}
|
||||
|
||||
func TestDeleteCheckpointRejectsChangedOrDeletedDocument(t *testing.T) {
|
||||
for _, action := range []string{"moved", "deleted"} {
|
||||
t.Run(action, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
plan, err := f.svc.planKnowledgeDelete(f.ctx, []string{"doc"})
|
||||
require.NoError(t, err)
|
||||
if action == "moved" {
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error,
|
||||
)
|
||||
} else {
|
||||
require.NoError(t, f.db.Where("id = ?", "doc").Delete(&types.Knowledge{}).Error)
|
||||
}
|
||||
require.Error(t, f.svc.executeKnowledgeDelete(plan, true))
|
||||
require.Zero(t, f.graph.calls)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Empty(t, f.files.deleted)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
if action == "moved" {
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", row.KnowledgeBaseID)
|
||||
require.Equal(t, types.ManualKnowledgeStatusDraft, row.ParseStatus)
|
||||
} else {
|
||||
require.Error(t, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDocumentWritesRejectMoveInProgress(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
|
||||
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("metadata", row.Metadata).Error)
|
||||
for _, op := range []func() error{
|
||||
func() error { return f.svc.DeleteKnowledge(f.ctx, "doc") },
|
||||
func() error { return f.svc.UpdateKnowledge(f.ctx, &types.Knowledge{ID: "doc", Title: "changed"}) },
|
||||
func() error { return f.chunks.DeleteChunk(f.ctx, "chunk") },
|
||||
func() error { return f.svc.DeleteKnowledgeList(f.ctx, []string{"doc"}) },
|
||||
} {
|
||||
var appErr *apperrors.AppError
|
||||
require.ErrorAs(t, op(), &appErr)
|
||||
require.Equal(t, 409, appErr.HTTPCode)
|
||||
}
|
||||
f.requireNoWrites(t)
|
||||
}
|
||||
|
|
@ -363,7 +363,10 @@ func (e *EvaluationService) EvalDataset(ctx context.Context, detail *types.Evalu
|
|||
// Setup cleanup of temporary resources
|
||||
defer func() {
|
||||
logger.Infof(ctx, "Cleaning up resources - deleting knowledge: %s", knowledge.ID)
|
||||
if err := e.knowledgeService.DeleteKnowledge(ctx, knowledge.ID); err != nil {
|
||||
if err := deleteReferencedKnowledge(ctx,
|
||||
e.knowledgeService,
|
||||
knowledgeBaseID,
|
||||
[]string{knowledge.ID}); err != nil {
|
||||
logger.Errorf(ctx, "Failed to delete knowledge: %v, knowledge ID: %s", err, knowledge.ID)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import (
|
|||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/agent/tools"
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
chatpipeline "github.com/Tencent/WeKnora/internal/application/service/chat_pipeline"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
"github.com/Tencent/WeKnora/internal/config"
|
||||
|
|
@ -468,7 +469,7 @@ func (s *DataTableSummaryService) Handle(ctx context.Context, t *asynq.Task) err
|
|||
|
||||
ctx = logger.WithRequestID(ctx, uuid.New().String())
|
||||
ctx = logger.WithField(ctx, "knowledge", payload.KnowledgeID)
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
|
||||
logger.Infof(ctx, "Processing table extraction for knowledge: %s", payload.KnowledgeID)
|
||||
|
||||
|
|
@ -478,6 +479,12 @@ func (s *DataTableSummaryService) Handle(ctx context.Context, t *asynq.Task) err
|
|||
return err
|
||||
}
|
||||
|
||||
ctx, err = access.WithKBTaskWrite(ctx, resources.knowledgeBase, payload.TenantID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx = context.WithValue(ctx, types.TenantInfoContextKey, resources.tenant)
|
||||
|
||||
// 3. 加载表格数据并生成摘要
|
||||
chunks, err := s.processTableData(ctx, resources)
|
||||
if err != nil {
|
||||
|
|
@ -508,12 +515,20 @@ type extractionResources struct {
|
|||
// 思路:集中加载所有依赖,统一错误处理,避免分散的资源获取逻辑
|
||||
func (s *DataTableSummaryService) prepareResources(ctx context.Context, payload DataTableSummaryPayload) (*extractionResources, error) {
|
||||
// 获取并验证知识文件
|
||||
knowledge, err := s.knowledgeService.GetKnowledgeByID(ctx, payload.KnowledgeID)
|
||||
knowledge, err := s.knowledgeService.GetRepository().GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "failed to get knowledge: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if knowledge == nil || knowledge.ID != payload.KnowledgeID || knowledge.TenantID != payload.TenantID {
|
||||
return nil, fmt.Errorf("invalid table summary knowledge scope")
|
||||
}
|
||||
kb, err := knowledgeWriteKB(ctx, s.knowledgeBaseService, knowledge)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// 验证文件类型
|
||||
fileType := strings.ToLower(knowledge.FileType)
|
||||
if fileType != "csv" && fileType != "xlsx" && fileType != "xls" {
|
||||
|
|
@ -542,13 +557,6 @@ func (s *DataTableSummaryService) prepareResources(ctx context.Context, payload
|
|||
return nil, err
|
||||
}
|
||||
|
||||
// Load the KB to discover its VectorStoreID binding so the factory can
|
||||
// route to the bound store (or fall back to tenant engines if unbound).
|
||||
kb, err := s.knowledgeBaseService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "failed to get knowledge base for vector store lookup: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
var vectorStoreID *string
|
||||
if kb != nil {
|
||||
vectorStoreID = kb.VectorStoreID
|
||||
|
|
@ -774,12 +782,12 @@ func (s *DataTableSummaryService) cleanupOnFailure(ctx context.Context, resource
|
|||
logger.Warnf(ctx, "Starting cleanup due to failure: %v", indexErr)
|
||||
|
||||
// 1. 更新知识状态为失败
|
||||
resources.knowledge.ParseStatus = types.ParseStatusFailed
|
||||
resources.knowledge.ErrorMessage = indexErr.Error()
|
||||
if err := s.knowledgeService.UpdateKnowledge(ctx, resources.knowledge); err != nil {
|
||||
logger.Errorf(ctx, "Failed to update knowledge status: %v", err)
|
||||
} else {
|
||||
logger.Infof(ctx, "Updated knowledge %s status to failed", resources.knowledge.ID)
|
||||
before, after := *resources.knowledge, *resources.knowledge
|
||||
after.ParseStatus = types.ParseStatusFailed
|
||||
after.ErrorMessage = indexErr.Error()
|
||||
if err := s.knowledgeService.GetRepository().UpdateKnowledgeForTransfer(ctx, &before, &after); err != nil {
|
||||
logger.Warnf(ctx, "Table summary cleanup skipped after knowledge changed: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
// 提取chunk IDs
|
||||
|
|
@ -790,7 +798,7 @@ func (s *DataTableSummaryService) cleanupOnFailure(ctx context.Context, resource
|
|||
|
||||
// 删除已创建的chunks
|
||||
if len(chunkIDs) > 0 {
|
||||
if err := s.chunkService.DeleteChunks(ctx, chunkIDs); err != nil {
|
||||
if err := s.chunkService.GetRepository().DeleteChunks(ctx, resources.knowledge.TenantID, chunkIDs); err != nil {
|
||||
logger.Errorf(ctx, "Failed to delete chunks: %v", err)
|
||||
} else {
|
||||
logger.Infof(ctx, "Deleted %d chunks", len(chunkIDs))
|
||||
|
|
|
|||
|
|
@ -0,0 +1,69 @@
|
|||
package file
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
||||
// ResolveTenantFileServiceWithFallback applies the storage resolution and
|
||||
// fallback rule shared by file proxies and artifact downloads: when tenant-level
|
||||
// resolution fails and the requested provider matches the process-global
|
||||
// STORAGE_TYPE, serve through the global file service instead of failing.
|
||||
// Returns ok=false (already logged) when no service can be resolved; the
|
||||
// caller decides the HTTP status.
|
||||
func ResolveTenantFileServiceWithFallback(
|
||||
ctx context.Context,
|
||||
logTag string,
|
||||
tenant *types.Tenant,
|
||||
backendID, provider, absDir string,
|
||||
storageResolver interfaces.StorageBackendResolver,
|
||||
globalFileService interfaces.FileService,
|
||||
) (fileSvc interfaces.FileService, resolvedProvider string, ok bool) {
|
||||
var err error
|
||||
if storageResolver != nil {
|
||||
fileSvc, resolvedProvider, err = storageResolver.ResolveFileService(ctx, tenant, backendID, provider, absDir)
|
||||
} else if tenant.StorageEngineConfig != nil {
|
||||
fileSvc, resolvedProvider, err = NewFileServiceFromStorageConfig(provider, tenant.StorageEngineConfig, absDir)
|
||||
} else {
|
||||
err = http.ErrMissingFile
|
||||
}
|
||||
if err == nil {
|
||||
return fileSvc, resolvedProvider, true
|
||||
}
|
||||
|
||||
globalStorageType := strings.ToLower(strings.TrimSpace(os.Getenv("STORAGE_TYPE")))
|
||||
if globalStorageType == "" {
|
||||
globalStorageType = "local"
|
||||
}
|
||||
if provider == globalStorageType && globalFileService != nil {
|
||||
logger.Warnf(
|
||||
ctx,
|
||||
"[Router] %s tenant storage config missing or invalid, fallback to global file "+
|
||||
"service: tenant_id=%d provider=%s err=%v",
|
||||
|
||||
logTag,
|
||||
tenant.ID,
|
||||
provider,
|
||||
err,
|
||||
)
|
||||
return globalFileService, globalStorageType, true
|
||||
}
|
||||
logger.Warnf(
|
||||
ctx,
|
||||
"[Router] %s resolve file service failed without fallback: tenant_id=%d "+
|
||||
"provider=%s global_storage_type=%s err=%v",
|
||||
|
||||
logTag,
|
||||
tenant.ID,
|
||||
provider,
|
||||
globalStorageType,
|
||||
err,
|
||||
)
|
||||
return nil, "", false
|
||||
}
|
||||
|
|
@ -342,7 +342,7 @@ func (s *ImageMultimodalService) Handle(ctx context.Context, task *asynq.Task) e
|
|||
}
|
||||
|
||||
// Persist chunks
|
||||
if err := s.chunkService.CreateChunks(ctx, newChunks); err != nil {
|
||||
if err := s.chunkService.GetRepository().CreateChunks(ctx, newChunks); err != nil {
|
||||
handleErr = fmt.Errorf("create multimodal chunks: %w", err)
|
||||
return handleErr
|
||||
}
|
||||
|
|
@ -446,7 +446,7 @@ func (s *ImageMultimodalService) indexChunks(ctx context.Context, payload types.
|
|||
continue
|
||||
}
|
||||
dbChunk.Status = int(types.ChunkStatusIndexed)
|
||||
if uerr := s.chunkService.UpdateChunk(ctx, dbChunk); uerr != nil {
|
||||
if uerr := s.chunkService.GetRepository().UpdateChunk(ctx, dbChunk); uerr != nil {
|
||||
logger.Warnf(ctx, "[ImageMultimodal] Failed to update chunk %s status to indexed: %v", chunk.ID, uerr)
|
||||
}
|
||||
}
|
||||
|
|
@ -503,7 +503,7 @@ func (s *ImageMultimodalService) indexChunks(ctx context.Context, payload types.
|
|||
continue
|
||||
}
|
||||
dbChunk.Status = int(types.ChunkStatusIndexed)
|
||||
if err := s.chunkService.UpdateChunk(ctx, dbChunk); err != nil {
|
||||
if err := s.chunkService.GetRepository().UpdateChunk(ctx, dbChunk); err != nil {
|
||||
logger.Warnf(ctx, "[ImageMultimodal] Failed to update chunk %s status to indexed: %v", chunk.ID, err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"errors"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
|
|
@ -489,14 +490,7 @@ func (s *kbShareService) CheckTenantKBPermission(ctx context.Context, kbID strin
|
|||
// HasTenantKBPermission is a thin "do I have at least N" wrapper over
|
||||
// CheckTenantKBPermission for callers that don't need the granular role.
|
||||
func (s *kbShareService) HasTenantKBPermission(ctx context.Context, kbID string, callerTenantID uint64, callerTenantRole types.TenantRole, requiredRole types.OrgMemberRole) (bool, error) {
|
||||
role, isShared, err := s.CheckTenantKBPermission(ctx, kbID, callerTenantID, callerTenantRole)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !isShared {
|
||||
return false, nil
|
||||
}
|
||||
return role.HasPermission(requiredRole), nil
|
||||
return access.NewKBSharePermissions(ctx, s, callerTenantID, callerTenantRole).Check(kbID, requiredRole)
|
||||
}
|
||||
|
||||
// GetKBSourceTenant gets the source tenant ID for a shared knowledge base
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import (
|
|||
"fmt"
|
||||
"io"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
|
@ -592,7 +593,18 @@ func (s *knowledgeService) MoveKnowledgeToFolder(ctx context.Context,
|
|||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
rows, err := loadKnowledgeWriteBatch(ctx, s.repo, s.kbService, ids)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
ids = make([]string, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
if row.KnowledgeBaseID != kbID {
|
||||
return 0, werrors.NewForbiddenError("knowledge outside target KB")
|
||||
}
|
||||
ids = append(ids, row.ID)
|
||||
}
|
||||
tenantID := rows[0].TenantID
|
||||
affected, err := s.repo.UpdateKnowledgeFolderPath(ctx, tenantID, kbID, ids, normalized)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to move knowledge to folder %q: %v", normalized, err)
|
||||
|
|
@ -627,7 +639,18 @@ func (s *knowledgeService) RenameKnowledgeFolder(ctx context.Context,
|
|||
return 0, werrors.NewBadRequestError("不能将文件夹移动到它自己的子目录下")
|
||||
}
|
||||
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, kbID)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if kb == nil || kb.ID != kbID {
|
||||
return 0, werrors.NewNotFoundError("knowledge base not found")
|
||||
}
|
||||
ctx, err = requireKBWrite(ctx, kb)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
tenantID := kb.TenantID
|
||||
affected, err := s.repo.RenameKnowledgeFolderPath(ctx, tenantID, kbID, source, target)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to rename folder %q to %q: %v", source, target, err)
|
||||
|
|
@ -689,7 +712,10 @@ func (s *knowledgeService) GetKnowledgeFile(ctx context.Context, id string) (io.
|
|||
}
|
||||
|
||||
func (s *knowledgeService) UpdateKnowledge(ctx context.Context, knowledge *types.Knowledge) error {
|
||||
record, err := s.repo.GetKnowledgeByID(ctx, ctx.Value(types.TenantIDContextKey).(uint64), knowledge.ID)
|
||||
if knowledge == nil {
|
||||
return werrors.NewBadRequestError("knowledge cannot be nil")
|
||||
}
|
||||
record, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledge.ID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get knowledge record: %v", err)
|
||||
return err
|
||||
|
|
@ -769,48 +795,48 @@ func (s *knowledgeService) GetKnowledgeBatchWithSharedAccess(ctx context.Context
|
|||
if len(ids) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
ownList, err := s.repo.GetKnowledgeBatch(ctx, tenantID, ids)
|
||||
permissions := kbReadPermissions(ctx, s.kbShareService)
|
||||
rows, err := s.repo.GetKnowledgeBatch(ctx, tenantID, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ownList := make([]*types.Knowledge, 0, len(rows))
|
||||
foundSet := make(map[string]bool)
|
||||
for _, k := range ownList {
|
||||
appendAllowed := func(k *types.Knowledge) {
|
||||
if k == nil || foundSet[k.ID] {
|
||||
return
|
||||
}
|
||||
allowed, err := permissions.Check(k.KnowledgeBaseID, k.TenantID, types.OrgRoleViewer)
|
||||
if err == nil && allowed {
|
||||
ownList = append(ownList, k)
|
||||
foundSet[k.ID] = true
|
||||
}
|
||||
}
|
||||
for _, k := range rows {
|
||||
appendAllowed(k)
|
||||
if k != nil {
|
||||
foundSet[k.ID] = true
|
||||
}
|
||||
}
|
||||
userIDVal := ctx.Value(types.UserIDContextKey)
|
||||
if userIDVal == nil {
|
||||
return ownList, nil
|
||||
}
|
||||
userID, ok := userIDVal.(string)
|
||||
if !ok || userID == "" {
|
||||
return ownList, nil
|
||||
}
|
||||
// Plan 3: shared-KB permission is keyed on (tenant, tenant_role)
|
||||
// rather than user. callerTenantRole drives the 3-D cap.
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
for _, id := range ids {
|
||||
if foundSet[id] {
|
||||
continue
|
||||
}
|
||||
k, err := s.repo.GetKnowledgeByIDOnly(ctx, id)
|
||||
if err != nil || k == nil || k.KnowledgeBaseID == "" {
|
||||
continue
|
||||
if err != nil && !errors.Is(err, repository.ErrKnowledgeNotFound) {
|
||||
return nil, err
|
||||
}
|
||||
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, k.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
|
||||
if err != nil || !hasPermission {
|
||||
continue
|
||||
if err == nil {
|
||||
appendAllowed(k)
|
||||
}
|
||||
foundSet[k.ID] = true
|
||||
ownList = append(ownList, k)
|
||||
foundSet[id] = true
|
||||
}
|
||||
return ownList, nil
|
||||
}
|
||||
|
||||
// SetKnowledgeTags replaces all tags for a single knowledge entry.
|
||||
func (s *knowledgeService) SetKnowledgeTags(ctx context.Context, knowledgeID string, tagIDs []string) error {
|
||||
return s.repo.SetKnowledgeTags(ctx, knowledgeID, tagIDs)
|
||||
return s.UpdateKnowledgeTag(ctx, knowledgeID, tagIDs)
|
||||
}
|
||||
|
||||
// ListKnowledgeIDsByTagIDs returns document knowledge IDs carrying any of the
|
||||
|
|
@ -909,11 +935,11 @@ func (s *knowledgeService) GetKnowledgeTags(ctx context.Context, knowledgeIDs []
|
|||
|
||||
// UpdateKnowledgeTag updates the tags assigned to a knowledge document.
|
||||
func (s *knowledgeService) UpdateKnowledgeTag(ctx context.Context, knowledgeID string, tagIDs []string) error {
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
|
||||
knowledge, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tenantID := knowledge.TenantID
|
||||
|
||||
// Validate all tag IDs
|
||||
if err := s.validateKnowledgeTagIDs(ctx, tenantID, knowledge.KnowledgeBaseID, tagIDs); err != nil {
|
||||
|
|
@ -925,29 +951,25 @@ func (s *knowledgeService) UpdateKnowledgeTag(ctx context.Context, knowledgeID s
|
|||
|
||||
// UpdateKnowledgeTagBatch updates tags for document knowledge items in batch.
|
||||
// authorizedKBID restricts all updates to knowledge items belonging to this KB;
|
||||
// pass empty string to skip the check (caller must ensure authorization by other means).
|
||||
func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authorizedKBID string, updates map[string][]string) error {
|
||||
// an empty value allows multiple KBs only when every KB has an explicit write grant.
|
||||
func (s *knowledgeService) UpdateKnowledgeTagBatch(
|
||||
ctx context.Context,
|
||||
authorizedKBID string,
|
||||
updates map[string][]string,
|
||||
) error {
|
||||
if len(updates) == 0 {
|
||||
return nil
|
||||
}
|
||||
tenantIDVal := ctx.Value(types.TenantIDContextKey)
|
||||
if tenantIDVal == nil {
|
||||
return werrors.NewUnauthorizedError("workspace ID not found in context")
|
||||
}
|
||||
tenantID, ok := tenantIDVal.(uint64)
|
||||
if !ok {
|
||||
return werrors.NewUnauthorizedError("invalid workspace ID in context")
|
||||
}
|
||||
|
||||
// Get all knowledge items in batch
|
||||
knowledgeIDs := make([]string, 0, len(updates))
|
||||
for knowledgeID := range updates {
|
||||
knowledgeIDs = append(knowledgeIDs, knowledgeID)
|
||||
for id := range updates {
|
||||
knowledgeIDs = append(knowledgeIDs, id)
|
||||
}
|
||||
knowledgeList, err := s.repo.GetKnowledgeBatch(ctx, tenantID, knowledgeIDs)
|
||||
sort.Strings(knowledgeIDs)
|
||||
knowledgeList, err := loadKnowledgeWriteBatch(ctx, s.repo, s.kbService, knowledgeIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tenantID := knowledgeList[0].TenantID
|
||||
|
||||
// Validate all requested IDs were found and belong to the authorized KB
|
||||
if authorizedKBID != "" {
|
||||
|
|
@ -1002,15 +1024,15 @@ func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authoriz
|
|||
if !ok {
|
||||
return werrors.NewBadRequestError(fmt.Sprintf("标签 %s 不存在", tagID))
|
||||
}
|
||||
if tag.KnowledgeBaseID != knowledge.KnowledgeBaseID {
|
||||
if tag.TenantID != tenantID || tag.KnowledgeBaseID != knowledge.KnowledgeBaseID {
|
||||
return werrors.NewBadRequestError(fmt.Sprintf("标签 %s 不属于知识库 %s", tagID, knowledge.KnowledgeBaseID))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Set tags for each knowledge
|
||||
for knowledgeID, tagIDs := range updates {
|
||||
if err := s.repo.SetKnowledgeTags(ctx, knowledgeID, tagIDs); err != nil {
|
||||
for _, knowledgeID := range knowledgeIDs {
|
||||
if err := s.repo.SetKnowledgeTags(ctx, knowledgeID, updates[knowledgeID]); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
|
@ -1021,10 +1043,12 @@ func (s *knowledgeService) UpdateKnowledgeTagBatch(ctx context.Context, authoriz
|
|||
// SearchKnowledge searches knowledge items by keyword across the tenant and shared knowledge bases.
|
||||
// fileTypes: optional list of file extensions to filter by (e.g., ["csv", "xlsx"])
|
||||
func (s *knowledgeService) SearchKnowledge(ctx context.Context, keyword string, offset, limit int, fileTypes []string) ([]*types.Knowledge, bool, int64, error) {
|
||||
tenantID, ok := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
if !ok {
|
||||
caller := types.CallerFromContext(ctx)
|
||||
tenantID := caller.TenantID
|
||||
if tenantID == 0 {
|
||||
return nil, false, 0, werrors.NewUnauthorizedError("Workspace ID not found in context")
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, tenantID)
|
||||
|
||||
scopes := make([]types.KnowledgeSearchScope, 0)
|
||||
|
||||
|
|
@ -1041,9 +1065,9 @@ func (s *knowledgeService) SearchKnowledge(ctx context.Context, keyword string,
|
|||
// Shared knowledge bases (document type only). Plan 3 of #1303 keys
|
||||
// the share lookup on (tenantID, callerTenantRole); userID is no
|
||||
// longer load-bearing for org-share access.
|
||||
if userIDVal := ctx.Value(types.UserIDContextKey); userIDVal != nil {
|
||||
if userID, ok := userIDVal.(string); ok && userID != "" {
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
if s.kbShareService != nil {
|
||||
if caller.UserID != "" {
|
||||
callerTenantRole := caller.Role
|
||||
sharedList, err := s.kbShareService.ListSharedKnowledgeBases(ctx, tenantID, callerTenantRole)
|
||||
if err == nil {
|
||||
for _, info := range sharedList {
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import (
|
|||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/common"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
|
|
@ -113,7 +114,7 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
|
|||
if err := json.Unmarshal(task.Payload(), &payload); err != nil {
|
||||
return fmt.Errorf("unmarshal knowledge auto tag payload: %w", err)
|
||||
}
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
if payload.Language != "" {
|
||||
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
|
||||
}
|
||||
|
|
@ -153,6 +154,10 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
|
|||
if kb == nil || kb.TenantID != payload.TenantID || kb.Type != types.KnowledgeBaseTypeDocument {
|
||||
return skip("knowledge_base_not_eligible", nil)
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
config := kb.AutoTagConfig
|
||||
if config == nil || !config.Enabled {
|
||||
return skip("auto_tag_disabled", nil)
|
||||
|
|
@ -251,6 +256,9 @@ func (s *KnowledgeAutoTagService) Handle(ctx context.Context, task *asynq.Task)
|
|||
if len(added) == 0 {
|
||||
return skip("all_matches_already_attached", types.JSONMap{"matched_tag_count": len(validIDs)})
|
||||
}
|
||||
if err := access.RequireKBWrite(ctx, kb); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.knowledgeRepo.AddKnowledgeTagRelations(
|
||||
ctx,
|
||||
payload.TenantID,
|
||||
|
|
|
|||
|
|
@ -7,6 +7,7 @@ import (
|
|||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/models/chat"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
|
|
@ -36,6 +37,7 @@ type autoTagKnowledgeRepo struct {
|
|||
existing map[string][]*types.KnowledgeTag
|
||||
added []string
|
||||
addErr error
|
||||
writeCtx context.Context
|
||||
}
|
||||
|
||||
func (r *autoTagKnowledgeRepo) GetKnowledgeByIDOnly(context.Context, string) (*types.Knowledge, error) {
|
||||
|
|
@ -49,8 +51,9 @@ func (r *autoTagKnowledgeRepo) GetKnowledgeTags(
|
|||
}
|
||||
|
||||
func (r *autoTagKnowledgeRepo) AddKnowledgeTagRelations(
|
||||
_ context.Context, _ uint64, _, _ string, tagIDs []string,
|
||||
ctx context.Context, _ uint64, _, _ string, tagIDs []string,
|
||||
) error {
|
||||
r.writeCtx = ctx
|
||||
if r.addErr != nil {
|
||||
return r.addErr
|
||||
}
|
||||
|
|
@ -173,6 +176,9 @@ func TestAutoTagHandleAttachesMatchedTags(t *testing.T) {
|
|||
fixture := newAutoTagFixture(t, `{"matches":[{"index":1,"confidence":0.9}]}`)
|
||||
require.NoError(t, fixture.handle(t))
|
||||
assert.Equal(t, []string{"tag-a"}, fixture.repo.added)
|
||||
require.Zero(t, types.CallerFromContext(fixture.repo.writeCtx).TenantID)
|
||||
require.NoError(t, access.RequireKBWrite(fixture.repo.writeCtx, &types.KnowledgeBase{ID: "kb", TenantID: 7}))
|
||||
require.Error(t, access.RequireKBWrite(fixture.repo.writeCtx, &types.KnowledgeBase{ID: "other", TenantID: 7}))
|
||||
}
|
||||
|
||||
// A missing confidence must not be read as zero: models frequently omit the
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
|
|
@ -81,18 +82,21 @@ func TestReparseKnowledgeManualEnqueueFailureIsVisible(t *testing.T) {
|
|||
repo := &reparseFailureKnowledgeRepo{knowledge: knowledge}
|
||||
svc := &knowledgeService{
|
||||
repo: repo,
|
||||
kbService: &reparseFailureKBService{kb: &types.KnowledgeBase{ID: "kb-1"}},
|
||||
kbService: &reparseFailureKBService{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 7}},
|
||||
task: failingReparseTaskEnqueuer{err: enqueueErr},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
|
||||
ctx, grantErr := access.WithKBTaskWrite(ctx, &types.KnowledgeBase{ID: "kb-1", TenantID: 7}, 7)
|
||||
require.NoError(t, grantErr)
|
||||
|
||||
got, err := svc.ReparseKnowledge(ctx, knowledge.ID, nil)
|
||||
|
||||
require.Error(t, err)
|
||||
require.Same(t, knowledge, got)
|
||||
require.Equal(t, types.ParseStatusFailed, knowledge.ParseStatus)
|
||||
require.Equal(t, "disabled", knowledge.EnableStatus)
|
||||
require.Equal(t, "Failed to enqueue processing task", knowledge.ErrorMessage)
|
||||
require.NotNil(t, got)
|
||||
require.Equal(t, types.ParseStatusFailed, got.ParseStatus)
|
||||
require.Equal(t, "disabled", got.EnableStatus)
|
||||
require.Equal(t, "Failed to enqueue processing task", got.ErrorMessage)
|
||||
require.GreaterOrEqual(t, repo.updateCalls, 2, "pending and failed states must both be persisted")
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,186 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type callerScopeShares struct {
|
||||
interfaces.KBShareService
|
||||
callers []uint64
|
||||
}
|
||||
|
||||
func (s *callerScopeShares) CheckTenantKBPermission(
|
||||
_ context.Context,
|
||||
kbID string,
|
||||
caller uint64,
|
||||
_ types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
s.callers = append(s.callers, caller)
|
||||
// Tenant 2 has access to a third tenant's KB; caller 1 does not.
|
||||
return types.OrgRoleViewer, kbID == "onward" && caller == 2, nil
|
||||
}
|
||||
|
||||
func TestSharedExecutionFiltersDocumentsChunksAndSearchScope(t *testing.T) {
|
||||
shares := &callerScopeShares{}
|
||||
svc, db := newKnowledgeSharedAccessService(t, shares)
|
||||
require.NoError(t, db.AutoMigrate(&types.Chunk{}))
|
||||
kbs := []*types.KnowledgeBase{
|
||||
{ID: "own", TenantID: 1},
|
||||
{ID: "granted", TenantID: 2},
|
||||
{ID: "private", TenantID: 2},
|
||||
{ID: "onward", TenantID: 3},
|
||||
}
|
||||
var ids []string
|
||||
for _, kb := range kbs {
|
||||
ids = append(ids, kb.ID)
|
||||
seedKnowledge(t, db, &types.Knowledge{ID: kb.ID, TenantID: kb.TenantID, KnowledgeBaseID: kb.ID, Type: "file"})
|
||||
require.NoError(
|
||||
t,
|
||||
db.Create(
|
||||
&types.Chunk{
|
||||
ID: kb.ID,
|
||||
TenantID: kb.TenantID,
|
||||
KnowledgeBaseID: kb.ID,
|
||||
KnowledgeID: kb.ID,
|
||||
Content: kb.ID,
|
||||
},
|
||||
).Error,
|
||||
)
|
||||
}
|
||||
search := &knowledgeBaseService{
|
||||
kgRepo: svc.repo,
|
||||
chunkRepo: repository.NewChunkRepository(db),
|
||||
kbShareService: shares,
|
||||
}
|
||||
kbService := &suggestionKBService{kbs: make(map[string]*types.KnowledgeBase)}
|
||||
for _, kb := range kbs {
|
||||
kbService.kbs[kb.ID] = kb
|
||||
}
|
||||
session := &sessionService{knowledgeBaseService: kbService, knowledgeService: svc, kbShareService: shares}
|
||||
base := newSharedAccessContext()
|
||||
grant := &access.KBAccess{
|
||||
KnowledgeBase: kbs[1],
|
||||
Caller: types.CallerFromContext(base),
|
||||
EffectiveTenantID: 2,
|
||||
Permission: types.OrgRoleViewer,
|
||||
}
|
||||
for _, execution := range []uint64{1, 2, 3} {
|
||||
ctx := logger.CloneContext(types.WithExecutionTenant(grant.Context(base), execution))
|
||||
rows, err := svc.GetKnowledgeBatchWithSharedAccess(ctx, execution, ids)
|
||||
require.NoError(t, err)
|
||||
got := make([]string, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
got = append(got, row.ID)
|
||||
}
|
||||
require.ElementsMatch(t, []string{"own", "granted"}, got)
|
||||
knowledgeMap, err := search.fetchKnowledgeDataWithShared(ctx, execution, ids)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, knowledgeMap, 2)
|
||||
require.NotNil(t, knowledgeMap["own"])
|
||||
require.NotNil(t, knowledgeMap["granted"])
|
||||
chunks, err := search.listChunksByIDWithShared(ctx, execution, ids)
|
||||
require.NoError(t, err)
|
||||
got = got[:0]
|
||||
for _, chunk := range chunks {
|
||||
got = append(got, chunk.ID)
|
||||
}
|
||||
require.ElementsMatch(t, []string{"own", "granted"}, got)
|
||||
require.NoError(t, search.authorizeKBAccess(ctx, kbs[:2]))
|
||||
require.Error(t, search.authorizeKBAccess(ctx, kbs[2:3]), "execution tenant is not ownership")
|
||||
require.Error(t, search.authorizeKBAccess(ctx, kbs[3:]), "source tenant shares are not inherited")
|
||||
targets, err := session.buildSearchTargets(
|
||||
ctx,
|
||||
execution,
|
||||
ids,
|
||||
nil,
|
||||
[]types.TagScope{{KnowledgeBaseID: "private", TagIDs: []string{"tag"}}},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, targets, 2, "denied KBs must not become search or tag targets")
|
||||
got = got[:0]
|
||||
for _, target := range targets {
|
||||
got = append(got, target.KnowledgeBaseID)
|
||||
}
|
||||
require.ElementsMatch(t, []string{"own", "granted"}, got)
|
||||
_, err = resolveKBReadTenant(ctx, kbs[1], shares)
|
||||
require.NoError(t, err, "FAQ/tag reads accept the exact upstream grant")
|
||||
_, err = resolveKBReadTenant(ctx, kbs[2], shares)
|
||||
require.Error(t, err, "FAQ/tag reads cannot access another KB in the execution tenant")
|
||||
}
|
||||
for _, caller := range shares.callers {
|
||||
require.Equal(t, uint64(1), caller)
|
||||
}
|
||||
}
|
||||
|
||||
type callerScopeKBList struct {
|
||||
interfaces.KnowledgeBaseService
|
||||
tenant uint64
|
||||
}
|
||||
|
||||
func (s *callerScopeKBList) ListKnowledgeBases(ctx context.Context) ([]*types.KnowledgeBase, error) {
|
||||
s.tenant = types.MustTenantIDFromContext(ctx)
|
||||
return []*types.KnowledgeBase{{ID: "own", TenantID: s.tenant, Type: types.KnowledgeBaseTypeDocument}}, nil
|
||||
}
|
||||
|
||||
type callerScopeSearchRepo struct {
|
||||
interfaces.KnowledgeRepository
|
||||
scopes []types.KnowledgeSearchScope
|
||||
}
|
||||
|
||||
func (r *callerScopeSearchRepo) SearchKnowledgeInScopes(
|
||||
_ context.Context,
|
||||
scopes []types.KnowledgeSearchScope,
|
||||
_ string,
|
||||
_, _ int,
|
||||
_ []string,
|
||||
) ([]*types.Knowledge, bool, int64, error) {
|
||||
r.scopes = scopes
|
||||
return nil, false, 0, nil
|
||||
}
|
||||
|
||||
func TestDocumentSearchListsCallerTenantAfterExecutionSwitch(t *testing.T) {
|
||||
kbs := &callerScopeKBList{}
|
||||
repo := &callerScopeSearchRepo{}
|
||||
svc := &knowledgeService{kbService: kbs, repo: repo}
|
||||
ctx := types.WithExecutionTenant(newSharedAccessContext(), 2)
|
||||
_, _, _, err := svc.SearchKnowledge(ctx, "query", 0, 10, nil)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(1), kbs.tenant)
|
||||
require.Equal(t, []types.KnowledgeSearchScope{{TenantID: 1, KBID: "own"}}, repo.scopes)
|
||||
require.Equal(t, uint64(2), types.MustTenantIDFromContext(ctx), "search must not mutate the parent context")
|
||||
}
|
||||
|
||||
func TestSharedAgentBatchUsesGrantInsteadOfSourceTenantOwnership(t *testing.T) {
|
||||
svc, db := newKnowledgeSharedAccessService(t, &callerScopeShares{})
|
||||
for _, kbID := range []string{"selected", "private"} {
|
||||
seedKnowledge(t, db, &types.Knowledge{ID: kbID, TenantID: 2, KnowledgeBaseID: kbID, Type: "file"})
|
||||
}
|
||||
agent := &types.CustomAgent{
|
||||
TenantID: 2,
|
||||
Config: types.CustomAgentConfig{KBSelectionMode: "selected", KnowledgeBases: []string{"selected"}},
|
||||
}
|
||||
ctx := logger.CloneContext(access.WithSharedAgent(newSharedAccessContext(), agent))
|
||||
rows, err := svc.GetKnowledgeBatchWithSharedAccess(ctx, 2, []string{"selected", "private"})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rows, 1)
|
||||
require.Equal(t, "selected", rows[0].ID)
|
||||
ctx = types.WithTenantAPIKeyScope(ctx, types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"private"}})
|
||||
search := &knowledgeBaseService{}
|
||||
app, ok := apperrors.IsAppError(
|
||||
search.authorizeKBAccess(ctx, []*types.KnowledgeBase{{ID: "selected", TenantID: 2}}),
|
||||
)
|
||||
require.True(t, ok)
|
||||
require.Equal(t, apperrors.ErrForbidden, app.Code, "API-key scope denial must not become a lookup failure")
|
||||
rows, err = svc.GetKnowledgeBatchWithSharedAccess(ctx, 2, []string{"selected", "private"})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, rows)
|
||||
}
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestDeleteTaskMovingConflictStopsRetries(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
beforeWrites := f.repo.writes
|
||||
raw, err := json.Marshal(
|
||||
types.KnowledgeListDeletePayload{TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc"}},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
err = f.svc.ProcessKnowledgeListDelete(context.Background(), asynq.NewTask(types.TypeKnowledgeListDelete, raw))
|
||||
require.ErrorIs(t, err, asynq.SkipRetry)
|
||||
require.ErrorContains(t, err, "unfinished move")
|
||||
require.Equal(t, beforeWrites, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
}
|
||||
|
||||
func TestDeleteWikiRemovesEveryChunkType(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
f.kbs.values["kb"].IndexingStrategy.WikiEnabled = true
|
||||
refs := types.StringArray{"chunk"}
|
||||
chunkTypes := []string{
|
||||
types.ChunkTypeParentText, types.ChunkTypeImageOCR, types.ChunkTypeImageCaption, types.ChunkTypeSummary,
|
||||
}
|
||||
for _, typ := range chunkTypes {
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Create(
|
||||
&types.Chunk{ID: typ, TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb", ChunkType: typ},
|
||||
).Error,
|
||||
)
|
||||
refs = append(refs, typ)
|
||||
}
|
||||
refs = append(refs, "other-chunk")
|
||||
page := &types.WikiPage{
|
||||
ID: "page",
|
||||
Slug: "concept/shared",
|
||||
SourceRefs: types.StringArray{"doc", "other-doc"},
|
||||
ChunkRefs: refs,
|
||||
}
|
||||
wiki := &moveWikiPageRepo{pages: []*types.WikiPage{page}}
|
||||
meta := &moveWikiMetaService{}
|
||||
f.svc.wikiRepo = wiki
|
||||
f.svc.wikiService = meta
|
||||
f.svc.taskPendingRepo = &moveWikiPendingRepo{}
|
||||
f.svc.task = &transferQueue{}
|
||||
require.NoError(t, f.svc.DeleteKnowledge(f.ctx, "doc"))
|
||||
require.Len(t, meta.updated, 1)
|
||||
require.Equal(t, types.StringArray{"other-chunk"}, meta.updated[0].ChunkRefs)
|
||||
require.Equal(t, types.StringArray{"other-doc"}, meta.updated[0].SourceRefs)
|
||||
}
|
||||
|
||||
type cleanupKBFailure struct {
|
||||
interfaces.KnowledgeBaseService
|
||||
value *types.KnowledgeBase
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *cleanupKBFailure) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
|
||||
return s.value, s.err
|
||||
}
|
||||
|
||||
func TestCleanupStopsBeforeSideEffectsWhenKBUnavailable(t *testing.T) {
|
||||
for _, scenario := range []string{"storage error", "nil KB", "wrong KB", "wrong tenant"} {
|
||||
t.Run(scenario, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
lookup := &cleanupKBFailure{}
|
||||
switch scenario {
|
||||
case "storage error":
|
||||
lookup.err = errors.New("database unavailable")
|
||||
case "wrong KB":
|
||||
lookup.value = &types.KnowledgeBase{ID: "other", TenantID: 7}
|
||||
case "wrong tenant":
|
||||
lookup.value = &types.KnowledgeBase{ID: "kb", TenantID: 8}
|
||||
}
|
||||
f.svc.kbService = lookup
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
row.EmbeddingModelID = "bound-model"
|
||||
// Engine/model dependencies remain absent: touching a fallback backend
|
||||
// would panic instead of preserving the lookup error and original data.
|
||||
err = f.svc.cleanupKnowledgeResources(f.ctx, row)
|
||||
require.Error(t, err)
|
||||
if lookup.err != nil {
|
||||
require.ErrorIs(t, err, lookup.err)
|
||||
}
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
require.Empty(t, f.files.deleted)
|
||||
require.Empty(t, f.tenants.adjustments)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoveReparseKBLookupFailurePreservesSourceCheckpoint(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
row.SetManualMetadata(types.NewManualKnowledgeMetadata("content", types.ManualKnowledgeStatusPublish, 1)),
|
||||
)
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
failure := errors.New("source KB load failed")
|
||||
f.svc.kbService = &cleanupKBFailure{err: failure}
|
||||
err = f.svc.moveOneKnowledge(f.ctx, "doc", f.kbs.values["kb"], f.kbs.values["other"], "reparse")
|
||||
require.ErrorIs(t, err, failure)
|
||||
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "kb", row.KnowledgeBaseID)
|
||||
state, err := transferState(row)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "moving", state.Phase)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1017,21 +1017,18 @@ func (s *knowledgeService) UpdateManualKnowledge(ctx context.Context,
|
|||
return nil, werrors.NewValidationError("状态仅支持 draft 或 publish")
|
||||
}
|
||||
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
existing, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
|
||||
existing, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to load knowledge: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
if !existing.IsManual() {
|
||||
return nil, werrors.NewBadRequestError("仅支持手工知识的在线编辑")
|
||||
}
|
||||
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, existing.KnowledgeBaseID)
|
||||
ctx, err = withKBWriteTenantInfo(ctx, kb, s.tenantRepo)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get knowledge base for manual update: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
tenantID := existing.TenantID
|
||||
|
||||
var version int
|
||||
if meta, err := existing.ManualMetadata(); err == nil && meta != nil {
|
||||
|
|
@ -1115,7 +1112,7 @@ func (s *knowledgeService) UpdateManualKnowledge(ctx context.Context,
|
|||
|
||||
// enqueueManualProcessing enqueues a manual:process Asynq task for async cleanup + re-indexing.
|
||||
func (s *knowledgeService) enqueueManualProcessing(ctx context.Context,
|
||||
knowledge *types.Knowledge, content string, needCleanup bool,
|
||||
knowledge *types.Knowledge, content string, needCleanup bool, options ...asynq.Option,
|
||||
) (string, error) {
|
||||
requestID, _ := types.RequestIDFromContext(ctx)
|
||||
payload := types.ManualProcessPayload{
|
||||
|
|
@ -1134,7 +1131,7 @@ func (s *knowledgeService) enqueueManualProcessing(ctx context.Context,
|
|||
|
||||
task := asynq.NewTask(types.TypeManualProcess, payloadBytes,
|
||||
asynq.Queue(types.QueueDefault), asynq.MaxRetry(3), asynq.Timeout(30*time.Minute))
|
||||
info, err := s.task.Enqueue(task)
|
||||
info, err := s.task.Enqueue(task, options...)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to enqueue manual process task: %w", err)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,10 +4,11 @@ import (
|
|||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
|
|
@ -120,150 +121,11 @@ func deleteExtractedImages(
|
|||
|
||||
// DeleteKnowledge deletes a knowledge entry and all related resources
|
||||
func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error {
|
||||
// Get the knowledge entry
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, ctx.Value(types.TenantIDContextKey).(uint64), id)
|
||||
plan, err := s.planKnowledgeDelete(ctx, []string{id})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Mark as deleting first to prevent async task conflicts
|
||||
// This ensures that any running async tasks will detect the deletion and abort
|
||||
originalStatus := knowledge.ParseStatus
|
||||
knowledge.ParseStatus = types.ParseStatusDeleting
|
||||
knowledge.UpdatedAt = time.Now()
|
||||
if err := s.repo.UpdateKnowledge(ctx, knowledge); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge failed to mark as deleting")
|
||||
// Continue with deletion even if marking fails
|
||||
} else {
|
||||
logger.Infof(ctx, "Marked knowledge %s as deleting (previous status: %s)", id, originalStatus)
|
||||
}
|
||||
|
||||
// Best-effort: purge any queued downstream tasks for this knowledge
|
||||
// (multimodal / post-process / question / summary / graph extract).
|
||||
// Worker checkpoints already drop them on the floor, but dequeuing
|
||||
// here avoids waking workers just to no-op when the parse was still
|
||||
// in flight at delete time. No-op in Lite mode and on completed rows
|
||||
// (no queued descendants anyway).
|
||||
if originalStatus == types.ParseStatusPending ||
|
||||
originalStatus == types.ParseStatusProcessing {
|
||||
s.dequeueKnowledgeTasks(ctx, id)
|
||||
}
|
||||
|
||||
// Resolve file service for this KB before spawning goroutines
|
||||
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
kbFileSvc := s.resolveFileService(ctx, kb)
|
||||
|
||||
// Collect image URLs before chunks are deleted (ImageInfo references are lost after deletion)
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
chunkImageInfos, err := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantID, []string{id})
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to collect image URLs for cleanup: %v", err)
|
||||
}
|
||||
var imageInfoStrs []string
|
||||
for _, ci := range chunkImageInfos {
|
||||
imageInfoStrs = append(imageInfoStrs, ci.ImageInfo)
|
||||
}
|
||||
imageURLs := collectImageURLs(ctx, imageInfoStrs)
|
||||
|
||||
wg := errgroup.Group{}
|
||||
// Delete knowledge embeddings from vector store.
|
||||
// Skip entirely when the knowledge has no embedding model (e.g. Wiki-only KB):
|
||||
// nothing was ever written to the vector store, so there is nothing to delete,
|
||||
// and GetEmbeddingModel would fail with "model ID cannot be empty".
|
||||
if strings.TrimSpace(knowledge.EmbeddingModelID) != "" {
|
||||
wg.Go(func() error {
|
||||
// kb was already loaded above for resolveFileService — reuse its
|
||||
// VectorStoreID for engine routing.
|
||||
var boundStoreID *string
|
||||
if kb != nil {
|
||||
boundStoreID = kb.VectorStoreID
|
||||
}
|
||||
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
|
||||
ctx,
|
||||
s.retrieveEngine,
|
||||
s.ownership,
|
||||
tenantID,
|
||||
boundStoreID,
|
||||
)
|
||||
if err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
|
||||
return err
|
||||
}
|
||||
embeddingModel, err := s.modelService.GetEmbeddingModel(ctx, knowledge.EmbeddingModelID)
|
||||
if err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
|
||||
return err
|
||||
}
|
||||
if err := retrieveEngine.DeleteByKnowledgeIDList(ctx, []string{knowledge.ID}, embeddingModel.GetDimensions(), knowledge.Type); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge embedding failed")
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
})
|
||||
} else {
|
||||
logger.Infof(ctx, "Knowledge %s has no embedding model, skipping vector store cleanup", knowledge.ID)
|
||||
}
|
||||
|
||||
// Clean wiki pages before deleting chunks so cleanup can still identify
|
||||
// which chunk_refs belonged to this source document.
|
||||
if kb != nil && kb.IsWikiEnabled() {
|
||||
s.cleanupWikiOnKnowledgeDelete(ctx, knowledge)
|
||||
}
|
||||
|
||||
// Delete all chunks associated with this knowledge
|
||||
wg.Go(func() error {
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete chunks failed")
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
// Delete the knowledge graph
|
||||
wg.Go(func() error {
|
||||
namespace := types.NameSpace{KnowledgeBase: knowledge.KnowledgeBaseID, Knowledge: knowledge.ID}
|
||||
if err := s.graphEngine.DelGraph(ctx, []types.NameSpace{namespace}); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete knowledge graph failed")
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
})
|
||||
|
||||
if err = wg.Wait(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.repo.DeleteKnowledgeTagRelations(ctx, id); err != nil {
|
||||
logger.Warnf(ctx, "Failed to delete tag relations for knowledge %s: %v", id, err)
|
||||
}
|
||||
// Delete the knowledge row FIRST, then drop its physical file. Physical
|
||||
// cleanup is deliberately deferred until the row is gone: if any of the
|
||||
// index/chunk/graph cleanups above failed we already returned early with the
|
||||
// row (and its file) intact, so the queued retry — or a user-triggered
|
||||
// reparse — can still read the original file. Deleting the file before the
|
||||
// row could leave a "file missing but row present" zombie that can neither be
|
||||
// reparsed nor cleanly re-deleted (issue #2192). Orphaning a file after the
|
||||
// row is gone is the tolerable failure mode instead.
|
||||
if err := s.repo.DeleteKnowledge(ctx, tenantID, id); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Best-effort physical cleanup. Errors here only leak storage; they must not
|
||||
// fail the delete now that the row is already gone.
|
||||
if knowledge.FilePath != "" {
|
||||
if err := kbFileSvc.DeleteFile(ctx, knowledge.FilePath); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete file failed")
|
||||
}
|
||||
}
|
||||
deleteExtractedImages(ctx, kbFileSvc, knowledgeResourceOwners(s.resourceCatalog, knowledge.ID), imageURLs)
|
||||
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
tenantInfo.StorageUsed -= knowledge.StorageSize
|
||||
if err := s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, -knowledge.StorageSize); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge update tenant storage used failed")
|
||||
}
|
||||
recordKBActivity(ctx, s.audit, tenantID, knowledge.KnowledgeBaseID, types.AuditActionKnowledgeDeleted,
|
||||
"knowledge", knowledge.ID, types.AuditOutcomeSuccess,
|
||||
map[string]any{"title": knowledge.Title, "type": knowledge.Type})
|
||||
return nil
|
||||
return s.executeKnowledgeDelete(plan, true)
|
||||
}
|
||||
|
||||
// cleanupWikiOnKnowledgeDelete handles wiki pages when a source document is deleted.
|
||||
|
|
@ -277,9 +139,9 @@ func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error
|
|||
// - It writes a tombstone + scrubs pending ingest ops so new pages cannot be
|
||||
// born with a stale source_ref (guards (a) queued ingest and (b) ingest
|
||||
// tasks mid-LLM call — both consult the tombstone before writing).
|
||||
// - It immediately reconciles any pages already present (delete-if-only-ref
|
||||
// or strip-ref-if-multi).
|
||||
// - It *unconditionally* enqueues a retract task. Crucially we DO NOT gate
|
||||
// - It persists a retract task before reconciling existing pages (delete
|
||||
// if only source, or strip the source if shared), preserving retry evidence.
|
||||
// Crucially we DO NOT gate
|
||||
// enqueue on "pages currently exist": in the ingest/delete race the
|
||||
// knowledge may have pages that exist only after this function returns
|
||||
// (the ingest task fires later and, absent the tombstone, would have
|
||||
|
|
@ -287,20 +149,42 @@ func (s *knowledgeService) DeleteKnowledge(ctx context.Context, id string) error
|
|||
// run time, so even with an empty PageSlugs it will do the right thing —
|
||||
// and at worst it's a cheap no-op.
|
||||
func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, knowledge *types.Knowledge) {
|
||||
if err := s.cleanupWikiReferences(ctx, knowledge, s.wikiChunkRefsForKnowledge(ctx, knowledge)); err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *knowledgeService) cleanupWikiReferences(
|
||||
ctx context.Context,
|
||||
knowledge *types.Knowledge,
|
||||
sourceChunkRefs map[string]bool,
|
||||
) error {
|
||||
if knowledge == nil {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
kbID := knowledge.KnowledgeBaseID
|
||||
knowledgeID := knowledge.ID
|
||||
if kbID == "" || knowledgeID == "" {
|
||||
return
|
||||
return nil
|
||||
}
|
||||
|
||||
var cleanupErr error
|
||||
|
||||
// (1) Tombstone + scrub pending ingest — must happen first so any
|
||||
// wiki_ingest task that wakes up between here and the retract enqueue
|
||||
// below sees "knowledge gone" and bails out.
|
||||
s.markKnowledgeDeletedForWiki(ctx, kbID, knowledgeID)
|
||||
s.scrubWikiPendingIngest(ctx, kbID, knowledgeID, "cleanup")
|
||||
if s.redisClient != nil {
|
||||
key := WikiDeletedTombstoneKey(kbID, knowledgeID)
|
||||
if err := s.redisClient.Set(ctx, key, "1", wikiDeletedTTL).Err(); err != nil {
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
}
|
||||
}
|
||||
if s.taskPendingRepo != nil {
|
||||
err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest)
|
||||
if err != nil {
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Pull title/summary from the knowledge itself — do NOT read them from
|
||||
// existing wiki pages. In the race window wiki pages may not exist yet,
|
||||
|
|
@ -322,9 +206,8 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
|
|||
pages, err := s.wikiRepo.ListBySourceRef(ctx, kbID, knowledgeID)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup: failed to list pages by source ref %s: %v", knowledgeID, err)
|
||||
pages = nil
|
||||
return errors.Join(cleanupErr, err)
|
||||
}
|
||||
sourceChunkRefs := s.wikiChunkRefsForKnowledge(ctx, knowledge)
|
||||
|
||||
// Prefer the on-disk summary if the summary page already exists (it's
|
||||
// richer than the raw user-provided description). Leave docSummary
|
||||
|
|
@ -336,22 +219,37 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
|
|||
}
|
||||
}
|
||||
|
||||
// Persist the retract page set before removing source refs. Otherwise an
|
||||
// enqueue failure would leave the retry unable to rediscover those pages.
|
||||
var pageSlugs, folderIDs []string
|
||||
for _, page := range pages {
|
||||
if page.PageType != types.WikiPageTypeIndex {
|
||||
pageSlugs = append(pageSlugs, page.Slug)
|
||||
folderIDs = append(folderIDs, page.FolderID)
|
||||
}
|
||||
}
|
||||
lang := types.LanguageFromContextOrDefault(ctx)
|
||||
tenantID, _ := types.TenantIDFromContext(ctx)
|
||||
if err := enqueueWikiRetract(ctx, s.task, s.taskPendingRepo, WikiRetractPayload{
|
||||
TenantID: tenantID, KnowledgeBaseID: kbID, KnowledgeID: knowledgeID,
|
||||
DocTitle: docTitle, DocSummary: docSummary, Language: lang,
|
||||
PageSlugs: pageSlugs, FolderIDs: uniqueWikiFolderIDs(folderIDs),
|
||||
}); err != nil {
|
||||
return errors.Join(cleanupErr, err)
|
||||
}
|
||||
|
||||
var deletedSlugs []string
|
||||
var retractSlugs []string
|
||||
var affectedFolderIDs []string
|
||||
for _, page := range pages {
|
||||
if page.PageType == types.WikiPageTypeIndex {
|
||||
continue
|
||||
}
|
||||
if page.FolderID != "" {
|
||||
affectedFolderIDs = append(affectedFolderIDs, page.FolderID)
|
||||
}
|
||||
|
||||
remaining := removeSourceRef(page.SourceRefs, knowledgeID)
|
||||
|
||||
if len(remaining) == 0 {
|
||||
if err := s.wikiService.DeletePage(ctx, kbID, page.Slug); err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup: failed to delete page %s: %v", page.Slug, err)
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
} else {
|
||||
deletedSlugs = append(deletedSlugs, page.Slug)
|
||||
}
|
||||
|
|
@ -360,8 +258,7 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
|
|||
page.ChunkRefs = removeChunkRefs(page.ChunkRefs, sourceChunkRefs)
|
||||
if err := s.wikiService.UpdatePageMeta(ctx, page); err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup: failed to update source refs for page %s: %v", page.Slug, err)
|
||||
} else {
|
||||
retractSlugs = append(retractSlugs, page.Slug)
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -371,34 +268,14 @@ func (s *knowledgeService) cleanupWikiOnKnowledgeDelete(ctx context.Context, kno
|
|||
len(deletedSlugs), knowledgeID, deletedSlugs)
|
||||
}
|
||||
|
||||
allAffectedSlugs := append(retractSlugs, deletedSlugs...)
|
||||
|
||||
// (3) Unconditionally enqueue the retract task. See function comment —
|
||||
// an empty PageSlugs is not a bug, it's the signal "re-query at run
|
||||
// time". The handler will ListPagesBySourceRef again, pick up any
|
||||
// pages that materialised after we looked, and also rebuild the index
|
||||
// so the knowledge's disappearance is reflected in the UI.
|
||||
lang := types.LanguageFromContextOrDefault(ctx)
|
||||
tenantID, _ := types.TenantIDFromContext(ctx)
|
||||
EnqueueWikiRetract(ctx, s.task, s.taskPendingRepo, WikiRetractPayload{
|
||||
TenantID: tenantID,
|
||||
KnowledgeBaseID: kbID,
|
||||
KnowledgeID: knowledgeID,
|
||||
DocTitle: docTitle,
|
||||
DocSummary: docSummary,
|
||||
Language: lang,
|
||||
PageSlugs: allAffectedSlugs,
|
||||
FolderIDs: uniqueWikiFolderIDs(affectedFolderIDs),
|
||||
})
|
||||
logger.Infof(ctx, "wiki cleanup: enqueued retract task for knowledge %s (%d known slugs: %v)",
|
||||
knowledgeID, len(allAffectedSlugs), allAffectedSlugs)
|
||||
return cleanupErr
|
||||
}
|
||||
|
||||
func (s *knowledgeService) wikiChunkRefsForKnowledge(ctx context.Context, knowledge *types.Knowledge) map[string]bool {
|
||||
if knowledge == nil || s.chunkRepo == nil {
|
||||
return nil
|
||||
}
|
||||
chunks, err := s.chunkRepo.ListChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID)
|
||||
chunks, err := s.chunkRepo.ListAllChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup: failed to list chunks for knowledge %s: %v", knowledge.ID, err)
|
||||
return nil
|
||||
|
|
@ -413,19 +290,6 @@ func (s *knowledgeService) wikiChunkRefsForKnowledge(ctx context.Context, knowle
|
|||
return refs
|
||||
}
|
||||
|
||||
// markKnowledgeDeletedForWiki writes a short-TTL tombstone so any wiki_ingest
|
||||
// task still running or queued for this knowledge can short-circuit before
|
||||
// resurrecting a page with a stale source_ref. No-op when Redis is absent.
|
||||
func (s *knowledgeService) markKnowledgeDeletedForWiki(ctx context.Context, kbID, knowledgeID string) {
|
||||
if s.redisClient == nil || kbID == "" || knowledgeID == "" {
|
||||
return
|
||||
}
|
||||
key := WikiDeletedTombstoneKey(kbID, knowledgeID)
|
||||
if err := s.redisClient.Set(ctx, key, "1", wikiDeletedTTL).Err(); err != nil {
|
||||
logger.Warnf(ctx, "wiki cleanup: failed to write tombstone %s: %v", key, err)
|
||||
}
|
||||
}
|
||||
|
||||
// scrubWikiPendingIngest removes queued WikiOpIngest entries for a knowledge
|
||||
// from task_pending_ops. Used by both the delete path (we're about to
|
||||
// soft-delete the doc, no point ingesting it) and the reparse path (the
|
||||
|
|
@ -441,7 +305,8 @@ func (s *knowledgeService) scrubWikiPendingIngest(ctx context.Context, kbID, kno
|
|||
if s.taskPendingRepo == nil || kbID == "" || knowledgeID == "" {
|
||||
return
|
||||
}
|
||||
if err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest); err != nil {
|
||||
err := s.taskPendingRepo.DeleteByDedupKey(ctx, wikiTaskType, wikiTaskScope, kbID, knowledgeID, WikiOpIngest)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "wiki %s: failed to scrub pending ingest ops for knowledge %s: %v", reason, knowledgeID, err)
|
||||
return
|
||||
}
|
||||
|
|
@ -552,15 +417,20 @@ func buildKnowledgeVectorDeleteGroups(
|
|||
|
||||
// DeleteKnowledgeList deletes a knowledge entry and all related resources
|
||||
func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string) error {
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
// 1. Get the knowledge entry
|
||||
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
knowledgeList, err := s.repo.GetKnowledgeBatch(ctx, tenantInfo.ID, ids)
|
||||
plan, err := s.planKnowledgeDelete(ctx, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.executeKnowledgeDelete(plan, false)
|
||||
}
|
||||
|
||||
func (s *knowledgeService) executeKnowledgeDelete(plan *knowledgeDeletePlan, single bool) error {
|
||||
if len(plan.ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
ctx, ids, knowledgeList := plan.ctx, plan.ids, plan.knowledge
|
||||
knowledgeBases, kbFileServices := plan.kbs, plan.files
|
||||
tenantInfo, _ := types.TenantInfoFromContext(ctx)
|
||||
|
||||
// Mark all as deleting first to prevent async task conflicts.
|
||||
// Remember which entries still had queued / in-flight downstream tasks
|
||||
|
|
@ -568,12 +438,12 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
var inFlightIDs []string
|
||||
for _, knowledge := range knowledgeList {
|
||||
prev := knowledge.ParseStatus
|
||||
before := *knowledge
|
||||
knowledge.ParseStatus = types.ParseStatusDeleting
|
||||
knowledge.UpdatedAt = time.Now()
|
||||
if err := s.repo.UpdateKnowledge(ctx, knowledge); err != nil {
|
||||
if err := s.repo.UpdateKnowledgeForTransfer(ctx, &before, knowledge); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).WithField("knowledge_id", knowledge.ID).
|
||||
Errorf("DeleteKnowledgeList failed to mark as deleting")
|
||||
// Continue with deletion even if marking fails
|
||||
return err
|
||||
}
|
||||
if prev == types.ParseStatusPending || prev == types.ParseStatusProcessing {
|
||||
inFlightIDs = append(inFlightIDs, knowledge.ID)
|
||||
|
|
@ -582,28 +452,14 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
logger.Infof(ctx, "Marked %d knowledge entries as deleting", len(knowledgeList))
|
||||
|
||||
// Best-effort dequeue of downstream tasks for in-flight entries.
|
||||
// See DeleteKnowledge for the rationale; loop is per-knowledge because
|
||||
// Workers also check deletion state; this avoids waking them unnecessarily.
|
||||
// The loop is per-knowledge because
|
||||
// the inspector only filters by knowledge_id, not by ID set.
|
||||
for _, kid := range inFlightIDs {
|
||||
s.dequeueKnowledgeTasks(ctx, kid)
|
||||
}
|
||||
|
||||
// Pre-resolve KB metadata and file services so goroutines don't need DB access.
|
||||
knowledgeBases := make(map[string]*types.KnowledgeBase)
|
||||
kbFileServices := make(map[string]interfaces.FileService)
|
||||
for _, knowledge := range knowledgeList {
|
||||
if _, ok := kbFileServices[knowledge.KnowledgeBaseID]; !ok {
|
||||
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
knowledgeBases[knowledge.KnowledgeBaseID] = kb
|
||||
kbFileServices[knowledge.KnowledgeBaseID] = s.resolveFileService(ctx, kb)
|
||||
}
|
||||
}
|
||||
|
||||
// Collect image URLs before chunks are deleted
|
||||
chunkImageInfos, err := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantInfo.ID, ids)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to collect image URLs for batch cleanup: %v", err)
|
||||
}
|
||||
chunkImageInfos := plan.imageInfo
|
||||
knowledgeToKB := make(map[string]string)
|
||||
for _, k := range knowledgeList {
|
||||
knowledgeToKB[k.ID] = k.KnowledgeBaseID
|
||||
|
|
@ -664,7 +520,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
// 3. Clean wiki pages before deleting chunks so cleanup can still identify
|
||||
// which chunk_refs belonged to each source document.
|
||||
for _, knowledge := range knowledgeList {
|
||||
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
kb := knowledgeBases[knowledge.KnowledgeBaseID]
|
||||
if kb != nil && kb.IsWikiEnabled() {
|
||||
s.cleanupWikiOnKnowledgeDelete(ctx, knowledge)
|
||||
}
|
||||
|
|
@ -672,7 +528,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
|
||||
// 4. Delete all chunks associated with this knowledge
|
||||
wg.Go(func() error {
|
||||
if err := s.chunkService.DeleteByKnowledgeList(ctx, ids); err != nil {
|
||||
if err := s.chunkRepo.DeleteByKnowledgeList(ctx, tenantInfo.ID, ids); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge delete chunks failed")
|
||||
return err
|
||||
}
|
||||
|
|
@ -695,7 +551,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
return nil
|
||||
})
|
||||
|
||||
if err = wg.Wait(); err != nil {
|
||||
if err := wg.Wait(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, knowledgeID := range ids {
|
||||
|
|
@ -704,7 +560,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
}
|
||||
}
|
||||
// 6. Delete the knowledge rows FIRST, then drop their physical files. See
|
||||
// DeleteKnowledge for the rationale: deferring file removal until the rows are
|
||||
// Deferring file removal until the rows are
|
||||
// gone avoids "file missing but row present" zombies that break reparse /
|
||||
// re-delete when an earlier cleanup step failed (issue #2192). A failure below
|
||||
// only orphans storage.
|
||||
|
|
@ -731,7 +587,7 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
}
|
||||
deleteExtractedImages(ctx, fSvc, knowledgeResourceOwners(s.resourceCatalog, kbKnowledgeIDs[kbID]...), urls)
|
||||
}
|
||||
tenantInfo.StorageUsed += storageAdjust
|
||||
// TenantInfo can be shared by concurrent cleanup branches; update only storage accounting in the repository.
|
||||
if err := s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, storageAdjust); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("DeleteKnowledge update tenant storage used failed")
|
||||
}
|
||||
|
|
@ -752,6 +608,13 @@ func (s *knowledgeService) DeleteKnowledgeList(ctx context.Context, ids []string
|
|||
details["knowledge_ids"] = knowledgeIDs
|
||||
}
|
||||
kbActivityAppendSampleTitles(details, titles...)
|
||||
if single {
|
||||
knowledge := knowledges[0]
|
||||
recordKBActivity(ctx, s.audit, tenantInfo.ID, kbID, types.AuditActionKnowledgeDeleted,
|
||||
"knowledge", knowledge.ID, types.AuditOutcomeSuccess,
|
||||
map[string]any{"title": knowledge.Title, "type": knowledge.Type})
|
||||
continue
|
||||
}
|
||||
recordKBActivity(ctx, s.audit, tenantInfo.ID, kbID, types.AuditActionKnowledgeBatchDeleted,
|
||||
"knowledge", "", types.AuditOutcomeSuccess, details)
|
||||
}
|
||||
|
|
@ -768,23 +631,14 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
|
|||
return nil
|
||||
}
|
||||
|
||||
kb, err := knowledgeWriteKB(ctx, s.kbService, knowledge)
|
||||
if err != nil {
|
||||
return fmt.Errorf("resolve cleanup knowledge base: %w", err)
|
||||
}
|
||||
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
if knowledge.EmbeddingModelID != "" {
|
||||
// Load KB to discover its VectorStoreID binding. Falls back to tenant
|
||||
// effective engines if the KB has no binding or the load fails.
|
||||
//
|
||||
// Silent fallback risk: if a bound KB fails to load here due to a
|
||||
// transient DB error, the cleanup will delete from env engines and
|
||||
// leave orphan vectors in the bound store. Warn so operators can spot it.
|
||||
var boundStoreID *string
|
||||
if kb, loadErr := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID); loadErr == nil && kb != nil {
|
||||
boundStoreID = kb.VectorStoreID
|
||||
} else if loadErr != nil {
|
||||
logger.GetLogger(ctx).WithField("error", loadErr).WithField("knowledge_base_id", knowledge.KnowledgeBaseID).
|
||||
Warnf("cleanupKnowledgeResources: failed to load KB for vector store resolution; falling back to tenant effective engines")
|
||||
}
|
||||
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
|
||||
ctx, s.retrieveEngine, s.ownership, tenantInfo.ID, boundStoreID)
|
||||
ctx, s.retrieveEngine, s.ownership, tenantInfo.ID, kb.VectorStoreID)
|
||||
if err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Error("Failed to init retrieve engine during cleanup")
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
|
|
@ -803,7 +657,6 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
|
|||
}
|
||||
|
||||
// Collect image URLs before chunks are deleted
|
||||
kb, _ := s.kbService.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
fileSvc := s.resolveFileService(ctx, kb)
|
||||
chunkImageInfos, imgErr := s.chunkService.GetRepository().ListImageInfoByKnowledgeIDs(ctx, tenantInfo.ID, []string{knowledge.ID})
|
||||
if imgErr != nil {
|
||||
|
|
@ -816,7 +669,7 @@ func (s *knowledgeService) cleanupKnowledgeResources(ctx context.Context, knowle
|
|||
}
|
||||
imageURLs := collectImageURLs(ctx, imageInfoStrs)
|
||||
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Error("Failed to delete manual knowledge chunks")
|
||||
cleanupErr = errors.Join(cleanupErr, err)
|
||||
}
|
||||
|
|
@ -860,20 +713,46 @@ func (s *knowledgeService) ProcessKnowledgeListDelete(ctx context.Context, t *as
|
|||
|
||||
logger.Infof(ctx, "Processing knowledge list delete task for %d knowledge items", len(payload.KnowledgeIDs))
|
||||
|
||||
// Get tenant info
|
||||
tenant, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get tenant %d: %v", payload.TenantID, err)
|
||||
return err
|
||||
if payload.TenantID == 0 {
|
||||
return fmt.Errorf("invalid delete task tenant: %w", asynq.SkipRetry)
|
||||
}
|
||||
|
||||
// Set context values
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = context.WithValue(ctx, types.TenantInfoContextKey, tenant)
|
||||
|
||||
// Delete knowledge list
|
||||
if err := s.DeleteKnowledgeList(ctx, payload.KnowledgeIDs); err != nil {
|
||||
logger.Errorf(ctx, "Failed to delete knowledge list: %v", err)
|
||||
ids, err := writeResourceIDs(payload.KnowledgeIDs)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid delete task IDs: %w", asynq.SkipRetry)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return nil
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
kbID := payload.KnowledgeBaseID
|
||||
if kbID == "" {
|
||||
// Pre-upgrade tasks were admitted by single-KB endpoints but omitted
|
||||
// the KB ID. Reconstruct only an unambiguous, same-tenant scope.
|
||||
rows, err := s.repo.GetKnowledgeBatch(ctx, payload.TenantID, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil
|
||||
}
|
||||
for _, row := range rows {
|
||||
if row == nil || row.TenantID != payload.TenantID || row.KnowledgeBaseID == "" ||
|
||||
(kbID != "" && row.KnowledgeBaseID != kbID) {
|
||||
return fmt.Errorf("ambiguous legacy delete scope: %w", asynq.SkipRetry)
|
||||
}
|
||||
kbID = row.KnowledgeBaseID
|
||||
}
|
||||
}
|
||||
bindings := make(map[string]string, len(ids))
|
||||
for _, id := range ids {
|
||||
bindings[id] = kbID
|
||||
}
|
||||
ctx = withKnowledgeCleanup(ctx, payload.TenantID, bindings)
|
||||
if err := s.DeleteKnowledgeList(ctx, ids); err != nil {
|
||||
var appErr *apperrors.AppError
|
||||
if errors.As(err, &appErr) && (appErr.HTTPCode == 403 || appErr.HTTPCode == 400 || appErr.HTTPCode == 409) {
|
||||
return fmt.Errorf("invalid delete task scope: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,169 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
||||
type knowledgeCleanupKey struct{}
|
||||
|
||||
// Cleanup continues a previously admitted operation on exact persisted
|
||||
// references. It never grants general KB write access to a session owner.
|
||||
type knowledgeCleanupScope struct {
|
||||
caller types.Caller
|
||||
tenant uint64
|
||||
bindings map[string]string // knowledge ID -> KB ID, copied when admitted
|
||||
}
|
||||
|
||||
func withKnowledgeCleanup(ctx context.Context, tenant uint64, bindings map[string]string) context.Context {
|
||||
ctx = types.WithExecutionTenant(ctx, tenant)
|
||||
copyOfBindings := make(map[string]string, len(bindings))
|
||||
for id, kb := range bindings {
|
||||
copyOfBindings[id] = kb
|
||||
}
|
||||
return context.WithValue(ctx, knowledgeCleanupKey{}, knowledgeCleanupScope{
|
||||
caller: types.CallerFromContext(ctx), tenant: tenant, bindings: copyOfBindings,
|
||||
})
|
||||
}
|
||||
|
||||
// Only call with IDs obtained from an already authorized persisted relationship
|
||||
// (session messages, a temporary KB, or a clone diff), never a request body.
|
||||
// expectedKB additionally pins workflows with an independently known KB.
|
||||
func deleteReferencedKnowledge(
|
||||
ctx context.Context,
|
||||
svc interfaces.KnowledgeService,
|
||||
expectedKB string,
|
||||
ids []string,
|
||||
) error {
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ids, err = writeResourceIDs(ids)
|
||||
if err != nil || len(ids) == 0 {
|
||||
return err
|
||||
}
|
||||
rows, err := svc.GetRepository().GetKnowledgeBatch(ctx, tenant, ids)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
wanted := make(map[string]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
wanted[id] = true
|
||||
}
|
||||
bindings := make(map[string]string, len(rows))
|
||||
remaining := make([]string, 0, len(rows))
|
||||
for _, row := range rows {
|
||||
if row == nil || !wanted[row.ID] || row.TenantID != tenant || row.KnowledgeBaseID == "" ||
|
||||
(expectedKB != "" && row.KnowledgeBaseID != expectedKB) {
|
||||
return apperrors.NewForbiddenError("cleanup resource binding changed")
|
||||
}
|
||||
bindings[row.ID] = row.KnowledgeBaseID
|
||||
remaining = append(remaining, row.ID)
|
||||
}
|
||||
if len(remaining) == 0 {
|
||||
return nil
|
||||
}
|
||||
return svc.DeleteKnowledgeList(withKnowledgeCleanup(ctx, tenant, bindings), remaining)
|
||||
}
|
||||
|
||||
type knowledgeDeletePlan struct {
|
||||
ctx context.Context
|
||||
ids []string
|
||||
knowledge []*types.Knowledge
|
||||
kbs map[string]*types.KnowledgeBase
|
||||
files map[string]interfaces.FileService
|
||||
imageInfo []interfaces.ChunkImageInfo
|
||||
}
|
||||
|
||||
func (s *knowledgeService) planKnowledgeDelete(ctx context.Context, ids []string) (*knowledgeDeletePlan, error) {
|
||||
ids, err := writeResourceIDs(ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
plan := &knowledgeDeletePlan{
|
||||
ctx: ctx,
|
||||
kbs: make(map[string]*types.KnowledgeBase),
|
||||
files: make(map[string]interfaces.FileService),
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return plan, nil
|
||||
}
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cleanup, isCleanup := ctx.Value(knowledgeCleanupKey{}).(knowledgeCleanupScope)
|
||||
if isCleanup {
|
||||
if cleanup.tenant != tenant || cleanup.caller != types.CallerFromContext(ctx) {
|
||||
return nil, apperrors.NewForbiddenError("invalid cleanup scope")
|
||||
}
|
||||
for _, id := range ids {
|
||||
if cleanup.bindings[id] == "" {
|
||||
return nil, apperrors.NewForbiddenError("knowledge outside cleanup scope")
|
||||
}
|
||||
}
|
||||
}
|
||||
rows, err := s.repo.GetKnowledgeBatch(ctx, tenant, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byID := make(map[string]*types.Knowledge, len(rows))
|
||||
for _, row := range rows {
|
||||
if row != nil && row.TenantID == tenant {
|
||||
byID[row.ID] = row
|
||||
}
|
||||
}
|
||||
for _, id := range ids {
|
||||
row := byID[id]
|
||||
if row == nil {
|
||||
if isCleanup {
|
||||
// An earlier attempt already removed this exact resource.
|
||||
continue
|
||||
}
|
||||
return nil, apperrors.NewNotFoundError("knowledge not found")
|
||||
}
|
||||
if isCleanup && cleanup.bindings[id] != row.KnowledgeBaseID {
|
||||
return nil, apperrors.NewForbiddenError("cleanup resource binding changed")
|
||||
}
|
||||
if err := access.RejectMovingKnowledge(row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, ok := plan.kbs[row.KnowledgeBaseID]; !ok {
|
||||
kb, err := knowledgeWriteKB(ctx, s.kbService, row)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !isCleanup {
|
||||
if _, err := requireKBWrite(ctx, kb); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
plan.kbs[kb.ID] = kb
|
||||
}
|
||||
copyOfRow := *row
|
||||
plan.knowledge = append(plan.knowledge, ©OfRow)
|
||||
plan.ids = append(plan.ids, id)
|
||||
}
|
||||
if len(plan.ids) == 0 {
|
||||
return plan, nil
|
||||
}
|
||||
// Resolve owner storage/index context and all KB routing before mutations.
|
||||
ctx, err = withKBWriteTenantInfo(ctx, plan.kbs[plan.knowledge[0].KnowledgeBaseID], s.tenantRepo)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
plan.ctx = ctx
|
||||
for id, kb := range plan.kbs {
|
||||
plan.files[id] = s.resolveFileService(ctx, kb)
|
||||
}
|
||||
plan.imageInfo, err = s.chunkRepo.ListImageInfoByKnowledgeIDs(ctx, tenant, plan.ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
|
@ -32,32 +32,9 @@ func (s *knowledgeService) ListFAQEntries(ctx context.Context,
|
|||
return nil, err
|
||||
}
|
||||
|
||||
// Check if this is a shared knowledge base access
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
effectiveTenantID := tenantID
|
||||
|
||||
// If the kb belongs to a different tenant, check for shared access
|
||||
if kb.TenantID != tenantID {
|
||||
// Get user ID from context
|
||||
userIDVal := ctx.Value(types.UserIDContextKey)
|
||||
if userIDVal == nil {
|
||||
return nil, werrors.NewForbiddenError("无权访问该知识库")
|
||||
}
|
||||
_ = userIDVal.(string) // userID retained only for legacy log fields
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
|
||||
// Check if the caller's tenant has at least viewer permission via org sharing.
|
||||
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, kbID, tenantID, callerTenantRole, types.OrgRoleViewer)
|
||||
if err != nil || !hasPermission {
|
||||
return nil, werrors.NewForbiddenError("无权访问该知识库")
|
||||
}
|
||||
|
||||
// Use the source tenant ID for data access
|
||||
sourceTenantID, err := s.kbShareService.GetKBSourceTenant(ctx, kbID)
|
||||
if err != nil {
|
||||
return nil, werrors.NewForbiddenError("无权访问该知识库")
|
||||
}
|
||||
effectiveTenantID = sourceTenantID
|
||||
effectiveTenantID, err := resolveKBReadTenant(ctx, kb, s.kbShareService)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
faqKnowledge, err := s.findFAQKnowledge(ctx, effectiveTenantID, kb.ID)
|
||||
|
|
@ -141,7 +118,7 @@ func (s *knowledgeService) CreateFAQEntry(ctx context.Context,
|
|||
return nil, werrors.NewBadRequestError("请求体不能为空")
|
||||
}
|
||||
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -342,7 +319,7 @@ func (s *knowledgeService) UpdateFAQEntry(ctx context.Context,
|
|||
if payload == nil {
|
||||
return nil, werrors.NewBadRequestError("请求体不能为空")
|
||||
}
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -510,7 +487,7 @@ func (s *knowledgeService) AddSimilarQuestions(ctx context.Context,
|
|||
return nil, werrors.NewBadRequestError("相似问列表不能为空")
|
||||
}
|
||||
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -655,7 +632,7 @@ func (s *knowledgeService) AddSimilarQuestions(ctx context.Context,
|
|||
func (s *knowledgeService) UpdateFAQEntryStatus(ctx context.Context,
|
||||
kbID string, entryID string, isEnabled bool,
|
||||
) error {
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -704,7 +681,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
if req == nil || (len(req.ByID) == 0 && len(req.ByTag) == 0) {
|
||||
return nil
|
||||
}
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -713,25 +690,17 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
enabledUpdates := make(map[string]bool)
|
||||
tagUpdates := make(map[string]string)
|
||||
|
||||
// Convert exclude seq_ids to UUIDs
|
||||
excludeUUIDs := make([]string, 0, len(req.ExcludeIDs))
|
||||
if len(req.ExcludeIDs) > 0 {
|
||||
excludeChunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, req.ExcludeIDs)
|
||||
if err == nil {
|
||||
for _, c := range excludeChunks {
|
||||
excludeUUIDs = append(excludeUUIDs, c.ID)
|
||||
}
|
||||
}
|
||||
plan, err := s.planFAQFields(ctx, kb, req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
excludeUUIDs := plan.excludeIDs
|
||||
|
||||
// Handle ByTag updates first (by tag seq_id)
|
||||
if len(req.ByTag) > 0 {
|
||||
for tagSeqID, update := range req.ByTag {
|
||||
// Convert tag seq_id to UUID
|
||||
tag, err := s.tagRepo.GetBySeqID(ctx, tenantID, tagSeqID)
|
||||
if err != nil {
|
||||
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", tagSeqID))
|
||||
}
|
||||
for _, tagSeqID := range sortedFAQIDs(req.ByTag) {
|
||||
update := req.ByTag[tagSeqID]
|
||||
tag := plan.tags[tagSeqID]
|
||||
|
||||
var setFlags, clearFlags types.ChunkFlags
|
||||
|
||||
|
|
@ -748,10 +717,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
var newTagUUID *string
|
||||
if update.TagID != nil {
|
||||
if *update.TagID > 0 {
|
||||
newTag, err := s.tagRepo.GetBySeqID(ctx, tenantID, *update.TagID)
|
||||
if err != nil {
|
||||
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", *update.TagID))
|
||||
}
|
||||
newTag := plan.tags[*update.TagID]
|
||||
newTagUUID = &newTag.ID
|
||||
} else {
|
||||
emptyStr := ""
|
||||
|
|
@ -768,6 +734,19 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
return err
|
||||
}
|
||||
|
||||
// Preserve group changes when an explicit ByID patch follows it.
|
||||
for _, id := range affectedIDs {
|
||||
if chunk := plan.chunksByID[id]; chunk != nil {
|
||||
if update.IsEnabled != nil {
|
||||
chunk.IsEnabled = *update.IsEnabled
|
||||
}
|
||||
chunk.Flags = (chunk.Flags | setFlags) &^ clearFlags
|
||||
if newTagUUID != nil {
|
||||
chunk.TagID = *newTagUUID
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Collect affected IDs for retriever sync
|
||||
if len(affectedIDs) > 0 {
|
||||
if update.IsEnabled != nil {
|
||||
|
|
@ -786,33 +765,15 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
|
||||
// Handle ByID updates (by entry seq_id)
|
||||
if len(req.ByID) > 0 {
|
||||
entrySeqIDs := make([]int64, 0, len(req.ByID))
|
||||
for entrySeqID := range req.ByID {
|
||||
entrySeqIDs = append(entrySeqIDs, entrySeqID)
|
||||
}
|
||||
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, entrySeqIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Build chunk seq_id to chunk map
|
||||
chunkBySeqID := make(map[int64]*types.Chunk)
|
||||
for _, chunk := range chunks {
|
||||
chunkBySeqID[chunk.SeqID] = chunk
|
||||
}
|
||||
chunkBySeqID := plan.chunks
|
||||
|
||||
setFlags := make(map[string]types.ChunkFlags)
|
||||
clearFlags := make(map[string]types.ChunkFlags)
|
||||
chunksToUpdate := make([]*types.Chunk, 0)
|
||||
|
||||
for entrySeqID, update := range req.ByID {
|
||||
chunk, exists := chunkBySeqID[entrySeqID]
|
||||
if !exists {
|
||||
continue
|
||||
}
|
||||
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
|
||||
continue
|
||||
}
|
||||
for _, entrySeqID := range sortedFAQIDs(req.ByID) {
|
||||
update := req.ByID[entrySeqID]
|
||||
chunk := chunkBySeqID[entrySeqID]
|
||||
|
||||
needUpdate := false
|
||||
|
||||
|
|
@ -839,10 +800,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
if update.TagID != nil {
|
||||
var newTagID string
|
||||
if *update.TagID > 0 {
|
||||
newTag, err := s.tagRepo.GetBySeqID(ctx, tenantID, *update.TagID)
|
||||
if err != nil {
|
||||
return werrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", *update.TagID))
|
||||
}
|
||||
newTag := plan.tags[*update.TagID]
|
||||
newTagID = newTag.ID
|
||||
}
|
||||
if chunk.TagID != newTagID {
|
||||
|
|
@ -900,7 +858,7 @@ func (s *knowledgeService) UpdateFAQEntryFieldsBatch(ctx context.Context,
|
|||
|
||||
// UpdateFAQEntryTag updates the tag assigned to an FAQ entry.
|
||||
func (s *knowledgeService) UpdateFAQEntryTag(ctx context.Context, kbID string, entryID string, tagID *string) error {
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -948,104 +906,15 @@ func (s *knowledgeService) UpdateFAQEntryTag(ctx context.Context, kbID string, e
|
|||
// UpdateFAQEntryTagBatch updates tags for FAQ entries in batch.
|
||||
// Key: entry seq_id, Value: tag seq_id (nil to remove tag)
|
||||
func (s *knowledgeService) UpdateFAQEntryTagBatch(ctx context.Context, kbID string, updates map[int64]*int64) error {
|
||||
if len(updates) == 0 {
|
||||
return nil
|
||||
req := &types.FAQEntryFieldsBatchUpdate{ByID: make(map[int64]types.FAQEntryFieldsUpdate, len(updates))}
|
||||
for id, tag := range updates {
|
||||
value := int64(0) // nil in the tag API means remove the tag.
|
||||
if tag != nil {
|
||||
value = *tag
|
||||
}
|
||||
req.ByID[id] = types.FAQEntryFieldsUpdate{TagID: &value}
|
||||
}
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
|
||||
// Get all chunks in batch by seq_id
|
||||
entrySeqIDs := make([]int64, 0, len(updates))
|
||||
for entrySeqID := range updates {
|
||||
entrySeqIDs = append(entrySeqIDs, entrySeqID)
|
||||
}
|
||||
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, tenantID, entrySeqIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Build chunk seq_id to chunk map
|
||||
chunkBySeqID := make(map[int64]*types.Chunk)
|
||||
for _, chunk := range chunks {
|
||||
chunkBySeqID[chunk.SeqID] = chunk
|
||||
}
|
||||
|
||||
// Build tag seq_id set for validation
|
||||
tagSeqIDSet := make(map[int64]bool)
|
||||
for _, tagSeqID := range updates {
|
||||
if tagSeqID != nil && *tagSeqID > 0 {
|
||||
tagSeqIDSet[*tagSeqID] = true
|
||||
}
|
||||
}
|
||||
|
||||
// Validate all tags in batch by seq_id
|
||||
tagMap := make(map[int64]*types.KnowledgeTag)
|
||||
if len(tagSeqIDSet) > 0 {
|
||||
tagSeqIDs := make([]int64, 0, len(tagSeqIDSet))
|
||||
for tagSeqID := range tagSeqIDSet {
|
||||
tagSeqIDs = append(tagSeqIDs, tagSeqID)
|
||||
}
|
||||
tags, err := s.tagRepo.GetBySeqIDs(ctx, tenantID, tagSeqIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, tag := range tags {
|
||||
if tag.KnowledgeBaseID != kb.ID {
|
||||
return werrors.NewBadRequestError(fmt.Sprintf("标签 %d 不属于当前知识库", tag.SeqID))
|
||||
}
|
||||
tagMap[tag.SeqID] = tag
|
||||
}
|
||||
}
|
||||
|
||||
// Update chunks
|
||||
chunksToUpdate := make([]*types.Chunk, 0)
|
||||
for entrySeqID, tagSeqID := range updates {
|
||||
chunk, exists := chunkBySeqID[entrySeqID]
|
||||
if !exists {
|
||||
continue
|
||||
}
|
||||
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
|
||||
continue
|
||||
}
|
||||
|
||||
var resolvedTagID string
|
||||
if tagSeqID != nil && *tagSeqID > 0 {
|
||||
tag, ok := tagMap[*tagSeqID]
|
||||
if !ok {
|
||||
return werrors.NewBadRequestError(fmt.Sprintf("标签 %d 不存在", *tagSeqID))
|
||||
}
|
||||
resolvedTagID = tag.ID
|
||||
}
|
||||
|
||||
chunk.TagID = resolvedTagID
|
||||
chunk.UpdatedAt = time.Now()
|
||||
chunksToUpdate = append(chunksToUpdate, chunk)
|
||||
}
|
||||
|
||||
if len(chunksToUpdate) > 0 {
|
||||
if err := s.chunkRepo.UpdateChunks(ctx, chunksToUpdate); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Sync tag updates to retriever engines
|
||||
tagUpdates := make(map[string]string)
|
||||
for _, chunk := range chunksToUpdate {
|
||||
tagUpdates[chunk.ID] = chunk.TagID
|
||||
}
|
||||
retrieveEngine, err := retriever.CreateRetrieveEngineForKB(
|
||||
ctx, s.retrieveEngine, s.ownership, tenantID, kb.VectorStoreID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := retrieveEngine.BatchUpdateChunkTagID(ctx, tagUpdates); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
return s.UpdateFAQEntryFieldsBatch(ctx, kbID, req)
|
||||
}
|
||||
|
||||
// SearchFAQEntries searches FAQ entries using hybrid search.
|
||||
|
|
@ -1389,41 +1258,47 @@ func (s *knowledgeService) DeleteFAQEntries(ctx context.Context,
|
|||
if len(entrySeqIDs) == 0 {
|
||||
return werrors.NewBadRequestError("请选择需要删除的 FAQ 条目")
|
||||
}
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
var faqKnowledge *types.Knowledge
|
||||
chunksToRemove := make([]*types.Chunk, 0, len(entrySeqIDs))
|
||||
for _, seqID := range entrySeqIDs {
|
||||
if seqID <= 0 {
|
||||
continue
|
||||
}
|
||||
chunk, err := s.chunkRepo.GetChunkBySeqID(ctx, tenantID, seqID)
|
||||
if err != nil {
|
||||
return werrors.NewNotFoundError("FAQ条目不存在")
|
||||
}
|
||||
if chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
|
||||
return werrors.NewBadRequestError("包含无效的 FAQ 条目")
|
||||
}
|
||||
if err := s.chunkService.DeleteChunk(ctx, chunk.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
if faqKnowledge == nil {
|
||||
faqKnowledge, err = s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
|
||||
selected, err := s.loadFAQWriteChunks(ctx, kb, entrySeqIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
chunksToRemove := make([]*types.Chunk, 0, len(selected))
|
||||
knowledges := make(map[string]*types.Knowledge)
|
||||
groups := make(map[string][]*types.Chunk)
|
||||
for _, id := range sortedFAQIDs(selected) {
|
||||
chunk := selected[id]
|
||||
if knowledges[chunk.KnowledgeID] == nil {
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if knowledge == nil || knowledge.TenantID != tenantID || knowledge.KnowledgeBaseID != kb.ID ||
|
||||
knowledge.Type != types.KnowledgeTypeFAQ {
|
||||
return werrors.NewForbiddenError("FAQ 文档不属于当前知识库")
|
||||
}
|
||||
knowledges[chunk.KnowledgeID] = knowledge
|
||||
}
|
||||
groups[chunk.KnowledgeID] = append(groups[chunk.KnowledgeID], chunk)
|
||||
chunksToRemove = append(chunksToRemove, chunk)
|
||||
}
|
||||
if len(chunksToRemove) > 0 && faqKnowledge != nil {
|
||||
if err := s.deleteFAQChunkVectors(ctx, kb, faqKnowledge, chunksToRemove); err != nil {
|
||||
// All entries and parent documents are authorized before any deletion.
|
||||
for _, chunk := range chunksToRemove {
|
||||
if err := s.chunkService.DeleteChunk(ctx, chunk.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for id, chunks := range groups {
|
||||
if err := s.deleteFAQChunkVectors(ctx, kb, knowledges[id], chunks); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
details := map[string]any{"count": len(chunksToRemove), "source_type": "faq"}
|
||||
titles := make([]string, 0, len(chunksToRemove))
|
||||
for _, chunk := range chunksToRemove {
|
||||
|
|
@ -1630,6 +1505,9 @@ func (s *knowledgeService) validateFAQKnowledgeBase(ctx context.Context, kbID st
|
|||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if kb == nil || kb.ID != kbID {
|
||||
return nil, werrors.NewNotFoundError("知识库不存在")
|
||||
}
|
||||
kb.EnsureDefaults()
|
||||
if kb.Type != types.KnowledgeBaseTypeFAQ {
|
||||
return nil, werrors.NewBadRequestError("仅 FAQ 知识库支持该操作")
|
||||
|
|
@ -1906,7 +1784,9 @@ func (s *knowledgeService) buildFAQTagResolver(
|
|||
if len(seqIDs) > 0 {
|
||||
if tags, err := s.tagRepo.GetBySeqIDs(ctx, tenantID, seqIDs); err == nil {
|
||||
for _, t := range tags {
|
||||
seqIDToUUID[t.SeqID] = t.ID
|
||||
if validateFAQTagScope(t, tenantID, kbID) == nil {
|
||||
seqIDToUUID[t.SeqID] = t.ID
|
||||
}
|
||||
}
|
||||
} else {
|
||||
logger.Warnf(ctx, "buildFAQTagResolver: batch GetBySeqIDs failed (%d ids), fallback per-entry: %v",
|
||||
|
|
@ -1953,6 +1833,9 @@ func (s *knowledgeService) resolveTagID(ctx context.Context, kbID string, payloa
|
|||
if err != nil {
|
||||
return "", fmt.Errorf("failed to find tag by seq_id %d: %w", payload.TagID, err)
|
||||
}
|
||||
if err := validateFAQTagScope(tag, tenantID, kbID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return tag.ID, nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,149 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
||||
func sortedFAQIDs[V any](values map[int64]V) []int64 {
|
||||
ids := make([]int64, 0, len(values))
|
||||
for id := range values {
|
||||
ids = append(ids, id)
|
||||
}
|
||||
sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] })
|
||||
return ids
|
||||
}
|
||||
|
||||
// loadFAQWriteChunks validates the entire selection before the first mutation.
|
||||
// Copies keep planning from mutating repository-owned snapshots on rejection.
|
||||
func (s *knowledgeService) loadFAQWriteChunks(
|
||||
ctx context.Context,
|
||||
kb *types.KnowledgeBase,
|
||||
ids []int64,
|
||||
) (map[int64]*types.Chunk, error) {
|
||||
wanted := make(map[int64]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
if id <= 0 {
|
||||
return nil, apperrors.NewBadRequestError("FAQ 条目 ID 必须为正整数")
|
||||
}
|
||||
wanted[id] = true
|
||||
}
|
||||
result := make(map[int64]*types.Chunk, len(wanted))
|
||||
if len(wanted) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
chunks, err := s.chunkRepo.ListChunksBySeqID(ctx, kb.TenantID, sortedFAQIDs(wanted))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, chunk := range chunks {
|
||||
if chunk == nil || !wanted[chunk.SeqID] {
|
||||
continue
|
||||
}
|
||||
if chunk.TenantID != kb.TenantID || chunk.KnowledgeBaseID != kb.ID || chunk.ChunkType != types.ChunkTypeFAQ {
|
||||
return nil, apperrors.NewForbiddenError("FAQ 条目不属于当前知识库")
|
||||
}
|
||||
snapshot := *chunk
|
||||
result[chunk.SeqID] = &snapshot
|
||||
}
|
||||
if len(result) != len(wanted) {
|
||||
return nil, apperrors.NewNotFoundError("FAQ 条目不存在")
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
type faqFieldPlan struct {
|
||||
chunks map[int64]*types.Chunk
|
||||
chunksByID map[string]*types.Chunk
|
||||
tags map[int64]*types.KnowledgeTag
|
||||
excludeIDs []string
|
||||
}
|
||||
|
||||
func (s *knowledgeService) planFAQFields(
|
||||
ctx context.Context,
|
||||
kb *types.KnowledgeBase,
|
||||
req *types.FAQEntryFieldsBatchUpdate,
|
||||
) (*faqFieldPlan, error) {
|
||||
ids := append(sortedFAQIDs(req.ByID), req.ExcludeIDs...)
|
||||
chunks, err := s.loadFAQWriteChunks(ctx, kb, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
plan := &faqFieldPlan{
|
||||
chunks: chunks,
|
||||
chunksByID: make(map[string]*types.Chunk),
|
||||
tags: make(map[int64]*types.KnowledgeTag),
|
||||
}
|
||||
for _, chunk := range chunks {
|
||||
plan.chunksByID[chunk.ID] = chunk
|
||||
}
|
||||
for _, id := range req.ExcludeIDs {
|
||||
plan.excludeIDs = append(plan.excludeIDs, chunks[id].ID)
|
||||
}
|
||||
tagIDs := make(map[int64]bool)
|
||||
for id := range req.ByTag {
|
||||
if id <= 0 {
|
||||
return nil, apperrors.NewBadRequestError("标签 ID 必须为正整数")
|
||||
}
|
||||
tagIDs[id] = true
|
||||
}
|
||||
for _, updates := range []map[int64]types.FAQEntryFieldsUpdate{req.ByTag, req.ByID} {
|
||||
for _, update := range updates {
|
||||
if update.TagID != nil && *update.TagID > 0 {
|
||||
tagIDs[*update.TagID] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(tagIDs) > 0 {
|
||||
tags, err := s.tagRepo.GetBySeqIDs(ctx, kb.TenantID, sortedFAQIDs(tagIDs))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, tag := range tags {
|
||||
if tag == nil || !tagIDs[tag.SeqID] {
|
||||
continue
|
||||
}
|
||||
if err := validateFAQTagScope(tag, kb.TenantID, kb.ID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
plan.tags[tag.SeqID] = tag
|
||||
}
|
||||
for _, id := range sortedFAQIDs(tagIDs) {
|
||||
if plan.tags[id] == nil {
|
||||
return nil, apperrors.NewNotFoundError(fmt.Sprintf("标签 %d 不存在", id))
|
||||
}
|
||||
}
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
func validateFAQTagScope(tag *types.KnowledgeTag, tenantID uint64, kbID string) error {
|
||||
if tag == nil {
|
||||
return apperrors.NewNotFoundError("标签不存在")
|
||||
}
|
||||
if tag.TenantID != tenantID || tag.KnowledgeBaseID != kbID {
|
||||
return apperrors.NewForbiddenError("标签不属于当前知识库")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *knowledgeService) validateFAQImportTags(
|
||||
ctx context.Context,
|
||||
kb *types.KnowledgeBase,
|
||||
entries []types.FAQEntryPayload,
|
||||
) error {
|
||||
// Import still reports per-entry content validation failures. Referenced
|
||||
// resources must all be in scope before admission or execution, however.
|
||||
req := &types.FAQEntryFieldsBatchUpdate{ByTag: make(map[int64]types.FAQEntryFieldsUpdate)}
|
||||
for _, entry := range entries {
|
||||
if entry.TagID != 0 {
|
||||
req.ByTag[entry.TagID] = types.FAQEntryFieldsUpdate{}
|
||||
}
|
||||
}
|
||||
_, err := s.planFAQFields(ctx, kb, req)
|
||||
return err
|
||||
}
|
||||
|
|
@ -11,6 +11,8 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
werrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
|
|
@ -39,10 +41,13 @@ func (s *knowledgeService) UpsertFAQEntries(ctx context.Context,
|
|||
}
|
||||
|
||||
// 验证知识库是否存在且有效
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := s.validateFAQImportTags(ctx, kb, payload.Entries); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
|
||||
|
|
@ -1386,7 +1391,7 @@ func (s *knowledgeService) executeFAQImport(ctx context.Context, taskID string,
|
|||
}
|
||||
}()
|
||||
|
||||
kb, err = s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
kb, ctx, err = s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -2245,7 +2250,29 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
|
|||
|
||||
ctx = logger.WithRequestID(ctx, uuid.New().String())
|
||||
ctx = logger.WithField(ctx, "faq_import", payload.TaskID)
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, payload.KBID)
|
||||
if err != nil {
|
||||
if errors.Is(err, repository.ErrKnowledgeBaseNotFound) {
|
||||
return fmt.Errorf("%w: FAQ task KB no longer exists", asynq.SkipRetry)
|
||||
}
|
||||
return err
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: FAQ task KB does not belong to its tenant", asynq.SkipRetry)
|
||||
}
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
|
||||
if err != nil {
|
||||
if errors.Is(err, repository.ErrKnowledgeNotFound) {
|
||||
return fmt.Errorf("%w: FAQ task document no longer exists", asynq.SkipRetry)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if knowledge == nil || knowledge.TenantID != payload.TenantID || knowledge.KnowledgeBaseID != payload.KBID ||
|
||||
knowledge.Type != types.KnowledgeTypeFAQ {
|
||||
return fmt.Errorf("%w: FAQ task document does not belong to its KB", asynq.SkipRetry)
|
||||
}
|
||||
|
||||
// 获取任务重试信息,用于判断是否是最后一次重试
|
||||
retryCount, _ := asynq.GetRetryCount(ctx)
|
||||
|
|
@ -2287,6 +2314,9 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
|
|||
|
||||
logger.Infof(ctx, "Processing FAQ import task: task_id=%s, kb_id=%s, total_entries=%d, dry_run=%v, retry=%d/%d",
|
||||
payload.TaskID, payload.KBID, len(payload.Entries), payload.DryRun, retryCount, maxRetry)
|
||||
if err := s.validateFAQImportTags(ctx, kb, payload.Entries); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 保存原始总数量
|
||||
originalTotalEntries := len(payload.Entries)
|
||||
|
|
@ -2361,31 +2391,6 @@ func (s *knowledgeService) ProcessFAQImport(ctx context.Context, t *asynq.Task)
|
|||
logger.Warnf(ctx, "Failed to update FAQ import progress: %v", err)
|
||||
}
|
||||
|
||||
// 幂等性检查:获取knowledge记录(FAQ任务使用knowledge ID作为taskID)
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, payload.TenantID, payload.KnowledgeID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "failed to get FAQ knowledge: %v", err)
|
||||
return nil
|
||||
}
|
||||
|
||||
if knowledge == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, payload.KBID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get knowledge base: %v", err)
|
||||
// 如果是最后一次重试,更新状态为失败
|
||||
if isLastRetry {
|
||||
if updateErr := s.updateFAQImportProgressStatus(ctx, payload.TaskID, payload.InstanceID, payload.EnqueuedAt, types.FAQImportStatusFailed, 0, originalTotalEntries, 0, "获取知识库失败", err.Error()); updateErr != nil {
|
||||
logger.Errorf(ctx, "Failed to update task status to failed: %v", updateErr)
|
||||
}
|
||||
s.recordFAQImportKBActivity(ctx, &payload, progress, originalTotalEntries, types.AuditActionFAQImportFailed, types.AuditOutcomeFailed)
|
||||
}
|
||||
s.cleanupFAQEntriesFileOnFinalFailure(ctx, payload.EntriesURL, retryCount, maxRetry)
|
||||
return fmt.Errorf("failed to get knowledge base: %w", err)
|
||||
}
|
||||
|
||||
// 检查任务状态 - 幂等性处理(复用之前获取的 existingProgress)
|
||||
var processedCount int
|
||||
if existingProgress != nil {
|
||||
|
|
@ -2825,8 +2830,11 @@ func (s *knowledgeService) UpdateLastFAQImportResultDisplayStatus(ctx context.Co
|
|||
return werrors.NewBadRequestError("invalid display status, must be 'open' or 'close'")
|
||||
}
|
||||
|
||||
// 获取当前空间ID
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
kb, ctx, err := s.writableFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tenantID := kb.TenantID
|
||||
|
||||
// 查找FAQ类型的knowledge
|
||||
knowledgeList, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, tenantID, kbID)
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
|
@ -40,12 +41,23 @@ func (r *folderMoveRepoStub) RenameKnowledgeFolderPath(
|
|||
}
|
||||
|
||||
func folderMoveContext() context.Context {
|
||||
return context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
|
||||
return (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 1,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 1,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
}
|
||||
|
||||
func TestMoveKnowledgeToFolderNormalizesDestination(t *testing.T) {
|
||||
repo := &folderMoveRepoStub{}
|
||||
svc := &knowledgeService{repo: repo}
|
||||
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
|
||||
|
||||
affected, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1", "k2"}, "/docs//spec/")
|
||||
require.NoError(t, err)
|
||||
|
|
@ -56,7 +68,7 @@ func TestMoveKnowledgeToFolderNormalizesDestination(t *testing.T) {
|
|||
|
||||
func TestMoveKnowledgeToFolderAcceptsRootAndRejectsEmptyBatch(t *testing.T) {
|
||||
repo := &folderMoveRepoStub{}
|
||||
svc := &knowledgeService{repo: repo}
|
||||
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
|
||||
|
||||
// The empty destination is meaningful: it files documents back at the top level.
|
||||
_, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1"}, "")
|
||||
|
|
@ -70,7 +82,7 @@ func TestMoveKnowledgeToFolderAcceptsRootAndRejectsEmptyBatch(t *testing.T) {
|
|||
|
||||
func TestMoveKnowledgeToFolderRejectsTraversalAndUnsafeInput(t *testing.T) {
|
||||
repo := &folderMoveRepoStub{}
|
||||
svc := &knowledgeService{repo: repo}
|
||||
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
|
||||
|
||||
_, err := svc.MoveKnowledgeToFolder(folderMoveContext(), "kb-1", []string{"k1"}, "../../etc/docs")
|
||||
require.NoError(t, err)
|
||||
|
|
@ -82,7 +94,7 @@ func TestMoveKnowledgeToFolderRejectsTraversalAndUnsafeInput(t *testing.T) {
|
|||
|
||||
func TestRenameKnowledgeFolderValidatesPaths(t *testing.T) {
|
||||
repo := &folderMoveRepoStub{}
|
||||
svc := &knowledgeService{repo: repo}
|
||||
svc := &knowledgeService{repo: repo, kbService: &writeKBLookup{kb: &types.KnowledgeBase{ID: "kb-1", TenantID: 1}}}
|
||||
ctx := folderMoveContext()
|
||||
|
||||
affected, err := svc.RenameKnowledgeFolder(ctx, "kb-1", "docs", "handbook")
|
||||
|
|
@ -112,3 +124,15 @@ func TestRenameKnowledgeFolderValidatesPaths(t *testing.T) {
|
|||
assert.Equal(t, int64(0), affected)
|
||||
assert.Equal(t, before, repo.renameCalls)
|
||||
}
|
||||
|
||||
func (r *folderMoveRepoStub) GetKnowledgeBatch(
|
||||
_ context.Context,
|
||||
tenant uint64,
|
||||
ids []string,
|
||||
) ([]*types.Knowledge, error) {
|
||||
rows := make([]*types.Knowledge, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
rows = append(rows, &types.Knowledge{ID: id, TenantID: tenant, KnowledgeBaseID: "kb-1"})
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
|
|
@ -74,7 +75,7 @@ type metadataUpdateKBService struct {
|
|||
func (metadataUpdateKBService) GetKnowledgeBaseByID(
|
||||
_ context.Context, id string,
|
||||
) (*types.KnowledgeBase, error) {
|
||||
return &types.KnowledgeBase{ID: id, SummaryModelID: "summary-model"}, nil
|
||||
return &types.KnowledgeBase{ID: id, TenantID: 7, SummaryModelID: "summary-model"}, nil
|
||||
}
|
||||
|
||||
type metadataUpdateTaskEnqueuer struct {
|
||||
|
|
@ -114,6 +115,17 @@ func TestUpdateKnowledgeMetadataDoesNotReindexChunks(t *testing.T) {
|
|||
repo: repo, chunkRepo: chunkRepo, kbService: metadataUpdateKBService{}, task: taskEnqueuer,
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
err := service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
@ -143,6 +155,17 @@ func TestUpdateKnowledgeMetadataDoesNotLeaveTasklessPendingStatus(t *testing.T)
|
|||
task: &metadataUpdateTaskEnqueuer{err: errors.New("queue unavailable")},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
require.NoError(t, service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1", CustomMetadata: types.JSON(`{"region":"Shanghai"}`),
|
||||
|
|
@ -162,6 +185,17 @@ func TestUpdateKnowledgeUnchangedMetadataDoesNotRefreshSummary(t *testing.T) {
|
|||
repo: repo, chunkRepo: &metadataUpdateChunkRepo{}, kbService: metadataUpdateKBService{}, task: taskEnqueuer,
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
require.NoError(t, service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
|
|||
|
|
@ -2,8 +2,11 @@ package service
|
|||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
|
@ -19,6 +22,7 @@ import (
|
|||
type moveWikiKnowledgeRepo struct {
|
||||
interfaces.KnowledgeRepository
|
||||
knowledge *types.Knowledge
|
||||
commitErr error
|
||||
}
|
||||
|
||||
func (r *moveWikiKnowledgeRepo) GetKnowledgeByID(
|
||||
|
|
@ -40,22 +44,33 @@ func (r *moveWikiKnowledgeRepo) DeleteKnowledgeTagRelations(_ context.Context, _
|
|||
|
||||
type moveWikiPageRepo struct {
|
||||
interfaces.WikiPageRepository
|
||||
listedKBs []string
|
||||
listedKBs []string
|
||||
listErr error
|
||||
beforeList func()
|
||||
pages []*types.WikiPage
|
||||
}
|
||||
|
||||
func (r *moveWikiPageRepo) ListBySourceRef(
|
||||
_ context.Context, kbID, _ string,
|
||||
) ([]*types.WikiPage, error) {
|
||||
r.listedKBs = append(r.listedKBs, kbID)
|
||||
return nil, nil
|
||||
if r.beforeList != nil {
|
||||
r.beforeList()
|
||||
}
|
||||
return r.pages, r.listErr
|
||||
}
|
||||
|
||||
type moveWikiPendingRepo struct {
|
||||
interfaces.TaskPendingOpsRepository
|
||||
ops []*types.TaskPendingOp
|
||||
ops []*types.TaskPendingOp
|
||||
err error
|
||||
failKB string
|
||||
}
|
||||
|
||||
func (r *moveWikiPendingRepo) Enqueue(_ context.Context, op *types.TaskPendingOp) error {
|
||||
if r.err != nil && (r.failKB == "" || r.failKB == op.ScopeID) {
|
||||
return r.err
|
||||
}
|
||||
r.ops = append(r.ops, op)
|
||||
return nil
|
||||
}
|
||||
|
|
@ -67,6 +82,8 @@ func (r *moveWikiPendingRepo) DeleteByDedupKey(_ context.Context, _, _, _, _, _
|
|||
type moveWikiChunkRepo struct {
|
||||
interfaces.ChunkRepository
|
||||
movedToKB string
|
||||
moves int
|
||||
moveErr error
|
||||
}
|
||||
|
||||
func (r *moveWikiChunkRepo) ListChunksByKnowledgeID(
|
||||
|
|
@ -78,6 +95,10 @@ func (r *moveWikiChunkRepo) ListChunksByKnowledgeID(
|
|||
func (r *moveWikiChunkRepo) MoveChunksByKnowledgeID(
|
||||
_ context.Context, _ uint64, _ string, targetKBID string,
|
||||
) error {
|
||||
r.moves++
|
||||
if r.moveErr != nil {
|
||||
return r.moveErr
|
||||
}
|
||||
r.movedToKB = targetKBID
|
||||
return nil
|
||||
}
|
||||
|
|
@ -85,6 +106,7 @@ func (r *moveWikiChunkRepo) MoveChunksByKnowledgeID(
|
|||
func wikiEnabledKB(id string) *types.KnowledgeBase {
|
||||
return &types.KnowledgeBase{
|
||||
ID: id,
|
||||
TenantID: 1,
|
||||
IndexingStrategy: types.IndexingStrategy{WikiEnabled: true},
|
||||
}
|
||||
}
|
||||
|
|
@ -124,25 +146,25 @@ func newMoveWikiService(t *testing.T) (
|
|||
}
|
||||
|
||||
func moveWikiCtx() context.Context {
|
||||
return context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1))
|
||||
ctx, _ := access.WithKBTransferTask(
|
||||
context.Background(),
|
||||
wikiEnabledKB("kb-src"),
|
||||
wikiEnabledKB("kb-dst"),
|
||||
1,
|
||||
access.KBTransferMove,
|
||||
"move-task",
|
||||
false,
|
||||
)
|
||||
return ctx
|
||||
}
|
||||
|
||||
func TestMoveOneKnowledgeRetractsWikiFromSourceKB(t *testing.T) {
|
||||
// An unknown mode makes the move itself a no-op, which pins the cleanup as
|
||||
// unconditional: it runs before the mode dispatch, while the knowledge still
|
||||
// belongs to the source KB.
|
||||
func TestMoveOneKnowledgeRejectsUnknownModeBeforeWikiCleanup(t *testing.T) {
|
||||
svc, wikiRepo, pendingRepo, _ := newMoveWikiService(t)
|
||||
|
||||
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1",
|
||||
wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "bogus")
|
||||
|
||||
require.Error(t, err)
|
||||
assert.Equal(t, []string{"kb-src"}, wikiRepo.listedKBs)
|
||||
|
||||
srcOps := opsFor(pendingRepo.ops, "kb-src")
|
||||
require.Len(t, srcOps, 1)
|
||||
assert.Equal(t, WikiOpRetract, srcOps[0].Op)
|
||||
assert.Equal(t, "kn-1", srcOps[0].DedupKey)
|
||||
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1", wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "bogus")
|
||||
require.ErrorContains(t, err, "unknown move mode")
|
||||
require.Empty(t, wikiRepo.listedKBs)
|
||||
require.Empty(t, pendingRepo.ops)
|
||||
require.Equal(t, types.ParseStatusCompleted, svc.repo.(*moveWikiKnowledgeRepo).knowledge.ParseStatus)
|
||||
}
|
||||
|
||||
func TestMoveOneKnowledgeReuseVectorsIngestsIntoTargetKB(t *testing.T) {
|
||||
|
|
@ -165,10 +187,187 @@ func TestMoveOneKnowledgeReuseVectorsIngestsIntoTargetKB(t *testing.T) {
|
|||
func TestMoveOneKnowledgeSkipsWikiWorkForNonWikiKBs(t *testing.T) {
|
||||
svc, wikiRepo, pendingRepo, _ := newMoveWikiService(t)
|
||||
|
||||
err := svc.moveOneKnowledge(moveWikiCtx(), "kn-1",
|
||||
&types.KnowledgeBase{ID: "kb-src"}, &types.KnowledgeBase{ID: "kb-dst"}, "reuse_vectors")
|
||||
err := svc.moveOneKnowledge(
|
||||
moveWikiCtx(),
|
||||
"kn-1",
|
||||
&types.KnowledgeBase{
|
||||
ID: "kb-src",
|
||||
TenantID: 1,
|
||||
},
|
||||
&types.KnowledgeBase{ID: "kb-dst", TenantID: 1},
|
||||
"reuse_vectors",
|
||||
)
|
||||
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, wikiRepo.listedKBs)
|
||||
assert.Empty(t, pendingRepo.ops)
|
||||
}
|
||||
|
||||
func (r *moveWikiKnowledgeRepo) UpdateKnowledgeForTransfer(
|
||||
ctx context.Context,
|
||||
_ *types.Knowledge,
|
||||
after *types.Knowledge,
|
||||
) error {
|
||||
if after.KnowledgeBaseID == "kb-dst" && r.commitErr != nil {
|
||||
return r.commitErr
|
||||
}
|
||||
return r.UpdateKnowledge(ctx, after)
|
||||
}
|
||||
|
||||
func (r *moveWikiChunkRepo) ListAllChunksByKnowledgeID(context.Context, uint64, string) ([]*types.Chunk, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type moveWikiMetaService struct {
|
||||
interfaces.WikiPageService
|
||||
updated []*types.WikiPage
|
||||
}
|
||||
|
||||
func (s *moveWikiMetaService) UpdatePageMeta(_ context.Context, page *types.WikiPage) error {
|
||||
s.updated = append(s.updated, page)
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestMoveWikiCleanupRunsOnlyAfterOwnershipCommit(t *testing.T) {
|
||||
for _, stage := range []string{"chunks", "CAS"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
svc, wiki, pending, chunks := newMoveWikiService(t)
|
||||
repo := svc.repo.(*moveWikiKnowledgeRepo)
|
||||
failure := errors.New("transfer failed")
|
||||
if stage == "chunks" {
|
||||
chunks.moveErr = failure
|
||||
} else {
|
||||
repo.commitErr = failure
|
||||
}
|
||||
err := svc.moveOneKnowledge(
|
||||
moveWikiCtx(),
|
||||
"kn-1",
|
||||
wikiEnabledKB("kb-src"),
|
||||
wikiEnabledKB("kb-dst"),
|
||||
"reuse_vectors",
|
||||
)
|
||||
require.ErrorIs(t, err, failure)
|
||||
require.Equal(t, "kb-src", repo.knowledge.KnowledgeBaseID)
|
||||
require.Empty(t, wiki.listedKBs)
|
||||
require.Empty(t, pending.ops)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoveWikiCleanupFailureRetriesWithoutMovingAgain(t *testing.T) {
|
||||
for _, stage := range []string{"lookup", "enqueue", "target enqueue"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
svc, wiki, pending, chunks := newMoveWikiService(t)
|
||||
repo := svc.repo.(*moveWikiKnowledgeRepo)
|
||||
failure := errors.New("wiki unavailable")
|
||||
if stage == "lookup" {
|
||||
wiki.listErr = failure
|
||||
} else {
|
||||
pending.err = failure
|
||||
if stage == "target enqueue" {
|
||||
pending.failKB = "kb-dst"
|
||||
}
|
||||
}
|
||||
wiki.beforeList = func() { require.Equal(t, "kb-dst", repo.knowledge.KnowledgeBaseID) }
|
||||
err := svc.moveOneKnowledge(
|
||||
moveWikiCtx(),
|
||||
"kn-1",
|
||||
wikiEnabledKB("kb-src"),
|
||||
wikiEnabledKB("kb-dst"),
|
||||
"reuse_vectors",
|
||||
)
|
||||
require.ErrorIs(t, err, failure)
|
||||
require.Equal(t, "kb-dst", repo.knowledge.KnowledgeBaseID)
|
||||
require.Equal(t, []string{"kb-src"}, wiki.listedKBs)
|
||||
require.Empty(t, opsFor(pending.ops, "kb-dst"))
|
||||
wiki.listErr = nil
|
||||
pending.err = nil
|
||||
require.NoError(
|
||||
t,
|
||||
svc.moveOneKnowledge(
|
||||
moveWikiCtx(),
|
||||
"kn-1",
|
||||
wikiEnabledKB("kb-src"),
|
||||
wikiEnabledKB("kb-dst"),
|
||||
"reuse_vectors",
|
||||
),
|
||||
)
|
||||
require.Equal(t, 1, chunks.moves)
|
||||
require.Equal(t, []string{"kb-src", "kb-src"}, wiki.listedKBs)
|
||||
require.NotEmpty(t, opsFor(pending.ops, "kb-dst"))
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoveReparseWikiUsesDurableSourceRefsAfterChunksDeleted(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
f.kbs.values["kb"].IndexingStrategy.WikiEnabled = true
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
row.SetManualMetadata(types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1)),
|
||||
)
|
||||
row.Description = "source summary"
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
page := &types.WikiPage{
|
||||
ID: "page",
|
||||
KnowledgeBaseID: "kb",
|
||||
SourceRefs: types.StringArray{"doc", "other-doc"},
|
||||
ChunkRefs: types.StringArray{"chunk", "other-chunk"},
|
||||
}
|
||||
wiki := &moveWikiPageRepo{pages: []*types.WikiPage{page}}
|
||||
meta := &moveWikiMetaService{}
|
||||
pending := &moveWikiPendingRepo{}
|
||||
wiki.beforeList = func() {
|
||||
saved, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", saved.KnowledgeBaseID)
|
||||
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, chunks)
|
||||
}
|
||||
f.svc.wikiRepo = wiki
|
||||
f.svc.wikiService = meta
|
||||
f.svc.taskPendingRepo = pending
|
||||
f.svc.task = &transferQueue{}
|
||||
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, "reparse")))
|
||||
require.Equal(t, []string{"kb"}, wiki.listedKBs)
|
||||
require.Len(t, meta.updated, 1)
|
||||
require.Equal(t, types.StringArray{"other-doc"}, meta.updated[0].SourceRefs)
|
||||
require.Equal(t, types.StringArray{"other-chunk"}, meta.updated[0].ChunkRefs)
|
||||
ops := opsFor(pending.ops, "kb")
|
||||
require.Len(t, ops, 1)
|
||||
var payload WikiPendingOp
|
||||
require.NoError(t, json.Unmarshal(ops[0].Payload, &payload))
|
||||
require.Equal(t, "source summary", payload.DocSummary)
|
||||
}
|
||||
|
||||
func TestMoveWikiRetractionIsDurableBeforeRemovingPageSources(t *testing.T) {
|
||||
svc, wiki, pending, chunks := newMoveWikiService(t)
|
||||
page := &types.WikiPage{ID: "page", Slug: "concept/source", SourceRefs: types.StringArray{"kn-1", "other"}}
|
||||
wiki.pages = []*types.WikiPage{page}
|
||||
meta := &moveWikiMetaService{}
|
||||
svc.wikiService = meta
|
||||
pending.err = errors.New("enqueue unavailable")
|
||||
err := svc.moveOneKnowledge(
|
||||
moveWikiCtx(),
|
||||
"kn-1",
|
||||
wikiEnabledKB("kb-src"),
|
||||
wikiEnabledKB("kb-dst"),
|
||||
"reuse_vectors",
|
||||
)
|
||||
require.ErrorIs(t, err, pending.err)
|
||||
require.Empty(t, meta.updated)
|
||||
require.Contains(t, page.SourceRefs, "kn-1")
|
||||
pending.err = nil
|
||||
require.NoError(
|
||||
t,
|
||||
svc.moveOneKnowledge(moveWikiCtx(), "kn-1", wikiEnabledKB("kb-src"), wikiEnabledKB("kb-dst"), "reuse_vectors"),
|
||||
)
|
||||
require.Equal(t, 1, chunks.moves)
|
||||
require.Len(t, meta.updated, 1)
|
||||
var op WikiPendingOp
|
||||
require.NoError(t, json.Unmarshal(opsFor(pending.ops, "kb-src")[0].Payload, &op))
|
||||
require.Equal(t, []string{"concept/source"}, op.PageSlugs)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import (
|
|||
"sort"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/tracing/langfuse"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
|
|
@ -83,11 +84,39 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
|
|||
|
||||
logger.Infof(ctx, "[KnowledgePostProcess] Orchestrating post processing for knowledge: %s", payload.KnowledgeID)
|
||||
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
if payload.Language != "" {
|
||||
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
|
||||
}
|
||||
|
||||
// 1. Fetch Knowledge and KB
|
||||
knowledge, err := s.knowledgeRepo.GetKnowledgeByIDOnly(ctx, payload.KnowledgeID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("get knowledge %s: %w", payload.KnowledgeID, err)
|
||||
}
|
||||
if knowledge == nil {
|
||||
logger.Warnf(ctx, "[KnowledgePostProcess] Knowledge %s not found, aborting.", payload.KnowledgeID)
|
||||
return nil
|
||||
}
|
||||
|
||||
if err := validateProcessingKnowledge(knowledge,
|
||||
payload.TenantID,
|
||||
payload.KnowledgeBaseID,
|
||||
payload.KnowledgeID); err != nil {
|
||||
return err
|
||||
}
|
||||
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
|
||||
if err != nil || kb == nil {
|
||||
return fmt.Errorf("get knowledge base %s: %w", payload.KnowledgeBaseID, err)
|
||||
}
|
||||
|
||||
if kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
|
||||
return fmt.Errorf("postprocess task KB owner changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid postprocess scope: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
// Resolve attempt: payload carries it from the upstream stage, but
|
||||
// fall back to the latest known attempt for compatibility with
|
||||
// in-flight tasks queued before this code shipped.
|
||||
|
|
@ -108,16 +137,6 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
|
|||
|
||||
postSpan := s.tracker().BeginStage(ctx, payload.KnowledgeID, attempt, types.StagePostProcess, nil)
|
||||
|
||||
// 1. Fetch Knowledge and KB
|
||||
knowledge, err := s.knowledgeRepo.GetKnowledgeByIDOnly(ctx, payload.KnowledgeID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("get knowledge %s: %w", payload.KnowledgeID, err)
|
||||
}
|
||||
if knowledge == nil {
|
||||
logger.Warnf(ctx, "[KnowledgePostProcess] Knowledge %s not found, aborting.", payload.KnowledgeID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Skip post-processing entirely when the knowledge has been cancelled
|
||||
// by the user or marked for deletion. We must NOT enqueue summary /
|
||||
// question / graph / wiki child tasks for an aborted knowledge. We
|
||||
|
|
@ -136,11 +155,6 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
|
|||
return nil
|
||||
}
|
||||
|
||||
kb, err := s.kbService.GetKnowledgeBaseByIDOnly(ctx, payload.KnowledgeBaseID)
|
||||
if err != nil || kb == nil {
|
||||
return fmt.Errorf("get knowledge base %s: %w", payload.KnowledgeBaseID, err)
|
||||
}
|
||||
|
||||
processOverrides, _ := knowledge.ProcessOverrides()
|
||||
eff := ResolveProcessConfig(kb, processOverrides)
|
||||
|
||||
|
|
@ -150,6 +164,13 @@ func (s *KnowledgePostProcessService) Handle(ctx context.Context, task *asynq.Ta
|
|||
return fmt.Errorf("list chunks for knowledge %s: %w", payload.KnowledgeID, err)
|
||||
}
|
||||
|
||||
for _, chunk := range chunks {
|
||||
if chunk == nil || chunk.TenantID != payload.TenantID || chunk.KnowledgeID != payload.KnowledgeID ||
|
||||
chunk.KnowledgeBaseID != payload.KnowledgeBaseID {
|
||||
return fmt.Errorf("postprocess chunk binding changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
}
|
||||
|
||||
// Gather all text-like chunks (including newly added OCR and Caption from multimodal tasks)
|
||||
var textChunks []*types.Chunk
|
||||
for _, c := range chunks {
|
||||
|
|
|
|||
|
|
@ -117,21 +117,30 @@ func newWikiEnqueueTestService(
|
|||
) (*KnowledgePostProcessService, *wikiEnqueueFailureKnowledgeRepo) {
|
||||
repo := &wikiEnqueueFailureKnowledgeRepo{
|
||||
knowledge: &types.Knowledge{
|
||||
ID: knowledgeID,
|
||||
ParseStatus: types.ParseStatusProcessing,
|
||||
ID: knowledgeID,
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb-wiki",
|
||||
ParseStatus: types.ParseStatusProcessing,
|
||||
},
|
||||
}
|
||||
pendingRepo.knowledgeRepo = repo
|
||||
return &KnowledgePostProcessService{
|
||||
knowledgeRepo: repo,
|
||||
kbService: &wikiEnqueueFailureKBService{kb: &types.KnowledgeBase{
|
||||
ID: "kb-wiki",
|
||||
ID: "kb-wiki",
|
||||
TenantID: 7,
|
||||
IndexingStrategy: types.IndexingStrategy{
|
||||
WikiEnabled: true,
|
||||
},
|
||||
}},
|
||||
chunkService: &wikiEnqueueFailureChunkService{chunks: []*types.Chunk{
|
||||
{ID: "chunk-1", ChunkType: types.ChunkTypeText},
|
||||
{
|
||||
ID: "chunk-1",
|
||||
TenantID: 7,
|
||||
KnowledgeID: knowledgeID,
|
||||
KnowledgeBaseID: "kb-wiki",
|
||||
ChunkType: types.ChunkTypeText,
|
||||
},
|
||||
}},
|
||||
taskEnqueuer: queue,
|
||||
pendingRepo: pendingRepo,
|
||||
|
|
@ -219,3 +228,13 @@ func TestKnowledgePostProcessRetriesWikiTriggerWithoutDoubleAccounting(t *testin
|
|||
queue.taskTypes,
|
||||
)
|
||||
}
|
||||
|
||||
func TestPostProcessRejectsMovedKnowledgeBeforeWikiOrSpanWrites(t *testing.T) {
|
||||
pending := &wikiEnqueueFailurePendingRepo{}
|
||||
queue := &wikiEnqueueFailureTaskQueue{}
|
||||
svc, repo := newWikiEnqueueTestService("moved-doc", pending, queue)
|
||||
repo.knowledge.KnowledgeBaseID = "other-kb"
|
||||
require.ErrorIs(t, svc.Handle(context.Background(), newWikiEnqueuePostProcessTask(t, "moved-doc")), asynq.SkipRetry)
|
||||
require.Zero(t, repo.expectedSubtasks)
|
||||
require.Equal(t, types.ParseStatusProcessing, repo.knowledge.ParseStatus)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
"github.com/Tencent/WeKnora/internal/common"
|
||||
werrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
|
|
@ -32,13 +34,21 @@ func (s *knowledgeService) cloneKnowledge(
|
|||
src *types.Knowledge,
|
||||
targetKB *types.KnowledgeBase,
|
||||
) (err error) {
|
||||
if src.ParseStatus != "completed" {
|
||||
logger.GetLogger(ctx).WithField("knowledge_id", src.ID).Errorf("MoveKnowledge parse status is not completed")
|
||||
return nil
|
||||
sourceKB, err := knowledgeWriteKB(ctx, s.kbService, src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := access.RequireKBTransfer(ctx, sourceKB, targetKB, access.KBTransferClone); err != nil {
|
||||
return err
|
||||
}
|
||||
if src.ParseStatus != types.ParseStatusCompleted {
|
||||
return fmt.Errorf("source knowledge %s is not completed", src.ID)
|
||||
}
|
||||
if _, err := s.transferChunks(ctx, src, sourceKB.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
tenantInfo := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
dst := &types.Knowledge{
|
||||
ID: uuid.New().String(),
|
||||
ID: uuid.NewString(),
|
||||
TenantID: targetKB.TenantID,
|
||||
KnowledgeBaseID: targetKB.ID,
|
||||
Type: src.Type,
|
||||
|
|
@ -55,8 +65,21 @@ func (s *knowledgeService) cloneKnowledge(
|
|||
FileSize: src.FileSize,
|
||||
FileHash: src.FileHash,
|
||||
FilePath: src.FilePath,
|
||||
StorageSize: src.StorageSize,
|
||||
StorageSize: 0,
|
||||
Metadata: src.Metadata,
|
||||
CustomMetadata: src.CustomMetadata,
|
||||
}
|
||||
|
||||
state := knowledgeTransferState{
|
||||
TaskID: access.TransferTaskID(ctx),
|
||||
Operation: access.KBTransferClone,
|
||||
SourceKB: sourceKB.ID,
|
||||
TargetKB: targetKB.ID,
|
||||
SourceID: src.ID,
|
||||
Phase: "cloning",
|
||||
}
|
||||
if err := setTransferState(dst, state); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Deep-copy the source document file into an object owned by the destination
|
||||
|
|
@ -80,37 +103,52 @@ func (s *knowledgeService) cloneKnowledge(
|
|||
}
|
||||
|
||||
defer func() {
|
||||
if err != nil {
|
||||
if len(copiedFilePaths) > 0 {
|
||||
cleanupCopiedObjects(ctx, s.resolveFileService(ctx, targetKB), copiedFilePaths)
|
||||
}
|
||||
dst.ParseStatus = "failed"
|
||||
dst.ErrorMessage = err.Error()
|
||||
_ = s.repo.UpdateKnowledge(ctx, dst)
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge failed to move knowledge")
|
||||
} else {
|
||||
dst.ParseStatus = "completed"
|
||||
dst.EnableStatus = "enabled"
|
||||
_ = s.repo.UpdateKnowledge(ctx, dst)
|
||||
logger.GetLogger(ctx).WithField("knowledge_id", dst.ID).Infof("MoveKnowledge move knowledge successfully")
|
||||
if err == nil {
|
||||
return
|
||||
}
|
||||
stored, loadErr := s.repo.GetKnowledgeByID(ctx, dst.TenantID, dst.ID)
|
||||
if errors.Is(loadErr, repository.ErrKnowledgeNotFound) {
|
||||
cleanupCopiedObjects(ctx, s.resolveFileService(ctx, targetKB), copiedFilePaths)
|
||||
return
|
||||
}
|
||||
// An uncertain save/read must not delete objects that may be referenced.
|
||||
if loadErr != nil || stored == nil || validateTransferKnowledge(stored, targetKB) != nil {
|
||||
return
|
||||
}
|
||||
marker, markerErr := transferState(stored)
|
||||
if markerErr != nil || !matchesTransfer(ctx, marker, sourceKB, targetKB, access.KBTransferClone, src.ID, "") ||
|
||||
stored.ParseStatus == types.ParseStatusCompleted {
|
||||
return
|
||||
}
|
||||
before, after := *stored, *stored
|
||||
after.ParseStatus = types.ParseStatusFailed
|
||||
after.ErrorMessage = err.Error()
|
||||
_ = s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after)
|
||||
}()
|
||||
|
||||
if err = s.repo.CreateKnowledge(ctx, dst); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge create knowledge failed")
|
||||
return
|
||||
return err
|
||||
}
|
||||
tenantInfo.StorageUsed += dst.StorageSize
|
||||
if err = s.tenantRepo.AdjustStorageUsed(ctx, tenantInfo.ID, dst.StorageSize); err != nil {
|
||||
logger.GetLogger(ctx).WithField("error", err).Errorf("MoveKnowledge update tenant storage used failed")
|
||||
return
|
||||
// Create timestamps can be rounded by the database before the first CAS.
|
||||
persisted, err := s.repo.GetKnowledgeByID(ctx, dst.TenantID, dst.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateTransferKnowledge(persisted, targetKB); err != nil {
|
||||
return err
|
||||
}
|
||||
dst.UpdatedAt = persisted.UpdatedAt
|
||||
if err = s.CloneChunk(ctx, src, dst); err != nil {
|
||||
logger.GetLogger(ctx).WithField("knowledge_id", dst.ID).
|
||||
WithField("error", err).Errorf("MoveKnowledge move chunks failed")
|
||||
return
|
||||
return err
|
||||
}
|
||||
return
|
||||
before := *dst
|
||||
state.Phase = "done"
|
||||
if err = setTransferState(dst, state); err != nil {
|
||||
return err
|
||||
}
|
||||
dst.ParseStatus = types.ParseStatusCompleted
|
||||
dst.EnableStatus = "enabled"
|
||||
dst.StorageSize = src.StorageSize
|
||||
return s.repo.UpdateKnowledgeForTransfer(ctx, &before, dst)
|
||||
}
|
||||
|
||||
// processDocumentFromPassage handles asynchronous processing of text passages
|
||||
|
|
@ -294,7 +332,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
|
|||
logger.Infof(ctx, "Cleaning up existing chunks and index data for knowledge: %s", knowledge.ID)
|
||||
|
||||
// 删除旧的chunks
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
|
||||
logger.Warnf(ctx, "Failed to delete existing chunks (may not exist): %v", err)
|
||||
// 不返回错误,继续处理(可能没有旧数据)
|
||||
}
|
||||
|
|
@ -494,7 +532,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
|
|||
s.beginStage(ctx, knowledge.ID, types.StageChunking, types.JSONMap{
|
||||
"chunks_planned": len(insertChunks),
|
||||
})
|
||||
if err := s.chunkService.CreateChunks(ctx, insertChunks); err != nil {
|
||||
if err := s.chunkRepo.CreateChunks(ctx, insertChunks); err != nil {
|
||||
knowledge.ParseStatus = types.ParseStatusFailed
|
||||
knowledge.ErrorMessage = err.Error()
|
||||
knowledge.UpdatedAt = time.Now()
|
||||
|
|
@ -574,7 +612,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
|
|||
if aborted, status := s.isKnowledgeAborted(ctx, knowledge.TenantID, knowledge.ID); aborted {
|
||||
logger.Infof(ctx, "Knowledge aborted (%s) before indexing: %s", status, knowledge.ID)
|
||||
if status == types.ParseStatusDeleting {
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
|
||||
logger.Warnf(ctx, "Failed to cleanup chunks after deletion detected: %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -589,7 +627,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
|
|||
s.repo.UpdateKnowledge(ctx, knowledge)
|
||||
|
||||
// delete failed chunks
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
|
||||
logger.Errorf(ctx, "Delete chunks failed: %v", err)
|
||||
}
|
||||
|
||||
|
|
@ -622,7 +660,7 @@ func (s *knowledgeService) processChunks(ctx context.Context,
|
|||
if aborted, status := s.isKnowledgeAborted(ctx, knowledge.TenantID, knowledge.ID); aborted {
|
||||
logger.Infof(ctx, "Knowledge aborted (%s) after indexing: %s", status, knowledge.ID)
|
||||
if status == types.ParseStatusDeleting {
|
||||
if err := s.chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID); err != nil {
|
||||
if err := s.chunkRepo.DeleteChunksByKnowledgeID(ctx, knowledge.TenantID, knowledge.ID); err != nil {
|
||||
logger.Warnf(ctx, "Failed to cleanup chunks after deletion detected: %v", err)
|
||||
}
|
||||
if err := retrieveEngine.DeleteByKnowledgeIDList(ctx, []string{knowledge.ID}, embeddingModel.GetDimensions(), kb.Type); err != nil {
|
||||
|
|
@ -1319,7 +1357,7 @@ func (s *knowledgeService) ProcessSummaryGeneration(ctx context.Context, t *asyn
|
|||
}
|
||||
|
||||
// Save summary chunk
|
||||
if err := s.chunkService.CreateChunks(ctx, []*types.Chunk{summaryChunk}); err != nil {
|
||||
if err := s.chunkRepo.CreateChunks(ctx, []*types.Chunk{summaryChunk}); err != nil {
|
||||
logger.Errorf(ctx, "Failed to create summary chunk: %v", err)
|
||||
summaryErr = err
|
||||
return fmt.Errorf("failed to create summary chunk: %w", err)
|
||||
|
|
@ -1726,7 +1764,7 @@ func (s *knowledgeService) processQuestionGenerationForKnowledge(ctx context.Con
|
|||
}
|
||||
|
||||
// Update chunk in database
|
||||
if err := s.chunkService.UpdateChunk(ctx, chunk); err != nil {
|
||||
if err := s.chunkRepo.UpdateChunk(ctx, chunk); err != nil {
|
||||
chunkUpdateFailed++
|
||||
logger.Warnf(ctx, "Failed to update chunk %s: %v", chunk.ID, err)
|
||||
continue
|
||||
|
|
@ -2059,7 +2097,7 @@ func (s *knowledgeService) processQuestionGenerationForChunks(ctx context.Contex
|
|||
logger.Warnf(ctx, "Failed to set document metadata for chunk %s: %v", chunk.ID, err)
|
||||
continue
|
||||
}
|
||||
if err := s.chunkService.UpdateChunk(ctx, chunk); err != nil {
|
||||
if err := s.chunkRepo.UpdateChunk(ctx, chunk); err != nil {
|
||||
logger.Warnf(ctx, "Failed to update chunk %s: %v", chunk.ID, err)
|
||||
continue
|
||||
}
|
||||
|
|
@ -2180,13 +2218,12 @@ func (s *knowledgeService) RegenerateChunkQuestions(
|
|||
return nil, fmt.Errorf("questions can only be generated for text chunks")
|
||||
}
|
||||
generationRevision := chunk.ContentRevision
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, chunk.KnowledgeID)
|
||||
knowledge, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, chunk.KnowledgeID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, chunk.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
if knowledge.KnowledgeBaseID != chunk.KnowledgeBaseID || chunk.TenantID != knowledge.TenantID {
|
||||
return nil, werrors.NewForbiddenError("chunk does not belong to its knowledge document")
|
||||
}
|
||||
if kb.SummaryModelID == "" {
|
||||
return nil, fmt.Errorf("summary model is required for question generation")
|
||||
|
|
@ -2200,7 +2237,7 @@ func (s *knowledgeService) RegenerateChunkQuestions(
|
|||
return ""
|
||||
}
|
||||
neighbor, getErr := s.chunkRepo.GetChunkByID(ctx, tenantID, id)
|
||||
if getErr != nil {
|
||||
if getErr != nil || !sameChunkDocument(chunk, neighbor) {
|
||||
return ""
|
||||
}
|
||||
return neighbor.Content
|
||||
|
|
@ -2414,13 +2451,14 @@ func (s *knowledgeService) ReparseKnowledge(
|
|||
) (*types.Knowledge, error) {
|
||||
logger.Info(ctx, "Start re-parsing knowledge")
|
||||
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
existing, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
|
||||
existing, kb, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to load knowledge: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
tenantID := existing.TenantID
|
||||
|
||||
// Allocate a fresh span tree attempt up front. Doing this BEFORE
|
||||
// the cleanup + enqueue means: (a) the UI immediately sees a new
|
||||
// attempt with all five stages back to "pending" instead of the
|
||||
|
|
@ -2434,13 +2472,6 @@ func (s *knowledgeService) ReparseKnowledge(
|
|||
logger.Warnf(ctx, "[Reparse] OpenAttempt failed for %s: %v (will fall back in worker)", existing.ID, err)
|
||||
}
|
||||
|
||||
// Get knowledge base configuration
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, existing.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get knowledge base for reparse: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// When the caller supplies new overrides (e.g. via the reparse confirm
|
||||
// dialog), validate them against this knowledge's file type, then persist
|
||||
// to metadata so both this call's enqueue and the worker re-read the same
|
||||
|
|
@ -2914,6 +2945,10 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
chunkID string,
|
||||
imageInfo string,
|
||||
) error {
|
||||
knowledge, _, err := loadKnowledgeWrite(ctx, s.repo, s.kbService, knowledgeID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
imageInfo = common.CleanInvalidUTF8(imageInfo)
|
||||
var images []*types.ImageInfo
|
||||
if err := json.Unmarshal([]byte(imageInfo), &images); err != nil {
|
||||
|
|
@ -2932,6 +2967,11 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
logger.Errorf(ctx, "Failed to get chunk: %v", err)
|
||||
return err
|
||||
}
|
||||
if chunk == nil || chunk.ID != chunkID || chunk.KnowledgeID != knowledge.ID ||
|
||||
chunk.TenantID != knowledge.TenantID ||
|
||||
chunk.KnowledgeBaseID != knowledge.KnowledgeBaseID {
|
||||
return werrors.NewForbiddenError("chunk does not belong to its knowledge document")
|
||||
}
|
||||
chunk.ImageInfo = imageInfo
|
||||
tenantID := ctx.Value(types.TenantIDContextKey).(uint64)
|
||||
chunkChildren, err := s.chunkService.ListChunkByParentID(ctx, tenantID, chunkID)
|
||||
|
|
@ -2942,6 +2982,11 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
})
|
||||
return err
|
||||
}
|
||||
for _, child := range chunkChildren {
|
||||
if !sameChunkDocument(chunk, child) {
|
||||
return werrors.NewForbiddenError("image child does not belong to its document")
|
||||
}
|
||||
}
|
||||
logger.Infof(ctx, "Found %d chunks with parent chunk ID: %s", len(chunkChildren), chunkID)
|
||||
|
||||
// Iterate through each chunk and update its content based on the image information
|
||||
|
|
@ -3029,7 +3074,7 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
logger.Infof(ctx, "Updated %d chunks out of %d total chunks", len(updateChunk), len(chunkChildren)+1)
|
||||
|
||||
if len(addChunk) > 0 {
|
||||
err := s.chunkService.CreateChunks(ctx, addChunk)
|
||||
err := s.chunkRepo.CreateChunks(ctx, addChunk)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
"add_chunk_size": len(addChunk),
|
||||
|
|
@ -3040,7 +3085,7 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
|
||||
// Update the chunks
|
||||
for _, c := range updateChunk {
|
||||
err := s.chunkService.UpdateChunk(ctx, c)
|
||||
err := s.chunkRepo.UpdateChunk(ctx, c)
|
||||
if err != nil {
|
||||
logger.ErrorWithFields(ctx, err, map[string]interface{}{
|
||||
"chunk_id": c.ID,
|
||||
|
|
@ -3061,7 +3106,7 @@ func (s *knowledgeService) UpdateImageInfo(
|
|||
}
|
||||
|
||||
// Update the knowledge file hash
|
||||
knowledge, err := s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
|
||||
knowledge, err = s.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get knowledge: %v", err)
|
||||
return err
|
||||
|
|
@ -3088,7 +3133,7 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
|
|||
|
||||
ctx = logger.WithRequestID(ctx, payload.RequestId)
|
||||
ctx = logger.WithField(ctx, "manual_process", payload.KnowledgeID)
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
|
||||
tenantInfo, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
|
||||
if err != nil {
|
||||
|
|
@ -3107,6 +3152,13 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
|
|||
return nil
|
||||
}
|
||||
|
||||
if err := validateProcessingKnowledge(knowledge,
|
||||
payload.TenantID,
|
||||
payload.KnowledgeBaseID,
|
||||
payload.KnowledgeID); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Skip if already completed or being deleted
|
||||
if knowledge.ParseStatus == types.ParseStatusCompleted {
|
||||
logger.Infof(ctx, "ProcessManualUpdate: already completed, skipping: %s", payload.KnowledgeID)
|
||||
|
|
@ -3130,6 +3182,13 @@ func (s *knowledgeService) ProcessManualUpdate(ctx context.Context, t *asynq.Tas
|
|||
s.repo.UpdateKnowledge(ctx, knowledge)
|
||||
return nil
|
||||
}
|
||||
if kb == nil || kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
|
||||
return fmt.Errorf("processing task KB owner changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid processing scope: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
|
||||
// Re-check abort status right before marking processing — see the same
|
||||
// note in ProcessDocument for the cancel race this guards.
|
||||
|
|
@ -3185,7 +3244,7 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
|
|||
|
||||
ctx = logger.WithRequestID(ctx, payload.RequestId)
|
||||
ctx = logger.WithField(ctx, "document_process", payload.KnowledgeID)
|
||||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
if payload.Language != "" {
|
||||
ctx = context.WithValue(ctx, types.LanguageContextKey, payload.Language)
|
||||
}
|
||||
|
|
@ -3216,6 +3275,13 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
|
|||
return nil
|
||||
}
|
||||
|
||||
if err := validateProcessingKnowledge(knowledge,
|
||||
payload.TenantID,
|
||||
payload.KnowledgeBaseID,
|
||||
payload.KnowledgeID); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// 检查是否正在删除 / 已被用户取消 - 如果是则直接退出
|
||||
if knowledge.ParseStatus == types.ParseStatusDeleting {
|
||||
logger.Infof(ctx, "Knowledge is being deleted, aborting processing: %s", payload.KnowledgeID)
|
||||
|
|
@ -3261,6 +3327,13 @@ func (s *knowledgeService) ProcessDocument(ctx context.Context, t *asynq.Task) e
|
|||
s.repo.UpdateKnowledge(ctx, knowledge)
|
||||
return nil
|
||||
}
|
||||
if kb == nil || kb.ID != payload.KnowledgeBaseID || kb.TenantID != payload.TenantID {
|
||||
return fmt.Errorf("processing task KB owner changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid processing scope: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
|
||||
processOverrides, _ := knowledge.ProcessOverrides()
|
||||
eff := ResolveProcessConfig(kb, processOverrides)
|
||||
|
|
@ -3926,6 +3999,11 @@ func (s *knowledgeService) ProcessKnowledgeListReparse(ctx context.Context, t *a
|
|||
|
||||
logger.Infof(ctx, "Processing knowledge list reparse task for %d knowledge items", len(payload.KnowledgeIDs))
|
||||
|
||||
ctx, ids, err := s.reparseTaskScope(ctx, payload)
|
||||
if err != nil || len(ids) == 0 {
|
||||
return err
|
||||
}
|
||||
|
||||
tenant, err := s.tenantRepo.GetTenantByID(ctx, payload.TenantID)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Failed to get tenant %d: %v", payload.TenantID, err)
|
||||
|
|
@ -3935,7 +4013,7 @@ func (s *knowledgeService) ProcessKnowledgeListReparse(ctx context.Context, t *a
|
|||
ctx = context.WithValue(ctx, types.TenantIDContextKey, payload.TenantID)
|
||||
ctx = context.WithValue(ctx, types.TenantInfoContextKey, tenant)
|
||||
|
||||
outcome, err := runKnowledgeListReparseSubmissions(payload.KnowledgeIDs, func(id string) error {
|
||||
outcome, err := runKnowledgeListReparseSubmissions(ids, func(id string) error {
|
||||
_, err := s.ReparseKnowledge(ctx, id, payload.ProcessConfig)
|
||||
return err
|
||||
})
|
||||
|
|
|
|||
|
|
@ -29,11 +29,11 @@ func (r *parentChildKnowledgeRepo) UpdateKnowledge(
|
|||
}
|
||||
|
||||
type parentChildChunkService struct {
|
||||
interfaces.ChunkService
|
||||
interfaces.ChunkRepository
|
||||
created []*types.Chunk
|
||||
}
|
||||
|
||||
func (s *parentChildChunkService) DeleteChunksByKnowledgeID(context.Context, string) error {
|
||||
func (s *parentChildChunkService) DeleteChunksByKnowledgeID(context.Context, uint64, string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -160,7 +160,7 @@ func TestProcessChunksIndexesEveryTextChild(t *testing.T) {
|
|||
ctx := context.WithValue(context.Background(), types.TenantInfoContextKey, tenant)
|
||||
svc := &knowledgeService{
|
||||
repo: &parentChildKnowledgeRepo{knowledge: knowledge},
|
||||
chunkService: chunkService,
|
||||
chunkRepo: chunkService,
|
||||
modelService: parentChildModelService{embedder: parentChildEmbedder{}},
|
||||
retrieveEngine: parentChildRetrieveRegistry{engine: retrieveEngine},
|
||||
graphEngine: parentChildGraphRepo{},
|
||||
|
|
|
|||
|
|
@ -0,0 +1,60 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/hibiken/asynq"
|
||||
)
|
||||
|
||||
// Validate the whole batch before any destructive submission. The exact KB
|
||||
// grant is consumed again when ReparseKnowledge reloads each document.
|
||||
func (s *knowledgeService) reparseTaskScope(
|
||||
ctx context.Context, payload types.KnowledgeListReparsePayload,
|
||||
) (context.Context, []string, error) {
|
||||
if payload.TenantID == 0 {
|
||||
return ctx, nil, fmt.Errorf("invalid reparse task tenant: %w", asynq.SkipRetry)
|
||||
}
|
||||
ids, err := writeResourceIDs(payload.KnowledgeIDs)
|
||||
if err != nil {
|
||||
return ctx, nil, fmt.Errorf("invalid reparse task IDs: %w", asynq.SkipRetry)
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return ctx, ids, nil
|
||||
}
|
||||
ctx = types.WithExecutionTenant(ctx, payload.TenantID)
|
||||
rows, err := s.repo.GetKnowledgeBatch(ctx, payload.TenantID, ids)
|
||||
if err != nil {
|
||||
return ctx, nil, err
|
||||
}
|
||||
kbID := payload.KnowledgeBaseID
|
||||
seen := make(map[string]bool, len(rows))
|
||||
for _, row := range rows {
|
||||
if row == nil || row.TenantID != payload.TenantID || row.KnowledgeBaseID == "" ||
|
||||
(kbID != "" && row.KnowledgeBaseID != kbID) {
|
||||
return ctx, nil, fmt.Errorf("reparse task binding changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
if err := access.RejectMovingKnowledge(row); err != nil {
|
||||
return ctx, nil, fmt.Errorf("reparse task document unavailable: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
// Legacy payloads can reconstruct only their current unambiguous KB.
|
||||
kbID = row.KnowledgeBaseID
|
||||
seen[row.ID] = true
|
||||
}
|
||||
for _, id := range ids {
|
||||
if !seen[id] {
|
||||
return ctx, nil, fmt.Errorf("reparse task document missing: %w", asynq.SkipRetry)
|
||||
}
|
||||
}
|
||||
kb, err := s.kbService.GetKnowledgeBaseByID(ctx, kbID)
|
||||
if err != nil {
|
||||
return ctx, nil, err
|
||||
}
|
||||
if kb == nil || kb.ID != kbID || kb.TenantID != payload.TenantID {
|
||||
return ctx, nil, fmt.Errorf("reparse task KB binding changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
ctx, err = access.WithKBTaskWrite(ctx, kb, payload.TenantID)
|
||||
return ctx, ids, err
|
||||
}
|
||||
|
|
@ -21,45 +21,64 @@ type fakeKBShareService struct {
|
|||
func (f *fakeKBShareService) ShareKnowledgeBase(context.Context, string, string, string, uint64, types.OrgMemberRole) (*types.KnowledgeBaseShare, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) UpdateSharePermission(context.Context, string, types.OrgMemberRole, string, uint64) error {
|
||||
return errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) RemoveShare(context.Context, string, string, uint64) error {
|
||||
return errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) ListSharesByKnowledgeBase(context.Context, string, uint64) ([]*types.KnowledgeBaseShare, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) ListSharesByOrganization(context.Context, string) ([]*types.KnowledgeBaseShare, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) ListSharedKnowledgeBases(context.Context, uint64, types.TenantRole) ([]*types.SharedKnowledgeBaseInfo, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) ListSharedKnowledgeBasesInOrganization(context.Context, string, uint64, types.TenantRole) ([]*types.OrganizationSharedKnowledgeBaseItem, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) ListSharedKnowledgeBaseIDsByOrganizations(context.Context, []string, uint64) (map[string][]string, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) GetShare(context.Context, string) (*types.KnowledgeBaseShare, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) GetShareByKBAndOrg(context.Context, string, string) (*types.KnowledgeBaseShare, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
func (f *fakeKBShareService) CheckTenantKBPermission(context.Context, string, uint64, types.TenantRole) (types.OrgMemberRole, bool, error) {
|
||||
return "", false, errors.New("not implemented")
|
||||
|
||||
func (f *fakeKBShareService) CheckTenantKBPermission(
|
||||
_ context.Context,
|
||||
kbID string,
|
||||
_ uint64,
|
||||
_ types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
return types.OrgRoleViewer, f.allowedKBs[kbID], nil
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) HasTenantKBPermission(ctx context.Context, kbID string, callerTenantID uint64, callerTenantRole types.TenantRole, requiredRole types.OrgMemberRole) (bool, error) {
|
||||
return f.allowedKBs[kbID], nil
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) GetKBSourceTenant(context.Context, string) (uint64, error) {
|
||||
return 0, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) CountSharesByKnowledgeBaseIDs(context.Context, []string) (map[string]int64, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
||||
func (f *fakeKBShareService) CountByOrganizations(context.Context, []string) (map[string]int64, error) {
|
||||
return nil, errors.New("not implemented")
|
||||
}
|
||||
|
|
@ -153,3 +172,57 @@ func TestGetKnowledgeBatchWithSharedAccess_ExcludesSharedKnowledgeWithoutPermiss
|
|||
}
|
||||
|
||||
var _ interfaces.KBShareService = (*fakeKBShareService)(nil)
|
||||
|
||||
type countingKBShareService struct {
|
||||
fakeKBShareService
|
||||
calls int
|
||||
}
|
||||
|
||||
func (s *countingKBShareService) CheckTenantKBPermission(
|
||||
ctx context.Context,
|
||||
id string,
|
||||
tenantID uint64,
|
||||
role types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
s.calls++
|
||||
return s.fakeKBShareService.CheckTenantKBPermission(ctx, id, tenantID, role)
|
||||
}
|
||||
|
||||
func TestGetKnowledgeBatchWithSharedAccessChecksEachKBOnceAndRechecksNextRequest(t *testing.T) {
|
||||
shares := &countingKBShareService{
|
||||
fakeKBShareService: fakeKBShareService{allowedKBs: map[string]bool{"kb-shared": true}},
|
||||
}
|
||||
svc, db := newKnowledgeSharedAccessService(t, shares)
|
||||
for _, id := range []string{"first", "second"} {
|
||||
seedKnowledge(t, db, &types.Knowledge{ID: id, TenantID: 2, KnowledgeBaseID: "kb-shared", Type: "file"})
|
||||
}
|
||||
got, err := svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"first", "second"})
|
||||
require.NoError(t, err)
|
||||
require.Len(t, got, 2)
|
||||
require.Equal(t, 1, shares.calls)
|
||||
shares.allowedKBs["kb-shared"] = false
|
||||
got, err = svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"first", "second"})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, got)
|
||||
require.Equal(t, 2, shares.calls)
|
||||
}
|
||||
|
||||
func TestResolveKBReadTenantPreservesServiceBoundary(t *testing.T) {
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: 2}
|
||||
shares := &fakeKBShareService{allowedKBs: map[string]bool{"kb": true}}
|
||||
tenant, err := resolveKBReadTenant(newSharedAccessContext(), kb, shares)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(2), tenant)
|
||||
_, err = resolveKBReadTenant(
|
||||
context.WithValue(context.Background(), types.TenantIDContextKey, uint64(1)),
|
||||
kb,
|
||||
shares,
|
||||
)
|
||||
require.Error(t, err, "direct cross-tenant calls still require a user identity")
|
||||
_, err = resolveKBReadTenant(newSharedAccessContext(), kb, nil)
|
||||
require.Error(t, err, "a missing share service cannot grant access")
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(2))
|
||||
tenant, err = resolveKBReadTenant(ctx, kb, nil)
|
||||
require.NoError(t, err, "a service execution context already scoped to the KB remains usable")
|
||||
require.Equal(t, uint64(2), tenant)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type failingSharedKnowledgeRepo struct {
|
||||
interfaces.KnowledgeRepository
|
||||
err error
|
||||
}
|
||||
|
||||
func (r *failingSharedKnowledgeRepo) GetKnowledgeBatch(context.Context, uint64, []string) ([]*types.Knowledge, error) {
|
||||
return []*types.Knowledge{{ID: "own", TenantID: 1, KnowledgeBaseID: "own-kb"}}, nil
|
||||
}
|
||||
|
||||
func (r *failingSharedKnowledgeRepo) GetKnowledgeByIDOnly(context.Context, string) (*types.Knowledge, error) {
|
||||
return nil, r.err
|
||||
}
|
||||
|
||||
func TestSharedDocumentReadsPropagateStorageFailure(t *testing.T) {
|
||||
for _, failure := range []error{errors.New("database unavailable"), repository.ErrKnowledgeNotFound} {
|
||||
repo := &failingSharedKnowledgeRepo{err: failure}
|
||||
svc := &knowledgeService{repo: repo}
|
||||
search := &knowledgeBaseService{kgRepo: repo}
|
||||
rows, err := svc.GetKnowledgeBatchWithSharedAccess(newSharedAccessContext(), 1, []string{"own", "shared"})
|
||||
found, searchErr := search.fetchKnowledgeDataWithShared(newSharedAccessContext(), 1, []string{"own", "shared"})
|
||||
if errors.Is(failure, repository.ErrKnowledgeNotFound) {
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, searchErr)
|
||||
require.Len(t, rows, 1)
|
||||
require.Len(t, found, 1)
|
||||
} else {
|
||||
require.ErrorIs(t, err, failure)
|
||||
require.ErrorIs(t, searchErr, failure)
|
||||
require.Nil(t, rows)
|
||||
require.Nil(t, found)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,525 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/hibiken/asynq"
|
||||
)
|
||||
|
||||
// Transfer state lives with the document, so queue retries do not depend on
|
||||
// an expiring progress cache. It is server metadata, never custom metadata.
|
||||
type knowledgeTransferState struct {
|
||||
TaskID string `json:"task_id"`
|
||||
Operation access.KBTransferOperation `json:"operation"`
|
||||
SourceKB string `json:"source_kb"`
|
||||
TargetKB string `json:"target_kb"`
|
||||
SourceID string `json:"source_id"`
|
||||
Mode string `json:"mode,omitempty"`
|
||||
WikiChunkIDs []string `json:"wiki_chunk_ids,omitempty"`
|
||||
WikiSummary string `json:"wiki_summary,omitempty"`
|
||||
Phase string `json:"phase"`
|
||||
}
|
||||
|
||||
func transferState(k *types.Knowledge) (*knowledgeTransferState, error) {
|
||||
fields := map[string]json.RawMessage{}
|
||||
if len(k.Metadata) != 0 {
|
||||
if err := json.Unmarshal(k.Metadata, &fields); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
raw := fields[types.KnowledgeTransferMetadataKey]
|
||||
if len(raw) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
var state knowledgeTransferState
|
||||
if err := json.Unmarshal(raw, &state); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &state, nil
|
||||
}
|
||||
|
||||
func setTransferState(k *types.Knowledge, state knowledgeTransferState) error {
|
||||
fields := map[string]json.RawMessage{}
|
||||
if len(k.Metadata) != 0 {
|
||||
if err := json.Unmarshal(k.Metadata, &fields); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if fields == nil {
|
||||
fields = map[string]json.RawMessage{}
|
||||
}
|
||||
raw, err := json.Marshal(state)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fields[types.KnowledgeTransferMetadataKey] = raw
|
||||
k.Metadata, err = json.Marshal(fields)
|
||||
return err
|
||||
}
|
||||
|
||||
func matchesTransfer(
|
||||
ctx context.Context,
|
||||
state *knowledgeTransferState,
|
||||
source, target *types.KnowledgeBase,
|
||||
operation access.KBTransferOperation,
|
||||
sourceID, mode string,
|
||||
) bool {
|
||||
return state != nil && state.TaskID != "" && state.TaskID == access.TransferTaskID(ctx) &&
|
||||
state.Operation == operation &&
|
||||
state.SourceKB == source.ID &&
|
||||
state.TargetKB == target.ID &&
|
||||
state.SourceID == sourceID &&
|
||||
state.Mode == mode
|
||||
}
|
||||
|
||||
func validateTransferKnowledge(k *types.Knowledge, kb *types.KnowledgeBase) error {
|
||||
if k == nil || k.ID == "" || k.TenantID != kb.TenantID || k.KnowledgeBaseID != kb.ID {
|
||||
return fmt.Errorf("knowledge binding does not match the transfer scope")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Read every chunk type and validate persisted document/KB/tag relationships
|
||||
// before a clone or move can delete, index, create tags, or copy files.
|
||||
func (s *knowledgeService) transferChunks(
|
||||
ctx context.Context,
|
||||
k *types.Knowledge,
|
||||
allowedKBs ...string,
|
||||
) ([]*types.Chunk, error) {
|
||||
allowed := map[string]bool{}
|
||||
for _, id := range allowedKBs {
|
||||
allowed[id] = true
|
||||
}
|
||||
var all []*types.Chunk
|
||||
seen := map[string]bool{}
|
||||
tags := map[string]string{}
|
||||
chunks, err := s.chunkRepo.ListAllChunksByKnowledgeID(ctx, k.TenantID, k.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, c := range chunks {
|
||||
if c == nil || c.ID == "" || seen[c.ID] || c.TenantID != k.TenantID || c.KnowledgeID != k.ID ||
|
||||
!allowed[c.KnowledgeBaseID] {
|
||||
return nil, fmt.Errorf("chunk binding does not match the transfer scope")
|
||||
}
|
||||
seen[c.ID] = true
|
||||
if c.TagID != "" && tags[c.TagID] != c.KnowledgeBaseID {
|
||||
tag, err := s.tagRepo.GetByID(ctx, k.TenantID, c.TagID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if tag == nil || tag.ID != c.TagID || tag.TenantID != k.TenantID ||
|
||||
tag.KnowledgeBaseID != c.KnowledgeBaseID {
|
||||
return nil, fmt.Errorf("chunk tag belongs to another knowledge base")
|
||||
}
|
||||
tags[c.TagID] = c.KnowledgeBaseID
|
||||
}
|
||||
all = append(all, c)
|
||||
}
|
||||
|
||||
for _, c := range all {
|
||||
if c.ParentChunkID != "" && !seen[c.ParentChunkID] {
|
||||
return nil, fmt.Errorf("chunk parent belongs to another document")
|
||||
}
|
||||
}
|
||||
return all, nil
|
||||
}
|
||||
|
||||
func (s *knowledgeService) planKnowledgeMove(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
ids []string,
|
||||
mode string,
|
||||
) ([]*types.Knowledge, error) {
|
||||
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferMove); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
if err := access.ValidateKBTransferCompatibility(source, target, access.KBTransferMove, mode, tenant); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ids, err := writeResourceIDs(ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return nil, fmt.Errorf("knowledge IDs cannot be empty")
|
||||
}
|
||||
rows, err := s.repo.GetKnowledgeBatch(ctx, source.TenantID, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byID := map[string]*types.Knowledge{}
|
||||
for _, row := range rows {
|
||||
if row != nil {
|
||||
byID[row.ID] = row
|
||||
}
|
||||
}
|
||||
engineChecked := false
|
||||
result := make([]*types.Knowledge, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
row := byID[id]
|
||||
if row == nil || row.TenantID != source.TenantID {
|
||||
return nil, fmt.Errorf("knowledge %s not found in transfer tenant", id)
|
||||
}
|
||||
if err := validateMoveItem(ctx, row, source, target, mode); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
allowed := []string{row.KnowledgeBaseID}
|
||||
state, _ := transferState(row)
|
||||
if matchesTransfer(ctx, state, source, target, access.KBTransferMove, row.ID, mode) && state.Phase == "moving" {
|
||||
allowed = []string{source.ID, target.ID}
|
||||
}
|
||||
_, err = s.transferChunks(ctx, row, allowed...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if mode == "reuse_vectors" && row.EmbeddingModelID != "" {
|
||||
if row.EmbeddingModelID != source.EmbeddingModelID {
|
||||
return nil, fmt.Errorf("knowledge %s uses a different embedding model", row.ID)
|
||||
}
|
||||
if !engineChecked {
|
||||
engine, err := retriever.CreateRetrieveEngineForKB(
|
||||
ctx,
|
||||
s.retrieveEngine,
|
||||
s.ownership,
|
||||
source.TenantID,
|
||||
source.VectorStoreID,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := engine.ValidateKnowledgeIndexMove(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
engineChecked = true
|
||||
}
|
||||
}
|
||||
snapshot := *row
|
||||
result = append(result, &snapshot)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func validateMoveItem(ctx context.Context, k *types.Knowledge, source, target *types.KnowledgeBase, mode string) error {
|
||||
if k == nil || k.TenantID != source.TenantID {
|
||||
return access.ErrForbidden
|
||||
}
|
||||
state, err := transferState(k)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if matchesTransfer(ctx, state, source, target, access.KBTransferMove, k.ID, mode) {
|
||||
switch state.Phase {
|
||||
case "moving":
|
||||
if k.KnowledgeBaseID == source.ID &&
|
||||
(k.ParseStatus == types.ParseStatusProcessing || k.ParseStatus == types.ParseStatusFailed) {
|
||||
return nil
|
||||
}
|
||||
case "reparse_pending", "done":
|
||||
if k.KnowledgeBaseID == target.ID {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("knowledge %s changed during move", k.ID)
|
||||
}
|
||||
if k.KnowledgeBaseID != source.ID || k.ParseStatus != types.ParseStatusCompleted {
|
||||
return fmt.Errorf("knowledge %s must be completed in the source knowledge base", k.ID)
|
||||
}
|
||||
if state != nil && state.Phase != "done" {
|
||||
return fmt.Errorf("knowledge %s has an unfinished transfer", k.ID)
|
||||
}
|
||||
if mode == "reparse" {
|
||||
if k.IsManual() {
|
||||
meta, err := k.ManualMetadata()
|
||||
if err != nil || meta == nil || meta.Content == "" {
|
||||
return fmt.Errorf("knowledge %s has no manual content to reparse", k.ID)
|
||||
}
|
||||
} else if k.FilePath == "" {
|
||||
return fmt.Errorf("knowledge %s has no stored file to reparse", k.ID)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type knowledgeClonePlan struct {
|
||||
add []*types.Knowledge
|
||||
remove []string
|
||||
}
|
||||
|
||||
func (s *knowledgeService) planKnowledgeClone(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
) (*knowledgeClonePlan, error) {
|
||||
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferClone); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
src, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, source.TenantID, source.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
dst, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, target.TenantID, target.ID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, pair := range []struct {
|
||||
rows []*types.Knowledge
|
||||
kb *types.KnowledgeBase
|
||||
}{{src, source}, {dst, target}} {
|
||||
for _, k := range pair.rows {
|
||||
if err := validateTransferKnowledge(k, pair.kb); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := s.transferChunks(ctx, k, pair.kb.ID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(src, func(i, j int) bool { return src[i].ID < src[j].ID })
|
||||
sort.Slice(dst, func(i, j int) bool { return dst[i].ID < dst[j].ID })
|
||||
plan := &knowledgeClonePlan{}
|
||||
used := map[string]bool{}
|
||||
for _, k := range src {
|
||||
if k.ParseStatus != types.ParseStatusCompleted {
|
||||
return nil, fmt.Errorf("source knowledge %s is not completed", k.ID)
|
||||
}
|
||||
matched := false
|
||||
// Prefer exact source bindings created by this task; unlike file hashes,
|
||||
// these also identify manual/empty-hash documents across retries.
|
||||
for _, other := range dst {
|
||||
state, err := transferState(other)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !used[other.ID] && matchesTransfer(ctx, state, source, target, access.KBTransferClone, k.ID, "") {
|
||||
used[other.ID] = true
|
||||
if state.Phase == "done" && other.ParseStatus == types.ParseStatusCompleted {
|
||||
matched = true
|
||||
} else {
|
||||
plan.remove = append(plan.remove, other.ID)
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if !matched && k.FileHash != "" {
|
||||
for _, other := range dst {
|
||||
if !used[other.ID] && other.FileHash == k.FileHash && other.ParseStatus == types.ParseStatusCompleted {
|
||||
used[other.ID] = true
|
||||
matched = true
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if !matched {
|
||||
plan.add = append(plan.add, k)
|
||||
}
|
||||
}
|
||||
for _, k := range dst {
|
||||
if used[k.ID] {
|
||||
continue
|
||||
}
|
||||
if k.ParseStatus == types.ParseStatusProcessing || k.ParseStatus == types.ParseStatusPending ||
|
||||
k.ParseStatus == types.ParseStatusDeleting {
|
||||
return nil, fmt.Errorf("target knowledge %s is busy", k.ID)
|
||||
}
|
||||
plan.remove = append(plan.remove, k.ID)
|
||||
}
|
||||
return plan, nil
|
||||
}
|
||||
|
||||
func (s *knowledgeService) executeKnowledgeClone(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
progress func(int, int),
|
||||
) error {
|
||||
plan, err := s.planKnowledgeClone(ctx, source, target)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
total, done := len(plan.add)+len(plan.remove), 0
|
||||
if progress != nil {
|
||||
progress(done, total)
|
||||
}
|
||||
if len(plan.remove) > 0 {
|
||||
// The complete binding/preflight plan above precedes the first deletion.
|
||||
if err := deleteReferencedKnowledge(ctx, s, target.ID, plan.remove); err != nil {
|
||||
return err
|
||||
}
|
||||
done += len(plan.remove)
|
||||
if progress != nil {
|
||||
progress(done, total)
|
||||
}
|
||||
}
|
||||
// Serial execution keeps storage accounting, tag creation and progress
|
||||
// coherent; each completed document is independently resumable.
|
||||
for _, k := range plan.add {
|
||||
if err := s.cloneKnowledge(ctx, k, target); err != nil {
|
||||
return err
|
||||
}
|
||||
done++
|
||||
if progress != nil {
|
||||
progress(done, total)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// acknowledgeMovedReparse completes transfer admission, not document parsing.
|
||||
// The target processing pipeline owns ParseStatus after its task is enqueued.
|
||||
func (s *knowledgeService) acknowledgeMovedReparse(
|
||||
ctx context.Context,
|
||||
knowledge *types.Knowledge,
|
||||
source, target *types.KnowledgeBase,
|
||||
) error {
|
||||
// The processing worker can already have updated the row after enqueue.
|
||||
// Reload before checkpointing; never overwrite its parse status or metadata.
|
||||
current, err := s.repo.GetKnowledgeByID(ctx, knowledge.TenantID, knowledge.ID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := validateTransferKnowledge(current, target); err != nil {
|
||||
return err
|
||||
}
|
||||
state, err := transferState(current)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !matchesTransfer(ctx, state, source, target, access.KBTransferMove, knowledge.ID, "reparse") {
|
||||
return access.ErrForbidden
|
||||
}
|
||||
if state.Phase == "done" {
|
||||
return nil
|
||||
}
|
||||
before := *current
|
||||
after := *current
|
||||
state.Phase = "done"
|
||||
if err := setTransferState(&after, *state); err != nil {
|
||||
return err
|
||||
}
|
||||
return s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after)
|
||||
}
|
||||
|
||||
func (s *knowledgeService) preflightFAQClone(
|
||||
ctx context.Context,
|
||||
source, target *types.KnowledgeBase,
|
||||
) (map[string]*types.Chunk, map[string]*types.Chunk, error) {
|
||||
if err := access.RequireKBTransfer(ctx, source, target, access.KBTransferClone); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
maps := []map[string]*types.Chunk{{}, {}}
|
||||
for i, kb := range []*types.KnowledgeBase{source, target} {
|
||||
rows, err := s.repo.ListKnowledgeByKnowledgeBaseID(ctx, kb.TenantID, kb.ID)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if len(rows) > 1 {
|
||||
return nil, nil, fmt.Errorf("FAQ knowledge base has multiple document containers")
|
||||
}
|
||||
for _, row := range rows {
|
||||
if err := validateTransferKnowledge(row, kb); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if row.Type != types.KnowledgeTypeFAQ {
|
||||
return nil, nil, fmt.Errorf("invalid FAQ container")
|
||||
}
|
||||
chunks, err := s.transferChunks(ctx, row, kb.ID)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
for _, chunk := range chunks {
|
||||
if chunk.ChunkType != types.ChunkTypeFAQ {
|
||||
return nil, nil, fmt.Errorf("invalid FAQ chunk type")
|
||||
}
|
||||
maps[i][chunk.ID] = chunk
|
||||
}
|
||||
}
|
||||
}
|
||||
return maps[0], maps[1], nil
|
||||
}
|
||||
|
||||
func (s *knowledgeService) markMoveItemFailed(
|
||||
ctx context.Context,
|
||||
id string,
|
||||
source, target *types.KnowledgeBase,
|
||||
mode string,
|
||||
taskErr error,
|
||||
) {
|
||||
if access.RequireKBTransfer(ctx, source, target, access.KBTransferMove) != nil {
|
||||
return
|
||||
}
|
||||
row, err := s.repo.GetKnowledgeByID(ctx, source.TenantID, id)
|
||||
if err != nil || row == nil || row.ID != id || row.TenantID != source.TenantID {
|
||||
return
|
||||
}
|
||||
state, err := transferState(row)
|
||||
if err != nil || !matchesTransfer(ctx, state, source, target, access.KBTransferMove, id, mode) {
|
||||
return
|
||||
}
|
||||
sourceFailed := row.KnowledgeBaseID == source.ID && state.Phase == "moving" &&
|
||||
row.ParseStatus == types.ParseStatusProcessing
|
||||
enqueueFailed := row.KnowledgeBaseID == target.ID && state.Phase == "reparse_pending" &&
|
||||
row.ParseStatus == types.ParseStatusPending
|
||||
if !sourceFailed && !enqueueFailed {
|
||||
return
|
||||
}
|
||||
before, after := *row, *row
|
||||
after.ParseStatus = types.ParseStatusFailed
|
||||
after.ErrorMessage = taskErr.Error()
|
||||
if err := s.repo.UpdateKnowledgeForTransfer(ctx, &before, &after); err != nil {
|
||||
logger.Errorf(ctx, "Failed to record move failure for knowledge %s, task %s: %v (move error: %v)",
|
||||
row.ID, state.TaskID, err, taskErr)
|
||||
}
|
||||
}
|
||||
|
||||
// A stale parser must not start after a move has claimed a completed document,
|
||||
// including the interval before its knowledge_base_id changes. Reparse tasks
|
||||
// emitted by the move are admitted only once the destination checkpoint exists.
|
||||
func validateProcessingKnowledge(knowledge *types.Knowledge, tenant uint64, kbID, knowledgeID string) error {
|
||||
if knowledge == nil || tenant == 0 || kbID == "" || knowledge.ID != knowledgeID || knowledge.TenantID != tenant ||
|
||||
knowledge.KnowledgeBaseID != kbID {
|
||||
return fmt.Errorf("processing task binding changed: %w", asynq.SkipRetry)
|
||||
}
|
||||
state, err := transferState(knowledge)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid processing metadata: %v: %w", err, asynq.SkipRetry)
|
||||
}
|
||||
if state != nil && state.Operation == access.KBTransferMove && state.Phase == "moving" {
|
||||
return fmt.Errorf("knowledge is being moved: %w", asynq.SkipRetry)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Cleanup consumes the durable source references only after target ownership
|
||||
// is committed. A done transfer is retried here when wiki reconciliation fails.
|
||||
func (s *knowledgeService) cleanupMovedSourceWiki(ctx context.Context, knowledge *types.Knowledge,
|
||||
source, target *types.KnowledgeBase,
|
||||
) error {
|
||||
if !source.IsWikiEnabled() {
|
||||
return nil
|
||||
}
|
||||
if knowledge.KnowledgeBaseID != target.ID {
|
||||
return access.ErrForbidden
|
||||
}
|
||||
state, err := transferState(knowledge)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if state == nil || state.SourceKB != source.ID || state.TargetKB != target.ID {
|
||||
return access.ErrForbidden
|
||||
}
|
||||
old := *knowledge
|
||||
old.KnowledgeBaseID = source.ID
|
||||
old.Description = state.WikiSummary
|
||||
refs := make(map[string]bool, len(state.WikiChunkIDs))
|
||||
for _, id := range state.WikiChunkIDs {
|
||||
refs[id] = true
|
||||
}
|
||||
return s.cleanupWikiReferences(ctx, &old, refs)
|
||||
}
|
||||
|
|
@ -0,0 +1,457 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/alicebob/miniredis/v2"
|
||||
"github.com/hibiken/asynq"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func transferFixture(t *testing.T, operation access.KBTransferOperation) *documentWriteFixture {
|
||||
t.Helper()
|
||||
f := newDocumentWriteFixture(t)
|
||||
require.NoError(t, f.db.AutoMigrate(&types.Tenant{}))
|
||||
require.NoError(t, f.db.Create(&types.Tenant{ID: 7, Name: "tenant", StorageUsed: 5}).Error)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("parse_status", types.ParseStatusCompleted).Error,
|
||||
)
|
||||
ctx, err := access.WithKBTransferTask(
|
||||
context.Background(),
|
||||
f.kbs.values["kb"],
|
||||
f.kbs.values["other"],
|
||||
7,
|
||||
operation,
|
||||
"transfer-task",
|
||||
false,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
f.ctx = context.WithValue(ctx, types.TenantInfoContextKey, &types.Tenant{ID: 7, StorageUsed: 5})
|
||||
mr := miniredis.RunT(t)
|
||||
f.svc.redisClient = redis.NewClient(&redis.Options{Addr: mr.Addr()})
|
||||
t.Cleanup(func() { require.NoError(t, f.svc.redisClient.Close()) })
|
||||
return f
|
||||
}
|
||||
|
||||
func moveTask(t *testing.T, ids []string, mode string) *asynq.Task {
|
||||
t.Helper()
|
||||
raw, err := json.Marshal(
|
||||
types.KnowledgeMovePayload{
|
||||
TenantID: 7,
|
||||
TaskID: "transfer-task",
|
||||
SourceKBID: "kb",
|
||||
TargetKBID: "other",
|
||||
KnowledgeIDs: ids,
|
||||
Mode: mode,
|
||||
},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
return asynq.NewTask(types.TypeKnowledgeMove, raw)
|
||||
}
|
||||
|
||||
func TestMoveBatchPreflightHasNoPartialMutations(t *testing.T) {
|
||||
for _, ids := range [][]string{{"doc", "missing"}, {"doc", "other-doc"}, {"doc", ""}} {
|
||||
t.Run(ids[1], func(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
err := f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, ids, "reuse_vectors"))
|
||||
require.Error(t, err)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.ParseStatusCompleted, row.ParseStatus)
|
||||
require.Equal(t, "kb", row.KnowledgeBaseID)
|
||||
require.Empty(t, row.Metadata)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoveRejectsInvalidPairAndModeBeforeWrites(t *testing.T) {
|
||||
for _, change := range []string{"tenant", "same KB", "mode", "chunk", "tag"} {
|
||||
t.Run(change, func(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
mode := "reuse_vectors"
|
||||
switch change {
|
||||
case "tenant":
|
||||
f.kbs.values["other"].TenantID = 8
|
||||
case "same KB":
|
||||
f.kbs.values["other"].ID = "kb"
|
||||
case "mode":
|
||||
mode = "bogus"
|
||||
case "chunk":
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("knowledge_base_id", "third").Error,
|
||||
)
|
||||
case "tag":
|
||||
require.NoError(t, f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("tag_id", "tag").Error)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.KnowledgeTag{}).Where("id = ?", "tag").Update("knowledge_base_id", "other").Error,
|
||||
)
|
||||
}
|
||||
require.Error(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, mode)))
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.ParseStatusCompleted, row.ParseStatus)
|
||||
require.Empty(t, row.Metadata)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type transferFaultRepo struct {
|
||||
interfaces.KnowledgeRepository
|
||||
failID string
|
||||
claims int
|
||||
}
|
||||
|
||||
func (r *transferFaultRepo) DeleteKnowledgeTagRelations(ctx context.Context, id string) error {
|
||||
if id == r.failID {
|
||||
return errors.New("temporary tag cleanup failure")
|
||||
}
|
||||
return r.KnowledgeRepository.DeleteKnowledgeTagRelations(ctx, id)
|
||||
}
|
||||
|
||||
func (r *transferFaultRepo) UpdateKnowledgeForTransfer(ctx context.Context, before, after *types.Knowledge) error {
|
||||
r.claims++
|
||||
return r.KnowledgeRepository.UpdateKnowledgeForTransfer(ctx, before, after)
|
||||
}
|
||||
|
||||
func TestMovePartialFailureResumesAndSkipsCompletedDocuments(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Create(
|
||||
&types.Knowledge{ID: "second", TenantID: 7, KnowledgeBaseID: "kb", ParseStatus: types.ParseStatusCompleted},
|
||||
).Error,
|
||||
)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Create(
|
||||
&types.Chunk{
|
||||
ID: "second-chunk",
|
||||
TenantID: 7,
|
||||
KnowledgeID: "second",
|
||||
KnowledgeBaseID: "kb",
|
||||
ChunkType: types.ChunkTypeImageOCR,
|
||||
},
|
||||
).Error,
|
||||
)
|
||||
fault := &transferFaultRepo{KnowledgeRepository: f.repo, failID: "second"}
|
||||
f.svc.repo = fault
|
||||
task := moveTask(t, []string{"doc", "doc", "second"}, "reuse_vectors")
|
||||
require.ErrorContains(t, f.svc.ProcessKnowledgeMove(context.Background(), task), "temporary tag cleanup failure")
|
||||
first, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", first.KnowledgeBaseID)
|
||||
firstUpdated := first.UpdatedAt
|
||||
second, err := f.repo.GetKnowledgeByID(f.ctx, 7, "second")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "kb", second.KnowledgeBaseID)
|
||||
state, err := transferState(second)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "moving", state.Phase)
|
||||
fault.failID = ""
|
||||
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), task))
|
||||
first, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, firstUpdated, first.UpdatedAt, "completed item must not be moved again")
|
||||
second, err = f.repo.GetKnowledgeByID(f.ctx, 7, "second")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", second.KnowledgeBaseID)
|
||||
require.Equal(t, types.ParseStatusCompleted, second.ParseStatus)
|
||||
progress, err := f.svc.GetKnowledgeMoveProgress(f.ctx, "transfer-task")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 2, progress.Total)
|
||||
require.Zero(t, progress.Failed)
|
||||
require.Equal(t, types.KBCloneStatusCompleted, progress.Status)
|
||||
}
|
||||
|
||||
type transferQueue struct {
|
||||
fail bool
|
||||
ids []string
|
||||
}
|
||||
|
||||
func (q *transferQueue) Enqueue(_ *asynq.Task, options ...asynq.Option) (*asynq.TaskInfo, error) {
|
||||
for _, option := range options {
|
||||
if option.Type() == asynq.TaskIDOpt {
|
||||
q.ids = append(q.ids, option.Value().(string))
|
||||
}
|
||||
}
|
||||
if q.fail {
|
||||
return nil, errors.New("queue unavailable")
|
||||
}
|
||||
return &asynq.TaskInfo{ID: "processing-task"}, nil
|
||||
}
|
||||
|
||||
func TestMoveReparseRetryOnlyReenqueuesAndAccountsOnce(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
row.SetManualMetadata(types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1)),
|
||||
)
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
queue := &transferQueue{fail: true}
|
||||
f.svc.task = queue
|
||||
task := moveTask(t, []string{"doc"}, "reparse")
|
||||
require.ErrorContains(t, f.svc.ProcessKnowledgeMove(context.Background(), task), "queue unavailable")
|
||||
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", row.KnowledgeBaseID)
|
||||
require.Zero(t, row.StorageSize)
|
||||
cleanupCalls := f.graph.calls
|
||||
var tenant types.Tenant
|
||||
require.NoError(t, f.db.First(&tenant, 7).Error)
|
||||
require.Zero(t, tenant.StorageUsed)
|
||||
// A parser may already have written manual metadata before the move's ack.
|
||||
meta, err := row.ManualMetadata()
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, row.SetManualMetadata(meta))
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
queue.fail = false
|
||||
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), task))
|
||||
require.Equal(t, cleanupCalls, f.graph.calls)
|
||||
require.Len(t, queue.ids, 2)
|
||||
require.Equal(t, queue.ids[0], queue.ids[1])
|
||||
require.NoError(t, f.db.First(&tenant, 7).Error)
|
||||
require.Zero(t, tenant.StorageUsed)
|
||||
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
state, err := transferState(row)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "done", state.Phase)
|
||||
}
|
||||
|
||||
type cloneFaultChunks struct {
|
||||
interfaces.ChunkRepository
|
||||
fail bool
|
||||
}
|
||||
|
||||
func (r *cloneFaultChunks) CreateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
if err := r.ChunkRepository.CreateChunks(ctx, chunks); err != nil {
|
||||
return err
|
||||
}
|
||||
if r.fail {
|
||||
return errors.New("lost clone acknowledgement")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestCloneRetryReplacesIncompleteCopyWithoutDuplicatingEmptyHashes(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferClone)
|
||||
fault := &cloneFaultChunks{ChunkRepository: f.chunkRepo, fail: true}
|
||||
f.svc.chunkRepo = fault
|
||||
require.ErrorContains(
|
||||
t,
|
||||
f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil),
|
||||
"lost clone acknowledgement",
|
||||
)
|
||||
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rows, 1)
|
||||
require.Equal(t, types.ParseStatusFailed, rows[0].ParseStatus)
|
||||
failedChunks, chunkErr := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, rows[0].ID)
|
||||
require.NoError(t, chunkErr)
|
||||
require.Empty(t, failedChunks, "a lost create acknowledgement must also be rolled back")
|
||||
fault.fail = false
|
||||
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
|
||||
rows, err = f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rows, 1)
|
||||
require.Equal(t, types.ParseStatusCompleted, rows[0].ParseStatus)
|
||||
id := rows[0].ID
|
||||
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
|
||||
rows, err = f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rows, 1)
|
||||
require.Equal(t, id, rows[0].ID)
|
||||
var tenant types.Tenant
|
||||
require.NoError(t, f.db.First(&tenant, 7).Error)
|
||||
require.EqualValues(t, 10, tenant.StorageUsed)
|
||||
}
|
||||
|
||||
func TestClonePreflightDoesNotDeleteTargetForUnreadySource(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferClone)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("parse_status", types.ParseStatusProcessing).Error,
|
||||
)
|
||||
require.Error(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
|
||||
_, err := f.repo.GetKnowledgeByID(f.ctx, 7, "other-doc")
|
||||
require.NoError(t, err)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
}
|
||||
|
||||
func TestTransferCheckpointCannotResurrectOrOverwriteChangedDocument(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
before, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
after := *before
|
||||
after.StorageSize = 0
|
||||
after.KnowledgeBaseID = "other"
|
||||
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "third").Error)
|
||||
require.Error(t, f.repo.UpdateKnowledgeForTransfer(f.ctx, before, &after))
|
||||
var tenant types.Tenant
|
||||
require.NoError(t, f.db.First(&tenant, 7).Error)
|
||||
require.EqualValues(t, 5, tenant.StorageUsed)
|
||||
require.NoError(t, f.db.Where("id = ?", "doc").Delete(&types.Knowledge{}).Error)
|
||||
require.Error(t, f.repo.UpdateKnowledgeForTransfer(f.ctx, before, &after))
|
||||
_, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestProcessingTasksRejectOldKnowledgeBaseAfterMove(t *testing.T) {
|
||||
for _, taskType := range []string{types.TypeDocumentProcess, types.TypeManualProcess} {
|
||||
t.Run(taskType, func(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).
|
||||
Where("id = ?", "doc").
|
||||
Updates(map[string]any{"knowledge_base_id": "other", "parse_status": types.ParseStatusPending}).
|
||||
Error,
|
||||
)
|
||||
payload, err := json.Marshal(
|
||||
types.DocumentProcessPayload{TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb"},
|
||||
)
|
||||
require.NoError(t, err)
|
||||
task := asynq.NewTask(taskType, payload)
|
||||
if taskType == types.TypeDocumentProcess {
|
||||
err = f.svc.ProcessDocument(context.Background(), task)
|
||||
} else {
|
||||
err = f.svc.ProcessManualUpdate(context.Background(), task)
|
||||
}
|
||||
require.ErrorIs(t, err, asynq.SkipRetry)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCopyServiceResumesReservedTargetAndRequiresTransferGrant(t *testing.T) {
|
||||
repo := newFakeKBRepo()
|
||||
repo.rows["src"] = &types.KnowledgeBase{ID: "src", TenantID: 1}
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
require.Len(t, repo.rows, 1)
|
||||
ctx, err := access.WithKBTransferTask(
|
||||
context.Background(),
|
||||
repo.rows["src"],
|
||||
&types.KnowledgeBase{ID: "reserved", TenantID: 1, CreatorID: "creator"},
|
||||
1,
|
||||
access.KBTransferClone,
|
||||
"task",
|
||||
true,
|
||||
)
|
||||
require.NoError(t, err)
|
||||
for range 2 {
|
||||
_, target, err := svc.CopyKnowledgeBase(ctx, "src", "reserved")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "reserved", target.ID)
|
||||
require.Equal(t, "creator", target.CreatorID)
|
||||
}
|
||||
require.Len(t, repo.rows, 2)
|
||||
_, _, err = svc.CopyKnowledgeBase(ctx, "src", "other")
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
}
|
||||
|
||||
func TestProcessingCannotEnterClaimedMoveBeforeKBBindingChanges(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
setTransferState(
|
||||
row,
|
||||
knowledgeTransferState{
|
||||
TaskID: "transfer-task",
|
||||
Operation: access.KBTransferMove,
|
||||
SourceKB: "kb",
|
||||
TargetKB: "other",
|
||||
SourceID: "doc",
|
||||
Mode: "reuse_vectors",
|
||||
Phase: "moving",
|
||||
},
|
||||
),
|
||||
)
|
||||
row.ParseStatus = types.ParseStatusProcessing
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
f.repo.writes = 0
|
||||
payload, err := json.Marshal(types.DocumentProcessPayload{TenantID: 7, KnowledgeID: "doc", KnowledgeBaseID: "kb"})
|
||||
require.NoError(t, err)
|
||||
require.ErrorIs(
|
||||
t,
|
||||
f.svc.ProcessDocument(context.Background(), asynq.NewTask(types.TypeDocumentProcess, payload)),
|
||||
asynq.SkipRetry,
|
||||
)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
}
|
||||
|
||||
func TestAdmissionProgressCannotOverwriteFinishedWorker(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
require.NoError(
|
||||
t,
|
||||
f.svc.saveKnowledgeMoveProgress(
|
||||
f.ctx,
|
||||
&types.KnowledgeMoveProgress{TaskID: "task", Status: types.KBCloneStatusCompleted, Progress: 100},
|
||||
),
|
||||
)
|
||||
require.NoError(
|
||||
t,
|
||||
f.svc.SaveKnowledgeMoveProgress(
|
||||
f.ctx,
|
||||
&types.KnowledgeMoveProgress{TaskID: "task", Status: types.KBCloneStatusPending},
|
||||
),
|
||||
)
|
||||
progress, err := f.svc.GetKnowledgeMoveProgress(f.ctx, "task")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.KBCloneStatusCompleted, progress.Status)
|
||||
require.Equal(t, 100, progress.Progress)
|
||||
}
|
||||
|
||||
func TestClonePreservesFailedChunkIndexStatus(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferClone)
|
||||
require.NoError(t, f.db.Exec(`CREATE TRIGGER truncate_clone_created_time AFTER INSERT ON knowledges
|
||||
BEGIN UPDATE knowledges SET updated_at = strftime('%Y-%m-%d %H:%M:%S', NEW.updated_at) || '+00:00'
|
||||
WHERE id = NEW.id; END`).Error)
|
||||
require.NoError(t, f.db.Model(&types.Chunk{}).Where("id = ?", "chunk").Update("index_status", "failed").Error)
|
||||
require.NoError(t, f.svc.executeKnowledgeClone(f.ctx, f.kbs.values["kb"], f.kbs.values["other"], nil))
|
||||
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
|
||||
require.NoError(t, err)
|
||||
require.Len(t, rows, 1)
|
||||
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, rows[0].ID)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, chunks, 1)
|
||||
require.Equal(t, "failed", chunks[0].IndexStatus)
|
||||
}
|
||||
|
||||
func TestTableSummaryCleanupDoesNotFollowMovedDocument(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
chunk, err := f.chunkRepo.GetChunkByID(f.ctx, 7, "chunk")
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error)
|
||||
worker := &DataTableSummaryService{knowledgeService: f.svc, chunkService: f.chunks}
|
||||
worker.cleanupOnFailure(
|
||||
f.ctx, &extractionResources{knowledge: row}, []*types.Chunk{chunk}, errors.New("index failed"),
|
||||
)
|
||||
f.requireNoWrites(t)
|
||||
current, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", current.KnowledgeBaseID)
|
||||
require.Equal(t, row.ParseStatus, current.ParseStatus)
|
||||
}
|
||||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
|
@ -21,6 +22,17 @@ func TestUpdateKnowledgeExplicitEmptyDescriptionClearsSummary(t *testing.T) {
|
|||
repo: repo, kbService: metadataUpdateKBService{},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
err := service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
@ -45,6 +57,17 @@ func TestUpdateKnowledgeOmittedDescriptionPreservesExisting(t *testing.T) {
|
|||
repo: repo, kbService: metadataUpdateKBService{},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
err := service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
@ -68,6 +91,17 @@ func TestUpdateKnowledgeManualDescriptionSetsCompletedStatus(t *testing.T) {
|
|||
repo: repo, kbService: metadataUpdateKBService{},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
err := service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
@ -93,6 +127,17 @@ func TestUpdateKnowledgeDescriptionOnlyDoesNotTouchMetadata(t *testing.T) {
|
|||
repo: repo, kbService: metadataUpdateKBService{},
|
||||
}
|
||||
ctx := context.WithValue(context.Background(), types.TenantIDContextKey, uint64(7))
|
||||
ctx = (&access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{
|
||||
ID: "kb-1",
|
||||
TenantID: 7,
|
||||
},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}).Context(
|
||||
ctx,
|
||||
)
|
||||
|
||||
err := service.UpdateKnowledge(ctx, &types.Knowledge{
|
||||
ID: "knowledge-1",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,141 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
||||
type knowledgeBaseWriteLookup interface {
|
||||
GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error)
|
||||
}
|
||||
|
||||
func writeResourceIDs(ids []string) ([]string, error) {
|
||||
result := make([]string, 0, len(ids))
|
||||
seen := make(map[string]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
if strings.TrimSpace(id) == "" {
|
||||
return nil, apperrors.NewBadRequestError("resource ID cannot be empty")
|
||||
}
|
||||
if !seen[id] {
|
||||
seen[id] = true
|
||||
result = append(result, id)
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func writeExecutionTenant(ctx context.Context) (uint64, error) {
|
||||
tenant, ok := types.TenantIDFromContext(ctx)
|
||||
if !ok || tenant == 0 {
|
||||
return 0, apperrors.NewUnauthorizedError("workspace context unavailable")
|
||||
}
|
||||
return tenant, nil
|
||||
}
|
||||
|
||||
// Resolve persisted bindings before consuming grants. An input object's KB or
|
||||
// tenant must never select the scope used to authorize an existing resource.
|
||||
func knowledgeWriteKB(
|
||||
ctx context.Context,
|
||||
lookup knowledgeBaseWriteLookup,
|
||||
knowledge *types.Knowledge,
|
||||
) (*types.KnowledgeBase, error) {
|
||||
if knowledge == nil || knowledge.ID == "" || knowledge.KnowledgeBaseID == "" || knowledge.TenantID == 0 {
|
||||
return nil, apperrors.NewNotFoundError("knowledge not found")
|
||||
}
|
||||
kb, err := lookup.GetKnowledgeBaseByID(ctx, knowledge.KnowledgeBaseID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if kb == nil || kb.ID != knowledge.KnowledgeBaseID || kb.TenantID != knowledge.TenantID {
|
||||
return nil, apperrors.NewForbiddenError("knowledge does not belong to its knowledge base")
|
||||
}
|
||||
return kb, nil
|
||||
}
|
||||
|
||||
func loadKnowledgeWrite(
|
||||
ctx context.Context,
|
||||
repo interfaces.KnowledgeRepository,
|
||||
lookup knowledgeBaseWriteLookup,
|
||||
id string,
|
||||
) (*types.Knowledge, *types.KnowledgeBase, error) {
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
knowledge, err := repo.GetKnowledgeByID(ctx, tenant, id)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if knowledge == nil || knowledge.ID != id || knowledge.TenantID != tenant {
|
||||
return nil, nil, apperrors.NewNotFoundError("knowledge not found")
|
||||
}
|
||||
if err := access.RejectMovingKnowledge(knowledge); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
kb, err := knowledgeWriteKB(ctx, lookup, knowledge)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if _, err := requireKBWrite(ctx, kb); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
copyOfKnowledge := *knowledge
|
||||
return ©OfKnowledge, kb, nil
|
||||
}
|
||||
|
||||
// loadKnowledgeWriteBatch validates every requested ID, including missing
|
||||
// entries, before any caller can apply changes. Each KB needs its own grant.
|
||||
func loadKnowledgeWriteBatch(
|
||||
ctx context.Context,
|
||||
repo interfaces.KnowledgeRepository,
|
||||
lookup knowledgeBaseWriteLookup,
|
||||
ids []string,
|
||||
) ([]*types.Knowledge, error) {
|
||||
ids, err := writeResourceIDs(ids)
|
||||
if err != nil || len(ids) == 0 {
|
||||
return nil, err
|
||||
}
|
||||
tenant, err := writeExecutionTenant(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rows, err := repo.GetKnowledgeBatch(ctx, tenant, ids)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
byID := make(map[string]*types.Knowledge, len(rows))
|
||||
for _, row := range rows {
|
||||
if row != nil && row.TenantID == tenant {
|
||||
byID[row.ID] = row
|
||||
}
|
||||
}
|
||||
checked := make(map[string]bool)
|
||||
result := make([]*types.Knowledge, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
row := byID[id]
|
||||
if row == nil {
|
||||
return nil, apperrors.NewNotFoundError("knowledge not found")
|
||||
}
|
||||
if err := access.RejectMovingKnowledge(row); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !checked[row.KnowledgeBaseID] {
|
||||
kb, err := knowledgeWriteKB(ctx, lookup, row)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := requireKBWrite(ctx, kb); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
checked[row.KnowledgeBaseID] = true
|
||||
}
|
||||
copyOfRow := *row
|
||||
result = append(result, ©OfRow)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
|
@ -0,0 +1,442 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
"github.com/stretchr/testify/require"
|
||||
"gorm.io/driver/sqlite"
|
||||
"gorm.io/gorm"
|
||||
)
|
||||
|
||||
type writeKBLookup struct {
|
||||
interfaces.KnowledgeBaseService
|
||||
kb *types.KnowledgeBase
|
||||
}
|
||||
|
||||
func (s *writeKBLookup) GetKnowledgeBaseByID(context.Context, string) (*types.KnowledgeBase, error) {
|
||||
return s.kb, nil
|
||||
}
|
||||
|
||||
type writeChunkSpy struct {
|
||||
interfaces.ChunkRepository
|
||||
writes int
|
||||
}
|
||||
|
||||
func (s *writeChunkSpy) UpdateChunkFieldsByTagID(
|
||||
ctx context.Context,
|
||||
tenant uint64,
|
||||
kb, tag string,
|
||||
enabled *bool,
|
||||
set, clearFlags types.ChunkFlags,
|
||||
next *string,
|
||||
exclude []string,
|
||||
) ([]string, error) {
|
||||
s.writes++
|
||||
return s.ChunkRepository.UpdateChunkFieldsByTagID(ctx, tenant, kb, tag, enabled, set, clearFlags, next, exclude)
|
||||
}
|
||||
|
||||
func (s *writeChunkSpy) UpdateChunks(ctx context.Context, chunks []*types.Chunk) error {
|
||||
s.writes++
|
||||
return s.ChunkRepository.UpdateChunks(ctx, chunks)
|
||||
}
|
||||
|
||||
func (s *writeChunkSpy) UpdateChunkFlagsBatch(
|
||||
ctx context.Context,
|
||||
tenant uint64,
|
||||
kb string,
|
||||
set, clearFlags map[string]types.ChunkFlags,
|
||||
) error {
|
||||
s.writes++
|
||||
return s.ChunkRepository.UpdateChunkFlagsBatch(ctx, tenant, kb, set, clearFlags)
|
||||
}
|
||||
|
||||
type writeChunkDeleter struct {
|
||||
interfaces.ChunkService
|
||||
deleted []string
|
||||
}
|
||||
|
||||
func (s *writeChunkDeleter) DeleteChunk(_ context.Context, id string) error {
|
||||
s.deleted = append(s.deleted, id)
|
||||
return nil
|
||||
}
|
||||
|
||||
type writeIndexSpy struct {
|
||||
interfaces.RetrieveEngineService
|
||||
enabled map[string]bool
|
||||
tags map[string]string
|
||||
}
|
||||
|
||||
func (*writeIndexSpy) EngineType() types.RetrieverEngineType {
|
||||
return types.PostgresRetrieverEngineType
|
||||
}
|
||||
|
||||
func (*writeIndexSpy) Support() []types.RetrieverType {
|
||||
return []types.RetrieverType{types.VectorRetrieverType}
|
||||
}
|
||||
|
||||
func (s *writeIndexSpy) BatchUpdateChunkEnabledStatus(_ context.Context, values map[string]bool) error {
|
||||
s.enabled = values
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *writeIndexSpy) BatchUpdateChunkTagID(_ context.Context, values map[string]string) error {
|
||||
s.tags = values
|
||||
return nil
|
||||
}
|
||||
|
||||
type faqWriteFixture struct {
|
||||
svc *knowledgeService
|
||||
tags *knowledgeTagService
|
||||
db *gorm.DB
|
||||
ctx context.Context
|
||||
chunks *writeChunkSpy
|
||||
deletes *writeChunkDeleter
|
||||
index *writeIndexSpy
|
||||
kb *types.KnowledgeBase
|
||||
}
|
||||
|
||||
func newFAQWriteFixture(t *testing.T) *faqWriteFixture {
|
||||
t.Helper()
|
||||
db, err := gorm.Open(sqlite.Open("file:"+t.Name()+"?mode=memory&cache=shared"), &gorm.Config{})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, db.AutoMigrate(&types.Knowledge{}, &types.Chunk{}, &types.KnowledgeTag{}))
|
||||
store := "write-store"
|
||||
kb := &types.KnowledgeBase{ID: "kb", TenantID: 7, Type: types.KnowledgeBaseTypeFAQ, VectorStoreID: &store}
|
||||
lookup := &writeKBLookup{kb: kb}
|
||||
chunks := &writeChunkSpy{ChunkRepository: repository.NewChunkRepository(db)}
|
||||
deletes := &writeChunkDeleter{}
|
||||
index := &writeIndexSpy{}
|
||||
svc := &knowledgeService{
|
||||
kbService: lookup, chunkRepo: chunks, chunkService: deletes,
|
||||
tagRepo: repository.NewKnowledgeTagRepository(db), repo: repository.NewKnowledgeRepository(db),
|
||||
retrieveEngine: &fakeFanoutRegistry{byStore: map[string]interfaces.RetrieveEngineService{store: index}},
|
||||
ownership: &fakeOwnership{owned: map[string]uint64{store: 7}},
|
||||
}
|
||||
base := types.WithCaller(
|
||||
context.Background(),
|
||||
types.Caller{TenantID: 1, UserID: "caller", Role: types.TenantRoleAdmin},
|
||||
)
|
||||
grant := &access.KBAccess{
|
||||
KnowledgeBase: kb,
|
||||
Caller: types.CallerFromContext(base),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleEditor,
|
||||
}
|
||||
ctx := context.WithValue(grant.Context(base), types.TenantInfoContextKey, &types.Tenant{ID: 7})
|
||||
for _, tag := range []*types.KnowledgeTag{
|
||||
{ID: "tag", SeqID: 11, TenantID: 7, KnowledgeBaseID: "kb", Name: "tag"},
|
||||
{ID: "next", SeqID: 12, TenantID: 7, KnowledgeBaseID: "kb", Name: "next"},
|
||||
{ID: "foreign", SeqID: 22, TenantID: 7, KnowledgeBaseID: "other", Name: "foreign"},
|
||||
} {
|
||||
require.NoError(t, db.Create(tag).Error)
|
||||
}
|
||||
for _, chunk := range []*types.Chunk{
|
||||
{
|
||||
ID: "one",
|
||||
SeqID: 1,
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb",
|
||||
KnowledgeID: "faq",
|
||||
ChunkType: types.ChunkTypeFAQ,
|
||||
TagID: "tag",
|
||||
IsEnabled: true,
|
||||
},
|
||||
|
||||
{
|
||||
ID: "two",
|
||||
SeqID: 2,
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "other",
|
||||
KnowledgeID: "foreign-faq",
|
||||
ChunkType: types.ChunkTypeFAQ,
|
||||
TagID: "foreign",
|
||||
IsEnabled: true,
|
||||
},
|
||||
} {
|
||||
require.NoError(t, db.Create(chunk).Error)
|
||||
}
|
||||
require.NoError(
|
||||
t,
|
||||
db.Create(&types.Knowledge{ID: "faq", TenantID: 7, KnowledgeBaseID: "kb", Type: types.KnowledgeTypeFAQ}).Error,
|
||||
)
|
||||
tags := &knowledgeTagService{kbService: lookup, repo: svc.tagRepo}
|
||||
tags.chunkRepo = chunks
|
||||
svc.tagService = tags
|
||||
return &faqWriteFixture{
|
||||
svc: svc,
|
||||
tags: tags,
|
||||
db: db,
|
||||
ctx: ctx,
|
||||
chunks: chunks,
|
||||
deletes: deletes,
|
||||
index: index,
|
||||
kb: kb,
|
||||
}
|
||||
}
|
||||
|
||||
func TestFAQAndTagWritesRejectUnscopedServiceCalls(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
ctx := types.WithExecutionTenant(types.WithCaller(context.Background(), types.Caller{TenantID: 1}), 7)
|
||||
operations := map[string]func(context.Context) error{
|
||||
"create FAQ": func(ctx context.Context) error {
|
||||
_, err := f.svc.CreateFAQEntry(ctx, "kb", &types.FAQEntryPayload{})
|
||||
return err
|
||||
},
|
||||
"update FAQ": func(ctx context.Context) error {
|
||||
_, err := f.svc.UpdateFAQEntry(ctx, "kb", 1, &types.FAQEntryPayload{})
|
||||
return err
|
||||
},
|
||||
"similar questions": func(ctx context.Context) error {
|
||||
_, err := f.svc.AddSimilarQuestions(ctx, "kb", 1, []string{"question"})
|
||||
return err
|
||||
},
|
||||
"status": func(ctx context.Context) error { return f.svc.UpdateFAQEntryStatus(ctx, "kb", "one", false) },
|
||||
"fields": func(ctx context.Context) error {
|
||||
return f.svc.UpdateFAQEntryFieldsBatch(
|
||||
ctx,
|
||||
"kb",
|
||||
&types.FAQEntryFieldsBatchUpdate{ByID: map[int64]types.FAQEntryFieldsUpdate{1: {}}},
|
||||
)
|
||||
},
|
||||
"single tag": func(ctx context.Context) error { return f.svc.UpdateFAQEntryTag(ctx, "kb", "one", nil) },
|
||||
"batch tags": func(ctx context.Context) error {
|
||||
return f.svc.UpdateFAQEntryTagBatch(ctx, "kb", map[int64]*int64{1: nil})
|
||||
},
|
||||
"delete": func(ctx context.Context) error { return f.svc.DeleteFAQEntries(ctx, "kb", []int64{1}) },
|
||||
"import": func(ctx context.Context) error {
|
||||
_, err := f.svc.UpsertFAQEntries(
|
||||
ctx,
|
||||
"kb",
|
||||
&types.FAQBatchUpsertPayload{Entries: []types.FAQEntryPayload{{}}},
|
||||
)
|
||||
return err
|
||||
},
|
||||
"import display": func(ctx context.Context) error {
|
||||
return f.svc.UpdateLastFAQImportResultDisplayStatus(ctx, "kb", "close")
|
||||
},
|
||||
"create tag": func(ctx context.Context) error {
|
||||
_,
|
||||
err := f.tags.CreateTag(ctx,
|
||||
"kb",
|
||||
"new",
|
||||
"",
|
||||
0)
|
||||
return err
|
||||
},
|
||||
|
||||
"find/create tag": func(ctx context.Context) error {
|
||||
_,
|
||||
err := f.tags.FindOrCreateTagByName(ctx,
|
||||
"kb",
|
||||
"tag")
|
||||
return err
|
||||
},
|
||||
|
||||
"update tag": func(ctx context.Context) error {
|
||||
_,
|
||||
err := f.tags.UpdateTag(ctx,
|
||||
"tag",
|
||||
nil,
|
||||
nil,
|
||||
nil)
|
||||
return err
|
||||
},
|
||||
|
||||
"delete tag": func(ctx context.Context) error { return f.tags.DeleteTag(ctx, "tag", true, false, nil) },
|
||||
}
|
||||
for name, operation := range operations {
|
||||
t.Run(name, func(t *testing.T) { require.Error(t, operation(ctx)) })
|
||||
}
|
||||
require.Zero(t, f.chunks.writes)
|
||||
require.Empty(t, f.deletes.deleted)
|
||||
// The same service accepts a correctly scoped shared Editor operation.
|
||||
tag, err := f.tags.CreateTag(f.ctx, "kb", "created", "", 0)
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, uint64(7), tag.TenantID)
|
||||
}
|
||||
|
||||
func TestFAQBatchPreflightRejectsBeforeAnyWrite(t *testing.T) {
|
||||
no := false
|
||||
foreign := int64(22)
|
||||
for name, req := range map[string]*types.FAQEntryFieldsBatchUpdate{
|
||||
"foreign entry after group": {
|
||||
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
|
||||
ByID: map[int64]types.FAQEntryFieldsUpdate{2: {IsEnabled: &no}},
|
||||
},
|
||||
|
||||
"missing entry after group": {
|
||||
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
|
||||
ByID: map[int64]types.FAQEntryFieldsUpdate{999: {}},
|
||||
},
|
||||
|
||||
"foreign source tag": {ByTag: map[int64]types.FAQEntryFieldsUpdate{
|
||||
11: {IsEnabled: &no},
|
||||
22: {IsEnabled: &no},
|
||||
}},
|
||||
|
||||
"foreign destination tag": {ByID: map[int64]types.FAQEntryFieldsUpdate{1: {TagID: &foreign}}},
|
||||
"foreign exclusion": {
|
||||
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
|
||||
ExcludeIDs: []int64{2},
|
||||
},
|
||||
|
||||
"missing exclusion": {
|
||||
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {IsEnabled: &no}},
|
||||
ExcludeIDs: []int64{999},
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
require.Error(t, f.svc.UpdateFAQEntryFieldsBatch(f.ctx, "kb", req))
|
||||
require.Zero(t, f.chunks.writes)
|
||||
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
|
||||
require.NoError(t, err)
|
||||
require.True(t, chunk.IsEnabled)
|
||||
require.Equal(t, "tag", chunk.TagID)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFAQDeleteValidatesEntireSelectionAndParents(t *testing.T) {
|
||||
for _, ids := range [][]int64{{1, 2}, {1, 999}, {1, 0}} {
|
||||
t.Run(string(mustWriteJSON(t, ids)), func(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
require.Error(t, f.svc.DeleteFAQEntries(f.ctx, "kb", ids))
|
||||
require.Empty(t, f.deletes.deleted)
|
||||
})
|
||||
}
|
||||
t.Run("foreign parent", func(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "faq").Update("knowledge_base_id", "other").Error,
|
||||
)
|
||||
require.Error(t, f.svc.DeleteFAQEntries(f.ctx, "kb", []int64{1}))
|
||||
require.Empty(t, f.deletes.deleted)
|
||||
})
|
||||
}
|
||||
|
||||
func TestFAQFieldGroupsAndExplicitUpdatesPreservePrecedence(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
yes, no := true, false
|
||||
next := int64(12)
|
||||
req := &types.FAQEntryFieldsBatchUpdate{
|
||||
ByTag: map[int64]types.FAQEntryFieldsUpdate{11: {TagID: &next, IsRecommended: &yes}},
|
||||
ByID: map[int64]types.FAQEntryFieldsUpdate{1: {IsEnabled: &no, IsRecommended: &no}},
|
||||
}
|
||||
require.NoError(t, f.svc.UpdateFAQEntryFieldsBatch(f.ctx, "kb", req))
|
||||
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
|
||||
require.NoError(t, err)
|
||||
require.False(t, chunk.IsEnabled)
|
||||
require.Equal(t, "next", chunk.TagID, "an explicit enabled patch must preserve the group's tag change")
|
||||
require.False(t, chunk.Flags.HasFlag(types.ChunkFlagRecommended))
|
||||
require.Equal(t, map[string]string{"one": "next"}, f.index.tags)
|
||||
require.Equal(t, map[string]bool{"one": false}, f.index.enabled)
|
||||
}
|
||||
|
||||
func TestFAQTagOnlyBatchUsesValidatedPlanAndSynchronizesIndex(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
foreign := int64(22)
|
||||
require.Error(t, f.svc.UpdateFAQEntryTagBatch(f.ctx, "kb", map[int64]*int64{1: &foreign}))
|
||||
require.Zero(t, f.chunks.writes)
|
||||
require.NoError(t, f.svc.UpdateFAQEntryTagBatch(f.ctx, "kb", map[int64]*int64{1: nil}))
|
||||
require.Equal(t, map[string]string{"one": ""}, f.index.tags)
|
||||
}
|
||||
|
||||
func TestFAQImportRejectsForeignTagsBeforeEnqueue(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
_, err := f.svc.UpsertFAQEntries(
|
||||
f.ctx,
|
||||
"kb",
|
||||
&types.FAQBatchUpsertPayload{Entries: []types.FAQEntryPayload{{TagID: 22}}},
|
||||
)
|
||||
require.Error(t, err) // Redis, file storage and the queue are intentionally unwired.
|
||||
require.Zero(t, f.chunks.writes)
|
||||
_, err = f.svc.resolveTagID(f.ctx, "kb", &types.FAQEntryPayload{TagID: 22})
|
||||
require.Error(t, err)
|
||||
resolver := f.svc.buildFAQTagResolver(f.ctx, "kb", []types.FAQEntryPayload{{TagID: 22}})
|
||||
_, err = resolver(&types.FAQEntryPayload{TagID: 22})
|
||||
require.Error(t, err)
|
||||
}
|
||||
|
||||
func TestFAQImportWorkerRejectsMismatchedScopeBeforeSideEffects(t *testing.T) {
|
||||
for _, tenant := range []uint64{7, 8} {
|
||||
t.Run(string(mustWriteJSON(t, tenant)), func(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Create(
|
||||
&types.Knowledge{
|
||||
ID: "foreign-faq",
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "other",
|
||||
Type: types.KnowledgeTypeFAQ,
|
||||
},
|
||||
).Error,
|
||||
)
|
||||
payload := types.FAQImportPayload{TenantID: tenant, KBID: "kb", KnowledgeID: "foreign-faq", TaskID: "task"}
|
||||
err := f.svc.ProcessFAQImport(
|
||||
context.Background(),
|
||||
asynq.NewTask(types.TypeFAQImport, mustWriteJSON(t, payload)),
|
||||
)
|
||||
require.ErrorIs(t, err, asynq.SkipRetry)
|
||||
require.Zero(t, f.chunks.writes)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagDeleteRejectsInvalidExclusionsBeforeDeletion(t *testing.T) {
|
||||
for _, id := range []string{"two", "missing"} {
|
||||
t.Run(id, func(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
require.Error(t, f.tags.DeleteTag(f.ctx, "tag", true, false, []string{id}))
|
||||
tag, err := f.svc.tagRepo.GetByID(f.ctx, 7, "tag")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, tag)
|
||||
chunk, err := f.chunks.GetChunkByID(f.ctx, 7, "one")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, chunk)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func mustWriteJSON(t *testing.T, value interface{}) []byte {
|
||||
t.Helper()
|
||||
data, err := json.Marshal(value)
|
||||
require.NoError(t, err)
|
||||
return data
|
||||
}
|
||||
|
||||
func TestSharedFAQWriteLoadsOwnerTenantInfoWithoutReplacingCaller(t *testing.T) {
|
||||
f := newFAQWriteFixture(t)
|
||||
f.svc.tenantRepo = &processSyncTenantRepo{tenant: &types.Tenant{ID: 7}}
|
||||
ctx := context.WithValue(f.ctx, types.TenantInfoContextKey, &types.Tenant{ID: 1})
|
||||
_, scoped, err := f.svc.writableFAQKnowledgeBase(ctx, "kb")
|
||||
require.NoError(t, err)
|
||||
owner, _ := types.TenantInfoFromContext(scoped)
|
||||
require.Equal(t, uint64(7), owner.ID)
|
||||
require.Equal(t, uint64(1), types.CallerFromContext(scoped).TenantID)
|
||||
original, _ := types.TenantInfoFromContext(ctx)
|
||||
require.Equal(t, uint64(1), original.ID)
|
||||
}
|
||||
|
||||
func TestDataSourceTagCreationReceivesOnlyItsTaskKBGrant(t *testing.T) {
|
||||
h := newSyncDeletionHarness(t, false, "ds-tag-scope", "log-tag-scope", nil, nil)
|
||||
tags := h.svc.tagService.(*processSyncTagService)
|
||||
_, err := h.run(t)
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, tags.ctx)
|
||||
require.Zero(t, types.CallerFromContext(tags.ctx).TenantID)
|
||||
require.NoError(
|
||||
t,
|
||||
access.RequireKBWrite(tags.ctx, &types.KnowledgeBase{ID: h.ds.KnowledgeBaseID, TenantID: h.ds.TenantID}),
|
||||
)
|
||||
require.Error(t, access.RequireKBWrite(tags.ctx, &types.KnowledgeBase{ID: "other", TenantID: h.ds.TenantID}))
|
||||
}
|
||||
|
|
@ -9,6 +9,8 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
"github.com/Tencent/WeKnora/internal/datasource"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
|
|
@ -1099,72 +1101,57 @@ func (s *knowledgeBaseService) SetEmbeddingModel(ctx context.Context, id string,
|
|||
// CopyKnowledgeBase copies a knowledge base to a new knowledge base (shallow copy).
|
||||
// Source and target must belong to the tenant in context; cross-tenant access is rejected.
|
||||
//
|
||||
// Defensive checks:
|
||||
//
|
||||
// - When dstKB != "" (clone into an existing target), the source's
|
||||
// EmbeddingModelID and VectorStoreID must match the target's. Mismatched
|
||||
// embedding models would silently mix incompatible vector spaces;
|
||||
// mismatched vector stores would require copying physical vector data
|
||||
// between stores, which is not yet supported.
|
||||
// - When dstKB == "" (create a new target), VectorStoreID is copied from
|
||||
// the source so the new KB shares the same physical vector index. GORM
|
||||
// `<-:create` allows INSERT, so the new row is well-formed.
|
||||
//
|
||||
// The handler's CopyKnowledgeBase endpoint runs the same checks synchronously
|
||||
// before enqueueing the async clone task, so the 400 errors here are
|
||||
// defense-in-depth for the worker entry point.
|
||||
// The transfer grant fixes both resource IDs and whether a new destination
|
||||
// may be created. Existing targets must be compatible with the source. A
|
||||
// worker retry reloads the reserved target instead of creating another KB.
|
||||
func (s *knowledgeBaseService) CopyKnowledgeBase(ctx context.Context,
|
||||
srcKB string, dstKB string,
|
||||
) (*types.KnowledgeBase, *types.KnowledgeBase, error) {
|
||||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
// Load source KB with tenant scope to prevent cross-tenant cloning
|
||||
sourceKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, srcKB, tenantID)
|
||||
destinationID, create, creatorID, err := access.CloneDestination(ctx, srcKB)
|
||||
if err != nil {
|
||||
logger.Errorf(ctx, "Get source knowledge base failed: %v", err)
|
||||
return nil, nil, err
|
||||
}
|
||||
if dstKB != "" && dstKB != destinationID {
|
||||
return nil, nil, access.ErrForbidden
|
||||
}
|
||||
tenantID := types.MustTenantIDFromContext(ctx)
|
||||
sourceKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, srcKB, tenantID)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if sourceKB == nil || sourceKB.ID != srcKB || sourceKB.TenantID != tenantID {
|
||||
return nil, nil, access.ErrForbidden
|
||||
}
|
||||
sourceCopy := *sourceKB
|
||||
sourceKB = &sourceCopy
|
||||
sourceKB.EnsureDefaults()
|
||||
var targetKB *types.KnowledgeBase
|
||||
if dstKB != "" {
|
||||
// Load target KB with tenant scope so we only clone into the caller's tenant
|
||||
targetKB, err = s.repo.GetKnowledgeBaseByIDAndTenant(ctx, dstKB, tenantID)
|
||||
if err != nil {
|
||||
targetKB, err := s.repo.GetKnowledgeBaseByIDAndTenant(ctx, destinationID, tenantID)
|
||||
if err != nil && (!create || !errors.Is(err, repository.ErrKnowledgeBaseNotFound)) {
|
||||
return nil, nil, err
|
||||
}
|
||||
if targetKB != nil {
|
||||
targetCopy := *targetKB
|
||||
targetKB = &targetCopy
|
||||
targetKB.EnsureDefaults()
|
||||
if err := access.RequireKBTransfer(ctx, sourceKB, targetKB, access.KBTransferClone); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
// Defense 1: embedding model must match. Mixing incompatible
|
||||
// vector spaces would produce semantically broken search results.
|
||||
if sourceKB.EmbeddingModelID != targetKB.EmbeddingModelID {
|
||||
return nil, nil, apperrors.NewBadRequestError(
|
||||
"source and target knowledge bases use different embedding models; " +
|
||||
"clone into a target with the same embedding model")
|
||||
}
|
||||
|
||||
// Defense 2: vector store binding must match. Cross-store cloning
|
||||
// would require copying physical vector data between stores.
|
||||
// (both nil → equal; both same UUID → equal; otherwise → rejected)
|
||||
if !sourceKB.SharesStoreWith(targetKB) {
|
||||
return nil, nil, apperrors.NewBadRequestError(
|
||||
"source and target knowledge bases are bound to different vector stores; " +
|
||||
"cross-store cloning is not yet supported")
|
||||
}
|
||||
|
||||
// Defense 3: the concrete storage instance must match. Comparing only
|
||||
// provider names would incorrectly allow COS-A -> COS-B clones.
|
||||
if tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant); tenant != nil {
|
||||
defaultID, defaultProvider := "", ""
|
||||
if tenant.DefaultStorageBackendID != nil {
|
||||
defaultID = *tenant.DefaultStorageBackendID
|
||||
}
|
||||
if tenant.StorageEngineConfig != nil {
|
||||
defaultProvider = tenant.StorageEngineConfig.DefaultProvider
|
||||
}
|
||||
if !sourceKB.SharesStorageBackendWith(targetKB, defaultID, defaultProvider) {
|
||||
return nil, nil, apperrors.NewBadRequestError(
|
||||
"source and target knowledge bases use different storage instances; cross-storage-backend cloning is not supported")
|
||||
}
|
||||
tenant, _ := ctx.Value(types.TenantInfoContextKey).(*types.Tenant)
|
||||
if err := access.ValidateKBTransferCompatibility(sourceKB,
|
||||
targetKB,
|
||||
access.KBTransferClone,
|
||||
"",
|
||||
tenant); err != nil {
|
||||
return nil, nil, apperrors.NewBadRequestError(err.Error())
|
||||
}
|
||||
} else {
|
||||
reserved := &types.KnowledgeBase{ID: destinationID, TenantID: tenantID}
|
||||
if !create {
|
||||
return nil, nil, access.ErrNotFound
|
||||
}
|
||||
if err := access.RequireKBTransfer(ctx, sourceKB, reserved, access.KBTransferClone); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
var faqConfig *types.FAQConfig
|
||||
if sourceKB.FAQConfig != nil {
|
||||
cfg := *sourceKB.FAQConfig
|
||||
|
|
@ -1173,7 +1160,8 @@ func (s *knowledgeBaseService) CopyKnowledgeBase(ctx context.Context,
|
|||
// Preserve VectorStoreID so the cloned KB lands on the same
|
||||
// physical index. GORM `<-:create` permits the value at INSERT.
|
||||
targetKB = &types.KnowledgeBase{
|
||||
ID: uuid.New().String(),
|
||||
ID: destinationID,
|
||||
CreatorID: creatorID,
|
||||
Name: sourceKB.Name,
|
||||
Type: sourceKB.Type,
|
||||
Description: sourceKB.Description,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,74 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
)
|
||||
|
||||
// Service reads accept exact upstream grants. Otherwise cross-tenant reads
|
||||
// require a user and organization permission resolved for the original caller.
|
||||
func kbReadPermissions(ctx context.Context, shares access.KBShareLookup) *access.KBPermissions {
|
||||
if types.CallerFromContext(ctx).UserID == "" {
|
||||
shares = nil
|
||||
}
|
||||
return access.NewKBPermissions(ctx, shares)
|
||||
}
|
||||
|
||||
func resolveKBReadTenant(ctx context.Context, kb *types.KnowledgeBase, shares access.KBShareLookup) (uint64, error) {
|
||||
if kb != nil {
|
||||
allowed, err := kbReadPermissions(ctx, shares).Check(kb.ID, kb.TenantID, types.OrgRoleViewer)
|
||||
if err == nil && allowed {
|
||||
return kb.TenantID, nil
|
||||
}
|
||||
}
|
||||
return 0, apperrors.NewForbiddenError("无权访问该知识库")
|
||||
}
|
||||
|
||||
func requireKBWrite(ctx context.Context, kb *types.KnowledgeBase) (context.Context, error) {
|
||||
if err := access.RequireKBWrite(ctx, kb); err != nil {
|
||||
return ctx, apperrors.NewForbiddenError("无权修改该知识库")
|
||||
}
|
||||
return types.WithExecutionTenant(ctx, kb.TenantID), nil
|
||||
}
|
||||
|
||||
func (s *knowledgeService) writableFAQKnowledgeBase(
|
||||
ctx context.Context,
|
||||
kbID string,
|
||||
) (*types.KnowledgeBase, context.Context, error) {
|
||||
kb, err := s.validateFAQKnowledgeBase(ctx, kbID)
|
||||
if err != nil {
|
||||
return nil, ctx, err
|
||||
}
|
||||
ctx, err = requireKBWrite(ctx, kb)
|
||||
if err == nil {
|
||||
ctx, err = withKBWriteTenantInfo(ctx, kb, s.tenantRepo)
|
||||
}
|
||||
return kb, ctx, err
|
||||
}
|
||||
|
||||
// A shared KB mutation must also resolve models/index backends against its
|
||||
// owner. The authenticated caller remains unchanged when TenantInfo changes.
|
||||
func withKBWriteTenantInfo(
|
||||
ctx context.Context,
|
||||
kb *types.KnowledgeBase,
|
||||
tenants interfaces.TenantRepository,
|
||||
) (context.Context, error) {
|
||||
if tenant, ok := types.TenantInfoFromContext(ctx); ok && tenant != nil && tenant.ID == kb.TenantID {
|
||||
return ctx, nil
|
||||
}
|
||||
if tenants == nil {
|
||||
return ctx, apperrors.NewServiceUnavailableError("无法获取知识库所属空间")
|
||||
}
|
||||
tenant, err := tenants.GetTenantByID(ctx, kb.TenantID)
|
||||
if err != nil {
|
||||
return ctx, err
|
||||
}
|
||||
if tenant == nil || tenant.ID != kb.TenantID {
|
||||
return ctx, apperrors.NewNotFoundError("知识库所属空间不存在")
|
||||
}
|
||||
return context.WithValue(ctx, types.TenantInfoContextKey, tenant), nil
|
||||
}
|
||||
|
|
@ -6,6 +6,8 @@ import (
|
|||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/storageallowlist"
|
||||
|
|
@ -87,22 +89,27 @@ func (r *fakeKBRepo) CreateKnowledgeBase(_ context.Context, kb *types.KnowledgeB
|
|||
r.rows[kb.ID] = kb
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) GetKnowledgeBaseByID(_ context.Context, id string) (*types.KnowledgeBase, error) {
|
||||
return r.rows[id], nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) GetKnowledgeBaseByIDAndTenant(_ context.Context, id string, tenantID uint64) (*types.KnowledgeBase, error) {
|
||||
kb := r.rows[id]
|
||||
if kb == nil || kb.TenantID != tenantID {
|
||||
return nil, stderrors.New("not found")
|
||||
return nil, repository.ErrKnowledgeBaseNotFound
|
||||
}
|
||||
return kb, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) GetKnowledgeBaseByIDs(_ context.Context, _ []string) ([]*types.KnowledgeBase, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) ListKnowledgeBases(_ context.Context) ([]*types.KnowledgeBase, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) ListKnowledgeBasesByTenantID(_ context.Context, tenantID uint64) ([]*types.KnowledgeBase, error) {
|
||||
rows := make([]*types.KnowledgeBase, 0, len(r.rows))
|
||||
for _, kb := range r.rows {
|
||||
|
|
@ -112,6 +119,7 @@ func (r *fakeKBRepo) ListKnowledgeBasesByTenantID(_ context.Context, tenantID ui
|
|||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) UpdateKnowledgeBase(_ context.Context, _ *types.KnowledgeBase) error {
|
||||
return nil
|
||||
}
|
||||
|
|
@ -119,9 +127,11 @@ func (r *fakeKBRepo) DeleteKnowledgeBase(_ context.Context, _ string) error { re
|
|||
func (r *fakeKBRepo) TogglePinKnowledgeBase(_ context.Context, _ string, _ uint64) (*types.KnowledgeBase, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) CountByVectorStoreID(_ context.Context, _ *gorm.DB, _ uint64, _ string) (int64, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
||||
func (r *fakeKBRepo) CountByModelID(_ context.Context, _ uint64, _ string) (int64, error) {
|
||||
return 0, nil
|
||||
}
|
||||
|
|
@ -322,7 +332,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["src"] = mkKB("src", 1, "embed-1", &storeA)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
src, tgt, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
|
||||
src, tgt, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, ""), "src", "")
|
||||
require.NoError(t, err)
|
||||
require.NotNil(t, src.VectorStoreID)
|
||||
require.NotNil(t, tgt.VectorStoreID)
|
||||
|
|
@ -340,7 +350,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["src"] = mkKB("src", 1, "embed-1", nil)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, tgt, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "")
|
||||
_, tgt, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, ""), "src", "")
|
||||
require.NoError(t, err)
|
||||
assert.Nil(t, tgt.VectorStoreID)
|
||||
})
|
||||
|
|
@ -351,7 +361,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["dst"] = mkKB("dst", 1, "embed-1", nil)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
|
||||
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
|
||||
require.NoError(t, err)
|
||||
})
|
||||
|
||||
|
|
@ -361,7 +371,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeA)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
|
||||
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
|
||||
require.NoError(t, err)
|
||||
})
|
||||
|
||||
|
|
@ -371,7 +381,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["dst"] = mkKB("dst", 1, "embed-2", &storeA)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
|
||||
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "different embedding models")
|
||||
})
|
||||
|
|
@ -382,7 +392,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeB)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
|
||||
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "different vector stores")
|
||||
assert.NotContains(t, err.Error(), "Phase 4", "internal roadmap labels must not leak to end-user error messages")
|
||||
|
|
@ -394,7 +404,7 @@ func TestCopyKnowledgeBase_Defenses(t *testing.T) {
|
|||
repo.rows["dst"] = mkKB("dst", 1, "embed-1", &storeA)
|
||||
svc := newPR3KBService(repo, &fakeRegistry{}, &fakeOwnership{})
|
||||
|
||||
_, _, err := svc.CopyKnowledgeBase(ctxWithTenant(1), "src", "dst")
|
||||
_, _, err := svc.CopyKnowledgeBase(copyTaskTestContext(t, repo, "dst"), "src", "dst")
|
||||
require.Error(t, err)
|
||||
assert.Contains(t, err.Error(), "different vector stores")
|
||||
})
|
||||
|
|
@ -525,3 +535,22 @@ func TestDuplicateKnowledgeBase_UsesLocalizedEnglishSuffix(t *testing.T) {
|
|||
|
||||
assert.Equal(t, "Source KB Copy", target.Name)
|
||||
}
|
||||
|
||||
func copyTaskTestContext(t *testing.T, repo *fakeKBRepo, dst string) context.Context {
|
||||
t.Helper()
|
||||
target := repo.rows[dst]
|
||||
if target == nil {
|
||||
target = &types.KnowledgeBase{ID: "reserved-copy", TenantID: 1}
|
||||
}
|
||||
ctx, err := access.WithKBTransferTask(
|
||||
context.Background(),
|
||||
repo.rows["src"],
|
||||
target,
|
||||
1,
|
||||
access.KBTransferClone,
|
||||
"clone-task",
|
||||
dst == "",
|
||||
)
|
||||
require.NoError(t, err)
|
||||
return ctx
|
||||
}
|
||||
|
|
|
|||
|
|
@ -97,7 +97,7 @@ func (s *knowledgeBaseService) ResolveEmbeddingModelKeys(ctx context.Context, kb
|
|||
// Resolve each unique (modelID, tenantID) to a model identity key
|
||||
resolvedKeys := make(map[modelRef]string, len(uniqueRefs))
|
||||
for ref := range uniqueRefs {
|
||||
tenantCtx := context.WithValue(ctx, types.TenantIDContextKey, ref.TenantID)
|
||||
tenantCtx := types.WithExecutionTenant(ctx, ref.TenantID)
|
||||
model, err := s.modelService.GetModelByID(tenantCtx, ref.ModelID)
|
||||
if err != nil || model == nil {
|
||||
logger.Warnf(ctx, "ResolveEmbeddingModelKeys: cannot resolve model %s for tenant %d: %v", ref.ModelID, ref.TenantID, err)
|
||||
|
|
@ -143,7 +143,6 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
|
|||
searchKBIDs, secutils.SanitizeForLog(params.QueryText))
|
||||
|
||||
tenantInfo, _ := types.TenantInfoFromContext(ctx)
|
||||
requestTenantID := types.MustTenantIDFromContext(ctx)
|
||||
|
||||
// Batch-load every KB in scope. Required for store grouping,
|
||||
// embedding-model consistency validation, and FAQ type detection.
|
||||
|
|
@ -162,13 +161,13 @@ func (s *knowledgeBaseService) HybridSearch(ctx context.Context,
|
|||
return nil, apperrors.NewNotFoundError("knowledge base not found")
|
||||
}
|
||||
|
||||
// Authorize every KB the caller asked for. Same-tenant KBs are
|
||||
// always accessible; foreign-tenant KBs (Organization-shared) must
|
||||
// pass an explicit per-KB permission check. Without this guard, a
|
||||
// Authorize every KB for the original caller, using exact upstream
|
||||
// grants or organization permissions. Execution in a shared tenant
|
||||
// does not grant access to its other KBs. Without this guard, a
|
||||
// caller could pass arbitrary KB UUIDs in params.KnowledgeBaseIDs
|
||||
// and reach foreign tenants' bound vector stores via the per-group
|
||||
// engine resolution downstream.
|
||||
if err := s.authorizeKBAccess(ctx, kbs, requestTenantID); err != nil {
|
||||
if err := s.authorizeKBAccess(ctx, kbs); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -415,27 +415,35 @@ func (f *fakeRetrieveEngineService) Retrieve(ctx context.Context, p types.Retrie
|
|||
func (f *fakeRetrieveEngineService) Index(context.Context, embedding.Embedder, *types.IndexInfo, []types.RetrieverType) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) BatchIndex(context.Context, embedding.Embedder, []*types.IndexInfo, []types.RetrieverType) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) EstimateStorageSize(context.Context, embedding.Embedder, []*types.IndexInfo, []types.RetrieverType) int64 {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) CopyIndices(context.Context, string, map[string]string, map[string]string, string, int, string) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) DeleteByChunkIDList(context.Context, []string, int, string) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) DeleteBySourceIDList(context.Context, []string, int, string) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) DeleteByKnowledgeIDList(context.Context, []string, int, string) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) BatchUpdateChunkEnabledStatus(context.Context, map[string]bool) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
func (f *fakeRetrieveEngineService) BatchUpdateChunkTagID(context.Context, map[string]string) error {
|
||||
panic("unused")
|
||||
}
|
||||
|
|
@ -455,9 +463,11 @@ func (r *fakeFanoutRegistry) Register(interfaces.RetrieveEngineService) error {
|
|||
func (r *fakeFanoutRegistry) GetRetrieveEngineService(types.RetrieverEngineType) (interfaces.RetrieveEngineService, error) {
|
||||
return nil, stderrors.New("not used in fan-out tests")
|
||||
}
|
||||
|
||||
func (r *fakeFanoutRegistry) GetAllRetrieveEngineServices() []interfaces.RetrieveEngineService {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (r *fakeFanoutRegistry) GetByStoreID(id string) (interfaces.RetrieveEngineService, error) {
|
||||
if svc, ok := r.byStore[id]; ok {
|
||||
return svc, nil
|
||||
|
|
@ -775,7 +785,7 @@ func TestRetrieveFromStores_PerGroupTimeout(t *testing.T) {
|
|||
// ---------------------------------------------------------------------------
|
||||
|
||||
// fakeKBShareForAuth implements just enough of KBShareService for the
|
||||
// authorizeKBAccess test matrix. Only HasTenantKBPermission is exercised;
|
||||
// authorizeKBAccess test matrix. Only CheckTenantKBPermission is exercised;
|
||||
// the embedded interface keeps the type assignable.
|
||||
type fakeKBShareForAuth struct {
|
||||
// allowed maps kbID → tenantID → allowed. Mirrors the Plan 3 (#1303)
|
||||
|
|
@ -785,17 +795,10 @@ type fakeKBShareForAuth struct {
|
|||
interfaces.KBShareService
|
||||
}
|
||||
|
||||
func (f *fakeKBShareForAuth) HasTenantKBPermission(
|
||||
_ context.Context, kbID string, callerTenantID uint64,
|
||||
_ types.TenantRole, _ types.OrgMemberRole,
|
||||
) (bool, error) {
|
||||
if f.err != nil {
|
||||
return false, f.err
|
||||
}
|
||||
if perTenant, ok := f.allowed[kbID]; ok {
|
||||
return perTenant[callerTenantID], nil
|
||||
}
|
||||
return false, nil
|
||||
func (f *fakeKBShareForAuth) CheckTenantKBPermission(
|
||||
_ context.Context, kbID string, callerTenantID uint64, _ types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
return types.OrgRoleViewer, f.allowed[kbID][callerTenantID], f.err
|
||||
}
|
||||
|
||||
func ctxWithTenantForAuth(tenantID uint64) context.Context {
|
||||
|
|
@ -809,7 +812,7 @@ func TestAuthorizeKBAccess_SameTenantAllPass(t *testing.T) {
|
|||
{ID: "kb-1", TenantID: 7},
|
||||
{ID: "kb-2", TenantID: 7},
|
||||
}
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
|
|
@ -825,7 +828,7 @@ func TestAuthorizeKBAccess_ForeignTenantWithShare_OK(t *testing.T) {
|
|||
{ID: "kb-own", TenantID: 7},
|
||||
{ID: "kb-foreign", TenantID: 99},
|
||||
}
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
|
|
@ -840,7 +843,7 @@ func TestAuthorizeKBAccess_ForeignTenantNoShare_NotFound(t *testing.T) {
|
|||
kbs := []*types.KnowledgeBase{
|
||||
{ID: "kb-foreign", TenantID: 99},
|
||||
}
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
|
||||
require.Error(t, err)
|
||||
app, ok := apperrors.IsAppError(err)
|
||||
require.True(t, ok, "expected typed AppError, got %T", err)
|
||||
|
|
@ -853,7 +856,7 @@ func TestAuthorizeKBAccess_PermissionLookupError_500(t *testing.T) {
|
|||
share := &fakeKBShareForAuth{err: stderrors.New("share infra down")}
|
||||
s := &knowledgeBaseService{kbShareService: share}
|
||||
kbs := []*types.KnowledgeBase{{ID: "kb-foreign", TenantID: 99}}
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs, 7)
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), kbs)
|
||||
require.Error(t, err)
|
||||
app, ok := apperrors.IsAppError(err)
|
||||
require.True(t, ok)
|
||||
|
|
@ -863,7 +866,7 @@ func TestAuthorizeKBAccess_PermissionLookupError_500(t *testing.T) {
|
|||
func TestAuthorizeKBAccess_EmptyKBs_OK(t *testing.T) {
|
||||
t.Parallel()
|
||||
s := &knowledgeBaseService{kbShareService: &fakeKBShareForAuth{}}
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), nil, 7)
|
||||
err := s.authorizeKBAccess(ctxWithTenantForAuth(7), nil)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,9 @@ package service
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/repository"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
)
|
||||
|
|
@ -30,52 +32,42 @@ func (s *knowledgeBaseService) fetchKnowledgeData(ctx context.Context,
|
|||
}
|
||||
|
||||
// fetchKnowledgeDataWithShared gets knowledge data in batch, including knowledge
|
||||
// from shared KBs the user has access to.
|
||||
// from KBs authorized for the original caller. Initial tenant-scoped rows
|
||||
// require the same permission check as cross-tenant expansion.
|
||||
func (s *knowledgeBaseService) fetchKnowledgeDataWithShared(ctx context.Context,
|
||||
tenantID uint64,
|
||||
knowledgeIDs []string,
|
||||
) (map[string]*types.Knowledge, error) {
|
||||
knowledgeMap, err := s.fetchKnowledgeData(ctx, tenantID, knowledgeIDs)
|
||||
rows, err := s.fetchKnowledgeData(ctx, tenantID, knowledgeIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
missingIDs := s.findMissingIDs(knowledgeIDs, func(id string) bool {
|
||||
return knowledgeMap[id] != nil
|
||||
})
|
||||
if len(missingIDs) == 0 {
|
||||
return knowledgeMap, nil
|
||||
permissions := kbReadPermissions(ctx, s.kbShareService)
|
||||
knowledgeMap := make(map[string]*types.Knowledge, len(rows))
|
||||
appendAllowed := func(k *types.Knowledge) {
|
||||
if k == nil {
|
||||
return
|
||||
}
|
||||
allowed, err := permissions.Check(k.KnowledgeBaseID, k.TenantID, types.OrgRoleViewer)
|
||||
if err == nil && allowed {
|
||||
knowledgeMap[k.ID] = k
|
||||
}
|
||||
}
|
||||
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] %d knowledge IDs not found in current tenant, attempting shared KB lookup", len(missingIDs))
|
||||
|
||||
userID, ok := s.extractUserID(ctx)
|
||||
if !ok {
|
||||
logger.Warnf(ctx, "[fetchKnowledgeDataWithShared] userID not found or empty in context, skipping shared KB lookup")
|
||||
return knowledgeMap, nil
|
||||
for _, k := range rows {
|
||||
appendAllowed(k)
|
||||
}
|
||||
|
||||
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] Looking up %d missing knowledge IDs with userID=%s", len(missingIDs), userID)
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
for _, id := range missingIDs {
|
||||
for _, id := range knowledgeIDs {
|
||||
if rows[id] != nil {
|
||||
continue
|
||||
}
|
||||
k, err := s.kgRepo.GetKnowledgeByIDOnly(ctx, id)
|
||||
if err != nil || k == nil || k.KnowledgeBaseID == "" {
|
||||
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Knowledge %s not found or has no KB", id)
|
||||
continue
|
||||
if err != nil && !errors.Is(err, repository.ErrKnowledgeNotFound) {
|
||||
return nil, err
|
||||
}
|
||||
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, k.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
|
||||
if err != nil {
|
||||
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Permission check error for KB %s: %v", k.KnowledgeBaseID, err)
|
||||
continue
|
||||
if err == nil {
|
||||
appendAllowed(k)
|
||||
}
|
||||
if !hasPermission {
|
||||
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] No permission for KB %s", k.KnowledgeBaseID)
|
||||
continue
|
||||
}
|
||||
logger.Debugf(ctx, "[fetchKnowledgeDataWithShared] Found shared knowledge %s in KB %s", id, k.KnowledgeBaseID)
|
||||
knowledgeMap[k.ID] = k
|
||||
}
|
||||
|
||||
logger.Infof(ctx, "[fetchKnowledgeDataWithShared] After shared lookup, total knowledge found: %d", len(knowledgeMap))
|
||||
return knowledgeMap, nil
|
||||
}
|
||||
|
||||
|
|
@ -84,58 +76,41 @@ func (s *knowledgeBaseService) listChunksByIDWithShared(ctx context.Context,
|
|||
tenantID uint64,
|
||||
chunkIDs []string,
|
||||
) ([]*types.Chunk, error) {
|
||||
chunks, err := s.chunkRepo.ListChunksByID(ctx, tenantID, chunkIDs)
|
||||
rows, err := s.chunkRepo.ListChunksByID(ctx, tenantID, chunkIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
foundSet := make(map[string]bool, len(chunks))
|
||||
for _, c := range chunks {
|
||||
permissions := kbReadPermissions(ctx, s.kbShareService)
|
||||
chunks := make([]*types.Chunk, 0, len(rows))
|
||||
foundSet := make(map[string]bool)
|
||||
appendAllowed := func(c *types.Chunk) {
|
||||
if c == nil || foundSet[c.ID] {
|
||||
return
|
||||
}
|
||||
allowed, err := permissions.Check(c.KnowledgeBaseID, c.TenantID, types.OrgRoleViewer)
|
||||
if err == nil && allowed {
|
||||
chunks = append(chunks, c)
|
||||
foundSet[c.ID] = true
|
||||
}
|
||||
}
|
||||
for _, c := range rows {
|
||||
appendAllowed(c)
|
||||
if c != nil {
|
||||
foundSet[c.ID] = true
|
||||
}
|
||||
}
|
||||
|
||||
missing := s.findMissingIDs(chunkIDs, func(id string) bool {
|
||||
return foundSet[id]
|
||||
})
|
||||
missing := s.findMissingIDs(chunkIDs, func(id string) bool { return foundSet[id] })
|
||||
if len(missing) == 0 {
|
||||
return chunks, nil
|
||||
}
|
||||
logger.Infof(ctx, "[listChunksByIDWithShared] %d chunks not found in current tenant, attempting shared KB lookup", len(missing))
|
||||
|
||||
userID, ok := s.extractUserID(ctx)
|
||||
if !ok {
|
||||
logger.Warnf(ctx, "[listChunksByIDWithShared] userID not found or empty in context, skipping shared KB lookup")
|
||||
return chunks, nil
|
||||
}
|
||||
|
||||
logger.Infof(ctx, "[listChunksByIDWithShared] Looking up %d missing chunks with userID=%s", len(missing), userID)
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
crossChunks, err := s.chunkRepo.ListChunksByIDOnly(ctx, missing)
|
||||
if err != nil {
|
||||
logger.Warnf(ctx, "[listChunksByIDWithShared] Failed to fetch chunks by ID only: %v", err)
|
||||
return chunks, nil
|
||||
return nil, err
|
||||
}
|
||||
logger.Infof(ctx, "[listChunksByIDWithShared] Found %d chunks without tenant filter", len(crossChunks))
|
||||
|
||||
for _, c := range crossChunks {
|
||||
if c == nil || c.KnowledgeBaseID == "" {
|
||||
continue
|
||||
}
|
||||
hasPermission, err := s.kbShareService.HasTenantKBPermission(ctx, c.KnowledgeBaseID, tenantID, callerTenantRole, types.OrgRoleViewer)
|
||||
if err != nil {
|
||||
logger.Debugf(ctx, "[listChunksByIDWithShared] Permission check error for KB %s: %v", c.KnowledgeBaseID, err)
|
||||
continue
|
||||
}
|
||||
if !hasPermission {
|
||||
logger.Debugf(ctx, "[listChunksByIDWithShared] No permission for KB %s", c.KnowledgeBaseID)
|
||||
continue
|
||||
}
|
||||
chunks = append(chunks, c)
|
||||
appendAllowed(c)
|
||||
}
|
||||
|
||||
logger.Infof(ctx, "[listChunksByIDWithShared] After shared lookup, total chunks: %d", len(chunks))
|
||||
return chunks, nil
|
||||
}
|
||||
|
||||
|
|
@ -149,16 +124,3 @@ func (s *knowledgeBaseService) findMissingIDs(ids []string, exists func(string)
|
|||
}
|
||||
return missing
|
||||
}
|
||||
|
||||
// extractUserID extracts the user ID from context, returning ("", false) if not found.
|
||||
func (s *knowledgeBaseService) extractUserID(ctx context.Context) (string, bool) {
|
||||
userIDVal := ctx.Value(types.UserIDContextKey)
|
||||
if userIDVal == nil {
|
||||
return "", false
|
||||
}
|
||||
userID, ok := userIDVal.(string)
|
||||
if !ok || userID == "" {
|
||||
return "", false
|
||||
}
|
||||
return userID, true
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/application/service/retriever"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/logger"
|
||||
|
|
@ -184,9 +185,9 @@ func classifyFactoryError(
|
|||
}
|
||||
|
||||
// authorizeKBAccess rejects multi-KB searches whose scope includes a KB
|
||||
// that the caller is not entitled to read. Same-tenant KBs always pass.
|
||||
// Foreign-tenant KBs (Organization-shared) must pass an explicit
|
||||
// tenant-scoped permission check via kbShareService.HasTenantKBPermission,
|
||||
// that the original caller is not entitled to read. Exact upstream grants
|
||||
// support shared KB/agent execution; other foreign KBs need an organization
|
||||
// permission check for the caller via access.KBPermissions,
|
||||
// applying the 3-D cap (share role + caller's tenant-org role + tenant
|
||||
// Viewer cap) introduced in Plan 3 of #1303.
|
||||
//
|
||||
|
|
@ -198,20 +199,24 @@ func classifyFactoryError(
|
|||
func (s *knowledgeBaseService) authorizeKBAccess(
|
||||
ctx context.Context,
|
||||
kbs []*types.KnowledgeBase,
|
||||
requestTenantID uint64,
|
||||
) error {
|
||||
if len(kbs) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
callerTenantRole := types.TenantRoleFromContext(ctx)
|
||||
kbIDs := make([]string, 0, len(kbs))
|
||||
for _, kb := range kbs {
|
||||
kbIDs = append(kbIDs, kb.ID)
|
||||
}
|
||||
if err := types.AuthorizeTenantAPIKeyKnowledgeBases(ctx, kbIDs...); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
requestTenantID := types.CallerFromContext(ctx).TenantID
|
||||
permissions := access.NewKBPermissions(ctx, s.kbShareService)
|
||||
|
||||
for _, kb := range kbs {
|
||||
if kb.TenantID == requestTenantID {
|
||||
continue
|
||||
}
|
||||
hasPermission, permErr := s.kbShareService.HasTenantKBPermission(
|
||||
ctx, kb.ID, requestTenantID, callerTenantRole, types.OrgRoleViewer)
|
||||
hasPermission, permErr := permissions.Check(kb.ID, kb.TenantID, types.OrgRoleViewer)
|
||||
if permErr != nil {
|
||||
logger.ErrorWithFields(ctx, permErr, map[string]interface{}{
|
||||
"caller_tenant_id": requestTenantID,
|
||||
|
|
|
|||
|
|
@ -446,7 +446,7 @@ func (s *messageService) DeleteMessageKnowledge(ctx context.Context, knowledgeID
|
|||
return
|
||||
}
|
||||
logger.Infof(ctx, "Deleting chat history knowledge entry: %s", knowledgeID)
|
||||
if err := s.knowService.DeleteKnowledge(ctx, knowledgeID); err != nil {
|
||||
if err := deleteReferencedKnowledge(ctx, s.knowService, "", []string{knowledgeID}); err != nil {
|
||||
logger.Warnf(ctx, "Failed to delete chat history knowledge %s: %v", knowledgeID, err)
|
||||
}
|
||||
}
|
||||
|
|
@ -466,7 +466,7 @@ func (s *messageService) DeleteSessionKnowledge(ctx context.Context, sessionID s
|
|||
}
|
||||
|
||||
logger.Infof(ctx, "Deleting %d chat history knowledge entries for session %s", len(knowledgeIDs), sessionID)
|
||||
if err := s.knowService.DeleteKnowledgeList(ctx, knowledgeIDs); err != nil {
|
||||
if err := deleteReferencedKnowledge(ctx, s.knowService, "", knowledgeIDs); err != nil {
|
||||
logger.Warnf(ctx, "Failed to batch delete chat history knowledge for session %s: %v", sessionID, err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -155,6 +155,47 @@ func (s *resourceCatalog) Bind(ctx context.Context, reference, ownerType, ownerI
|
|||
})
|
||||
}
|
||||
|
||||
func (s *resourceCatalog) IsReferencedByKnowledgeBase(
|
||||
ctx context.Context,
|
||||
tenantID uint64,
|
||||
kbID, reference string,
|
||||
) (bool, error) {
|
||||
physical, resource, err := s.ResolvePath(ctx, reference)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if resource == nil {
|
||||
resource, err = s.repo.GetByTenantLocation(ctx, tenantID, resourceLocationHash(physical))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
if resource == nil || resource.TenantID != tenantID {
|
||||
return false, nil
|
||||
}
|
||||
return s.repo.IsReferencedByKnowledgeBase(ctx, tenantID, kbID, resource.ID)
|
||||
}
|
||||
|
||||
// GetMessageFileBindings resolves registered aliases before reading authoritative origins.
|
||||
func (s *resourceCatalog) GetMessageFileBindings(
|
||||
ctx context.Context, tenantID uint64, reference, messageID string,
|
||||
) (*types.MessageFileBindings, error) {
|
||||
physical, resource, err := s.ResolvePath(ctx, reference)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if resource == nil {
|
||||
resource, err = s.repo.GetByTenantLocation(ctx, tenantID, resourceLocationHash(physical))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
if resource == nil || resource.TenantID != tenantID {
|
||||
return &types.MessageFileBindings{}, nil
|
||||
}
|
||||
return s.repo.GetMessageFileBindings(ctx, tenantID, resource.ID, messageID)
|
||||
}
|
||||
|
||||
// Release implements interfaces.ResourceCatalog.
|
||||
//
|
||||
// Unbinding and counting are deliberately not a single transaction. A racing
|
||||
|
|
|
|||
|
|
@ -0,0 +1,169 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type catalogFileShares struct{}
|
||||
|
||||
func (catalogFileShares) CheckTenantKBPermission(
|
||||
_ context.Context,
|
||||
kb string,
|
||||
_ uint64,
|
||||
_ types.TenantRole,
|
||||
) (types.OrgMemberRole, bool, error) {
|
||||
return types.OrgRoleViewer, kb == "shared", nil
|
||||
}
|
||||
|
||||
func (catalogFileShares) GetKnowledgeBasesByIDsOnly(context.Context, []string) ([]*types.KnowledgeBase, error) {
|
||||
return []*types.KnowledgeBase{{ID: "shared", TenantID: 7}}, nil
|
||||
}
|
||||
|
||||
type catalogFileAgent struct{ agent *types.CustomAgent }
|
||||
|
||||
func (a *catalogFileAgent) GetSharedAgentForTenant(
|
||||
context.Context,
|
||||
uint64,
|
||||
types.TenantRole,
|
||||
string,
|
||||
...uint64,
|
||||
) (*types.CustomAgent, error) {
|
||||
return a.agent, nil
|
||||
}
|
||||
|
||||
func TestCatalogFileAuthorizationRejectsPrivateHandlesInSharedText(t *testing.T) {
|
||||
catalog, db := newResourceCatalogForTest(t)
|
||||
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
|
||||
ctx := newSharedAccessContext()
|
||||
const physical = "local://7/exports/private.png"
|
||||
ref, err := catalog.Register(ctx, 7, physical, interfaces.ResourceRegistration{})
|
||||
require.NoError(t, err)
|
||||
for _, kb := range []string{"private", "shared"} {
|
||||
require.NoError(t, db.Create(&types.KnowledgeBase{ID: kb, TenantID: 7}).Error)
|
||||
require.NoError(
|
||||
t,
|
||||
db.Create(&types.Knowledge{ID: kb + "-doc", TenantID: 7, KnowledgeBaseID: kb, Type: "file"}).Error,
|
||||
)
|
||||
}
|
||||
require.NoError(
|
||||
t,
|
||||
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "private-doc", types.ResourceRelationSourceFile),
|
||||
)
|
||||
text := " "
|
||||
require.NoError(
|
||||
t,
|
||||
db.Create(
|
||||
&types.Chunk{ID: "chunk", TenantID: 7, KnowledgeBaseID: "shared", KnowledgeID: "shared-doc", Content: text},
|
||||
).Error,
|
||||
)
|
||||
require.NoError(
|
||||
t,
|
||||
db.Create(&types.WikiPage{ID: "wiki", TenantID: 7, KnowledgeBaseID: "shared", Content: text}).Error,
|
||||
)
|
||||
grant := &access.KBAccess{
|
||||
KnowledgeBase: &types.KnowledgeBase{ID: "shared", TenantID: 7},
|
||||
Caller: types.CallerFromContext(ctx),
|
||||
EffectiveTenantID: 7,
|
||||
Permission: types.OrgRoleViewer,
|
||||
}
|
||||
lookup := catalog.(interfaces.KBResourceLookup)
|
||||
message := &types.Message{
|
||||
ID: "message",
|
||||
AgentTenantID: 1,
|
||||
Content: text,
|
||||
KnowledgeReferences: []*types.SearchResult{{KnowledgeBaseID: "shared", Content: text}},
|
||||
}
|
||||
shares := access.MessageKBShareAuthorizer{ShareGuard: catalogFileShares{}, KBs: catalogFileShares{}}
|
||||
check := func(allowed bool) {
|
||||
t.Helper()
|
||||
for _, path := range []string{ref, physical} {
|
||||
_, err := access.ResolveKBFile(ctx, grant, "shared", path, catalog, lookup)
|
||||
if allowed {
|
||||
require.NoError(t, err)
|
||||
} else {
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
}
|
||||
_, err = access.AuthorizeMessageFile(ctx, message, path, nil, catalog, shares)
|
||||
if allowed && path == ref {
|
||||
require.NoError(t, err)
|
||||
} else {
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
check(false)
|
||||
require.NoError(
|
||||
t,
|
||||
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "shared-doc", types.ResourceRelationSourceFile),
|
||||
)
|
||||
check(true)
|
||||
require.NoError(t, db.Where("id = ?", "shared").Delete(&types.KnowledgeBase{}).Error)
|
||||
check(false)
|
||||
}
|
||||
|
||||
func TestCatalogSharedAgentFileRequiresCurrentSelectionOrExactArtifactBinding(t *testing.T) {
|
||||
catalog, db := newResourceCatalogForTest(t)
|
||||
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}))
|
||||
ctx := newSharedAccessContext()
|
||||
const physical = "local://7/exports/source.pdf"
|
||||
ref, err := catalog.Register(ctx, 7, physical, interfaces.ResourceRegistration{})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, db.Create(&types.KnowledgeBase{ID: "private", TenantID: 7}).Error)
|
||||
doc := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "private", Type: "file"}
|
||||
require.NoError(t, db.Create(doc).Error)
|
||||
require.NoError(t, catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "doc", types.ResourceRelationSourceFile))
|
||||
agents := &catalogFileAgent{agent: &types.CustomAgent{ID: "agent", TenantID: 7}}
|
||||
message := &types.Message{
|
||||
ID: "message",
|
||||
AgentID: "agent",
|
||||
AgentTenantID: 7,
|
||||
Content: ref + " " + physical,
|
||||
Artifacts: types.MessageArtifacts{{URL: ref}},
|
||||
}
|
||||
check := func(allowed bool) {
|
||||
t.Helper()
|
||||
for _, path := range []string{ref, physical} {
|
||||
_, err := access.AuthorizeMessageFile(
|
||||
ctx,
|
||||
message,
|
||||
path,
|
||||
agents,
|
||||
catalog,
|
||||
access.MessageKBShareAuthorizer{},
|
||||
)
|
||||
if allowed {
|
||||
require.NoError(t, err)
|
||||
} else {
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, mode := range []string{"none", "", "unknown", "selected", "all"} {
|
||||
agents.agent.Config = types.CustomAgentConfig{KBSelectionMode: mode, KnowledgeBases: []string{"private"}}
|
||||
check(mode == "selected" || mode == "all")
|
||||
}
|
||||
agents.agent.Config = types.CustomAgentConfig{KBSelectionMode: "selected", KnowledgeBases: []string{"other"}}
|
||||
check(false)
|
||||
agents.agent.Config.KnowledgeBases = []string{"private"}
|
||||
scoped := types.WithTenantAPIKeyScope(ctx, types.TenantAPIKeyScope{KnowledgeBaseIDs: types.StringArray{"other"}})
|
||||
_, err = access.AuthorizeMessageFile(scoped, message, ref, agents, catalog, access.MessageKBShareAuthorizer{})
|
||||
require.ErrorIs(t, err, access.ErrForbidden)
|
||||
require.NoError(t, db.Delete(doc).Error)
|
||||
check(false)
|
||||
agents.agent.Config.KBSelectionMode = "none"
|
||||
require.NoError(
|
||||
t,
|
||||
catalog.Bind(ctx, ref, types.ResourceOwnerMessage, "another-message", types.ResourceRelationArtifact),
|
||||
)
|
||||
check(false)
|
||||
require.NoError(t, catalog.Bind(ctx, ref, types.ResourceOwnerMessage, "message", types.ResourceRelationArtifact))
|
||||
check(true)
|
||||
agents.agent = nil
|
||||
check(false)
|
||||
}
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestKBFileBindingRequiresLiveKnowledgeInExactKB(t *testing.T) {
|
||||
catalog, db := newResourceCatalogForTest(t)
|
||||
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
|
||||
ctx := context.Background()
|
||||
ref, err := catalog.Register(ctx, 7, "local://7/exports/image.png", interfaces.ResourceRegistration{})
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, db.Create(&types.KnowledgeBase{ID: "kb", TenantID: 7}).Error)
|
||||
knowledge := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "kb", Type: "file"}
|
||||
require.NoError(t, db.Create(knowledge).Error)
|
||||
require.NoError(
|
||||
t,
|
||||
catalog.Bind(ctx, ref, types.ResourceOwnerKnowledge, "doc", types.ResourceRelationExtractedImage),
|
||||
)
|
||||
lookup := catalog.(interfaces.KBResourceLookup)
|
||||
for _, reference := range []string{ref, "local://7/exports/image.png"} {
|
||||
ok, err := lookup.IsReferencedByKnowledgeBase(ctx, 7, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.True(t, ok)
|
||||
ok, err = lookup.IsReferencedByKnowledgeBase(ctx, 7, "other-kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok)
|
||||
}
|
||||
require.NoError(t, db.Delete(knowledge).Error)
|
||||
ok, err := lookup.IsReferencedByKnowledgeBase(ctx, 7, "kb", ref)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok, "a stale binding cannot authorize a deleted document")
|
||||
}
|
||||
|
||||
func TestKBFileLegacyTextReferencesDoNotAuthorizeFiles(t *testing.T) {
|
||||
catalog, db := newResourceCatalogForTest(t)
|
||||
require.NoError(t, db.AutoMigrate(&types.KnowledgeBase{}, &types.Knowledge{}, &types.Chunk{}, &types.WikiPage{}))
|
||||
lookup := catalog.(interfaces.KBResourceLookup)
|
||||
const reference = "local://7/exports/image.png"
|
||||
chunk := &types.Chunk{
|
||||
ID: "chunk",
|
||||
TenantID: 7,
|
||||
KnowledgeBaseID: "kb",
|
||||
Content: "",
|
||||
}
|
||||
require.NoError(t, db.Create(chunk).Error)
|
||||
ok, err := lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok)
|
||||
require.NoError(t, db.Model(chunk).Update("image_info", `[{"url":"`+reference+`"}]`).Error)
|
||||
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok, "text is not an ownership binding")
|
||||
require.NoError(t, db.Delete(chunk).Error)
|
||||
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok)
|
||||
page := &types.WikiPage{ID: "page", TenantID: 7, KnowledgeBaseID: "kb", Content: ""}
|
||||
require.NoError(t, db.Create(page).Error)
|
||||
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 7, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok, "text is not an ownership binding")
|
||||
ok, err = lookup.IsReferencedByKnowledgeBase(context.Background(), 8, "kb", reference)
|
||||
require.NoError(t, err)
|
||||
require.False(t, ok)
|
||||
}
|
||||
|
|
@ -0,0 +1,284 @@
|
|||
package service
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/application/access"
|
||||
apperrors "github.com/Tencent/WeKnora/internal/errors"
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/Tencent/WeKnora/internal/types/interfaces"
|
||||
"github.com/hibiken/asynq"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestReparseRejectsMovingDocumentBeforeSideEffects(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(t, setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}))
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
f.repo.writes = 0
|
||||
_, err = f.svc.ReparseKnowledge(f.ctx, "doc", nil)
|
||||
var appErr *apperrors.AppError
|
||||
require.ErrorAs(t, err, &appErr)
|
||||
require.Equal(t, 409, appErr.HTTPCode)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
require.Zero(t, f.graph.calls)
|
||||
}
|
||||
|
||||
func TestReparseRequiresExplicitWriteGrant(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
ctx := types.WithExecutionTenant(context.Background(), 7)
|
||||
_, err := f.svc.ReparseKnowledge(ctx, "doc", nil)
|
||||
require.Error(t, err)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
}
|
||||
|
||||
func TestReparseTaskPinsOriginalKBBeforeAnySubmission(t *testing.T) {
|
||||
for _, scenario := range []string{"moved", "missing", "mixed legacy", "blank ID", "moving"} {
|
||||
t.Run(scenario, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
payload := types.KnowledgeListReparsePayload{
|
||||
TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc"},
|
||||
}
|
||||
switch scenario {
|
||||
case "moved":
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error,
|
||||
)
|
||||
case "missing":
|
||||
payload.KnowledgeIDs = append(payload.KnowledgeIDs, "missing")
|
||||
case "mixed legacy":
|
||||
payload.KnowledgeBaseID = ""
|
||||
payload.KnowledgeIDs = append(payload.KnowledgeIDs, "other-doc")
|
||||
case "blank ID":
|
||||
payload.KnowledgeIDs = append(payload.KnowledgeIDs, " ")
|
||||
case "moving":
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
setTransferState(row, knowledgeTransferState{Operation: access.KBTransferMove, Phase: "moving"}),
|
||||
)
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
}
|
||||
f.repo.writes = 0
|
||||
raw, err := json.Marshal(payload)
|
||||
require.NoError(t, err)
|
||||
err = f.svc.ProcessKnowledgeListReparse(
|
||||
context.Background(), asynq.NewTask(types.TypeKnowledgeListReparse, raw),
|
||||
)
|
||||
require.ErrorIs(t, err, asynq.SkipRetry)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReparseTaskGrantCannotFollowMoveAfterPreflight(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
ctx, ids, err := f.svc.reparseTaskScope(context.Background(), types.KnowledgeListReparsePayload{
|
||||
TenantID: 7, KnowledgeBaseID: "kb", KnowledgeIDs: []string{"doc", "doc"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, []string{"doc"}, ids)
|
||||
require.NoError(t, access.RequireKBWrite(ctx, f.kbs.values["kb"]))
|
||||
require.Error(t, access.RequireKBWrite(ctx, f.kbs.values["other"]))
|
||||
require.NoError(t, f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("knowledge_base_id", "other").Error)
|
||||
_, err = f.svc.ReparseKnowledge(ctx, "doc", nil)
|
||||
require.Error(t, err)
|
||||
require.Zero(t, f.repo.writes)
|
||||
require.Zero(t, f.chunkRepo.writes)
|
||||
}
|
||||
|
||||
type sharedChunkFailureRepo struct {
|
||||
interfaces.ChunkRepository
|
||||
failure error
|
||||
}
|
||||
|
||||
func (r sharedChunkFailureRepo) ListChunksByID(context.Context, uint64, []string) ([]*types.Chunk, error) {
|
||||
return []*types.Chunk{{ID: "own", TenantID: 1, KnowledgeBaseID: "own"}}, nil
|
||||
}
|
||||
|
||||
func (r sharedChunkFailureRepo) ListChunksByIDOnly(context.Context, []string) ([]*types.Chunk, error) {
|
||||
return nil, r.failure
|
||||
}
|
||||
|
||||
func TestSharedChunkReadPropagatesStorageFailure(t *testing.T) {
|
||||
failure := errors.New("shared chunk storage unavailable")
|
||||
svc := &knowledgeBaseService{chunkRepo: sharedChunkFailureRepo{failure: failure}}
|
||||
rows, err := svc.listChunksByIDWithShared(newSharedAccessContext(), 1, []string{"own", "shared"})
|
||||
require.ErrorIs(t, err, failure)
|
||||
require.Nil(t, rows)
|
||||
}
|
||||
|
||||
func TestFAQCloneDoesNotInheritTransferState(t *testing.T) {
|
||||
f := transferFixture(t, access.KBTransferClone)
|
||||
kb := &types.KnowledgeBase{ID: "new-faq", TenantID: 7}
|
||||
src := &types.Knowledge{
|
||||
Metadata: types.JSON(`{"custom":{"value":3},"_knowledge_transfer":{"operation":"move","phase":"moving"}}`),
|
||||
}
|
||||
before := string(src.Metadata)
|
||||
dst, err := f.svc.getOrCreateFAQKnowledge(f.ctx, kb, src)
|
||||
require.NoError(t, err)
|
||||
require.JSONEq(t, `{"custom":{"value":3}}`, string(dst.Metadata))
|
||||
require.Equal(t, before, string(src.Metadata))
|
||||
require.NoError(t, access.RejectMovingKnowledge(dst))
|
||||
}
|
||||
|
||||
type partialCloneEngine struct {
|
||||
parentChildRetrieveEngine
|
||||
vectors map[string]int
|
||||
cleanupFailure bool
|
||||
}
|
||||
|
||||
func (e *partialCloneEngine) Support() []types.RetrieverType {
|
||||
return []types.RetrieverType{types.VectorRetrieverType, types.KeywordsRetrieverType}
|
||||
}
|
||||
|
||||
func (e *partialCloneEngine) CopyIndices(
|
||||
_ context.Context, _ string, documents, _ map[string]string, _ string, _ int, _ string,
|
||||
) error {
|
||||
for _, id := range documents {
|
||||
e.vectors[id] = 1
|
||||
}
|
||||
return errors.New("partial vector copy")
|
||||
}
|
||||
|
||||
func (e *partialCloneEngine) DeleteByKnowledgeIDList(_ context.Context, ids []string, _ int, _ string) error {
|
||||
if e.cleanupFailure {
|
||||
return errors.New("vector cleanup unavailable")
|
||||
}
|
||||
for _, id := range ids {
|
||||
delete(e.vectors, id)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestClonePartialIndexFailureCleansOnlyItsDestination(t *testing.T) {
|
||||
for _, cleanupFailure := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "cleaned", true: "cleanup fails visibly"}[cleanupFailure], func(t *testing.T) {
|
||||
t.Setenv("RETRIEVE_DRIVER", "postgres")
|
||||
f := transferFixture(t, access.KBTransferClone)
|
||||
engine := &partialCloneEngine{vectors: map[string]int{"doc": 1}, cleanupFailure: cleanupFailure}
|
||||
f.svc.retrieveEngine = parentChildRetrieveRegistry{engine: engine}
|
||||
f.svc.modelService = parentChildModelService{embedder: parentChildEmbedder{}}
|
||||
f.kbs.values["other"].EmbeddingModelID = "model"
|
||||
src, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
err = f.svc.cloneKnowledge(f.ctx, src, f.kbs.values["other"])
|
||||
require.ErrorContains(t, err, "partial vector copy")
|
||||
rows, err := f.repo.ListKnowledgeByKnowledgeBaseID(f.ctx, 7, "other")
|
||||
require.NoError(t, err)
|
||||
var dst *types.Knowledge
|
||||
for _, row := range rows {
|
||||
if row.ID != "other-doc" {
|
||||
dst = row
|
||||
}
|
||||
}
|
||||
require.NotNil(t, dst)
|
||||
require.Equal(t, types.ParseStatusFailed, dst.ParseStatus)
|
||||
chunks, err := f.chunkRepo.ListAllChunksByKnowledgeID(f.ctx, 7, dst.ID)
|
||||
require.NoError(t, err)
|
||||
if cleanupFailure {
|
||||
require.Len(t, chunks, 1, "retain image references when index cleanup is uncertain")
|
||||
require.Equal(t, 1, engine.vectors[dst.ID])
|
||||
require.Contains(t, dst.ErrorMessage, "vector cleanup unavailable")
|
||||
} else {
|
||||
require.Empty(t, chunks)
|
||||
require.Zero(t, engine.vectors[dst.ID])
|
||||
}
|
||||
require.Equal(t, 1, engine.vectors["doc"])
|
||||
var tenant types.Tenant
|
||||
require.NoError(t, f.db.First(&tenant, 7).Error)
|
||||
require.EqualValues(t, 5, tenant.StorageUsed, "failed clones must not change storage accounting")
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestReparseWorkerSubmitsPinnedAndLegacySingleKBTasks(t *testing.T) {
|
||||
for _, kbID := range []string{"kb", ""} {
|
||||
t.Run("scope="+kbID, func(t *testing.T) {
|
||||
f := newDocumentWriteFixture(t)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
row.SetManualMetadata(
|
||||
types.NewManualKnowledgeMetadata("# content", types.ManualKnowledgeStatusPublish, 1),
|
||||
),
|
||||
)
|
||||
require.NoError(t, f.repo.UpdateKnowledge(f.ctx, row))
|
||||
f.svc.task = &transferQueue{}
|
||||
raw, err := json.Marshal(types.KnowledgeListReparsePayload{
|
||||
TenantID: 7, KnowledgeBaseID: kbID, KnowledgeIDs: []string{"doc"},
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NoError(
|
||||
t,
|
||||
f.svc.ProcessKnowledgeListReparse(
|
||||
context.Background(), asynq.NewTask(types.TypeKnowledgeListReparse, raw),
|
||||
),
|
||||
)
|
||||
row, err = f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, types.ParseStatusPending, row.ParseStatus)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMovedReparseTransferPhaseDoesNotBlockItsParser(t *testing.T) {
|
||||
row := &types.Knowledge{ID: "doc", TenantID: 7, KnowledgeBaseID: "target", ParseStatus: types.ParseStatusPending}
|
||||
for _, phase := range []string{"reparse_pending", "done"} {
|
||||
require.NoError(
|
||||
t,
|
||||
setTransferState(row, knowledgeTransferState{
|
||||
Operation: access.KBTransferMove, Mode: "reparse", Phase: phase,
|
||||
}),
|
||||
)
|
||||
require.NoError(t, validateProcessingKnowledge(row, 7, "target", "doc"))
|
||||
require.ErrorIs(t, validateProcessingKnowledge(row, 7, "source", "doc"), asynq.SkipRetry)
|
||||
require.Equal(t, types.ParseStatusPending, row.ParseStatus, "transfer state is independent of parse completion")
|
||||
}
|
||||
}
|
||||
|
||||
type orphanMoveEngine struct {
|
||||
partialCloneEngine
|
||||
calls int
|
||||
chunks []string
|
||||
}
|
||||
|
||||
func (e *orphanMoveEngine) MoveKnowledgeIndices(
|
||||
_ context.Context, _, _, _ string, chunks []string, _ int, _ string,
|
||||
) error {
|
||||
e.calls++
|
||||
e.chunks = append([]string(nil), chunks...)
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestMoveStillVisitsVectorBackendWithoutDatabaseChunks(t *testing.T) {
|
||||
t.Setenv("RETRIEVE_DRIVER", "postgres")
|
||||
f := transferFixture(t, access.KBTransferMove)
|
||||
engine := &orphanMoveEngine{}
|
||||
f.svc.retrieveEngine = parentChildRetrieveRegistry{engine: engine}
|
||||
f.svc.modelService = parentChildModelService{embedder: parentChildEmbedder{}}
|
||||
f.kbs.values["kb"].EmbeddingModelID = "model"
|
||||
f.kbs.values["other"].EmbeddingModelID = "model"
|
||||
require.NoError(
|
||||
t,
|
||||
f.db.Model(&types.Knowledge{}).Where("id = ?", "doc").Update("embedding_model_id", "model").Error,
|
||||
)
|
||||
require.NoError(t, f.db.Where("knowledge_id = ?", "doc").Delete(&types.Chunk{}).Error)
|
||||
require.NoError(t, f.svc.ProcessKnowledgeMove(context.Background(), moveTask(t, []string{"doc"}, "reuse_vectors")))
|
||||
require.Equal(t, 1, engine.calls)
|
||||
require.Empty(t, engine.chunks)
|
||||
row, err := f.repo.GetKnowledgeByID(f.ctx, 7, "doc")
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, "other", row.KnowledgeBaseID)
|
||||
}
|
||||
|
|
@ -312,3 +312,42 @@ func (c *CompositeRetrieveEngine) EstimateStorageSize(ctx context.Context,
|
|||
}
|
||||
return sum.Load()
|
||||
}
|
||||
|
||||
// ValidateKnowledgeIndexMove checks all stores before the first mutation.
|
||||
func (c *CompositeRetrieveEngine) ValidateKnowledgeIndexMove(ctx context.Context) error {
|
||||
for _, info := range c.engineInfos {
|
||||
if _, ok := info.retrieveEngine.(interfaces.KnowledgeIndexMover); !ok {
|
||||
return fmt.Errorf("retriever %s does not support moving indices", info.retrieveEngine.EngineType())
|
||||
}
|
||||
if validator, ok := info.retrieveEngine.(interface{ ValidateKnowledgeIndexMove(context.Context) error }); ok {
|
||||
if err := validator.ValidateKnowledgeIndexMove(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// MoveKnowledgeIndices updates each validated store with retry-safe operations.
|
||||
func (c *CompositeRetrieveEngine) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
chunkIDs []string,
|
||||
dimension int,
|
||||
knowledgeType string,
|
||||
) error {
|
||||
if err := c.ValidateKnowledgeIndexMove(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
return c.concurrentExecWithError(ctx, func(ctx context.Context, info *engineInfo) error {
|
||||
return info.retrieveEngine.(interfaces.KnowledgeIndexMover).MoveKnowledgeIndices(
|
||||
ctx,
|
||||
sourceKB,
|
||||
targetKB,
|
||||
knowledgeID,
|
||||
chunkIDs,
|
||||
dimension,
|
||||
knowledgeType,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package retriever
|
|||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
|
@ -346,3 +347,37 @@ func (v *KeywordsVectorHybridRetrieveEngineService) BatchUpdateChunkTagID(
|
|||
) error {
|
||||
return v.indexRepository.BatchUpdateChunkTagID(ctx, chunkTagMap)
|
||||
}
|
||||
|
||||
// ValidateKnowledgeIndexMove checks backend support before any mutation.
|
||||
func (v *KeywordsVectorHybridRetrieveEngineService) ValidateKnowledgeIndexMove(ctx context.Context) error {
|
||||
if _, ok := v.indexRepository.(interfaces.KnowledgeIndexMover); !ok {
|
||||
return fmt.Errorf("retriever %s does not support moving indices", v.EngineType())
|
||||
}
|
||||
if validator, ok := v.indexRepository.(interface{ ValidateKnowledgeIndexMove(context.Context) error }); ok {
|
||||
return validator.ValidateKnowledgeIndexMove(ctx)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// MoveKnowledgeIndices delegates metadata relocation to the supported backend.
|
||||
func (v *KeywordsVectorHybridRetrieveEngineService) MoveKnowledgeIndices(
|
||||
ctx context.Context,
|
||||
sourceKB, targetKB, knowledgeID string,
|
||||
chunkIDs []string,
|
||||
dimension int,
|
||||
knowledgeType string,
|
||||
) error {
|
||||
if err := v.ValidateKnowledgeIndexMove(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
return v.indexRepository.(interfaces.KnowledgeIndexMover).MoveKnowledgeIndices(
|
||||
ctx,
|
||||
sourceKB,
|
||||
targetKB,
|
||||
knowledgeID,
|
||||
chunkIDs,
|
||||
dimension,
|
||||
knowledgeType,
|
||||
)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,38 @@
|
|||
package retriever
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/types"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type moveTestEngine struct {
|
||||
fakeEngine
|
||||
calls int
|
||||
}
|
||||
|
||||
func (e *moveTestEngine) MoveKnowledgeIndices(context.Context, string, string, string, []string, int, string) error {
|
||||
e.calls++
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestCompositeMoveChecksAllEnginesBeforeMutation(t *testing.T) {
|
||||
supported := &moveTestEngine{fakeEngine: fakeEngine{engineType: types.PostgresRetrieverEngineType}}
|
||||
unsupported := &fakeEngine{engineType: types.ElasticsearchRetrieverEngineType}
|
||||
c := &CompositeRetrieveEngine{
|
||||
engineInfos: []*engineInfo{{retrieveEngine: supported}, {retrieveEngine: unsupported}},
|
||||
}
|
||||
require.Error(
|
||||
t,
|
||||
c.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
|
||||
)
|
||||
require.Zero(t, supported.calls)
|
||||
c.engineInfos = c.engineInfos[:1]
|
||||
require.NoError(
|
||||
t,
|
||||
c.MoveKnowledgeIndices(context.Background(), "source", "target", "doc", []string{"chunk"}, 2, "document"),
|
||||
)
|
||||
require.Equal(t, 1, supported.calls)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue