16 KiB
16 KiB
Knowledge Base Processing Flow - Detailed with Line Numbers
Complete Document Ingestion Flow
┌──────────────────────────────────────────────────────────────────────────────┐
│ PHASE 1: DOCUMENT UPLOAD & CONVERSION │
└──────────────────────────────────────────────────────────────────────────────┘
File: internal/application/service/knowledge.go
Function: ProcessDocument(ctx, *asynq.Task) error
Lines: 7732-8108+
STEP 1: Parse Task Payload (7733-7737)
├─ Unmarshal task.Payload() → types.DocumentProcessPayload
├─ Extract: KnowledgeID, KnowledgeBaseID, FilePath, FileURL, FileType
└─ Log context setup
STEP 2: Idempotency Checks (7761-7801)
├─ Line 7762: Get Knowledge by ID
│ └─ knowledgeService.repo.GetKnowledgeByID(ctx, tenantID, knowledgeID)
├─ Line 7773: Check if already deleted
│ └─ if knowledge.ParseStatus == types.ParseStatusDeleting → return nil
├─ Line 7779: Check if already completed
│ └─ if knowledge.ParseStatus == types.ParseStatusCompleted → return nil
└─ Line 7814: Update status to "processing"
└─ knowledge.ParseStatus = "processing"
STEP 3: Fetch Knowledge Base Config (7803-7812)
├─ Line 7804: Get KB by ID
│ └─ knowledgeService.kbService.GetKnowledgeBaseByID(ctx, kbID)
├─ Validate: EmbeddingModelID must be configured
└─ Store KB config in context
STEP 4: Validate Multimodal Config (7821-7841)
├─ Line 7822: Check if image file without multimodal enabled
│ └─ IsImageType(payload.FileType) && !payload.EnableMultimodel
├─ Line 7833: Check if audio file without ASR enabled
│ └─ IsAudioType(payload.FileType) && !kb.ASRConfig.IsASREnabled()
└─ Line 7844: Reject video files (not supported)
└─ IsVideoType(payload.FileType) → fail
STEP 5: Handle File Input (7858-7931)
├─ If FileURL provided (7858-7921):
│ ├─ Line 7860: Validate URL for SSRF protection
│ │ └─ secutils.ValidateURLForSSRF(payload.FileURL)
│ ├─ Line 7871: Download file from URL
│ │ └─ downloadFileFromURL(ctx, url, &fileName, &fileType)
│ ├─ Line 7900: Save downloaded file
│ │ └─ fileSvc.SaveBytes(ctx, contentBytes, tenantID, fileName, true)
│ └─ Line 7915: Delegate to convert()
│
├─ Else if URL string provided (7922-7931):
│ └─ Call convert() directly
│
└─ Call convert(ctx, payload, kb, knowledge, isLastRetry)
└─ Returns: types.ReadResult + error
STEP 6: Parse Document to Chunks (after convert())
├─ convertResult contains: text chunks, images, OCR, etc.
├─ Validate conversion success
└─ Proceed to chunk processing
┌──────────────────────────────────────────────────────────────────────────────┐
│ PHASE 2: CHUNK CREATION & INDEXING │
└──────────────────────────────────────────────────────────────────────────────┘
File: internal/application/service/knowledge.go
Function: processChunks(ctx, kb, knowledge, chunks, opts)
Lines: 1681-2074
STEP 1: Setup (1681-1714)
├─ Line 1691: Create span for tracing
└─ Line 1708: Get embedding model
└─ embeddingModel = modelService.GetEmbeddingModel(ctx, kb.EmbeddingModelID)
STEP 2: Idempotency Cleanup (1716-1744)
├─ Line 1720: Delete old chunks
│ └─ chunkService.DeleteChunksByKnowledgeID(ctx, knowledge.ID)
├─ Line 1729: Delete old index data
│ └─ retrieveEngine.DeleteByKnowledgeIDList(ctx, [knowledgeID], ...)
└─ Line 1739: Delete old graph data
└─ graphEngine.DelGraph(ctx, [namespace])
STEP 3: Setup Parent-Child Chunking (if enabled) (1810-1876)
├─ Check: options.ParentChunks (1811)
├─ For each parent chunk (1815-1830):
│ └─ Create types.Chunk with ChunkType = types.ChunkTypeParentText
│ └─ Parent chunks NOT set up for indexing
│
├─ For each text chunk (1849-1877):
│ ├─ Create types.Chunk with ChunkType = types.ChunkTypeText
│ └─ Wire ParentChunkID if parent-child mode enabled
│ └─ textChunk.ParentChunkID = parentDBChunks[...].ID
│
└─ Result: insertChunks array contains parent + child chunks
STEP 4: Create Index Information (1908-1928)
├─ Line 1913: Get title prefix for semantic alignment
│ └─ titlePrefix = knowledge.Title + "\n"
│
├─ For each TEXT chunk (1917-1928):
│ ├─ indexContent = titlePrefix + chunk.Content
│ └─ Create IndexInfo:
│ ├─ Content: titlePrefix + chunk text
│ ├─ SourceID: chunk.ID
│ ├─ ChunkID: chunk.ID
│ ├─ KnowledgeID: knowledge.ID
│ └─ IsEnabled: true
│
└─ Result: indexInfoList (for indexing ONLY child/text chunks)
STEP 5: Storage Quota Check (1932-1955)
├─ Line 1934: Estimate storage size
│ └─ totalStorageSize = retrieveEngine.EstimateStorageSize(...)
│
└─ Line 1947: Check tenant quota
└─ if tenantInfo.StorageUsed + totalStorageSize > quota → fail
STEP 6: Save Chunks to Database (1965-1973)
├─ Line 1966: Insert all chunks (parent + text)
│ └─ chunkService.CreateChunks(ctx, insertChunks)
│
└─ Result: Chunks stored but not yet indexed
STEP 7: **CRITICAL - BATCH INDEXING** (1986-2008)
├─ Line 1987: ★ MAIN INDEXING CALL ★
│ └─ retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList)
│
├─ Key: indexInfoList contains ONLY text chunks (not parent chunks)
│
├─ BatchIndex() internally:
│ ├─ Gets tenant config: tenantInfo.GetEffectiveEngines()
│ │ └─ Returns: []RetrieverEngineParams
│ │ Each param: {RetrieverEngineType, RetrieverType}
│ │
│ ├─ For EACH configured retriever type:
│ │ ├─ If "vector": creates embeddings, stores in vector DB
│ │ │ └─ Uses embeddingModel.GetDimensions()
│ │ │
│ │ └─ If "keywords": indexes full-text/keywords
│ │ └─ Uses Elasticsearch, Postgres FTS, etc.
│ │
│ └─ Result: Content indexed in BOTH vector + keyword if configured
│
└─ Line 1988-2007: Error handling & cleanup if failed
STEP 8: Enqueue Post-Processing (2044-2066)
├─ Check: EnableMultimodel + StoredImages (2045-2046)
│
├─ If multimodal tasks pending (2045-2046):
│ └─ enqueueImageMultimodalTasks(ctx, knowledge, kb, images, ...)
│ └─ Chunks will be processed after multimodal tasks complete
│ └─ Those tasks will enqueue TypeKnowledgePostProcess when done
│
├─ Else (no multimodal):
│ ├─ Line 2050: Create KnowledgePostProcessPayload
│ ├─ Line 2057: Enqueue TypeKnowledgePostProcess task
│ │ └─ task.Enqueue(types.TypeKnowledgePostProcess, payload)
│ └─ This triggers Phase 3 immediately (no multimodal processing)
│
└─ Result: Next phase scheduled
STEP 9: Update Storage (2068-2073)
├─ Line 2069: Update tenant storage usage
│ └─ tenantInfo.StorageUsed += totalStorageSize
│
└─ Line 2070: Persist storage adjustment
└─ tenantRepo.AdjustStorageUsed(ctx, tenantID, size)
┌──────────────────────────────────────────────────────────────────────────────┐
│ PHASE 3: POST-PROCESSING ORCHESTRATION │
└──────────────────────────────────────────────────────────────────────────────┘
File: internal/application/service/knowledge_post_process.go
Type: KnowledgePostProcessService
Function: Handle(ctx, *asynq.Task) error
Lines: 42-128
STEP 1: Deserialize Input (44-54)
├─ Line 44: Unmarshal types.KnowledgePostProcessPayload
├─ Extract: TenantID, KnowledgeBaseID, KnowledgeID, Language
└─ Line 51-54: Setup context
STEP 2: Fetch Knowledge & KB (57-69)
├─ Line 57: Get knowledge
│ └─ knowledgeRepo.GetKnowledgeByIDOnly(ctx, knowledgeID)
│
└─ Line 66: Get knowledge base
└─ kbService.GetKnowledgeBaseByIDOnly(ctx, kbID)
STEP 3: Fetch Chunks (71-83)
├─ Line 72: Get all chunks for knowledge
│ └─ chunkService.ListChunksByKnowledgeID(ctx, knowledgeID)
│
├─ Line 79-82: Filter text-like chunks
│ ├─ types.ChunkTypeText
│ ├─ types.ChunkTypeImageOCR
│ └─ types.ChunkTypeImageCaption
│
└─ Result: textChunks array
STEP 4: Update ParseStatus (85-103)
├─ Line 87: Check current status (should be "processing")
├─ Line 88: knowledge.ParseStatus = "completed"
│
├─ Line 92-96: Set SummaryStatus
│ ├─ If textChunks exist:
│ │ └─ knowledge.SummaryStatus = SummaryStatusPending
│ │ → Summary generation will be enqueued
│ │
│ └─ Else:
│ └─ knowledge.SummaryStatus = SummaryStatusNone
│ → No summary needed (no content)
│
└─ Line 98: Update knowledge in DB
STEP 5: Enqueue Summary Generation (106-109)
├─ Check: len(textChunks) > 0 (106)
│
├─ Line 107: enqueueSummaryGenerationTask(ctx, payload)
│ ├─ Creates types.SummaryGenerationPayload
│ ├─ Task type: types.TypeSummaryGeneration
│ ├─ Queue: "low" priority
│ └─ Max retries: 3
│
└─ Result: Summary task enqueued
STEP 6: Enqueue Question Generation (if enabled) (106-109)
├─ Check: kb.QuestionGenerationConfig != nil (160)
├─ Check: kb.QuestionGenerationConfig.Enabled (160)
│
├─ Line 164-170: Validate & bound QuestionCount
│ ├─ Get: questionCount = kb.QuestionGenerationConfig.QuestionCount
│ ├─ Default: 3 (if <= 0)
│ └─ Max: 10 (cap at 10)
│
├─ Line 172-177: Create payload
│ └─ types.QuestionGenerationPayload{...}
│
├─ Line 185: Enqueue task
│ ├─ Task type: types.TypeQuestionGeneration
│ ├─ Queue: "low" priority
│ └─ Max retries: 3
│
└─ Result: Question generation task enqueued (if enabled)
STEP 7: Enqueue Graph RAG Extract (if enabled) (112-120)
├─ Line 112: Check kb.ExtractConfig != nil && kb.ExtractConfig.Enabled
│
├─ Line 113-119: For each text-like chunk
│ └─ NewChunkExtractTask(ctx, taskEnqueuer, tenantID, chunkID, summaryModelID)
│
├─ Uses: kb.SummaryModelID for LLM extraction
│
└─ Result: Graph extraction tasks enqueued (one per chunk)
STEP 8: **CRITICAL - WIKI INGEST TRIGGER** (122-126)
├─ Line 123: ★ WIKI INGEST CONDITION ★
│ └─ if kb.IsWikiEnabled() && kb.WikiConfig.AutoIngest && len(textChunks) > 0
│
├─ Breakdown:
│ ├─ kb.IsWikiEnabled():
│ │ └─ From knowledgebase.go:509
│ │ └─ kb.WikiConfig != nil && kb.WikiConfig.Enabled
│ │
│ ├─ kb.WikiConfig.AutoIngest:
│ │ └─ From wiki_page.go:98
│ │ └─ Must be explicitly true
│ │
│ └─ len(textChunks) > 0:
│ └─ Must have text content to ingest
│
├─ If condition TRUE (Line 124):
│ ├─ EnqueueWikiIngest(ctx, taskEnqueuer, redisClient, ...)
│ ├─ Task type: types.TypeWikiIngest (inferred)
│ └─ Line 125: Log success
│
└─ If condition FALSE:
└─ Wiki ingest skipped silently (common case)
STEP 9: Return Success (128)
└─ return nil
┌──────────────────────────────────────────────────────────────────────────────┐
│ PHASE 4: ASYNC POST-PROCESSING TASKS (Run in parallel) │
└──────────────────────────────────────────────────────────────────────────────┘
File: internal/application/service/knowledge.go
Task 1: Summary Generation
├─ Function: ProcessSummaryGeneration (line 2227)
├─ Input: types.SummaryGenerationPayload
├─ Process:
│ ├─ Fetch chunks for knowledge
│ ├─ Get summary model: kb.SummaryModelID
│ ├─ Generate summary via LLM
│ ├─ Create summary chunk (ChunkType = ChunkTypeSummary)
│ └─ Index summary chunk: retrieveEngine.BatchIndex()
│
└─ Result: knowledge.Description updated with summary
Task 2: Question Generation (if enabled)
├─ Function: ProcessQuestionGeneration (line 2416)
├─ Input: types.QuestionGenerationPayload
├─ Process:
│ ├─ For each text chunk
│ ├─ Use LLM to generate N questions
│ ├─ Create question chunks
│ └─ Index question chunks
│
└─ Result: Question chunks indexed for Q&A search
Task 3: Graph RAG Extraction (if enabled)
├─ Function: (Via NewChunkExtractTask)
├─ Input: ChunkID, SummaryModelID
├─ Process:
│ ├─ Extract entities and relationships from chunk
│ ├─ Build knowledge graph
│ └─ Store graph nodes/edges
│
└─ Result: Knowledge graph constructed from chunk content
Task 4: Wiki Ingest (if triggered)
├─ Function: (Via EnqueueWikiIngest)
├─ Input: KnowledgeBaseID, KnowledgeID, TenantID
├─ Process:
│ ├─ Fetch all chunks for knowledge
│ ├─ Use SynthesisModelID (kb.WikiConfig.SynthesisModelID)
│ ├─ Generate wiki pages from content
│ │ ├─ Summary pages (one per document)
│ │ ├─ Entity pages (auto-extracted)
│ │ ├─ Concept pages (auto-extracted)
│ │ └─ Interlinks between pages
│ │
│ └─ Store wiki pages in wiki_pages table
│
└─ Result: Wiki pages generated and interlinked
Configuration Decision Points
Vector vs Keyword Indexing
Decision Point: Line 1987 in processChunks()
retrieveEngine.BatchIndex(ctx, embeddingModel, indexInfoList)
How decided:
- BatchIndex() calls retrieveEngine (injected dependency)
- retrieveEngine gets tenant config:
tenantInfo.GetEffectiveEngines() - Returns:
[]RetrieverEngineParams - Each param specifies:
RetrieverEngineType: "postgres", "elasticsearch", etc.RetrieverType: "vector", "keywords", or both
- Implementation creates indices for ALL configured types
Example Configurations:
- Vector only:
{RetrieverEngineType: "postgres", RetrieverType: "vector"} - Keywords only:
{RetrieverEngineType: "elasticsearch", RetrieverType: "keywords"} - Hybrid: Both params included in array
Wiki Ingest Trigger
Decision Point: Line 123 in knowledge_post_process.go
if kb.IsWikiEnabled() && kb.WikiConfig.AutoIngest && len(textChunks) > 0
Configuration needed:
- KB type: Any (typically "document")
- WikiConfig.Enabled: true
- WikiConfig.AutoIngest: true
- WikiConfig.SynthesisModelID: valid model ID
- At least one text chunk in document
Question Generation Trigger
Decision Point: Line 160 in knowledge_post_process.go
if kb.QuestionGenerationConfig != nil && kb.QuestionGenerationConfig.Enabled
Configuration needed:
- QuestionGenerationConfig.Enabled: true
- QuestionGenerationConfig.QuestionCount: 1-10 (default 3)
Graph Extraction Trigger
Decision Point: Line 112 in knowledge_post_process.go
if kb.ExtractConfig != nil && kb.ExtractConfig.Enabled
Configuration needed:
- ExtractConfig.Enabled: true