WeKnora/internal/application/service
draix 7adf88766a fix(graph): recover JSON from malformed/truncated fences in extract_entity
Closes #1113

The graph extraction pipeline relies on a strict markdown fence regex to
pull JSON out of LLM responses. In production the regex misses ~84% of
real-world responses, in three ways:

  1. The LLM hits max_tokens mid-output and produces no closing fence.
  2. The opening fence is malformed or surrounded by prose, so the
     non-greedy regex fails to anchor a match.
  3. The model returns raw JSON with stray backticks but no real fence.

In every case extractContent fell through to returning the raw text,
which then failed json.Unmarshal with errors like
"invalid character '` + "`" + "' looking for beginning of value".

This change keeps the existing happy path untouched and adds a
conservative recovery step in the default branch of extractContent:

  - If an opening ``` is present, take everything after it, drop a
    likely language tag on the first line, cut at any trailing closing
    fence, and trim stray backticks/whitespace.
  - Otherwise, look for an outermost JSON object/array in the text using
    a small bracket-balanced scanner that respects string literals, so
    embedded {} or [] inside JSON strings don't confuse it.
  - Only fall back to the original raw-text behavior when neither
    strategy yields anything plausible.

The recovery helpers (stripFencesAndExtract, extractJSONLike,
isLikelyLanguageTag) are package-private and have table-driven tests in
extract_entity_test.go covering the three failure patterns described in
the issue, plus the previously-working fenced and bare-JSON shapes to
guard against regressions.
2026-05-07 15:40:19 +08:00
..
chat_pipeline fix(graph): recover JSON from malformed/truncated fences in extract_entity 2026-05-07 15:40:19 +08:00
file feat(ks3): add KS3 object storage support 2026-05-05 00:22:27 +08:00
llmcontext feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
memory feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
metric fix: resolve failing test and build errors 2026-04-22 21:17:29 +08:00
retriever feat(retriever): exponential backoff + safety truncation for embeddings 2026-05-06 17:17:07 +08:00
agent_service.go feat: correct data analysis errors using fileService. 2026-04-27 13:18:08 +08:00
agent_share.go fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
chunk.go feat: enhance document loading and state management in doc-content component 2026-03-02 21:21:49 +08:00
custom_agent.go feat: Enhance suggested questions functionality with wiki integration 2026-04-22 21:18:20 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource_service.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
evaluation.go feat(knowledge): add channel support to knowledge entries and related components 2026-03-25 22:08:29 +08:00
extract.go feat: correct data analysis errors using fileService. 2026-04-27 13:18:08 +08:00
graph.go feat(graph): enhance graph extraction prompt rendering and language context handling 2026-03-25 22:08:29 +08:00
image_multimodal.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
kbshare.go feat(api): integrate per-organization resource counts into organization listing 2026-02-09 11:40:25 +08:00
knowledge.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_clone_move.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_create.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_delete.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_faq.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_faq_import.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_image_cleanup_test.go fix(knowledge): delete extracted images from storage when knowledge is removed 2026-04-02 12:04:04 +08:00
knowledge_manual_test.go fix: sanitize tab chars and avoid double .md extension in manual download filename 2026-03-16 18:18:23 +08:00
knowledge_post_process.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
knowledge_process.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledge_shared_access_test.go fix: include shared knowledge in knowledge search 2026-05-05 01:04:49 +08:00
knowledge_storage_config_test.go fix(storage): merge tos/s3/oss/ks3 tenant configs in buildStorageConfig (#1117) 2026-05-06 17:09:42 +08:00
knowledge_summary_test.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
knowledge_util.go refactor(knowledge): split 9.8k-line service file by responsibility 2026-05-06 17:40:10 +08:00
knowledgebase.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
knowledgebase_search.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_faq.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
knowledgebase_search_fusion.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
knowledgebase_search_results.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
knowledgebase_search_shared.go feat: enhance input field and agent stream display with improved handling and sanitization 2026-03-16 02:26:47 +08:00
mcp_service.go feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
message.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
metric_hook.go refactor: Clean up code by removing unnecessary blank lines and improving comments for better readability and maintainability 2025-11-28 16:10:37 +08:00
model.go feat(models): support custom HTTP headers across all remote model calls 2026-04-23 23:36:40 +08:00
ocr_sanitizer.go feat: use language-neutral English prompts for multilingual LLM responses 2026-03-06 21:08:12 +08:00
ocr_sanitizer_test.go feat: introduce WeKnora Lite edition with lightweight configuration and deployment 2026-03-02 21:21:49 +08:00
organization.go feat(i18n): update localization for knowledge bases and organization features 2026-02-09 11:40:25 +08:00
session.go fix(sessions): review fixes for keyword search / pinning / IM titles 2026-04-30 16:23:00 +08:00
session_agent_qa.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
session_knowledge_qa.go fix(chat): include history in model fallback 2026-05-04 23:24:21 +08:00
session_knowledge_qa_test.go fix(chat): include history in model fallback 2026-05-04 23:24:21 +08:00
session_qa_helpers.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
skill_service.go feat: adapt preloaded skills to smart-reasoning agent 2026-02-04 20:08:49 +08:00
tag.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
tenant.go fix(tenant): return plaintext API key after create/reset 2026-04-29 19:46:21 +08:00
user.go simplify oidc callback result payload 2026-04-01 20:47:18 +08:00
vectorstore.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
vectorstore_healthcheck.go feat: add VectorStore service layer with types and connection testing 2026-04-13 22:57:17 +08:00
vectorstore_test.go feat: pass IndexConfig to repositories with scalability options 2026-04-17 16:52:24 +08:00
web_search.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_provider.go feat: search engine adds proxy addresses and adds SSRF verification. 2026-04-17 16:38:02 +08:00
web_search_state.go fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_ingest.go test(ingest): cover insufficient-content gate; allow threshold override 2026-05-06 11:23:32 +08:00
wiki_ingest_batch.go chore(comment): update slug source to knowledge ID 2026-05-06 11:23:32 +08:00
wiki_ingest_cite.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_ingest_cite_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_dedup_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00
wiki_ingest_test.go fix(ingest): QA follow-up — preserve OCR text and close remaining filename leaks 2026-05-06 11:23:32 +08:00
wiki_linkify.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_linkify_test.go feat: Refactor cross-link injection logic for wiki pages 2026-04-22 21:18:09 +08:00
wiki_lint.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page.go feat: Implement auto-linked content updates in wiki page management 2026-04-23 11:16:47 +08:00
wiki_page_test.go refactor: Update wiki link syntax and rendering in WikiBrowser component 2026-04-22 21:17:26 +08:00