WeKnora/docs
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
..
api feat: add VectorStore CRUD API endpoints and documentation 2026-04-14 17:34:57 +08:00
images chore: update architecture diagram 2026-03-30 21:54:50 +08:00
wiki docs: add wiki documentation with cross-referenced links 2026-04-15 20:38:31 +08:00
BUILTIN_MCP_SERVICES.md feat: add built-in MCP service support 2026-03-05 21:57:01 +08:00
BUILTIN_MODELS.md feat:add several mainstream model providers 2026-01-14 17:08:26 +08:00
IM集成开发文档.md add oidc 2026-03-30 11:13:44 +08:00
KnowledgeGraph.md docs: 更新多语言文档,新增知识图谱与MCP配置指南及目录结构 2025-11-19 16:56:49 +08:00
LITE.md feat: add LAN access configuration for desktop app, allowing users to enable public API access with corresponding UI updates and translations 2026-04-11 18:51:16 +08:00
Langfuse集成.md feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
MCP功能使用说明.md feat: Revamp conversation configuration management and enhance localization 2025-11-18 23:18:43 +08:00
OIDC认证调用流程.md simplify oidc callback result payload 2026-04-01 20:47:18 +08:00
QA.md feat: add LAN access configuration for desktop app, allowing users to enable public API access with corresponding UI updates and translations 2026-04-11 18:51:16 +08:00
ROADMAP.md feat(roadmap): update progress on document parsing and IM integration 2026-03-31 20:22:34 +08:00
agent-skills.md feat: adjust sandbox setting of agentConfig 2026-02-04 20:08:49 +08:00
docs.go feat(system): remove MinIO list buckets API and UI dependencies 2026-04-16 18:13:21 +08:00
swagger.json feat(system): remove MinIO list buckets API and UI dependencies 2026-04-16 18:13:21 +08:00
swagger.yaml feat(system): remove MinIO list buckets API and UI dependencies 2026-04-16 18:13:21 +08:00
使用其他向量数据库.md init commit 2025-08-05 15:08:07 +08:00
共享空间说明.md refactor(docs): streamline knowledge base access permissions documentation 2026-02-09 11:40:25 +08:00
开发指南.md feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
开启知识图谱功能.md feat: Update VLM configuration in knowledge base and API documentation 2025-11-20 20:08:24 +08:00
快速开发模式说明.md docs: update Air installation command from cosmtrek/air to air-verse/air 2026-02-04 19:27:44 +08:00
数据源导入开发文档.md docs: remove WeCom document integration from data source import documentation 2026-03-30 20:46:07 +08:00
添加新的网络搜索引擎.md feat(web-search): refactor web search provider 2026-03-31 20:22:34 +08:00