WeKnora/internal
wizardchen 492e92580b feat(observability): integrate Langfuse for LLM token tracking and tracing
Closes #620 #497. Add opt-in Langfuse observability covering all five
model types (chat, embedding, rerank, VLM, ASR) with HTTP-request-scoped
traces and Docker Compose support (both cloud and self-hosted).

Core package internal/tracing/langfuse:
- HTTP client with batched async ingestion (non-blocking in request path)
- Sampling, environment / release tagging, and graceful fallback when
  LANGFUSE_* env vars are absent (wrappers become no-ops)
- Gin middleware opens one trace per traced request and finishes it after
  the handler chain returns, attaching method / path / user / session
- Trace context is stored under a typed key exported from internal/types
  so logger.CloneContext can preserve it across handler / goroutine
  boundaries (otherwise each LLM call auto-created an orphan trace,
  fragmenting one request into many)

Per-model generation wrappers (opt-in via NewChat/NewEmbedder/...):
- chat: captures prompt, streaming output, token usage + TTFT
- embedding: approximates tokens when the provider omits usage
- rerank: previews query/docs, summarizes results to keep payload small
- vlm: records image count and total bytes, never uploads raw pixels
- asr: records file size and audio duration, never uploads audio bytes

Async title generation (GenerateTitleAsync) now forwards the trace key
into the goroutine so title calls appear under the parent chat trace.

Docker Compose:
- LANGFUSE_* env passthrough on the `app` service for cloud deployments
- Optional `langfuse` profile spins up a self-hosted Langfuse stack that
  reuses WeKnora's existing PostgreSQL (separate database via an idempotent
  init container that fixes ICU collation drift) and Redis (separate DB
  number), adding only ClickHouse, MinIO, web and worker containers
- web/worker entrypoints URL-encode DB_PASSWORD / REDIS_PASSWORD at start
  to avoid Prisma P1013 when passwords contain @ / # / etc.

Docs: docs/Langfuse集成.md covers cloud vs self-hosted, per-model usage
strategy, code map, and resource footprint.
2026-04-24 10:29:19 +08:00
..
agent fix(agent): terminate ReAct loop on final_answer regardless of arg parse success 2026-04-23 19:36:28 +08:00
application feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
assets feat(assets): add ASR test audio file and embed it in the application 2026-04-02 21:27:27 +08:00
common feat: Implement deadlock retry mechanism for chunk creation 2026-04-22 21:17:21 +08:00
config feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
container feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
database fix: update environment configuration for debugging and logging, enhance SQLite FTS5 table management, and improve API base URL handling 2026-04-11 18:51:16 +08:00
datasource fix: improve tenant access token logging and update test cases 2026-04-22 21:17:30 +08:00
errors feat: Add agent configuration and cleanup scripts for database migrations 2025-11-05 23:18:44 +08:00
event fix: improve tenant access token logging and update test cases 2026-04-22 21:17:30 +08:00
handler fix(agent-share): don't require rerank model for wiki-only agents 2026-04-24 10:21:05 +08:00
im fix : mattermost handle io.ReadAll errors in client file 2026-04-23 17:51:07 +08:00
infrastructure feat: Add pending tasks and active status to WikiStats 2026-04-22 21:17:31 +08:00
logger feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
mcp feat(mcp): implement reconnection logic for MCP tool calls and tool listing 2026-03-31 11:57:15 +08:00
middleware refactor: Update wiki prompts and ingestion logic for improved content handling 2026-04-22 21:17:31 +08:00
models feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
router feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
runtime init commit 2025-08-05 15:08:07 +08:00
sandbox feat: optimize security and deployment of agent skills 2026-02-04 20:08:49 +08:00
searchutil feat(prompt): enhance question rewriting and intent classification in prompt templates 2026-04-16 23:15:02 +08:00
stream feat(redis): add REDIS_USERNAME support for Redis ACL 2026-02-04 19:38:40 +08:00
tracing feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
types feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
utils feat(models): support custom HTTP headers across all remote model calls 2026-04-23 23:36:40 +08:00