WeKnora/internal/types
wizardchen dc634e863a feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling
Two encryption gaps and one fail-loud regression, fixed together because
they are inseparable in practice.

At-rest encryption gaps
-----------------------
Before this change, only Tenant.APIKey, Tenant.Credentials.WeKnoraCloud.AppSecret,
Model.Parameters.APIKey/AppSecret, and WebSearchProviderParameters.APIKey
were encrypted with SYSTEM_AES_KEY. MCP and DataSource were missed:

- MCPAuthConfig.Value/Scan went through bare json.Marshal/Unmarshal, so
  api_key and token were written to mcp_services.auth_config (json) in
  plaintext regardless of whether SYSTEM_AES_KEY was set.
- DataSource.Config (types.JSON) is a byte passthrough, so connector
  secrets (Notion token, GitHub PAT, Feishu app_secret, ...) inside the
  Credentials map were stored in plaintext too.

Now both follow the ModelParameters / WebSearchProviderParameters pattern.
Encryption operates on a local copy to avoid mutating the caller's
in-memory struct (otherwise the next read of the same *MCPAuthConfig
would surface ciphertext as if it were the API key). For DataSource the
ToJSON ↔ ParseConfig pair is the only application boundary that
materializes credentials, so encrypting there is sufficient and avoids
having to fork the generic types.JSON.

Backward compatibility
----------------------
DecryptStoredSecret returns rows without the enc:v1: prefix verbatim, so
historical plaintext records keep working. Next write re-saves them
encrypted. No migration step.

Graceful Scan-time degradation
------------------------------
The strict "fail loud on key rotation" behaviour of DecryptStoredSecret
turned into a footgun once it ran from GORM Scan: a single broken row
caused ListModels / ListMCPServices / ListDataSources to return zero
results, hiding all other rows and giving the operator no way to find
which credential needed re-entering.

New utils.DecryptStoredSecretLenient: same logic, but on failure returns
("", false). The strict variant is kept for hot paths that are about to
actually USE the credential (so a rotated key surfaces loudly there
rather than silently degrading to "missing api key").

All Scan paths switch to lenient + per-failure log line:
  ModelParameters.Scan, WebSearchProviderParameters.Scan,
  MCPAuthConfig.Scan, DataSource.ParseConfig, CredentialsConfig.Scan.

On lenient failure the field blanks out (never ciphertext), the row
still loads, the DTO reports configured=false so the UI shows "not
configured", and any upstream call attempted with the empty value fails
with a clear "missing api key" rather than sending ciphertext as the key.

Tenant.AfterFind (Tenant.APIKey) intentionally stays strict — it is the
primary HTTP auth credential and a silent empty value would change auth
semantics ambiguously.
2026-05-17 15:27:52 +08:00
..
interfaces refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
agent.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
agent_test.go Fix system prompt not being used when flag is false 2026-04-15 20:21:12 +08:00
agent_type_preset.go feat: Introduce agent type presets for smart-reasoning mode 2026-04-22 21:18:20 +08:00
builtin_agent_config.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
chat.go fix(agent): pass reasoning_content back to providers that require it 2026-05-13 12:29:10 +08:00
chat_history_config.go feat: implement chat history and retrieval settings 2026-03-09 11:46:56 +08:00
chat_manage.go feat(agent): support dedicated model for query understanding step 2026-05-12 20:27:20 +08:00
chunk.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
chunk_contextheader_test.go test(types): regression test that ContextHeader is hidden from JSON 2026-05-06 17:17:07 +08:00
cleanup.go init commit 2025-08-05 15:08:07 +08:00
const.go feat(observability): integrate Langfuse for LLM token tracking and tracing 2026-04-24 10:29:19 +08:00
context_helpers.go fix(i18n): use WEKNORA_LANGUAGE env for prompt language fallback instead of hardcoded zh-CN 2026-04-02 12:01:25 +08:00
context_helpers_test.go refactor: Replace hardcoded language logic with middleware infrastructure 2026-04-22 21:17:17 +08:00
custom_agent.go feat(agent): support dedicated model for query understanding step 2026-05-12 20:27:20 +08:00
dataset.go init commit 2025-08-05 15:08:07 +08:00
datasource.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
datasource_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
docparser.go fix(chunker): trim surrounding whitespace before embedding 2026-05-06 21:16:48 +08:00
embedding.go perf: 优化FAQ索引更新和删除性能 2026-01-14 17:30:04 +08:00
errors.go init commit 2025-08-05 15:08:07 +08:00
evaluation.go refactor: Update tenant ID handling across the application 2025-11-25 16:30:54 +08:00
event_bus.go refactor: Implement event-driven streaming and enhance session management 2025-11-07 22:41:41 +08:00
extract_graph.go Refactor knowledge post-processing into orchestrator task 2026-04-16 18:13:20 +08:00
faq.go feat(types): enhance FAQ metadata normalization and sanitization 2026-03-19 18:20:09 +08:00
faq_test.go feat(chunk): implement batch deletion for chunks to avoid MySQL placeholder limit 2026-03-19 18:20:09 +08:00
graph.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
indexing_strategy.go feat: Add comprehensive documentation for WeKnora knowledge base and wiki features 2026-04-22 21:18:10 +08:00
json.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
json_test.go test(types): add unit test for JSON unmarshalling and validation 2026-04-22 21:17:36 +08:00
knowledge.go feat(knowledge-base): document list filters and explicit batch-management UX 2026-05-12 18:29:47 +08:00
knowledgebase.go feat(chunker): recursive priority splitting + token-aware budget 2026-05-06 17:17:07 +08:00
knowledgebase_test.go feat: add vector_store_id column to knowledge_bases for per-KB vector store binding 2026-04-20 19:53:44 +08:00
mcp.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
mcp_test.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
memory.go feat: implement preview of user level memory graph 2026-02-25 14:29:54 +08:00
message.go feat(chat_pipeline): enhance query handling with image and attachment metadata 2026-04-14 18:21:12 +08:00
model.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
organization.go feat(api): integrate per-organization resource counts into organization listing 2026-02-09 11:40:25 +08:00
placeholder.go feat: enhance localization and configuration support 2026-03-17 22:23:24 +08:00
qa_request.go feat: attachment_support 2026-04-10 12:49:20 +08:00
retrieval_config.go feat(retrieval): configurable RRF parameters per tenant 2026-05-06 17:17:07 +08:00
retriever.go feat: add Tencent VectorDB retriever backend 2026-05-09 13:14:01 +08:00
search.go feat(prompt): enhance context and intent classification in templates 2026-03-26 18:57:12 +08:00
secret.go refactor(credentials): introduce /credentials subresource pattern 2026-05-17 15:27:52 +08:00
session.go feat(sessions): add keyword search, user-scoped pinning, and IM source visibility 2026-04-30 16:23:00 +08:00
tag.go feat(logger): implement LLM debug logging functionality 2026-04-16 23:15:02 +08:00
task.go refactor(retriever): introduce factory for KB-scoped retrieve engine resolution 2026-05-14 15:53:58 +08:00
task_dead_letter.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
task_pending_op.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
tenant.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
tenant_test.go feat: support Tencent VectorDB keyword retrieval 2026-05-10 00:02:05 +08:00
tracing.go feat(observability): extend Langfuse tracing across asynq pipeline 2026-04-24 13:16:47 +08:00
user.go add oidc 2026-03-30 11:13:44 +08:00
vectorstore.go fix: redact sensitive fields and prevent credential loss on edit 2026-05-17 11:37:53 +08:00
vectorstore_test.go fix: redact sensitive fields and prevent credential loss on edit 2026-05-17 11:37:53 +08:00
web_search.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
web_search_provider.go feat(crypto): AES-GCM at-rest for MCP and DataSource credentials with graceful key-rotation handling 2026-05-17 15:27:52 +08:00
web_search_test.go fix(web-search): normalize default tenant web search config at runtime 2026-04-30 15:28:35 +08:00
weknoracloud.go refactor(weknoracloud): update credential handling and configuration 2026-04-15 11:37:24 +08:00
wiki_log_entry.go perf(wiki): move ingest log to event table and index to on-demand API 2026-05-08 21:48:52 +08:00
wiki_page.go perf(wiki-ingest): scale to 4w-doc KBs + generic task queue / dead-letter 2026-05-10 00:01:06 +08:00
wiki_page_test.go feat: Enhance wiki extraction and configuration options 2026-04-22 21:18:32 +08:00