WeKnora/internal/common
wizardchen f1ef364820 fix(knowledge): harden auto-tagging and let it respect manual tags
Automatic tagging shipped with a few paths that quietly disabled it and a
prompt that spent most of its budget on data the model never needed.

Robustness:
- Treat an omitted confidence as a deliberate selection instead of zero.
  Models frequently drop the field, and the 0.75 floor then filtered out
  every match, silently turning the feature off. Percentage-scale values
  are rescaled rather than accepted as out-of-range certainty.
- Classify the first 500 tags when a knowledge base has more, logging a
  warning, instead of dropping the task entirely. Large knowledge bases
  previously got no auto-tagging at all with no user-visible signal.
- Reuse common.ParseLLMJsonResponse for the model reply, and give that
  helper a balanced-scan fallback. The local extraction cut the payload at
  the last "}" or "]", so a trailing citation such as "see [1]" produced
  invalid JSON and burned the task's retries.

Prompt:
- Address candidates by ordinal rather than UUID. Tag names are short while
  a UUID is 36 characters, so this cuts candidate tokens roughly tenfold,
  keeps the model from having to reproduce (and corrupt) an id, and is what
  makes the raised 500 ceiling affordable. Ordinals are range-checked and
  mapped back to ids server-side.
- Drop the rationale field. Nothing read it, and at max_tags=10 it competed
  with the 1024-token output budget and risked truncating the JSON.
- Fence document text in <document> so file content cannot be read as
  instructions.

Behaviour:
- Add auto_tag_config.skip_if_tagged, defaulting to true, so documents the
  user already tagged at upload time keep their classification. Existing
  tags are now loaded before the model call, so a skipped document costs no
  LLM round-trip. Configs stored without the field adopt the default.

Also promote AddKnowledgeTagRelations to KnowledgeRepository instead of
reaching for it through a type assertion that failed at runtime, and route
the max_tags bounds through AutoTagConfig.Normalize so the prompt, the match
filter and the persisted config cannot drift apart.

Adds handler-level coverage for scope, cancellation, idempotency, ordinal
validation and the new toggle, plus SDK, Swagger and API doc updates.
2026-08-07 15:35:58 +08:00
..
db_retry.go feat: Implement deadlock retry mechanism for chunk creation 2026-04-22 21:17:21 +08:00
redis_tls.go feat(redis): support TLS for Redis connections (#1930) 2026-07-06 23:02:02 +08:00
redis_tls_test.go fix(redis): harden TLS tests and surface TLS config at startup 2026-07-06 23:04:52 +08:00
tools.go fix(knowledge): harden auto-tagging and let it respect manual tags 2026-08-07 15:35:58 +08:00
tools_test.go fix(knowledge): harden auto-tagging and let it respect manual tags 2026-08-07 15:35:58 +08:00