* fix(knowledge): reject invalid JSON uploads before async parse
Validate .json payloads at HTTP upload/replace time with the same gate used
by the simple JSON reader, so JSONC or truncated files fail with HTTP 400
instead of after document-process retries. CreateKnowledgeFromFile itself
skips the gate so datasource sync and IM still create a knowledge row that
fails in async parse.
Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
* fix(knowledge): localize JSON upload validation messages
The upload queue shows the backend message verbatim and the rest of the
upload handler speaks Chinese. Map ValidateJSONContent's errors (now
sentinels with unchanged text, still shared with the async JSON reader)
to Chinese messages at the upload/replace boundary.
---------
Signed-off-by: xiaojunxiang <xiaojunxiang@kingsoft.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
Co-authored-by: lyingbug <xiayula2014@gmail.com>
Reparse:
- Open the new attempt only after override validation and cleanup pass.
A rejected reparse used to leave the previous run's subtasks treating
themselves as superseded, so the row stayed in finalizing for good.
- Cancel the previous run's queued tasks when it is still in flight.
ProcessDocument, processChunks, post-process, image tasks, wiki ops and
the dead-letter callback now skip work that belongs to a superseded
attempt.
- Summary tasks write only their own columns. A full-row save after the
LLM call could put finalizing over the new run's processing row.
Errors that were acked with the row still in flight:
- A failed post-process enqueue is retried in place and then returned,
so the task retries instead of leaving the row in processing.
- Transient tenant/knowledge reads and the processing write now retry.
- An image whose orphan check fails on its final attempt is still
counted. Empty manual content fails the row.
Counters:
- FinalizeSubtask runs its decrement and promote in one transaction.
Slot releases retry, and the post-process shortfall release keeps
going past a failed release.
Wiki:
- A synthesis model that cannot be built releases the KB's pending
documents on the final attempt instead of retrying forever.
- Panics in map/reduce workers become per-document failures.
- Housekeeping protects a row held by a wiki op for at most 48h.
Lite mode:
- SyncTaskExecutor honours options set on asynq.NewTask (MaxRetry,
Timeout, Deadline, ProcessIn/At), stops on SkipRetry, and runs the
same final-failure callback as the dead-letter middleware.
- The multimodal fan-in counter is kept in process, so post-process
waits for every image.
Also:
- Housekeeping recovers summaries left pending on finished rows. Cancel
and stall recovery close unfinished summaries.
- Housekeeping sweeps no longer overlap.
- Data source URL ingest writes only the metadata column.
- The document list drops cards the batch endpoint no longer returns.
Queued ProcessDocument tasks still held the old FilePath, and a crash
between the source UPDATE and reparse could leave completed knowledge
pointing at a new file. Dequeue first, flip pending in the same write,
and skip stale worker Saves when the stored path has moved.
Add KnowledgeService.ReplaceKnowledgeFile, which replaces the stored
source file of an existing file knowledge while preserving its knowledge
ID, then re-parses it through ReparseKnowledge. It is groundwork for the
local folder data source (#2873); no existing API, handler or connector
behavior changes.
Object storage is not transactional, so the steps are ordered to keep the
row pointing at an existing file: save the new file, update the source
columns in one UPDATE, reparse, and only then delete the old file. A
failed column update discards the new file; a failed reparse restores the
previous source columns (marked failed, since cleanup may already have
removed the old index) and discards the new file. The row is re-read
before a replacement file is deleted, so a write that committed despite
reporting an error never leaves the knowledge referencing a deleted file.
Identical content under the same name and folder is not re-parsed: changed
metadata is persisted and a DuplicateKnowledgeError is returned, matching
CreateKnowledgeFromFile's duplicate contract. Metadata is merged so
entries such as process overrides survive, and stored overrides are
re-validated against the new file type.