Taxonomy selection asked the embedding provider for every folder in one
request, and for every item in another, by calling BatchEmbed directly.
BatchEmbedWithPool is what reads BATCH_EMBED_SIZE and splits the inputs,
so these two calls sat outside the only control an operator has over
request width: a provider that caps inputs per request rejected them
however low that setting was, and the planner fell back to feeding all
folders (#3390).
Send both sets through the pool, the way every other embedding caller in
the service does. Nothing else changes: the pool reassembles the vectors
in input order, which is what the similarity ranking below depends on.
The regression test carries the real pooling implementation behind a
provider that records the width of every request it receives, so a call
that goes back to the provider directly shows up as one oversized
request rather than as a passing test.
* fix(security): close the low-severity shared-space findings
Verified and fixed from the shared-space permission review:
- Embed channels kept knowledge_ids, tag_ids, mentioned_items, skill_names
and summary_model_id from anonymous visitors. Mentions are only scoped
for shared agents, so a visitor could query any KB (and pick any model)
of the channel workspace by ID. The payload now drops them; the widget
never sends them.
- PUT /knowledgebase/:kb_id/wiki/issues/:issue_id/status updated any issue
by ID. The update is now scoped to the path's KB (404 otherwise).
- A direct org invite let the inviter attach any user of the target
workspace (or any user by ID) as its representative, whose username and
email the roster then showed to all members. Direct adds attach no
representative, and the roster returns emails only for the caller's own
workspace.
- COS and KS3 dropped the bucket (and COS the region) segment when turning
a path into an object key, while the tenant check scans every segment:
cos://42/<region>/prefix/7/... passed as tenant 42 and read tenant 7's
object from a shared bucket. Both must now match the service's config.
- Shared agent listings returned the full agent to receivers. Prompts and
the creator's user ID are withheld; capabilities and scope stay visible.
- Temporary document uploads let the caller pick the parser engine (and
its credentials) of a shared agent's workspace; only the agent's rules
choose one there now.
- Suggested questions used request / message-snapshot knowledge_ids
without a permission check. Each document is now resolved to its KB and
checked like KB IDs, including a shared agent's scope.
- IM channels accepted any knowledge_base_id (and any agent ID on create).
Both must belong to the channel's workspace, the KB also to a restricted
API key's allow-list.
- KB-restricted API keys could list org-shared KBs outside their allow-list
in the organization views; those listings are now filtered.
- With enable_rbac=false, RequireKBAccess let cross-workspace denials
through. That switch is for roles inside a workspace; the guard now
always denies another workspace's KB, since handlers behind it load by
ID without a tenant filter.
- The org member limit was counted before and enforced apart from the
insert; it is now checked in the insert's transaction with the
organization row locked (SQLite serializes writers instead).
- The UpdateOrganization route comment and API doc said owner-only; the
service and UI allow org admins. Comment and doc corrected.
Documented, not changed: issued /r/ and presigned links stay valid until
they expire after a share is revoked.
* fix(security): address review of the low-severity shared-space fixes
- Embed payloads also drop question_origin. It selects a KB like an
@mention for agents that retrieve only when mentioned, and the widget
never sends it; the selection stays inside the agent's scope, but an
anonymous visitor should not pick among its KBs. suggestion_attribution is
kept (the widget sends it and it is validated).
- The receiver view of a shared agent also withholds the follow-up
suggestion instruction, a prompt like the others. Starter questions are
shown to receivers anyway, and the sharer's name is displayed by design.
- Follow-up suggestion generation moves into the agent's workspace with
types.WithExecutionTenant instead of a bare tenant rewrite, so the
caller is pinned before the document permission check runs. The auth
middleware and the shared run already pinned it; this removes the
dependency on that.
- KS3 CopyFile copies only from the service's own bucket (as COS and S3
do); another bucket's path is a cross-backend copy.
Tests cover the dropped question_origin, the follow-up instruction and the
/shared-agents listing, a follow-up context with the tenant switched, a
repeated wiki issue status update, and the KS3 copy source.
* feat(agent): size agent history by the window and calibrate token estimates
Agent history was capped at history_turns (default 5) because compaction
could not survive a turn. With checkpoints persisted, load history by
tokens instead, and let compaction bound the session.
Budget. The loader drops the oldest turns that do not fit, and what it
drops is lost: neither replayed nor summarized. A budget equal to the
compaction threshold (the earlier attempt, since reverted) let the loader
trim first, so whenever one turn was larger than the system prompt plus
the new question the request never crossed the threshold; nothing was
summarized and the session became a sliding window with no checkpoint.
agent.HistoryTokenBudget is now the whole window: overflow reaches the
first round's compaction, which summarizes it into a new checkpoint.
A turn-by-turn regression test drives the real loader, budget and
compactor and fails as soon as a stored turn after the checkpoint is
missing from the loaded history; restoring the old budget fails it.
Summarizer input. History can now reach the window, so the part being
summarized can outgrow one request. An overflowing request degrades to a
raw archive that is never persisted, so every later turn would fail the
same way. Prepare keeps the newest messages that fit, the prompt states
how many were left out, and file paths still come from all of them.
Calibration. cl100k counts Chinese at about a token a character where
Qwen and DeepSeek spend about 0.6. The estimator now carries a scale
(clamped to 0.5-1.5, text only), learned from consecutive requests in a
turn: they share tools, system prompt and earlier messages, so the
provider's growth between them prices exactly the appended messages,
without mixing in how a provider renders tool schemas. The turn's scale
is stored on its usage (context_token_scale); the next turn's loader
prices history with it and hands it to the engine. Loading, the first
compaction check and compaction itself share one estimator, so they
agree; calibrating only the trigger would let the loader drop turns
again, which the regression test covers.
Also: history is priced as the engine sends it (redacted KB/Wiki results
cost one line); a checkpoint older than the rows read is matched by the
assistant rows' (created_at, id) order; a read ending on a turn's
question keeps that turn; a compaction discarded for freeing under 5%
still persists its checkpoint; a checkpoint write matching no row is an
error; a partial index backs the newest-checkpoint lookup. Agent mode no
longer reads history_turns: the field leaves AgentConfig, the editor
hides it in smart-reasoning mode, and quick-answer keeps its turn cap.
* fix(agent): address review of window-sized history loading
Summarization. The summarizer call had a fixed 60s total timeout. With
history loaded up to the window, a request could carry ~175k tokens and be
cut off while prefilling or writing normally; the degraded result left no
checkpoint, so the next turn reloaded the same history and failed the same
way. Summarization now streams and is cancelled only after no output for the
engine's stall timeout (120s by default, LLMCallTimeout overrides), with the
overall ceiling left to the transport, as for the engine's own rounds and as
in Codex and Pi. A stopped turn is not retried. When the summarizer still
fails, the raw archive is persisted as a checkpoint marked degraded so the
loop is broken, and the next compaction refines it as the previous summary.
The archive is held to the summary budget, newest messages first, so a
degraded compaction always frees room.
Loading. The loader priced turns as sent but returned and held them as
stored, so wiki pages stored in full and sent as one line could pile up in
memory. It now returns the as-sent history and releases each turn's stored
rows once replayed, keeping about a page of rows at a time.
Index. The partial checkpoint index is replaced by a composite
(session_id, created_at DESC, id DESC) index serving the backwards page,
the newest-checkpoint lookup and the recent-messages reads. It is built with
CREATE INDEX CONCURRENTLY so writes to messages are not blocked; the
migration holds that single statement, since a multi-statement query runs as
an implicit transaction. A PostgreSQL integration test, skipped without
WEKNORA_MIGRATION_TEST_POSTGRES_DSN, applies the chain, runs 000106 down and
up through golang-migrate, and checks both queries use the index without a
sort; the SQLite migration test checks the same plans.
Calibration. It no longer waits for a total token count, and a usage with
only a scale is persisted. A turn that measures nothing carries forward the
scale it started from, so the newest turn always has the latest one.
* fix(security): close cross-workspace access paths in shared spaces
A review of how shared-space (organization) members reach each other's
resources found six paths that bypass the org-share permission model:
- presigned-preview signed any resource handle or storage path, turning a
known resource:// handle into an anonymous /r/ URL that outlives share
revocation. It now requires the path to belong to the caller's tenant,
like /files.
- POST /initialization/initialize let a shared-KB editor rewrite the owner
workspace's models (base URL, API key) in place. Only the KB's own
workspace may call it, and rewriting a stored model needs the same
authority as PUT /models/:id.
- KB JSON carried the legacy inline credentials (storage_config
secret_id/secret_key, which initialization fills with the MinIO env
keys, and vlm_config api_key) to every share receiver. They are withheld
from JSON; their DB columns are unchanged.
- A chat without agent_source_tenant_id resolved shares before the
caller's own agent, so a member sharing a same-ID built-in agent took
over other workspaces' default agent. Own agents now win, built-in
agents can no longer be shared (existing such shares no longer resolve),
and embed channels drop a visitor-supplied source workspace.
- Any org-admin tenant (even a tenant Viewer inside it) could raise another
workspace's KB share to editor/admin, and the route guard was bypassable
via an unrelated :id. Share management is now: the sharer from the source
tenant (Contributor+), source-tenant Admin+, or an org-admin tenant's
Admin+ that may only remove or lower. share_id must belong to :id.
- Wiki write tools were registered for every wiki KB in the search targets,
which only need read access, so a viewer share (e.g. via the built-in
wiki fixer) could edit or delete another workspace's wiki pages. They now
act only on KBs the caller may edit.
* fix(security): close the KB settings path and owner details for share receivers
Follow-up to the shared-space review:
- PUT /initialization/config shares the route guard that admits share
editors, so an editor could still rebind the owner's KB models, chunking
and storage, and storage backends resolved against the caller's workspace
let it bind the owner's KB to its own backend. Another workspace now needs
an admin share (the only share level the frontend offers settings to) and
may not change the storage binding at all.
- GET /initialization/config showed model base URLs and bucket locations to
any admin of the caller's own workspace, including share receivers. They
now only see whether credentials are configured.
- Wiki write tools on the caller's own KBs had no role floor, so a tenant
Viewer (and the IM, embed and MCP endpoint principals that run as Viewer)
could write through an agent what the HTTP routes deny. Writes now need
Contributor+, or the ingest capability for a scoped API key, following
the RBAC rollout switch.
- resolveAgent logs the share lookup error instead of a nil own-agent error
when a shared selector fails.
* fix(security): close the medium shared-space access findings
Follow-up to the high-risk fixes, from the same review:
- Wiki fixer moved into a shared KB's workspace ran with the caller's own
config there: an unset MCP mode registered all of the owner's MCP
services, and an "all" KB scope reached the owner's other KBs. The scoped
run now uses the built-in config pinned to that one KB, without MCP,
skills, sandbox or web search, and falls back to the KB's own model.
- An unset MCP mode means "all" at runtime but is shown as none in the share
scope; shared runs now treat it as none.
- Quick-answer mode used the request's web_search_enabled alone, so a
receiver could turn on search the agent disables and spend the owner's
provider quota. It now requires the agent setting too, as agent mode does.
- summary_model_id picked any model of the workspace a run executes in. It
is ignored for shared agents and the relocated wiki fixer.
- A workspace that left or was removed from an organization kept sharing its
KBs and agents. Its shares are revoked on departure, and share queries and
permission checks require the source workspace's membership and a live
organization, which also covers rows left behind earlier.
- Sharing an agent exposes its KB scope to all members, so a Contributor
could share colleagues' KBs (or "all") through an agent. The sharer must
now be able to share each KB directly (creator or Admin+; "all" needs
Admin+), also when a shared agent's scope is widened later.
- A scoped API key with manage_channels could mint an MCP endpoint token
wider than itself. Endpoints it creates, updates or rotates must stay
within its KB allow-list and capabilities.
- A shared agent's run indexed the receiver's Q&A into the owner's chat
history KB; indexing now uses the session owner's configuration.
- Sharing an agent with skills keeps injecting the owner's skill
environment values (product decision); the share panel now warns that
members can read them. Preview of originals through read-only shares is
kept as designed and documented.
#3406 (000104_skill_served_version) and #3407
(000104_message_context_checkpoint) each took version 104 and were merged
eleven minutes apart. golang-migrate refuses to load a directory with a
duplicate version ("duplicate migration file"), so every PostgreSQL
deployment of main fails to migrate.
Renumber the later of the two. It is safe for every deployment that could
exist: one that migrated between the two merges applied the skill
migration as 104 and now applies this one as 105, and after the second
merge main could not load its migrations at all, so no deployment applied
this file as 104. The migration is ADD COLUMN IF NOT EXISTS either way.
Add a test that loads both migration directories. Pull request CI runs on
the merge with main, so a second PR taking an already merged number now
fails before it lands.
* fix(ocr): normalize inline HTML tables in VLM/OCR sanitizer
The VLM/OCR path left raw HTML <table> blocks in the body: looksLikeHTML
does not match markdown text with an embedded table, so ocrHTMLToMarkdown
(which also lacked the table plugin) never ran. The chunker then split the
raw markup mid-row, tripping the 7500-char hard limit and invalidating the
heading layer for the whole document.
Reuse docparser.NormalizeHTMLTables (already used by PaddleOCR-VL) instead:
export it and call it unconditionally from sanitizeOCRText, since it is a
safe no-op when no <table> is present.
* fix(ocr): keep merged-cell HTML tables chunkable
- Narrow span detection to values >1 so rowspan=1/colspan=1 tables
still convert to GFM instead of falling back to raw HTML.
- On the merged-cell fallback, split each <tr> onto its own line and
pad the block with blank lines so the chunker can split by "\n"
instead of force-cutting at the absolute max size.
* fix(knowledge): normalize inline HTML tables for all parser engines before chunking
Previously NormalizeHTMLTables was only wired into the paddleocr-vl
converters, so tables emitted by other engines (notably MinerU) reached
the chunker as raw single-line <table> blocks and were force-cut at the
7500 char hard limit.
Add the normalization at the two central pre-chunk boundaries instead:
- knowledge_process.go: after all converter/ASR/image markdown rewrites,
before the chunker step (covers every parser engine)
- knowledge_create.go: before building the splitter config for manual
markdown
Converters keep their existing calls (idempotent no-op).
* fix(chunker): bound Markdown link/image protected patterns to one line
The unbounded [^\]]* / [^)]+ character classes in the Markdown link and
image protectedPatterns also matched newlines, so a stray '[' left behind
by OCR (e.g. a full-width 【公章】 mis-read as half-width) could pair with
a distant '](' and swallow whole paragraphs as one protected atomic span.
buildUnitsWithProtection then treated the region as a single unit and
mergeUnits could not split it, defeating chunking (observed 1 chunk of
1753 runes where 16 chunks were expected).
Exclude newlines and bound the link text (<=200) / destination (<=500),
matching CommonMark's rule that neither may span a blank line.
Regression: protected_spans_test.go — same input now yields multiple
chunks with max 125 runes (chunkSize=128) instead of a single 1753-rune
chunk; well-formed single-line links/images stay protected.
* fix(ocr): keep non-GFM HTML tables chunkable across all ingest paths
Conversion fallbacks for <br>, nested lists, and one-column tables still
left single-line HTML, so MinerU output could hit the same oversized-chunk
bug. Split those rows, accept one-column GFM, skip fenced examples, and
normalize before image resolution so cell images are not escaped.
* style: wrap long lines and avoid shadowing max in table-chunk tests
Pre-push lll/revive failed on the PR diff versus origin/main.
---------
Co-authored-by: neoczr <neoczr@users.noreply.github.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
* feat(agent): persist compaction checkpoints across turns
Agent compaction only lived inside the turn that ran it: the summary was
discarded at turn end and LoadAgentHistory rebuilt raw history from the
messages table, so every later turn past the threshold paid for another
summarization of nearly the same history.
A compaction whose history summary ends exactly on a stored turn is now
written onto that turn's assistant message (messages.context_checkpoint).
LoadAgentHistory starts from the newest checkpoint: the summary replaces
that turn and everything before it, and later turns are replayed verbatim.
The compactor already treats a leading summary as the previous summary, so
checkpoints fold forward incrementally.
History messages carry their turn's assistant message ID (chat.Message
TurnID, engine-internal) so compaction can tell whether its cut lands on a
stored-turn boundary. Nothing is persisted when the cut ends inside a stored
turn, covers only the live turn, or the history summary degraded to a raw
archive; a split turn's prefix summary is never persisted because that turn
is replayed whole next time. Writes are column-scoped and best-effort, and a
failed lookup falls back to history without a checkpoint.
Storing the checkpoint on the covered turn keeps the boundary implicit, so
session forks copy it with the turn and deleting the turn drops it. The
HistoryTurns cap still applies, now to the turns after the checkpoint.
* test(skill): wait for the background install in the name-conflict test
InstallSkill queues the install on a goroutine, and the fixture row already
starts at installing, so reading the status right after the call raced the
mock install and failed whenever it landed on ready first. Wait for the run
to finish on the reused row instead, as the other install tests do.
* feat(agent): size agent history by tokens instead of turn count
Agent history was capped at history_turns (default 5) because every turn
used to rebuild raw history and compaction could not persist. With
compaction checkpoints persisted, the cap is no longer what keeps a session
inside the window, and it threw away context that would have fit.
LoadAgentHistory now loads the newest checkpoint summary plus the newest
turns after it that fit a token budget: the run's compaction threshold
(agent.HistoryTokenBudget, the window less the reply reserve). History
that large already triggers the first round's compaction, which persists a
new checkpoint, so compaction rather than a turn count bounds the session.
Loading more would only hand that compaction a summarizer input larger
than the model can read.
Rows are paged backwards by (created_at, id) and the read stops at the
checkpoint or once the budget is full, so a long session is never read
whole. Kept turns stay contiguous, the newest turn is kept even when it
alone exceeds the budget (compaction splits it), and a turn cut by the read
is left out rather than replayed without its original question.
Agent mode no longer reads history_turns: AgentConfig.HistoryTurns is
removed, the editor hides the input in smart-reasoning mode and explains
the automatic behavior, and the smart-reasoning builtins drop the setting.
Quick-answer (KnowledgeQA) keeps its turn cap since it has no compaction.
* Revert "feat(agent): size agent history by tokens instead of turn count"
This reverts commit d05fb6e5f1f161813a88e9127609f7a059f38f12.
Loading history up to the compaction threshold lets the loader drop the
oldest turns before compaction can trigger: whenever a single turn's
replay is larger than the system prompt plus the new question, the
trimmed history stays under the threshold, nothing is summarized, and the
session degrades into a large sliding window that never gets a
checkpoint. A turn-by-turn simulation of LoadAgentHistory with the real
compactor showed zero compactions over 60 turns in those cases.
Keep the persisted checkpoints with the HistoryTurns cap for now; token
budgeted history returns in a separate change with a budget that leaves
the overflow to compaction.
* fix(skills): keep retries and catalog installs from rewriting the skill definition
Every install path wrote its archive back to the workspace catalog. After a
skill was re-registered as v2, sandboxes still on v1 are pinned to v1's
archive, so retrying one of them (a failed retry, or a ready skill retried
with guidance) reinstalled v1 and also moved the catalog definition back to
v1 and deleted v2's archive.
Installs now carry where their archive came from. Only an upload or a source
pull is a new definition; a retry or a catalog install uses bytes the
workspace already stores and never rewrites the definition. A retry of a
pinned install keeps its pin and replays its own version, and bytes that
neither the definition nor the row holds are refused with a conflict.
The catalog listing also reports each installation's version so the UI can
show what an upgrade moves from and to.
* feat(frontend): upgrade outdated skill installs from the catalog, sandbox drawer and agent editor
An install whose archive differs from the catalog was flagged as outdated,
but nothing could move it to the catalog version: the install drawer hid
ready installs, and the manage drawer only offered a retry, which replays
the sandbox's own archive.
- Skill settings: cards with outdated installs show an upgrade button that
opens the install drawer with every outdated sandbox preselected. The
add wizard and install drawer also let outdated ready installs be picked,
labelled with the version they move from and to.
- Sandbox skill drawer: an outdated skill shows a "new version" row whose
upgrade button installs the catalog onto that one sandbox.
- Agent editor: skills on the chosen sandbox that are outdated show the
version change and an upgrade button, both for admins only, matching the
admin-only catalog install. The install progress entry, which opens a
drawer backed by admin-only endpoints, is now admin-only too.
- A skill that is being upgraded is no longer pruned from the agent's
selected skills while it is briefly not ready.
The outdated check is shared in utils/skillUpgrade.ts.
* feat(skills): keep the previous version serving while an upgrade runs or after it fails
An install rewrote the row to the new version and marked it installing as
soon as it started, and the agent is only offered ready rows. So an upgrade
took the skill away from every sandbox for its whole run, and a failed
upgrade kept it away, even though the image pointer only moves on success
and the previous version was in the image the whole time.
The row now records what it described when it was last ready (served:
version, description, instructions, digest, the archive its files come from,
and the snapshot that installed it). The agent's usable set, member env
acceptance and the @-mention list read TenantSkillEntity.ServedView, which
is that version marked ready while the row is installing or failed. A
successful install clears it and releases its archive; a failure or a stop
keeps it.
Keeping that consistent:
- The served archive counts as a reader when a bundle is released or
replaced by a re-registered catalog, so v1's files stay readable while v2
installs.
- The reaper and a failed removal put the served version back when the image
never moved past it, instead of labelling v1's files with the upgrade's
metadata. A stuck upgrade whose chain cannot be read is left alone, since
v1 keeps serving meanwhile.
- The env declaration is stored after the pointer moves rather than before
the snapshot, so an upgrade that fails late cannot drop the admin value of
a variable the previous version still reads.
Migration 000104 adds tenant_skills.served. The console says "upgrading,
still running 1.1.0" or "upgrade failed, still running 1.1.0", the agent
editor keeps such skills selectable, and the upgrade texts no longer warn
that the skill goes offline.
* fix(skills): offer only the upgrade to installs the catalog has moved past
Review follow-up on the upgrade flow.
- An install that differs from the catalog, ready or failed, is now offered
the upgrade and not the retries. The retry icon and the retry-with-guidance
box both replay the archive the install is pinned to, so next to an
Upgrade button they reinstalled the version the operator came to leave
behind. A failed first install of v1 whose catalog has since moved to v2
used to retry v1 from the drawer while the agent editor's Install went to
v2; every entry now goes to v2, and the agent editor labels it "Upgrade".
The drawer explains that the failed install is replaced by the catalog
version.
- A catalog install or a retry reuses the definition that already holds its
bytes instead of re-running the catalog upsert, which rewrote the row and
bumped its updated_at every time the skill was installed somewhere. The
upsert still runs for a definition that does not exist yet or whose object
was lost.
* feat(audit): show API key name on knowledge-base activity
API-key operations currently look like the first tenant user or
"system". Snapshot the key id/name into activity details so the
feed can tell which key initiated the change, without restoring
key authorization onto async workers.
Fixes#3402
* fix(frontend): show activity API key on a second line
The 132px actor column truncates "user · API Key · name" into one
ellipsis. Keep the person on the first line and the key name below.
---------
Co-authored-by: MercuialC <MercuialC@users.noreply.github.com>
Co-authored-by: root <root@localhost.localdomain>
* fix(agent): search bound knowledge bases before answering, and pass suggested-question origins
A user picked the suggested question "为什么在比较图形时需要谨慎处理?"
and the agent answered in round 1 with no tool call, claiming it had
searched the knowledge base and found nothing, then explained misleading
statistical charts. The question had been generated from "Corners Don't
Look Like That: Regarding Screenspace Ambient Occlusion" in a bound base,
where "comparing graphs" means SSAO falloff curves.
Two things let that happen: the grounding prompt said "Stable general
explanations need no lookup", and nothing told the agent where the
question came from - the base in question had no profile, only two
recent titles in the directory, and the click sent the text alone.
Prompt
- With knowledge tools available, search bound bases before answering a
question about a topic, even one that reads like general knowledge;
directory profiles and recent titles cannot prove a base lacks it.
- The general-explanation exemption applies only without bound bases.
- Say you searched or found nothing only when a search tool ran.
Question origin
- Suggested questions from FAQ/document chunks also return knowledge_id.
- Picking one on the home or chat page sends question_origin
{knowledge_base_id, knowledge_id} with that question only.
- The service keeps the origin only when its base is inside the turn's
resolved scope and the document belongs to that base; it never
changes the scope (unlike knowledge_base_ids, which narrows it).
- The engine renders it in runtime_context as <question_origin> with the
base and document handles and a note to search it before answering.
* fix(agent): keep question origins across file/tag scopes and bind them to the send
Review follow-up for the suggested-question origin.
- Scope check: the origin was kept only when its base was in the resolved
KB list, which is empty for file-only and tag-only scopes even though
their bases are searched. It is now checked against SearchTargets after
they are built; the document is kept only when a target for that base
covers it (unfiltered base, or a document scope listing it).
resolveQuestionOriginInfo re-checks SearchTargets, looks up the name of
a base that is not in kbInfos, and the renderer omits an unknown name.
- Frontend: the origin travelled through module-level "pending" state, so
a send that returned early (agent not ready, upload in progress,
steering) left it behind and a second pick overwrote it. It now rides
with the one send: triggerSend(text, options) -> createSession ->
send-msg -> sendMsg / changeFirstQuery.
- RetrieveKBOnlyWhenMentioned: suggestions are still generated from the
agent's bases, but picking one retrieved nothing. The origin now
selects its base, as an @mention would, when the agent may read it.
- API sync: question_origin in docs/api/chat.md and swagger;
QuestionOrigin on the Go SDK's KnowledgeQARequest/AgentQARequest and
knowledge_id on SuggestedQuestion.
Repeating the same query in another mode is usually wasted, but keyword
mode can still recall identifiers that hybrid never surfaced. Stop telling
the model that switching mode cannot help.
The agent searched seven times for "Echo 开放权重模型 降低成本". Hybrid
retrieval ranked the right chunk first (RRF rank 0) every time, but the
rerank model saw only the chunk text, which never names Echo or cost;
it scored 0.0025 and all 18 candidates fell below the 0.15 fallback
floor. The tool reported "No matching chunks", and the model cycled
hybrid -> semantic -> keyword, which cannot help since every mode shares
the same rerank check. Scoring the same chunk with its document title
("Show HN: Echo - ... at 1/3 the cost using open-weight models") gives
0.45, and 0.55 for the query written as a question.
- Rerank passages are title + enriched chunk (rerankPassage); FAQ chunks
keep their own question, and the pattern filter still sees chunk text
only.
- When retrieval found candidates but the rerank model rejected all of
them, the empty statement says so and that switching mode will not
help, instead of "No matching chunks".
- The tool description, query schema and agent system prompt ask for a
complete natural-language question in hybrid/semantic mode rather than
a keyword list, and drop "switch mode" as the retry advice. Every
keyword engine matches any token (OR + BM25), so a question costs the
keyword leg no recall; on the same 20 bases the Echo chunks rank the
same for both query styles.
read_document's in-document query was matched as one literal string, so a
keyword query such as "sunflower f(n,k)" found nothing in a 708-chunk
document even when both words were there. The zero-match result gave the
model no next step, and the UI rendered it as "0 / 708 chunks loaded",
which reads like a failed load.
Paging mode also had no size bound: the registry truncates
ToolResult.Output, but the model sees the result rendered from Data, so a
limit=100 page reached ~68k characters in the model context.
- A plain query is split on whitespace; a chunk matches when it contains
every word, in any order. regex=true still matches one expression.
- A zero-match result carries a <hint> suggesting fewer/different words
or reading a chunk from search_knowledge results.
- A page stops adding chunks at 80% of the output budget (counting
per-chunk rendering overhead); next_offset resumes at the first chunk
not returned, and an oversized first chunk is still returned.
- The UI summary shows in-document search results ("no matches" /
"N matches") and, for offset paging, the chunk range instead of a page
number that no longer holds.
* refactor(agent): consolidate knowledge retrieval tools into search_knowledge, read_document, list_documents
The agent's retrieval surface had grown to five overlapping tools whose
behaviour was scattered across long descriptions, interleaved "tips" in
tool output (most of which modelcontext discarded before the model saw
them), and a grep_chunks implementation that ran an unindexed regex scan
over the chunks table ordered by created_at, so large knowledge bases lost
recall and paid a full-table scan on every call.
Tool surface
- search_knowledge replaces knowledge_search and grep_chunks: one query,
mode = hybrid | semantic | keyword, optional knowledge_base_ids, limit.
Keyword mode runs through the retrieval engine's keyword index instead of
the DB regex scan. Modes unavailable for the scope are refused with a
reason. Rerank, MMR and de-duplication are kept; already_seen markers,
retrieval_statistics and the zero-result "CRITICAL" block are gone.
- read_document replaces list_knowledge_chunks, get_document_info and
wiki_read_source_doc: id accepts a dN or cN handle, always returns the
document header, pages by exact offset (fixing the old floor-to-page
bug), finds passages with query (literal by default, regex opt-in) and
reads neighbours around a chunk with context.
- list_documents pages through one knowledge base, honouring @file/@tag
scope.
- wiki_search takes a single query plus regex, knowledge_base_ids, limit;
literal queries are escaped so the model never reasons about JSON
escaping. Legacy queries / knowledge_base_id are still accepted.
- query_knowledge_graph is registered only when a graph-enabled KB is in
scope and its description/tips were trimmed.
Compatibility
- Retired names in stored allowlists, presets and API calls are mapped to
their successors by NormalizeAllowedTools at registration time; no data
migration. Stored histories with old names still decode, render and
redact. Calls to retired names return a replacement hint.
- modelcontext gains an "id" key space that decodes against both the chunk
and document handle tables (decode-only), plus rendering for the
document header, chunk roles, in-document matches and next_offset.
Prompts and surfaces
- The RAG, wiki researcher and hybrid templates now name the tools and the
search -> read -> cite order; tool descriptions are short and contain no
workflow instructions.
- Engine display names, history redaction, grounding prompt, IM rendering,
the built-in MCP server (search_knowledge gains mode/limit, grep_chunks
becomes keyword search, read_document gains query, wiki_search gains
regex), frontend editor/renderers/i18n, presets, website docs and the
changelog are updated accordingly.
Tests cover name mapping, mode validation and refusal, output shape,
read_document paging/context/query/scope, list_documents, wiki_search
escaping and legacy arguments, modelcontext decoding and rendering, and
registration-time aliasing and graph gating.
* fix(agent): address review findings on read_document, list_documents and model rendering
- read_document: fetch the second page whenever the first page did not
reach the end of the document, so an unaligned window near the end no
longer drops the trailing chunks (also fixes context windows around a
chunk near the end); a query passed together with a cN handle now
searches the owning document instead of being ignored; query mode stops
collecting once the output budget is nearly spent and reports truncated;
the document row is authorized in place instead of being fetched twice;
the context read takes the total from the window fetch instead of a
separate count query.
- list_documents: resolve the page's document scope in one batched tag
lookup instead of one call per row.
- modelcontext: render read_document output once and read next_offset
through the existing intValue helper.
- search_knowledge: point title/file-name lookups at list_documents.
- Tests for the boundary window, chunk context near the end, cN + query,
and budget-bound query results.
* fix(agent): close compatibility gaps in the consolidated retrieval tools
- Wiki-only scopes: read_document and list_documents are registered
whenever a wiki KB is in scope, since every KB stores chunks. Their
capability requirement includes wiki but is marked auxiliary, so they do
not widen a RAG agent's derived "all knowledge bases" filter to wiki-only
bases (Go and frontend derivations both honour it).
- search_knowledge resolves the retrieval mode per knowledge base instead
of refusing the call: FAQ and vector-only bases answer keyword requests
semantically, keyword-only bases answer semantic requests by keyword.
Results carry requested_mode and mode_fallbacks, the empty-result text
names the fallback, and the call is refused only when no base in scope
has a chunk index.
- The model rendering uses the mode the search actually ran with (it was
always labelled semantic) and emits <mode_fallback> per degraded base.
- list_documents applies the @file/@tag scope before paging: tags are
pushed into the database filter, pinned documents are loaded by ID, and
total/next_page count only readable documents.
- read_document resolves chunk context by chunk_index through a new
ChunkRepository.ListChunkNeighbors, so parent, summary and image chunks
sharing the index sequence no longer shift the neighbourhood; offset is
documented as a reading-order position, not a chunk index.
- wiki_search (agent and MCP) keeps its regular-expression default and
only matches literally when the text is not a valid expression;
regex=false forces a literal match and regex=true rejects invalid input.
- MCP grep_chunks keeps grep semantics: literal terms from the pattern
fetch candidates from the keyword index (semantic for bases without
one), and every returned chunk is verified against the pattern.
* docs(changelog): align the consolidation entry with regex wiki_search and grep_chunks semantics
* docs(mcp): note that grep_chunks recall is bounded by index candidates
* feat(artifacts): add an artifacts library with its own message_artifacts table
Skill-generated files were only reachable from inside the conversation that
produced them. This adds a sidebar entry and a page that lists them across
every conversation the user can see.
Storage: artifacts move out of the messages.artifacts JSONB column into a new
message_artifacts table (PostgreSQL 000103, SQLite 000023), backfilled on
upgrade. The table is now the only source of truth; the old column is kept and
the down migration writes the table back into it. The message repository
loads artifacts for every message it returns and writes them on create,
update (when non-nil) and fork, so callers still use Message.Artifacts.
mod_time is stored as RFC 3339 text because the collector matches files on
nanosecond mtimes, which PostgreSQL timestamps would truncate.
API: GET /api/v1/artifacts lists the latest version of each file (grouped by
session + source path, with version_count), scoped like the session list,
with keyword / file type filters and paging. Downloads keep using the
per-session endpoint.
Frontend: "产物/Artifacts" sidebar entry (hidden without sandbox support), a
library page with type tabs, search, date groups, infinite loading, preview
drawer, download and jump-to-conversation; strings in all five locales. The
sidebar's top-menu paths are now one shared list.
* fix(artifacts): keep the restore hash guard and count versions by URL
RecordRestoredArtifactMtime again only stamps rows whose content hash matches
the restored content (and ignores empty hashes), and never rewrites the hash,
matching MessageArtifacts.WithRestoredMtime. Without the guard a restore of
v1's content also moved v2's mtime and relabelled v2 with v1's hash.
The library's version_count now counts distinct URLs per (session, source
path). A later answer that references an earlier file stores another row with
the same URL, which COUNT(*) reported as an extra version.
* fix(artifacts): scope the library to web chats and clear the legacy column
The library used the session list's default scope, which counts sessions
without an owner as legacy tenant-level rows. IM sessions are created without
an owner, so every member saw the file names and titles of every IM chat
(downloads then 404 for non-admins). The library now applies the same "web"
bucket predicate as the home sidebar, extracted from QueryPaged so the two
cannot drift: no IM mapping (including cleared ones), no embed marker, no
API-key owner.
Migrations 000103 / SQLite 000023 now clear messages.artifacts after the
backfill. Message loads still SELECT *, so the stale copy was read on every
history load. Only rows that carried artifacts are rewritten; the column stays
for instances on the old binary during a rolling upgrade, and the down
migration still rebuilds it from the table.
* fix(migrations): keep the 000103 backfill alive on malformed artifact JSON
A single file_size that is not an integer ("12.5", "") or a created_at that
does not parse ("garbage", month 13) raised a cast error and failed the whole
migration, blocking startup. Casts now go through pg_temp helpers that return
NULL on bad input, falling back to 0 and the message's created_at; valid rows
are unchanged. SQLite casts never raise; its migration test now covers the
same malformed values.
* docs(artifacts): describe the web-only library scope and the cleared legacy column
response_type=error was being emitted when a tool execution failed,
but that type is reserved for internal agent failures (LLM/pipeline
errors), which clients treat as a crashed run. A failed tool call is
still a tool result: both the chat SSE handler and the skill install
transcript now emit response_type=tool_result with success=false and
the error string in metadata, so clients render it as a failed tool
card and let the agent recover instead of treating the run as dead.
Frontend needs no change: useChatStreamHandler already resolves tool
outcome from the success metadata field, history replay rebuilds from
agent_steps, and the API playground stops failing a whole request on
a recoverable tool error.
Second slice of #3338 (tracked as #3345).
Count alignment: CountKnowledgeByKnowledgeBaseID now excludes rows
mid-deletion, mirroring the document list's existing filter — a stranded
delete no longer shows as '4 documents, 3 listed' on the KB card. Failed
rows stay counted: they are visible, actionable cards.
Sweep C: housekeeping now recovers rows stuck at 'deleting' whose delete
task no longer exists. The delete dead-letter callback only flips rows
that are still 'deleting' when retries exhaust, so a worker death
mid-delete or a lost queue left the row hidden from the list forever.
Same recovery contract as the parse sweep: flip to failed with a clear
error so the row becomes visible and actionable; the delete executor is
idempotent and plans by row ID, so a late-arriving task still finishes
cleanly.
Liveness gate: a new TaskInspector probe (HasQueuedDeleteTasksForKnowledge)
matches knowledge:list_delete batch payloads — the per-parse matcher
cannot see them. A live queued/active delete defers the row (backlog,
not stranded); a probe error defers that row; a nil inspector defers the
whole sweep. Lite mode's noop returns false by design: inline deletes
never queue, so a stale deleting row there genuinely has no task.
Tests: sweep matrix (stranded recovered / backlogged kept / fresh
untouched / probe-error deferred / nil-inspector deferred), count
alignment, and the payload matcher table.
Refs #3345, #3338
* feat(mcp): add built-in MCP server with per-workspace endpoints
WeKnora previously exposed itself to MCP clients only through the Python
package under mcp-server/, which binds one process to one API key and
therefore to one workspace, and mirrors the REST API tool-for-tool.
This adds a native MCP server surface to the Go backend:
- New mcp_endpoints table (PostgreSQL 000100, SQLite 000020). A workspace
can publish any number of endpoints, each with its own bearer token
(SHA-256 stored, rotatable, shown once), knowledge-base allowlist, tool
allowlist, default agent and per-minute rate limit.
- Management API under /api/v1/mcp-endpoints (Viewer reads, Admin writes,
API keys need manage_channels) plus a tool catalog endpoint.
- internal/mcpserver hosts one mcp-go Streamable HTTP server in stateless
mode at /mcp/:endpoint_id. tools/list is filtered per endpoint and
tools/call re-checks the allowlist, applies rate limiting and touches
last_used_at. MCPEndpointAuth resolves the token and injects the
workspace, an mcp_endpoint principal and an API-key style scope derived
from the endpoint, so existing knowledge-base and capability checks
apply unchanged.
- Redesigned tool set (12 tools in four groups): retrieval and reading
reuse the agent's native tools; ask runs the default or a named agent
synchronously and returns a cited answer with a session_id for
follow-ups; wiki tools cover search, read and index; write tools are
off by default. Knowledge bases accept ids or names.
- Frontend: new "MCP Server" integrations tab (tab id mcpserver, to
avoid clashing with the existing MCP-client settings section) with
endpoint cards, a create/edit drawer, a one-time token dialog and
ready-to-paste snippets for Cursor/VS Code/Claude Desktop, Claude Code
and mcp-remote. Locales updated in all five languages.
- Deployment capability integrations.mcpserver wired on both sides.
- Docs: website-docs MCP page now leads with the built-in server; the
Python mcp-server README carries a deprecation notice.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(mcp): wire the MCP server into DI/router and tighten authorization
The previous commit lost eight staged files to the repo pre-commit hook,
which resolves ROOT to the main checkout and re-adds the main-checkout
copies of tracked files into the worktree index. This commit restores them
(container providers, RouterParams and route registration, context key,
principal, clone table entry, deployment capability key, SQLite parity
test) and addresses the review findings:
- Knowledge-base scope is authorized, not trusted: endpoint allowlists and
every tool call resolve each knowledge base through access.ResolveKB for
the calling principal (ownership or organization share). Foreign IDs are
rejected on create/update and dropped at call time.
- Write tools mint explicit editor grants via access.KBAccess.WithGrant
before calling the knowledge service, so add/update/delete go through the
same RequireKBWrite path as the REST API. add_document also checks the
grant itself.
- ask no longer accepts agent_id. It runs only the endpoint's configured
agent; tenant-owned agents and user-facing builtins qualify, internal
builtins (wiki fixer, skill installer) are refused both on save and at
call time. Web search is forced off for MCP calls.
- Tool annotations match behaviour: ask is not read-only, add_document is
a mutation.
- CORS allows MCP-Protocol-Version / Mcp-Session-Id / Last-Event-ID and
exposes Mcp-Session-Id for browser-based MCP clients.
- Rate limiter runs its local cleanup goroutine; SQLite migration gains the
deleted_at index for parity with PostgreSQL; non-ASCII endpoint names get
unique mcpServers keys derived from the endpoint id.
- Swagger annotations on the management handler.
- Tests: KB isolation (service and server), write grants, endpoint-agent
restriction, tool annotations, and a router test that mounts the public
/mcp/:endpoint_id surface and checks the deployment capability.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore(mcp): renumber migrations after rebase onto main
main took 000100 / 000020 for chunks_index_diet, so the mcp_endpoints
migrations move to 000101 (PostgreSQL) and 000021 (SQLite); the SQLite
parity test and docs follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(mcp): run document tools under the knowledge base owner tenant
Shared knowledge bases resolved through access.ResolveKB were only half
usable: documents of an organization-shared knowledge base are stored under
the owner tenant, so the document-level tools that queried by the calling
tenant found nothing, and add_document would have created rows under the
caller's tenant inside someone else's knowledge base.
- knowledgeInScope loads documents tenant-agnostically and then requires
the document to belong to an allowed knowledge base and to that base's
tenant.
- scopedKBContext replaces grantContext: it resolves the grant, switches
the execution tenant to the knowledge base owner (KBAccess.Context) and
swaps TenantInfo to the owner so storage/model settings resolve there.
list_documents, read_document and every write tool run under it.
- Locale text and docs no longer claim clients can pass agent_id.
- Go SDK gains MCP endpoint methods (client/mcp_endpoint.go) with docs, and
the Swagger artifacts are regenerated.
- Tests: shared knowledge base read/write under the owner tenant, spoofed
document tenant rejected, viewer share cannot write.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* chore(mcp): renumber migrations after rebase onto main
main now carries 000101 / 000021 (knowledge_profiles), so the
mcp_endpoints migrations move to 000102 (PostgreSQL) and 000022 (SQLite);
the SQLite parity test expects version 22 and the docs follow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(mcp): guard last_used touch without a repository and wrap SDK line
The transport tests build the server without a repository; the background
last_used_at write then dereferenced a nil interface and, on the slower CI
runner, took the test binary down. touchLastUsed now returns early when no
repository is wired, and the tests inject a recording repository and assert
the touch instead. Also wraps a 123-column SDK signature for the client
module's lll check.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(knowledge): derive knowledge base descriptions from document profiles
Knowledge bases only had a hand-written description, which most users
leave empty, so agents bound to several knowledge bases had nothing to
route on and searched all of them. This adds a generated description
that is derived from per-document data instead of maintained by a model.
Document profile: the summary call now returns JSON with a 2-3 sentence
summary plus a gist, 3-5 topic keywords, a document type and one
typical question (knowledges.profile). Input is capped at 8k chars and
output at 1024 tokens; plain-text replies from custom templates still
work and simply carry no profile. The summary chunk embeds the gist and
short summary.
Knowledge base aggregate: a database-only fold over those rows (counts
per file type, tag, normalized topic and doc type; stratified samples of
titles and questions; a hash of the inputs). Deleting, moving or
re-parsing a document changes nothing but the rows, so the next rebuild
is exact without any model judgement.
Description text: one small model call over the aggregate yields a
gist, merged topics and typical questions (knowledge_bases.
generated_profile). The call is skipped when the aggregate hash is
unchanged; failures keep the previous text and record the error; an
empty knowledge base clears it. The manual description is never
touched, and the agent runtime context renders both so the model can
pass knowledge_base_ids instead of fanning out.
Refreshes are requested from summary task exit, post-process completion,
deletion, cross-KB moves and enabling the option, debounced per
knowledge base in 30 s windows via kb:profile on the summary queue.
POST /knowledge-bases/:id/profile/generate regenerates synchronously.
profile_config (enabled, model_id, custom_instructions) opts a document
knowledge base in; the settings dialog shows the generated card with
regenerate and adopt-as-description actions in five locales. Search
results now log hits per chunk type so the value of indexing summary
chunks can be measured.
Migrations 000101 (PostgreSQL) and 000021 (SQLite) add the columns; the
SQLite schema test expects the new version and columns.
* fix(knowledge): keep generated descriptions out of duplicates and refresh clones
DuplicateKnowledgeBase copies the source row through JSON, which carried
generated_profile onto a knowledge base that has no documents yet, so
an empty duplicate advertised the source's topics. The duplicate now
starts without a generated description and keeps profile_config so it
derives its own once documents arrive.
CopyKnowledgeBase builds the target field by field and skipped
profile_config, and the clone task never asked for a description after
copying the documents, so a clone stayed undescribed until its next
upload. The target now carries a copy of profile_config and both clone
paths request a refresh on completion.
* fix(knowledge): refresh the loaded clone target and carry document profiles
ProcessKBClone requested the description refresh with the reservation
stub it builds before CopyKnowledgeBase runs. For the main create path
that stub has neither a type nor profile_config, so the refresh helper
rejected it and nothing was enqueued. The refresh now uses the loaded
target returned by CopyKnowledgeBase.
cloneKnowledge copied the summary text but not the structured profile,
so even a scheduled refresh would have aggregated titles without topics
until every clone was re-summarized. Cloned documents now carry an
independent copy of the profile.
* refactor(wiki): consolidate source_refs parsing and resolution
source_refs entries ("uuid" or "uuid|title") were parsed by ten
independent implementations that disagreed on whitespace handling, on
whether a model-supplied title could be trusted, and on how a leading
pipe was treated. The scope-enforced and legacy agent resolvers also
duplicated the lookup and the routing step looked every document up a
second time.
- types.ParseWikiSourceRef / FormatWikiSourceRef / WikiSourceRefMatchesKnowledge
are now the only readers and writer of the format.
- resolveWikiSourceDocuments replaces resolveAuthorizedSourceRefs,
resolveSourceRefs and wikiKnowledgeBasesForSourceRefs: one lookup per
document, titles always rebuilt from server data, dedupe by canonical ID,
unknown documents fail closed in every mode.
- removeSourceRef now matches padded or titled refs of a deleted document.
* refactor(data_analysis): expose the selected document as a fixed table name
The tool required the document ID inside the SQL text and blindly
string-replaced it with the physical DuckDB table name, which forced the
model-context codec to rewrite handles inside quoted SQL and let the
UUID-derived k_<uuid> table name reach the model through stored schema
summaries.
The document is now selected only by knowledge_id. Every query runs on a
dedicated connection where "dataset" is a temporary view over the
physical table (DuckDB temporary objects are connection-local, so
concurrent sessions stay isolated). The physical name remains accepted
for internal callers; any other table fails validation with an error
naming the table the model should use. Schema summaries, the schema
tool, the analyst prompt and the tool schema all name the same table, so
data_analysis.sql no longer carries source handles and its text codec
policy is removed.
* feat(modelcontext): report raw identifiers that survive encoding
Text-level compaction exists because raw UUIDs reach model context through
producers the registry never sees: error strings, slugs, results of
unknown shape. Retiring it requires knowing where those producers are.
Registry.LeakedIdentifiers scans an encoded message list for UUID-shaped
values in every application-composed field (user messages are skipped),
classifies each as registered (codec missed a rewrite) or unregistered
(producer emitted a raw ID), and SummarizeLeaks renders one line per
field. The agent engine and the three RAG pipeline call sites log the
report as a warning after EncodeMessages.
* fix(chat_pipeline): align RAG data analysis with the fixed table contract
The RAG data-analysis plugin still asked the planning model to fill in
knowledge_id and to write SQL from a prompt that exposed the raw
knowledge UUID, then passed the model's JSON straight to Execute. Under
the new data_analysis contract a query that names anything but the
dataset table is rejected, and the plugin logged that rejection at error
level and moved on, so the analysis vanished without any signal. A
"no analysis needed" plan took the same error path.
The pipeline now owns document selection: the model returns only
needs_analysis and sql, the prompt names the dataset table and never
shows the knowledge ID or physical table, and the plugin builds the tool
input itself. A rejected query is fed back to the model once for repair;
a query that still fails is reported through pipelineWarn with the SQL
and the tool error, and "not needed" is logged as such instead of as a
failure.
Covered by unit tests with a scripted model and executor: not-needed,
pipeline-owned knowledge_id, one-shot repair, persistent failure
surfaced, unparseable plan rejected.
* perf(chunk): speed up chunk ingestion and batch updates
Chunk writes were dominated by avoidable work in the repository layer:
- UpdateChunks built one CASE WHEN id = ? chain per column, so a batch of
N rows evaluated 5N branches per row (O(N²)). On PostgreSQL it now joins
against a VALUES list keyed by the primary key (O(N), batched by 1000);
other dialects run one small UPDATE per row inside a single transaction.
- CreateChunks copied Content into SourceContent for every new row, doubling
insert volume. SourceContent is only needed once a chunk is edited and the
edit path already backfills it lazily, so it is left empty on create.
Batch size raised from 100 to 500 and the insert (including the SQLite
seq_id pre-assignment) now runs inside one transaction.
- New UpdateChunkFieldsByIDs applies uniform field values with a single
UPDATE ... WHERE id IN per batch. FAQ import uses it to flip status after
indexing instead of shipping every row's content back, and resolves tag
names per batch instead of per entry.
- Drop three chunks indexes that only slow down writes: chunk_type (low
cardinality), content_hash (never used as a filter) and
(tenant_id, knowledge_id) (covered by idx_chunks_knowledge_enabled).
Migrations 000100 (PostgreSQL) and 000020 (SQLite) with down scripts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* perf(chunk): rewrite chunk batch writes and FAQ import status updates
Go changes belonging to the chunk write-path optimization. The previous
commit on this branch only carried the migrations because the repo's
pre-commit hook re-staged files from the main checkout when run inside a
git worktree, silently dropping the staged Go changes.
- UpdateChunks: O(N²) CASE chains replaced by a VALUES join on PostgreSQL
(batched by 1000) and per-row updates in one transaction elsewhere.
- CreateChunks: stop copying Content into SourceContent, batch 500 rows,
run the insert and the SQLite seq_id pre-assignment in one transaction.
- New UpdateChunkFieldsByIDs for uniform, tenant-scoped field updates.
- FAQ import flips status with UpdateChunkFieldsByIDs and resolves tag
names once per batch instead of once per entry.
- Drop the gorm index tag on ContentHash to match migration 000100;
SQLite schema test expects migration version 20.
- SQLite tests for the new and rewritten repository methods.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(session): add fork schema and history queries
为会话分叉增加数据模型与迁移,并提供截至分叉点的消息查询以及会话内的 request_id 配对。
* feat(sandbox): add git, fork snapshots, and bootstrap hook
沙箱镜像安装 git,Docker 快照使用独立的 fork 命名空间,并在首次创建会话沙箱时接入可选 bootstrap。
* feat(session): checkpoint workspace at the end of each turn
每轮结束后对沙箱 /workspace 打 git checkpoint,并避免 fork 后因 mtime 刷新重复挂上产物、以及附件仍绑在父会话而无法还原。
* feat(session): fork conversations and reap orphan snapshots
支持从历史 user 消息分叉会话、首次打开时从快照 bootstrap,并在失败或删除未打开的分叉时回收孤儿快照。
* feat(chat): add session fork entry, prefill, and badges
在用户消息上提供分叉入口,进入新会话时预填原问题,并展示降级提示与来源角标。
* style: add missing newline so session_fork_delete_test.go is gofmt-clean
* fix(session): prune later git history when rolling a fork workspace back
reset --hard 只还原工作区,分叉点之后的 commit 仍可被 checkout。校验 SHA 后丢掉其他 ref 并 gc,避免错误基线。
* style(session): clear golangci-lint findings on the session-fork diff
Pre-push lints the whole PR against main; wrap long lines, check c.Error,
and fix gofumpt/revive/staticcheck so the merge can be pushed.
* fix(chat): show the fork button on hover without shifting the thread
Overlay the action so revealing it no longer expands the user bubble and janks the messages below.
* fix(chat): drop the fork landing banners
Keep the original question prefilled; do not explain sandbox snapshot vs checkpoint on the new thread.
* fix(agent): dedupe artifacts by content hash, not size
Same-path same-size rewrites were skipped as restores. Keep path+mtime as the cheap skip, hash bytes when mtime moves, and stamp the restored mtime onto this session only.
* feat(session): allow forking from an assistant answer
User forks still copy history before the question and prefill it. Assistant forks copy through that reply and reuse its checkpoint, with a degrade tooltip when none exists.
* fix(session): resolve parent attachments from a forked chat
Fork copies attachment IDs onto the new messages but leaves temporary_documents on the parent. Preview and Get now look up those IDs on the fork's copied history instead of a scoped miss.
* fix(agent): rewrite sandbox refs after a content-hash skip
Fork restores change mtime on unchanged files, so Collect returns empty
and named citations were left as sandbox: paths. Resolve those names
against session artifacts, notify only real uploads, and stamp restored
mtime only on the matching content hash.
* fix(session): drop a dead fork snapshot and retry checkpoint rewrite
Create from a missing template now abandons bootstrap so the next
Resolve uses a normal sandbox. A failed checkpoint rewrite no longer
marks the bootstrap consumed, which previously made nested forks
SANDBOX_REPLACED.
* fix(session): load forked parent attachments into the prompt
ResolveForPrompt used session-scoped lookup, so copied parent attachment
IDs passed Get but never reached the model. Use the same resolveInSession
path as preview.
* fix(chat): keep the user-message fork button visible
The control was hover-only, which hid it on touch. Always show it, and
drop the unused degrade tooltip.
---------
Co-authored-by: hailongzhao <hailongzhao@tencent.com>
Co-authored-by: wizardchen <wizardchen@tencent.com>
processInitializationModels inserted models built by toModel(), which
carries no TenantID, so every model the initialization endpoint had to
create landed in the models table with tenant_id = 0. The read path
filters on (tenant_id = ? OR is_builtin = true), which means no tenant —
not even the one that just configured the KB — could ever resolve those
models; the KB stayed bound to unreadable model IDs and document
processing stalled forever (issue #3333).
Set the tenant from the KB that is already in scope before CreateModel.
The reuse path (updating an already-stored model) keeps its stored
tenant. Tests pin both paths.
Closes#3333
InstallSkill returns after starting a goroutine, so asserting
status==installing immediately is flaky on fast CI. Wait for the
fixture run to finish instead; a skip would never write a new snapshot.
SearchMessagesByKeyword filtered with `messages.content ILIKE ?`, which is
PostgreSQL-only syntax. On SQLite (the Lite build) and MySQL the query fails
with `near "ILIKE": syntax error`, so POST /api/v1/messages/search returned
HTTP 500 in both keyword and the default hybrid modes. The session-list
search (QueryPaged) was already made dialect-aware for the same reason; this
path was missed.
Pick the LIKE expression per dialect — ILIKE on Postgres, LOWER() LIKE
LOWER() elsewhere — and add an explicit `ESCAPE ?` bound to likeEscapeChar so
escapeLikeKeyword's backslash escapes are honored on SQLite, which has no
default LIKE escape character (a no-op on Postgres/MySQL, where `\` is
already the default).
Regression tests in message_search_test.go run against in-memory SQLite:
TestSearchMessagesByKeywordMatchesOnSQLite and
TestSearchMessagesByKeywordTreatsWildcardsLiterally both fail on the old code
with the ILIKE syntax error and pass with the fix.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CRrBenkCCr4pacMNwR4NuG
Serply returns Google results through a keyed JSON API, which fills the
gap between Google Custom Search (needs an engine ID and a restricted
scope) and the scrapers. The adapter follows brave.go: hardcoded
endpoint, redirects refused so the key never leaves the official host,
4MB response cap, and per-call country and freshness filters. country
maps to Google's gl parameter and pd/pw/pm/py map to tbs qdr:d/w/m/y.
Serply has no date-range filter, so that form returns an error instead
of being silently dropped. num is clamped to 10 because the API serves
one results page per call.
Wiring matches the Bocha provider: type constant and metadata, registry
factory, service validation, frontend union, swagger enum entries (edited
by hand; a full swag regen drags in unrelated drift), and the web search
docs table.
CheckKnowledgeExists excluded only parse_status='failed'. A row whose
async deletion was in flight (or stranded — worker death mid-delete, or
the dead-letter re-mark skipped because the status had moved on) stayed
at 'deleting' and kept the duplicate check matching forever, so
re-uploading the file returned "document already exists" with no path
back except manual SQL (issue #3338).
A deleting row is on its way out: an upload landing while the delete
task is queued or running ends with exactly one live row whichever way
the task concludes — success soft-deletes the old row, retry exhaustion
marks it failed. Excluding 'deleting' alongside 'failed' is therefore
semantically safe without a staleness threshold.
The duplicate-check status policy is now pinned by a matrix test
(failed/deleting pass, pending/processing/completed still duplicate)
covering both the hash path and the filename/size fallback.
Raw BM25 saturates compositeScore when vector recall is empty, so
normalize by max score after fusion while leaving retrieve traces intact.
Refs: https://github.com/Tencent/WeKnora/issues/3343
* feat(knowledge): add batch file downloads
* fix(knowledge): harden batch file downloads
Skip entries without original files, keep folder paths in the ZIP, map
client cancel to 400, and cap concurrent archives so a full-select
download is usable without 500s or disk pile-up.
* fix(knowledge): format batch-download tests for pre-push lint
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
Queued ProcessDocument tasks still held the old FilePath, and a crash
between the source UPDATE and reparse could leave completed knowledge
pointing at a new file. Dequeue first, flip pending in the same write,
and skip stale worker Saves when the stored path has moved.
Add KnowledgeService.ReplaceKnowledgeFile, which replaces the stored
source file of an existing file knowledge while preserving its knowledge
ID, then re-parses it through ReparseKnowledge. It is groundwork for the
local folder data source (#2873); no existing API, handler or connector
behavior changes.
Object storage is not transactional, so the steps are ordered to keep the
row pointing at an existing file: save the new file, update the source
columns in one UPDATE, reparse, and only then delete the old file. A
failed column update discards the new file; a failed reparse restores the
previous source columns (marked failed, since cleanup may already have
removed the old index) and discards the new file. The row is re-read
before a replacement file is deleted, so a write that committed despite
reporting an error never leaves the knowledge referencing a deleted file.
Identical content under the same name and folder is not re-parsed: changed
metadata is persisted and a DuplicateKnowledgeError is returned, matching
CreateKnowledgeFromFile's duplicate contract. Metadata is merged so
entries such as process overrides survive, and stored overrides are
re-validated against the new file type.
* feat(datasource): add Confluence sync
* fix(confluence): resume full sync and list Server pages without CQL
CQL search silently truncates large spaces and can delete pages that
dropped out of the index. Full-sync checkpoints now keep a deletion
baseline so Asynq retries continue instead of restarting.
* fix(confluence): satisfy package lint for pre-push against main
* fix(confluence): parse Server page lists and keep version expand
The default Server/DC edition unmarshaled /content/page as a nested
envelope, so sync imported zero pages. Pagination next links also drop
expand=version,space, which made later pages versionless and hid edits.
* fix(confluence): classify errors and fail closed on empty listings
Raw Confluence error bodies were copied into sync-log UI, and pagination
next links could leave the configured context path via `..`. Empty
listings against a non-empty baseline now abort instead of deleting.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
* fix(storage): use virtual-hosted addressing for OBS bucket requests
Huawei Cloud OBS has rejected path-style bucket addressing on domain
endpoints since 2023-12-30, so PutObject failed with 404 NoSuchKey while
HeadBucket still passed. The driver also pinned HostnameImmutable=true
via a deprecated custom endpoint resolver, which blocked the SDK from
injecting the bucket label into the hostname.
- Replace the custom resolver with BaseEndpoint and enable
virtual-hosted style for domain endpoints; keep path-style as a
defensive fallback for IP-literal or unparseable endpoints.
- Align checksum negotiation with the S3 driver
(RequestChecksumCalculation: WhenRequired).
- GetFileURL now emits {bucket}.{endpoint-host}/{key} URLs when no
proxy domain is configured, mirroring the client addressing.
- Share one obsS3Options constructor between the file service and the
connectivity check, and add obs_test.go (previously zero coverage).
Closes#3269
* test(storage): assert OBS PutObject hits virtual-hosted URLs
UsePathStyle=false is not sufficient — a HostnameImmutable resolver can still force path-style. Capture the signed request host/path so this cannot regress the way #3272 would have.
* test(storage): keep OBS client tests within lint limits
The new-from-rev pre-push lint also flags the original obs_test.go lines. Wrap long fatals and silence the deprecated EndpointResolver assertion so the fork push can run hooks.
---------
Co-authored-by: wizardchen <wizardchen@tencent.com>
parseYuqueConfig and parseIMAConfig validate DefaultBaseURL through
ValidateConnectorBaseURL, which resolves the host with net.LookupIP.
Their TestMain only whitelists loopback, so the config tests fail on
any machine where www.yuque.com or ima.qq.com resolves to a restricted
address (a local DNS sinkhole, for example), and the result depends on
the network the tests happen to run on.
Whitelist the default host as well, which is what the feishu tests
already do for open.feishu.cn.