CleanMarkdown regex patterns (script, onclick=, javascript:, etc.)
unconditionally delete matching substrings from chunk content, even when
they appear in code blocks or inline code.
JSON APIs should return raw stored data; XSS protection is the frontend
markdown renderer's responsibility.
SanitizeForDisplay and CleanMarkdown are retained in security.go.
The string-form 'members' key (unused dead code) conflicted with the
nested object form used by OrganizationSettingsModal. Removed the dead
string entries from en-US, ko-KR, and ru-RU locale files to fix esbuild
duplicate key warnings.
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.
Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
Markdown tables — searchable text, instead of being lost or (post-export)
rendered as unsearchable images. Bitable date/datetime cells render in the
table's timezone (configurable via Settings["timezone"], default GMT+8) rather
than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
pipeline, gated by the KB's multimodal setting: the image's external_id always
stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
records) break on an empty page so a malformed has_more response cannot spin
until the task deadline.
Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
"<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
means a single failed attachment can't wipe its siblings, and dupIsSameNode
guards the sweep so a file_hash-only dedup collision against a different node
cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
(never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
metadata external_id prefix scan.
Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
Introduce a reactive cache for source reference titles to improve display and reduce API calls. Implement a new function to parse source reference entries and update titles asynchronously. This change optimizes the rendering of source references in the wiki, ensuring that titles are resolved correctly without leaking file names into citation strings.
Align list hover/active states, spacing, and layout with the session sidebar;
fix revision drawer full-height list pane and top alignment with breadcrumbs.
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.
Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.
This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
Batch wiki reads used to lose middle pages when the registry applied
head/tail truncation to the combined output. wiki_read_page now budgets
per page, caps neighbour link summaries, and names omitted slugs explicitly.
Also align registry truncation with rune-based limits and raise the default
tool output ceiling to 24k characters.
router.go now holds only engine assembly (RouterParams / NewRouter /
trustedProxies, 300 lines); the Register* functions move to domain files —
routes_auth_tenant.go, routes_knowledge.go, routes_chat.go, routes_agent.go,
routes_infra.go — and the five file-proxy surfaces (/files, KB-scoped files,
presigned, presigned-preview, /r/:token grants) move to files.go with the
frontend SPA server in static.go. Pure moves; every route, guard chain and
API-key policy declaration is unchanged, as verified by the existing route
tests and the startup policy self-check.
files.go additionally consolidates the plumbing the four handlers each
hand-rolled: LOCAL_STORAGE_BASE_DIR resolution (localStorageAbsDir),
backend-ID/provider parsing (parseStorageTarget), file_path query validation
(requireFilePathQuery), resource-catalog tenancy mapping
(resolveCatalogResource), tenant-storage resolution with and without the
global-service fallback, and the shared success response including HEAD
short-circuit (streamStoredFile).
Also reattaches the RegisterMCPServiceRoutes doc comment that had been
stranded on top of RegisterSystemAdminRoutes, drops the swagger blank import
duplicated into every split file, and documents why AllowCredentials is
inert under the wildcard CORS origin.
The five near-identical "c.Set + context.WithValue" blocks (JWT branch,
tenantless branch, both API-key attach helpers, EmbedAuth) now go through
one applyAuthSession helper, so the c.Keys surface and the request-context
surface can no longer drift apart. The 140-line JWT branch inside Auth is
split into authenticateJWTUser / resolveTargetTenant, which also removes a
duplicate GetTenantByID roundtrip on the X-Tenant-ID switch path.
Fixes folded in:
- delete GetTenantIDFromContext: it read ctx.Value("tenantID") while the
value is stored under types.TenantIDContextKey ("TenantID"), so it could
never succeed (dead, broken, uncalled)
- isPlatformTenantOptionalAPI: match "/api/v1/system/admin" exactly or as
a "/"-delimited prefix instead of a bare HasPrefix that would also admit
sibling paths sharing the prefix
- 401 for a presented-but-invalid Bearer token now says "invalid or
expired token" instead of the misleading "missing authentication"
- replace the remaining log.Printf calls in the auth path with the
request-scoped structured logger
New tests lock the both-surfaces invariant of applyAuthSession, the
tenantless "no tenant/role keys" contract, and bearerToken parsing.
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
Internal-only rename of the remaining alias-era names (ChunkAlias ->
ChunkHandle, aliasForRealValue -> handleForDurable, realForAlias ->
durableForHandle, OrphanAliases -> OrphanHandles, shortSourceAliasRE ->
shortSourceHandleRE, locals and comments). Exported API names are
unchanged. The README now describes the single-package layout, the
handleTable/streamHold primitives and the sourceKeySpaces policy table.
streamHold captures the shared shape of streaming-safe handle decoding
(withhold a trailing byte run that could still become a handle, decode
what is released, decide the fate of a held tail at close). The resource
decoder, HandleStreamDecoder and the orphan-resource filter become
instances whose hold/flush functions are the original logic moved
verbatim; the citation stream expander stays a dedicated tag parser.
The duplicate res://\d+ regexes merge into resourceHandleShapeRE, all
stream types now live in stream.go, and orphan_stream.go is removed.
handleTable[M] is the single bidirectional primitive behind all model
handles: source chunks/documents/KBs/web pages, res:// resource handles,
and the exported HandleTable (iN issue handles, ingest ref-N/c000
handles). Metadata lives under the same lock as the mapping, web
references dedup on the canonical URL while decoding to the raw URL, and
the append-only counter preserves historical numbering exactly.
Behavior is unchanged and proven by the moved test suites; the per-space
text-rewrite strategies (plain substring for resources, word-boundary
regex for HandleTable, shape-regex lookup for sources) deliberately stay
in their wrappers because each is load-bearing wire behavior. The unused
ChunkReference.Alias field is dropped — handles are derived from the
table, never stored in metadata.
sourceKeySpaces becomes the single table of ID-bearing keys: it drives
handle registration (registerSourceIDByKey) and the alias-decode gate
(walkJSON / unresolved-handle audit), replacing the separate decodableKeys
list that had to be kept in sync by hand. The per-tool sourceIDKeys policy
remains the outer gate; a new test asserts every policy key has a
registration space.
The generic nil-policy wrappers (sourceRegistry.DecodeToolCalls,
EncodeMessages) are removed; the two tests that exercised anonymous-tool
decoding now go through the public Registry with real tool names.
Mechanical move: internal/llmresource/registry.go becomes
internal/modelcontext/resources.go with Registry renamed to
resourceRegistry and StreamDecoder to resourceStreamDecoder. No logic
changes; the unused DecodeResponse method is dropped. Tests move intact.
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.
Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
UnresolvedToolHandles)
Bump version to 0.7.1 across VERSION, frontend package manifests, and the
Helm chart appVersion. Add the 0.7.1 CHANGELOG entry and refresh all README
locales (badge, latest-updates, IM channel list with Yunzhijia).
Regenerate Swagger/OpenAPI docs, refresh the architecture diagram (IM Bots
9 channels), and extend docs/QA.md with entries for the Yunzhijia IM
integration, Volcengine rerank / Zhipu web search providers, and the Neo4j
conversation-memory removal.
This update addresses the issue where wiki summary-page slugs (summary/<knowledgeID>) were being mangled during the citation compaction process, leading to dead links. The changes include:
- Reordering the message encoding process to ensure resource references are aliased before source references, preventing the UUID from being replaced in the summary slug.
- Implementing a new function to selectively replace document IDs while preserving the integrity of summary slugs.
- Adding unit tests to verify that summary slugs remain intact and are not altered during compaction.
This fix enhances the reliability of cross-links in the wiki system and prevents potential dead links from being generated.
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:
- Ingest dedup now groups each new item with only its own trigram
candidates and validates merges against that per-item scope, rejecting
cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
system-owned summary pages, and auto-repairs dead [[slug]] links
(rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
storage handles so they round-trip safely through chat.
Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
Search tenants for invite by workspace name only (a user may belong to
multiple workspaces, so username/email matching was ambiguous). Align
organization i18n to distinguish shared spaces from workspaces, fix the
duplicate zh-CN members key, and wrap invite-code generation response in
data so the frontend and Go SDK can read it.
Avoid resetting upgrade form after destroy-on-close unmounts the textarea,
blur focus before closing the modal, and skip async form updates when closed.
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.
Fixes#2238
Follow-up fixes for the organization state sync work:
- reviewOrganizationJoinRequest no longer increments member_count when
approving a role-upgrade request (only genuine "join" approvals add a
member); callers pass the request_type so upgrades are excluded.
- setCurrentOrganization now merges the detail payload onto the existing
card instead of replacing it, so list-only aggregate badges
(share_count / agent_share_count / pending_join_request_count) are not
transiently blanked when a detail response omits them.
- OrganizationSettingsModal clears the previous organization context when
reopening in edit mode for a different org, avoiding a stale frame
before the detail request resolves.
Adds pure-helper unit tests for mergeById and reviewMemberCountDelta.
Downloading exposes the original source file, which is more sensitive than viewing parsed content or previews. Align backend route guards, handler checks, and frontend UI so tenant Viewers and org-shared read-only access cannot download.
Redesign share-to-space settings for knowledge bases and agents with member-style
search, add popup, and table list; simplify the org picker dropdown. Also refresh
organization settings join-request review with table layout and approval popup.
Redesign organization settings with grouped navigation, table-based member
and shared resource views, and anchored popups for invites and permission
upgrades. Refresh the join-space confirmation flow and soften global input
focus styling to a thin brand border without the heavy outer glow.
Use the deploy-time file size configuration for model debugger uploads
instead of a hardcoded 20 MB cap, and validate file size on the frontend
before sending the request.
Parse wiki search/read/edit/issue tool outputs into structured reference
cards so agent citations and the right-side drawer work consistently with
other retrieval tools.