Commit Graph

2480 Commits (c3bb3ff822d80247782ab1e780ed6cdbee63e99b)

Author SHA1 Message Date
wizardchen c3bb3ff822 refactor(knowledge): tighten chunk popup layout 2026-07-28 20:23:48 +08:00
wizardchen 378523c497 refactor(knowledge): move chunk details into header popups 2026-07-28 20:23:48 +08:00
wizardchen 1a26265611 refactor(knowledge): simplify chunk detail controls 2026-07-28 20:23:48 +08:00
wizardchen 6d39fc7551 refactor(knowledge): refine document chunk interactions 2026-07-28 20:23:48 +08:00
wizardchen 95775ae909 fix(knowledge): bound generated question source IDs 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
MidoriKurage 002434bbb3 fix: remove SanitizeForDisplay from chunk handler responses
CleanMarkdown regex patterns (script, onclick=, javascript:, etc.)
unconditionally delete matching substrings from chunk content, even when
they appear in code blocks or inline code.

JSON APIs should return raw stored data; XSS protection is the frontend
markdown renderer's responsibility.

SanitizeForDisplay and CleanMarkdown are retained in security.go.
2026-07-28 18:18:56 +08:00
wizardchen ffff3fe181 feat(knowledge): configure tags during upload confirmation 2026-07-28 11:23:25 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
wizardchen 6cebaa66f4 feat(frontend): refine user settings menu 2026-07-28 11:16:13 +08:00
wizardchen 9714eaa9a7 fix(im): handle disabled channels in IM callback and improve connection lifecycle management 2026-07-27 20:39:12 +08:00
wizardchen 53ddad57c0 fix(im): harden connection lifecycle across replicas 2026-07-27 20:39:12 +08:00
wizardchen 4826de44de fix(migrations): renumber knowledge metadata index migration to 000076
Avoid duplicate 000075 version conflict with wiki_page_revisions migration.
2026-07-27 17:31:28 +08:00
Binks Riverton cb6e18b952 fix(i18n): remove duplicate 'members' key in organization locale objects
The string-form 'members' key (unused dead code) conflicted with the
nested object form used by OrganizationSettingsModal. Removed the dead
string entries from en-US, ko-KR, and ru-RU locale files to fix esbuild
duplicate key warnings.
2026-07-27 16:09:45 +08:00
nullkey 6492bf6a15 feat(feishu): sync docx via blocks API with per-type downdrill (#2087)
Rework the Feishu datasource connector from whole-document export to the docx
blocks API, so embedded content that export silently dropped is now ingested,
and add the multi-item ingestion infrastructure it needs.

Feishu connector:
- Embedded online sheets and bitables are read via their APIs and inlined as
  Markdown tables — searchable text, instead of being lost or (post-export)
  rendered as unsearchable images. Bitable date/datetime cells render in the
  table's timezone (configurable via Settings["timezone"], default GMT+8) rather
  than a raw epoch-ms integer; a datetime formatter keeps the time component.
- Uploaded attachments (>2KB, parseable types) become standalone knowledge
  sub-items so their text is parsed and retrievable.
- Embedded images become standalone sub-items routed through the KB's VLM OCR
  pipeline, gated by the KB's multimodal setting: the image's external_id always
  stays in SubtreeKeep (toggling VLM off never sweeps a previously-OCR'd image)
  but the bytes are only downloaded/ingested when multimodal is enabled.
- Sub-documents (child wiki nodes) sync as their own nodes.
- The three paginated Feishu loops (document blocks, bitable fields, bitable
  records) break on an empty page so a malformed has_more response cannot spin
  until the task deadline.

Datasource ingestion infrastructure:
- Multi-item subtree reconciliation: a source node fans out into a parent plus
  "<parent>#<kind>#<token>" children (types.SubtreeChildID); on re-sync the
  sweep deletes prior children absent from SubtreeKeep. A per-child keep-set
  means a single failed attachment can't wipe its siblings, and dupIsSameNode
  guards the sweep so a file_hash-only dedup collision against a different node
  cannot delete this node's children.
- config.MultimodalEnabled mirrors the KB's VLM state to the connector per run
  (never persisted); an image a non-VLM KB rejects is skipped, not failed.
- A composite partial expression index (text_pattern_ops) backs the sweep's
  metadata external_id prefix scan.

Verified end-to-end against a live Feishu wiki: docx text, native tables,
embedded sheets/bitables (with timezone-correct dates), attachments, images
(VLM OCR), and sub-documents all sync, parse, and are retrievable.
2026-07-27 16:04:56 +08:00
wizardchen e7bd36f193 style(wiki): use gray hover background on source document links 2026-07-27 15:53:18 +08:00
wizardchen 2403d0f4a8 style(wiki): use neutral color for source document links by default
Match backlink tags: secondary text at rest, brand green only on hover.
2026-07-27 15:53:18 +08:00
wizardchen 6117dc2dce feat(wiki): enhance source reference handling in WikiBrowser
Introduce a reactive cache for source reference titles to improve display and reduce API calls. Implement a new function to parse source reference entries and update titles asynchronously. This change optimizes the rendering of source references in the wiki, ensuring that titles are resolved correctly without leaking file names into citation strings.
2026-07-27 15:53:18 +08:00
wizardchen f84f1c57e4 style(wiki): polish revision drawer and sidebar to match session list UX
Align list hover/active states, spacing, and layout with the session sidebar;
fix revision drawer full-height list pane and top alignment with breadcrumbs.
2026-07-27 15:53:18 +08:00
wizardchen 80a5003cc9 feat(wiki): add page revision history with diff, revert, and manual editing
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
2026-07-27 15:03:24 +08:00
wizardchen 1d64fc6234 fix(knowledge): reconcile wiki state when moving documents across KBs
Cross-KB moves updated knowledges/chunks but left wiki_pages source_refs
behind in the source wiki KB, so the folder tree and wiki graph still showed
moved documents. Run the same wiki cleanup as delete on the source KB and
enqueue wiki ingest on the target for reuse_vectors moves.
2026-07-27 15:02:02 +08:00
MidoriKurage 3b476c07df fix(pipeline): deterministic merge output ordering
Sort mergedChunks after groupAndMergeOverlapping concatenates results
from different merge units. The unit order depends on Go map iteration
in the grouping phase, which is non-deterministic.

Uses the same tie-breakers as filter_top_k.go to restore consistent
ordering at the merge source.
2026-07-27 13:30:21 +08:00
MidoriKurage dfb371154c test(pipeline): add failing test for deterministic merge output ordering
groupAndMergeOverlapping groups results through Go maps whose
iteration order is undefined. The concatenated output has no global
sort, so different runs can return chunks in different sequences
even for identical inputs.

This test exercises cross-unit tie-breaking (Score, KnowledgeID,
ChunkType) with chunks that land in separate merge units. The test
currently fails because the post-merge sort is missing.
2026-07-27 13:30:21 +08:00
wizardchen 4a5fc4487b fix(agent): keep batched wiki_read_page results within output budget
Batch wiki reads used to lose middle pages when the registry applied
head/tail truncation to the combined output. wiki_read_page now budgets
per page, caps neighbour link summaries, and names omitted slugs explicitly.
Also align registry truncation with rune-based limits and raise the default
tool output ceiling to 24k characters.
2026-07-27 13:22:16 +08:00
koishi 9e66389126 Add files via upload 2026-07-27 11:39:06 +08:00
xiaowang a7dc78810c fix(knowledge): clean bound vector stores in batch delete 2026-07-27 11:32:30 +08:00
wizardchen 234110b540 refactor(router): split the 2390-line router.go by domain and dedupe the file proxies
router.go now holds only engine assembly (RouterParams / NewRouter /
trustedProxies, 300 lines); the Register* functions move to domain files —
routes_auth_tenant.go, routes_knowledge.go, routes_chat.go, routes_agent.go,
routes_infra.go — and the five file-proxy surfaces (/files, KB-scoped files,
presigned, presigned-preview, /r/:token grants) move to files.go with the
frontend SPA server in static.go. Pure moves; every route, guard chain and
API-key policy declaration is unchanged, as verified by the existing route
tests and the startup policy self-check.

files.go additionally consolidates the plumbing the four handlers each
hand-rolled: LOCAL_STORAGE_BASE_DIR resolution (localStorageAbsDir),
backend-ID/provider parsing (parseStorageTarget), file_path query validation
(requireFilePathQuery), resource-catalog tenancy mapping
(resolveCatalogResource), tenant-storage resolution with and without the
global-service fallback, and the shared success response including HEAD
short-circuit (streamStoredFile).

Also reattaches the RegisterMCPServiceRoutes doc comment that had been
stranded on top of RegisterSystemAdminRoutes, drops the swagger blank import
duplicated into every split file, and documents why AllowCredentials is
inert under the wildcard CORS origin.
2026-07-27 11:25:23 +08:00
wizardchen a55b09ea82 refactor(auth): unify session-context attachment and split the Auth middleware
The five near-identical "c.Set + context.WithValue" blocks (JWT branch,
tenantless branch, both API-key attach helpers, EmbedAuth) now go through
one applyAuthSession helper, so the c.Keys surface and the request-context
surface can no longer drift apart. The 140-line JWT branch inside Auth is
split into authenticateJWTUser / resolveTargetTenant, which also removes a
duplicate GetTenantByID roundtrip on the X-Tenant-ID switch path.

Fixes folded in:
- delete GetTenantIDFromContext: it read ctx.Value("tenantID") while the
  value is stored under types.TenantIDContextKey ("TenantID"), so it could
  never succeed (dead, broken, uncalled)
- isPlatformTenantOptionalAPI: match "/api/v1/system/admin" exactly or as
  a "/"-delimited prefix instead of a bare HasPrefix that would also admit
  sibling paths sharing the prefix
- 401 for a presented-but-invalid Bearer token now says "invalid or
  expired token" instead of the misleading "missing authentication"
- replace the remaining log.Printf calls in the auth path with the
  request-scoped structured logger

New tests lock the both-surfaces invariant of applyAuthSession, the
tenantless "no tenant/role keys" contract, and bearerToken parsing.
2026-07-27 11:25:23 +08:00
wizardchen 718b7347d0 fix(agent): harden search-target scope and model-handle policies
Single SearchTarget must intersect document whitelists with tag scope
instead of treating tags as an independent authorization path. SQL scope
filters now AND document and tag constraints; Agent tools use an explicit
scopeEnforced flag so empty targets fail closed. Also add missing built-in
tool handle policies with drift tests, cache handle regexes at registration,
encode tool error text, and avoid wikiKBIDs slice aliasing.
2026-07-27 11:05:43 +08:00
wizardchen 6663268504 refactor(modelcontext): unify handle terminology and document the merged package
Internal-only rename of the remaining alias-era names (ChunkAlias ->
ChunkHandle, aliasForRealValue -> handleForDurable, realForAlias ->
durableForHandle, OrphanAliases -> OrphanHandles, shortSourceAliasRE ->
shortSourceHandleRE, locals and comments). Exported API names are
unchanged. The README now describes the single-package layout, the
handleTable/streamHold primitives and the sourceKeySpaces policy table.
2026-07-27 11:05:43 +08:00
wizardchen 69b792398e refactor(modelcontext): consolidate stream decoding on one suffix-hold primitive
streamHold captures the shared shape of streaming-safe handle decoding
(withhold a trailing byte run that could still become a handle, decode
what is released, decide the fate of a held tail at close). The resource
decoder, HandleStreamDecoder and the orphan-resource filter become
instances whose hold/flush functions are the original logic moved
verbatim; the citation stream expander stays a dedicated tag parser.

The duplicate res://\d+ regexes merge into resourceHandleShapeRE, all
stream types now live in stream.go, and orphan_stream.go is removed.
2026-07-27 11:05:43 +08:00
wizardchen 213ca0b42e refactor(modelcontext): rebuild every handle space on one generic table
handleTable[M] is the single bidirectional primitive behind all model
handles: source chunks/documents/KBs/web pages, res:// resource handles,
and the exported HandleTable (iN issue handles, ingest ref-N/c000
handles). Metadata lives under the same lock as the mapping, web
references dedup on the canonical URL while decoding to the raw URL, and
the append-only counter preserves historical numbering exactly.

Behavior is unchanged and proven by the moved test suites; the per-space
text-rewrite strategies (plain substring for resources, word-boundary
regex for HandleTable, shape-regex lookup for sources) deliberately stay
in their wrappers because each is load-bearing wire behavior. The unused
ChunkReference.Alias field is dropped — handles are derived from the
table, never stored in metadata.
2026-07-27 11:05:43 +08:00
wizardchen 286eadac0a refactor(modelcontext): unify source-key gating behind one dispatch table
sourceKeySpaces becomes the single table of ID-bearing keys: it drives
handle registration (registerSourceIDByKey) and the alias-decode gate
(walkJSON / unresolved-handle audit), replacing the separate decodableKeys
list that had to be kept in sync by hand. The per-tool sourceIDKeys policy
remains the outer gate; a new test asserts every policy key has a
registration space.

The generic nil-policy wrappers (sourceRegistry.DecodeToolCalls,
EncodeMessages) are removed; the two tests that exercised anonymous-tool
decoding now go through the public Registry with real tool names.
2026-07-27 11:05:43 +08:00
wizardchen 3b9de5de8e refactor(modelcontext): fold llmreference into modelcontext as source codec
Mechanical move: internal/llmreference becomes in-package code.
registry.go splits into sources.go (handle registration, tool-argument
codec, replay encoding) and citations.go (protocol prompts, <ref/> ->
<kb/>/<web/> expansion, stream expander); model_output.go moves verbatim.

Renames only: Registry -> sourceRegistry, StreamExpander ->
citationStreamExpander, ProtocolPrompt -> sourceProtocolPrompt,
ToolArgumentPolicy -> toolArgumentPolicy. ChunkReference becomes a native
modelcontext type (was an alias). Unreferenced DocumentAlias,
KnowledgeBaseAlias and ExpandResponse are dropped. Tests move intact.
2026-07-27 11:05:43 +08:00
wizardchen c01b00313b refactor(modelcontext): fold llmresource into modelcontext as resource codec
Mechanical move: internal/llmresource/registry.go becomes
internal/modelcontext/resources.go with Registry renamed to
resourceRegistry and StreamDecoder to resourceStreamDecoder. No logic
changes; the unused DecodeResponse method is dropped. Tests move intact.
2026-07-27 11:05:43 +08:00
wizardchen 5e413c6478 refactor(agent): introduce modelcontext registry and fix handle-codec bugs
Unify request-local model-handle handling behind a single
modelcontext.Registry (source citations, durable resource handles, wiki
issue handles) with per-tool argument policies and unresolved-handle
rejection before tool execution.

Also fixes issues found in review:
- decode wiki_read_issue's iN issue_id argument (was never resolved)
- preserve stored wiki page aliases when wiki_write_page omits aliases
- validate http/https scheme before registering url fields as web sources
- gate tool-role MultiContent text behind the tool result policy
- dedup KB IDs in NewWikiScopesFromKBIDs to avoid false ambiguity
- remove dead codec facades (decodeJSON, EncodeMessagesWithPolicy,
  UnresolvedToolHandles)
2026-07-27 11:05:43 +08:00
wizardchen c64a48647c chore(release): prepare v0.7.1
Bump version to 0.7.1 across VERSION, frontend package manifests, and the
Helm chart appVersion. Add the 0.7.1 CHANGELOG entry and refresh all README
locales (badge, latest-updates, IM channel list with Yunzhijia).

Regenerate Swagger/OpenAPI docs, refresh the architecture diagram (IM Bots
9 channels), and extend docs/QA.md with entries for the Yunzhijia IM
integration, Volcengine rerank / Zhipu web search providers, and the Neo4j
conversation-memory removal.
2026-07-24 16:53:25 +08:00
wizardchen 84f65a809f fix(agent): ensure wiki summary slugs are preserved during citation compaction
This update addresses the issue where wiki summary-page slugs (summary/<knowledgeID>) were being mangled during the citation compaction process, leading to dead links. The changes include:

- Reordering the message encoding process to ensure resource references are aliased before source references, preventing the UUID from being replaced in the summary slug.
- Implementing a new function to selectively replace document IDs while preserving the integrity of summary slugs.
- Adding unit tests to verify that summary slugs remain intact and are not altered during compaction.

This fix enhances the reliability of cross-links in the wiki system and prevents potential dead links from being generated.
2026-07-24 15:57:08 +08:00
wizardchen 5d0326f987 fix(wiki): harden slug integrity across ingest and agent write paths
Weak LLMs routinely mangle high-entropy wiki slugs (especially UUID-based
summary/<id> slugs), producing dead cross-links and cross-item dedup
hallucinations. This tightens integrity end to end:

- Ingest dedup now groups each new item with only its own trigram
  candidates and validates merges against that per-item scope, rejecting
  cross-item mispairings; candidate top-K narrowed 20 -> 5.
- Editor path aliases out-link slugs behind low-entropy ref-N tokens and
  translates them back after generation, removing the chance to mangle.
- Agent write tool normalizes/validates slugs, blocks manual creation of
  system-owned summary pages, and auto-repairs dead [[slug]] links
  (rewrite-only, never strips) via bigram/display-text resolution.
- llmresource registry aliases summary/<uuid> slugs the same way as
  storage handles so they round-trip safely through chat.

Adds unit tests for slug validation, aliasing round-trip, dead-link
repair, and dedup rejection rules.
2026-07-24 15:57:08 +08:00
wizardchen d283fabc99 fix(doris): harden Stream Load HTTP with SSRF-safe client
Doris Stream Load previously used a plain http.Client, leaving FE→BE
redirects and stored FE URLs without outbound SSRF checks. Use the shared
SSRFSafeHTTPClient, validate URLs at send time, and only forward Basic
credentials to whitelisted BE hosts after FE 307 redirects.
2026-07-24 15:10:44 +08:00
wizardchen 50eb48c344 fix(org): disambiguate shared-space terms and tighten invite search
Search tenants for invite by workspace name only (a user may belong to
multiple workspaces, so username/email matching was ambiguous). Align
organization i18n to distinguish shared spaces from workspaces, fix the
duplicate zh-CN members key, and wrap invite-code generation response in
data so the frontend and Go SDK can read it.
2026-07-24 11:42:27 +08:00
wizardchen 075b16d6ca fix: prevent TDesign textarea autosize crash in org settings popup
Avoid resetting upgrade form after destroy-on-close unmounts the textarea,
blur focus before closing the modal, and skip async form updates when closed.
2026-07-24 11:06:23 +08:00
Jack Wang 43839f7830 fix(agent): honor suggested-questions limit parameter
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.

Fixes #2238
2026-07-24 10:46:53 +08:00
wizardchen e0b6a71503 fix(frontend): correct optimistic organization state after review and detail loads
Follow-up fixes for the organization state sync work:

- reviewOrganizationJoinRequest no longer increments member_count when
  approving a role-upgrade request (only genuine "join" approvals add a
  member); callers pass the request_type so upgrades are excluded.
- setCurrentOrganization now merges the detail payload onto the existing
  card instead of replacing it, so list-only aggregate badges
  (share_count / agent_share_count / pending_join_request_count) are not
  transiently blanked when a detail response omits them.
- OrganizationSettingsModal clears the previous organization context when
  reopening in edit mode for a different org, avoiding a stale frame
  before the detail request resolves.

Adds pure-helper unit tests for mergeById and reviewMemberCountDelta.
2026-07-24 10:45:12 +08:00
ttommybot 014cf4e545 fix: synchronize organization state after writes 2026-07-24 10:40:33 +08:00
wizardchen 8cb43473e6 fix(knowledge): restrict source file download to Editor and above
Downloading exposes the original source file, which is more sensitive than viewing parsed content or previews. Align backend route guards, handler checks, and frontend UI so tenant Viewers and org-shared read-only access cannot download.
2026-07-23 22:19:39 +08:00
wizardchen 697e0ae04e feat(frontend): polish organization space UI and share-to-space panels
Redesign share-to-space settings for knowledge bases and agents with member-style
search, add popup, and table list; simplify the org picker dropdown. Also refresh
organization settings join-request review with table layout and approval popup.
2026-07-23 21:03:20 +08:00
wizardchen 265556950b feat(frontend): polish shared space settings and join preview UI
Redesign organization settings with grouped navigation, table-based member
and shared resource views, and anchored popups for invites and permission
upgrades. Refresh the join-space confirmation flow and soften global input
focus styling to a thin brand border without the heavy outer glow.
2026-07-23 20:32:29 +08:00
wizardchen c28cd05c0c fix(models): align model debug upload limit with global MAX_FILE_SIZE
Use the deploy-time file size configuration for model debugger uploads
instead of a hardcoded 20 MB cap, and validate file size on the frontend
before sending the request.
2026-07-23 19:43:06 +08:00
wizardchen b2ae22c4eb feat(chat): show wiki tool results in the references drawer
Parse wiki search/read/edit/issue tool outputs into structured reference
cards so agent citations and the right-side drawer work consistently with
other retrieval tools.
2026-07-23 19:41:34 +08:00