Commit Graph

550 Commits (267c847fb86a62c70b4e76ea8db69e8038b7d5ea)

Author SHA1 Message Date
ljl e8882fa403 fix(docs): correct swagger @Router path annotations for agent-chat and knowledge-chat 2026-08-06 14:31:20 +08:00
xiaowang 520be78766 fix(chunking): normalize line endings before splitting / 在分块前统一换行符 2026-08-05 19:24:45 +08:00
wizardchen 1985a59b69 fix(chunking): sync preview swagger and normalize splitter config
Centralize ingestion base splitter defaults in chunker.NormalizeSplitterConfig
so the preview endpoint applies the same normalization as upload. Regenerate
Swagger for parent-child preview fields and add regression tests for single-
level preview, default parent/child sizes, and context headers.
2026-08-05 16:17:48 +08:00
wizardchen 39f4220c13 fix(wiki): ensure prompt language is always resolved
Wiki generation sometimes rendered "Write in ." because async workers
read empty locale payloads without falling back to context or defaults.
Unify language resolution helpers and apply them at enqueue and consume
time so every wiki and related async prompt gets a non-empty language.
2026-08-05 16:14:50 +08:00
xiaowang d2888e9565 fix(chunking): align preview with parent-child splitting / 修复预览与父子分块不一致 2026-08-05 15:09:14 +08:00
wizardchen e71c2dce2e fix(api): address resource_urls review follow-ups
Harden public URL mode: stop logging signed URLs at INFO, copy messages
before rewriting history responses, reuse winning grants on insert races,
sync the Go client resource_urls query param, and add knowledge-search
coverage.
2026-08-05 14:42:51 +08:00
wizardchen eb6ac0110c fix(api): close the resource_urls=public gaps found in review
Follow-up on resource_urls=public. Review turned up one authorization gap plus a
set of correctness problems in the streaming path.

Authorization:

  - Embed channels were documented as excluded, but nothing enforced it. The
    delegating handlers hand the visitor's own gin context to the shared
    handlers, so ?resource_urls=public — and a deployment-wide
    RESOURCE_URL_MODE=public — reached anonymous traffic and produced
    shareable, credential-free URLs that bypass the channel-scoped
    /embed/:channel_id/files proxy. ensureEmbedSession now pins the request
    context to handle mode, which covers every delegated endpoint including
    future ones. The downgrade is silent so a client that forwards the
    parameter keeps working.

  - A knowledge-base-restricted API key is refused with 403. Such a key is
    already denied /files because a raw storage path cannot be bound to its
    allow-list, so handing it anonymous file URLs would reopen that hole from
    the other side. The default handle mode stays available to it.

Streaming:

  - Rewriter.ref holds its lock across resolution. Resolver implementations keep
    an unsynchronised per-provider cache, so the previous unlocked window was a
    real data race for any caller sharing a StreamRewriter between goroutines —
    which its own doc comment invited. Serialising also collapses a concurrent
    duplicate into one signature.

  - Release the holdback buffer on every path that ends a stream while the
    client is still connected: a user-requested stop, an error event, and
    giving up on the event store. Only completion flushed before, so the tail
    generated just before a stop was silently dropped.

  - A released tail carries the metadata of the event it was cut from instead of
    a bare event_id.

  - The maxHeldBytes safety valve aligns to a UTF-8 boundary rather than cutting
    a character in half, and an unclosed "![…](" only holds back a
    whitespace-free destination under 2048 bytes, so prose that merely mentions
    Markdown image syntax no longer stalls the stream behind the buffer.

Also resolves the mode before running a knowledge search instead of after, so an
invalid value no longer costs a full retrieval, and warns once per distinct bad
RESOURCE_URL_MODE value instead of once per process.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
lyingbug 061319581d refactor(chat): build SSE payload after the liveness check
In public resource URL mode, building the payload consumes the chunk into the
holdback buffer, so returning between build and write would drop it. Reusing
emitStreamEvent also removes the duplicated flush-before-completion logic.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
lyingbug ef1f0b36c4 docs(api): document resource_urls and RESOURCE_URL_MODE
Adds a cross-cutting "文件与图片引用" section to docs/api/README.md covering the
parameter, the deployment default, the endpoints it applies to, and the two
caveats that matter in practice: it needs APP_EXTERNAL_URL (or a publicly
reachable storage backend) to produce a link at all, and the links it produces
are time-limited but anonymously readable.

Also annotates the affected endpoints for Swagger and refreshes the generated
files for those parameters only, leaving unrelated drift in the committed
swagger output alone.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
lyingbug 96e28b2438 test(api): cover resource_urls=public across the rewriter, handlers and SSE wire
Includes an end-to-end ContinueStream test that asserts the full SSE wire
output. It caught a leak: the references event carries its results twice, and
the copy in Data holds the typed []*SearchResult that CopyData did not
traverse, so handles reached a caller that asked for public URLs. CopyData now
handles the typed slice, []string and map[string]string forms.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
lyingbug 877a6ebf9d feat(api): add resource_urls=public to return directly loadable file URLs
API responses reference stored files as opaque resource:// handles, so an
integrating app had to make a second authenticated call to /files for every
image before it could render anything.

Add an opt-in that resolves those references server-side into time-limited
HTTP(S) URLs, using the same mechanism the IM channels already rely on:

  - per request: ?resource_urls=public (default: handle, unchanged)
  - per deployment: RESOURCE_URL_MODE=public

Applied to the chat SSE endpoints (knowledge-chat, agent-chat,
continue-stream), message history load, and knowledge-search. Streamed answers
buffer a trailing incomplete reference so a handle split across two deltas is
still rewritten. References that cannot become an HTTP URL (for example local
storage with no APP_EXTERNAL_URL) stay handles, so clients keep the /files
fallback. Embed channels are deliberately excluded: their visitors are
anonymous.

SSE payloads share SearchResult pointers and metadata maps with the stream
replay buffer and the message being persisted, so those are rewritten as copies.

Co-authored-by: lyingbug <lyingbug@users.noreply.github.com>
2026-08-05 14:42:51 +08:00
qilifan 3507156929
feat(storage): support AWS S3 NoAk default credentials (#2008)
* feat(storage): support AWS S3 default credentials

* fix(storage): allow clearing S3 credentials

* fix static checks and trim locale diff

---------

Co-authored-by: qilifan <4359902+qilifan@users.noreply.github.com>
2026-08-05 11:59:02 +08:00
wizardchen bb23444e39 fix(knowledge): sync folder APIs and address review follow-ups
Document folder tree/move/rename endpoints in docs/api, Swagger, and the Go
client; align frontend path normalization with server caps; guard rename
no-ops and duplicate folder creation; enforce KB ownership on folder rename.
2026-08-04 20:45:29 +08:00
lyingbug 5fea7a1695 feat(knowledge): browse folders like a file manager and allow re-filing
The scope switch is gone. It existed because "select a folder" was ambiguous
in a list that could not show folders, and turning that ambiguity into a
control just moved the problem onto the user.

The list now shows what a folder contains: its sub-folders as entries first,
then the documents directly inside it. So the mode follows what the user is
already doing instead of a switch - browsing shows one level, and any active
filter searches the whole subtree flat, with a breadcrumb note saying so. A
document uploaded on its own needs no special row either: it sits at the top
level next to the folders, exactly where it is.

Folders can also be adjusted now, which they could not before - a mis-filed
upload previously had to be deleted and re-uploaded. Documents move via the
row menu or the batch bar into an existing or newly typed folder, and a
folder can be renamed in place, carrying its subtree. Both are plain
folder_path updates: nothing is re-parsed, re-chunked or re-embedded.
2026-08-04 20:45:29 +08:00
lyingbug fa67dfd44b feat(knowledge): store folder path for folder uploads and expose folder tree API
Folder uploads used to encode their relative directory inside file_name,
which made the document list render the whole path as the title and left
no way to browse or query a single folder. The directory now lives in a
dedicated folder_path column (with a backfill for existing rows), the
document list accepts folder_path / folder_recursive filters, and a new
GET /knowledge-bases/:id/knowledge/folders endpoint returns the folder
tree with per-folder document counts.
2026-08-04 20:45:29 +08:00
sunmh 59cbe58330 fix(sessions): 放行 embed 运行时读取自身会话,修复追问建议 500 与刷新丢对话 2026-07-30 17:51:34 +08:00
wizardchen cc846d8e29 fix: harden shared agent source selector validation
Reject invalid agent_source_tenant_id instead of silently falling back, fix a resolveAgent nil panic when a shared selector fails, and add regression tests for wiki write-tool filtering and agent resolution.
2026-07-30 12:28:23 +08:00
wizardchen ab9f705843 fix: harden shared agent source permissions 2026-07-30 12:28:23 +08:00
wizardchen 497e8312e3 fix(knowledge): harden summary refresh after review feedback
Discard stale async refreshes without clobbering newer summaries, queue manual refreshes instead of blocking HTTP, sort edited chunks by index for summaries, escape document metadata in chat headers, and distinguish lookup failures from true staleness.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-28 20:23:48 +08:00
wizardchen 48b661134b fix(chunks): retain questions across content edits 2026-07-28 20:23:48 +08:00
wizardchen 6a439a4c3b fix(chunks): keep edited retrieval content consistent 2026-07-28 20:23:48 +08:00
wizardchen 1b9858ecb8 fix(knowledge): enqueue stale summary refreshes 2026-07-28 20:23:48 +08:00
wizardchen ac3eccddf5 feat(knowledge): support editable chunks and metadata 2026-07-28 20:23:48 +08:00
MidoriKurage 002434bbb3 fix: remove SanitizeForDisplay from chunk handler responses
CleanMarkdown regex patterns (script, onclick=, javascript:, etc.)
unconditionally delete matching substrings from chunk content, even when
they appear in code blocks or inline code.

JSON APIs should return raw stored data; XSS protection is the frontend
markdown renderer's responsibility.

SanitizeForDisplay and CleanMarkdown are retained in security.go.
2026-07-28 18:18:56 +08:00
wizardchen e88564a7db Remove redundant Wiki operation log 2026-07-28 11:18:33 +08:00
wizardchen 9714eaa9a7 fix(im): handle disabled channels in IM callback and improve connection lifecycle management 2026-07-27 20:39:12 +08:00
wizardchen 53ddad57c0 fix(im): harden connection lifecycle across replicas 2026-07-27 20:39:12 +08:00
wizardchen 80a5003cc9 feat(wiki): add page revision history with diff, revert, and manual editing
Store superseded wiki page versions for diff and rollback, attribute edits
to user/agent/pipeline/revert sources, and expose revision APIs plus an
editor UI with optimistic locking on save.
2026-07-27 15:03:24 +08:00
wizardchen 50eb48c344 fix(org): disambiguate shared-space terms and tighten invite search
Search tenants for invite by workspace name only (a user may belong to
multiple workspaces, so username/email matching was ambiguous). Align
organization i18n to distinguish shared spaces from workspaces, fix the
duplicate zh-CN members key, and wrap invite-code generation response in
data so the frontend and Go SDK can read it.
2026-07-24 11:42:27 +08:00
Jack Wang 43839f7830 fix(agent): honor suggested-questions limit parameter
Treat limit as the authoritative upper bound instead of clamping it down
to the agent's Starters.Count, so ?limit=N no longer always returns 6.

Fixes #2238
2026-07-24 10:46:53 +08:00
wizardchen 8cb43473e6 fix(knowledge): restrict source file download to Editor and above
Downloading exposes the original source file, which is more sensitive than viewing parsed content or previews. Align backend route guards, handler checks, and frontend UI so tenant Viewers and org-shared read-only access cannot download.
2026-07-23 22:19:39 +08:00
wizardchen c28cd05c0c fix(models): align model debug upload limit with global MAX_FILE_SIZE
Use the deploy-time file size configuration for model debugger uploads
instead of a hardcoded 20 MB cap, and validate file size on the frontend
before sending the request.
2026-07-23 19:43:06 +08:00
wizardchen 6c794f64cf feat(runtime): add bulk purge for archived queue tasks and chat_attachment i18n
Operators can clear all finally-failed tasks in one queue from the runtime
drawer, with audit logging and scoped API access. Also localize the
chat_attachment queue label in the runtime queue details table.
2026-07-23 16:52:35 +08:00
ttommybot a287808597 fix: prevent API keys from assigning owner role 2026-07-23 16:04:42 +08:00
wizardchen 8ae82573ab fix(mcp): coordinate OAuth token refresh and authorization polling
Prevent concurrent refresh-token rotation across instances, distinguish
expired-but-refreshable tokens from active authorization, and bind OAuth
popup polling to a per-attempt ID so stale tokens cannot satisfy new consent.
2026-07-23 15:35:24 +08:00
zhangzherui 7fde37eeeb fix(search): validate hybrid search query text 2026-07-23 12:05:57 +08:00
wizardchen 8ca9c117bb feat(knowledge): enhance FAQ entry filtering and tagging
- Updated the listFAQEntries function to support multiple tag IDs for filtering.
- Modified the frontend components to accommodate the new tag_ids parameter.
- Enhanced the KbTagManageDrawer and FAQEntryManager components to reflect changes in tag handling.
- Improved API documentation to clarify the new tag filtering capabilities.

This commit enhances the flexibility of FAQ entry retrieval, allowing users to filter by multiple tags for more precise results.
2026-07-23 10:50:32 +08:00
wizardchen b6a3b347da feat(knowledge): enhance FAQ export functionality
- Updated the exportFAQEntries function to support exporting FAQ entries in both CSV and JSON formats.
- Modified the frontend to include options for selecting the export format.
- Enhanced localization files to reflect the new export options in English, Russian, and Chinese.
- Implemented new tests to validate the export functionality for both formats.

This commit improves the flexibility of FAQ exports, allowing users to choose their preferred format for better integration with other tools.
2026-07-23 10:50:32 +08:00
wizardchen a3a0308b62 feat(knowledge): add scoped activity audit trail 2026-07-23 10:50:32 +08:00
wizardchen 4c3ae69b49 feat(wiki): prune empty folders after document retract
Automatically remove wiki folders that become empty when knowledge is
retracted, using the durable finalize lane so pruning waits until ingest
drains. Repository DeleteFolder now atomically checks emptiness to avoid
races with concurrent page or child-folder writes.
2026-07-22 19:27:22 +08:00
wizardchen 0f2c6cee11 fix: normalize API key expiry timestamps to UTC (#2137)
Store authorization timestamps as timestamptz and normalize Unix expiry
inputs to UTC so tenant/platform API keys expire at the intended instant
even when the app process timezone differs from the PostgreSQL session.
2026-07-22 17:09:55 +08:00
wizardchen 23db592c4a feat(api-keys): add platform-scoped API keys for control-plane automation
Introduce platform API keys with capability-based access to tenant management,
system settings, runtime queues, and audit logs, plus admin UI for key lifecycle
and system audit log viewing.
2026-07-21 17:21:35 +08:00
jacentsao 150c07368b feat(im): add Yunzhijia integration 2026-07-21 12:24:33 +08:00
wizardchen 25ef7b6c68 feat(session): gate IM/embed/API channel sessions behind admin
Restrict channel-managed sessions (API keys, IM, embed) to tenant admin
in the Web console for list/read paths, hide channel sidebar buckets from
non-admins, and add CountSessionsBySource for aggregate stats without
exposing session rows to viewers.
2026-07-20 20:31:54 +08:00
KKtwo 3011f132e9 feat: support Volcengine rerank 2026-07-20 19:58:48 +08:00
wizardchen 29bec6c613 feat(session): implement strict owner scope for API-key sessions
Add GetOwnedSession method to enforce strict ownership checks for session mutations, ensuring tenant admins can read but not modify API-key sessions. Update relevant handlers to utilize this method for operations that alter session data, maintaining security and integrity of API-key sessions.
2026-07-20 19:31:02 +08:00
wizardchen 76a4ffe525 feat(session): add admin API session bucket and stabilize sidebar filter
Let tenant admins list and open API-key sessions from the Web console while
keeping per-key isolation for everyone else. Pin the session source filter so
it no longer jumps when switching buckets, and resolve session origin on read
so deep links stay in sync with the active folder after refresh.
2026-07-20 19:31:02 +08:00
wizardchen 380e371c2e refactor: remove Neo4j-based conversation memory feature
Drop the memory pipeline, API fields, settings UI, and embed toggles so chat no longer depends on Neo4j graph storage for episodic recall.
2026-07-18 23:41:48 +08:00
qilifan 9afb8b7b68 feat(im,datasource): support Lark, Feishu's international edition
Lark and Feishu are the same product deployed on two isolated clouds
(open.larksuite.com and open.feishu.cn). WeKnora touches both from two
places — the IM bot and the Wiki data-source connector — and this brings
Lark to parity across both. Neither API surface differs, so rather than
duplicate either implementation, each is parametrized by a Region and
registered twice.

IM (internal/im/feishu):
  Every API call routes through a.api(), which prefixes the region's
  host, and the WebSocket client passes larkws.WithDomain — the SDK
  defaults to open.feishu.cn, so Lark long connections would otherwise
  fail to authenticate.

Data source (internal/datasource/connector/feishu):
  Lark was half-supported: the client honoured a base_url override and a
  LarkBaseURL constant existed, but the connector registered only as
  "feishu" and the UI never exposed base_url, so Lark wikis could not be
  configured at all. Lark is now its own connector type. base_url is kept
  as an explicit override so data sources that pointed a "feishu"
  connector at open.larksuite.com keep working.

Bugs found and fixed along the way:

- Wiki resource links were hardcoded to https://feishu.cn/wiki/{token} in
  four places, so a Lark wiki linked into the wrong cloud entirely.
- Bot identity is now platform-prefixed (feishu:cli_x vs lark:cli_x).
  BotIdentity carries a unique index, so without this a Lark channel
  would be rejected as a duplicate of an unrelated Feishu bot sharing an
  app_id.
- The card image cache is now keyed by app_id. image_keys are issued per
  app on one cloud; a Feishu key handed to a Lark app fails the card
  update with code=200570.
- The datasource doc claimed Lark "adapts automatically" — it could not,
  since the UI had no way to reach base_url.

Permissions are where the clouds genuinely diverge. Identical APIs do not
mean identical permission catalogues: aily:file:read, aily:file:write and
corehr:file:download exist on Feishu but not on Lark, so pasting Feishu's
scope JSON into Lark fails to import. That JSON is also over-broad (it
covers products WeKnora never calls) and simultaneously incomplete (it
omits the drive:*/docx:* scopes the connector needs). Lark therefore gets
its own list, derived from the 17 endpoints the code actually calls and
checked identifier-by-identifier against Lark's own API reference, plus a
full Feishu→Lark mapping table marking each scope verified, unverified,
or absent. Both features use tenant_access_token only — no user_access_
token exists anywhere in the repo — so the user dimension stays empty.

Known limitation: Region.ThinkingText makes Lark's initial card
placeholder English, but once streaming starts the think-block headers
come from the shared MarkdownThinkStyle and are Chinese on every platform
(Lark, Slack, Telegram, Mattermost alike). Fixing that means threading a
locale through FormatIMDisplayContent, which has no platform context — a
cross-platform change, not a Lark one. Documented as such.

The channel-creation 400 message is now derived from validIMPlatforms so
the list cannot drift as platforms are added.

Tests: region invariants and per-region URL construction on both sides,
the Feishu/Lark bot-identity split, platform-to-channel mapping, base_url
override precedence, an end-to-end SendReply against a fake Open Platform
(token fetch, reply, thread-unsupported fallback, hard-error surfacing),
and a ListResources drive asserting wiki links never leak the other
cloud's host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:05:54 +08:00
wizardchen eedd7d4641 feat(agent): enhance suggested questions API with tag scopes
Update the SuggestedQuestionsRequest to include TagScopes for better filtering of knowledge items based on their parent knowledge bases. Modify related API documentation and frontend interfaces to accommodate the new structure. Ensure backend services handle the new tag scope logic correctly, including tests for the updated functionality.
2026-07-17 10:19:43 +08:00