WeKnora/pluginsdk/README.md

386 lines
14 KiB
Markdown

# WeKnora plugin SDK (Go)
Build code plugins for WeKnora: web search providers, data source
connectors and document parsers that run as their own process. The module has no dependencies.
| Package | What it is |
| --- | --- |
| `pluginapi` | The extension protocol v1: envelope, errors, stream events, messages. `openapi.yaml` describes it for other languages. |
| `pluginsdk` | Write a plugin: register contributions, then `Serve`. |
| `client` | Call a plugin, as WeKnora does. |
| `conformance`, `cmd/weknora-plugin-conformance` | Check that a plugin speaks the protocol. |
Writing Python? [`python/`](python/README.md) is the same SDK for Python
3.9+, with only the standard library.
## A connector in 30 lines
```go
package main
import (
"context"
"log"
"github.com/Tencent/WeKnora/pluginsdk"
"github.com/Tencent/WeKnora/pluginsdk/pluginapi"
)
type notes struct{}
func (notes) Validate(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig) error {
if cfg.Credentials["token"] == nil {
return pluginapi.InvalidConfig("token is required", map[string]string{"credentials.token": "required"})
}
return nil
}
func (notes) ListResources(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig, parent string) ([]pluginapi.Resource, error) {
return []pluginapi.Resource{{ExternalID: "inbox", Name: "Inbox"}}, nil
}
func (notes) Fetch(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig, in pluginapi.FetchInput, s *pluginsdk.Stream) (*pluginapi.Cursor, error) {
if err := s.Item(pluginapi.FetchedItem{ExternalID: "n1", Title: "First note", Content: []byte("# Hello")}); err != nil {
return nil, err
}
return &pluginapi.Cursor{State: map[string]any{"after": "n1"}}, nil
}
func main() {
p := pluginsdk.New(pluginsdk.Info{ID: "acme.notes", Version: "1.0.0"})
p.Connector("notes", notes{})
log.Fatal(p.Serve())
}
```
What the SDK does for you:
- **Handshake.** Started by WeKnora's plugin host, the plugin listens on the
socket it is given and accepts only the host's token.
- **Remote mode.** Run on its own, it listens on `WEKNORA_PLUGIN_ADDR` and
verifies signed requests with `WEKNORA_PLUGIN_SECRET`.
- **Per call.** Each call's deadline reaches your context. Returned
`pluginapi` errors keep their code; anything else, and any panic, becomes
`internal`.
Every call carries the tenant and the configuration, secrets already
decrypted, in three separate scopes:
- `call.Config.System`: set by the system administrator.
- `call.Config.Tenant`: set per workspace.
- the instance: `cfg`, or `call.DecodeInstance`.
Keep plugins stateless and never cache a tenant's credentials across calls.
## Parsers
`p.Parser(id, ...)` turns a document (`in.Content`, base64 on the wire) into
Markdown. WeKnora chunks the text and stores the images the Markdown
references: return them in `Images` with an `OriginalRef` matching the
`![](...)` target. Declare the extensions it handles in `plugin.yaml`
(`fileTypes: [srt, vtt]`). Return a retryable error (`unavailable`,
`rate_limited`) to have the document retried later; any other error fails
it for good.
## Pages
A plugin can add pages to the app:
- `pages`: toolbox tabs.
- `settingsSections`: sections of the settings dialog.
- `kbTabs`: tabs on each knowledge base.
Each is an HTML `entry` under `ui/` in the package. It runs in a sandboxed
iframe and talks to the app through
[`@weknora/plugin-ui`](../packages/plugin-ui). A page's requests reach the
plugin's `UI` handler:
```go
p.UI(func(ctx context.Context, call *pluginsdk.Call, req pluginapi.UIRequest) (*pluginapi.UIResponse, error) {
// req.Mount ("pages/links"), req.Method, req.Path, req.Body, req.Role
return pluginsdk.UIJSON(200, links)
})
```
WeKnora refuses a caller below the page's `minRole` before the call.
The handler can check `req.Role` for anything finer.
## Events
List the events a plugin wants in `permissions.events`. The administrator
sees them at install.
| Event | When | Data |
| --- | --- | --- |
| `knowledge.ingested` | A document finished processing | `KnowledgeEventData` |
| `knowledge.failed` | A document failed for good | `KnowledgeEventData` with `error` |
| `knowledge.deleted` | A document was deleted | `KnowledgeEventData` |
| `chat.answered` | An answer completed | `ChatEventData`, including the question and answer |
```go
p.OnEvent(func(ctx context.Context, call *pluginsdk.Call, ev pluginapi.EventDelivery) error {
// ev.ID stays the same across retries: deduplicate on it.
return nil
})
```
Delivery is at least once, in the background:
- A workspace receives events only while it has the plugin switched on.
- A retryable error (`unavailable`, `rate_limited`) or a timeout brings
the event back later with the same `ID` and a higher `Attempt`.
- Any other error drops the event.
- Ignore types you do not know: the list only grows.
## Webhooks
`contributes.webhooks` gives each workspace a secret URL per webhook,
under `/api/v1/plugin-callbacks/`. Workspace admins copy it from the plugin
center, and it is in `call.Context.Webhooks` when WeKnora knows its
public address.
```go
p.Webhook("events", func(ctx context.Context, call *pluginsdk.Call, req pluginapi.WebhookRequest) (*pluginapi.WebhookResponse, error) {
// call is the workspace the URL belongs to; verify the sender with its configuration.
return &pluginapi.WebhookResponse{Status: 204}, nil
})
```
WeKnora answers 404 unless all of these hold:
- the URL is genuine;
- the webhook exists;
- the workspace has the plugin on.
Bodies are limited to 1 MB, with 20 calls a second per URL.
## Agent tools
Agent tools are MCP tools. A plugin with code can serve them itself: declare
an `mcpServers` contribution without `mcp.url`, then add the tools.
```yaml
contributes:
mcpServers:
- id: tools
name: { en-US: ACME Issues }
toolViews: # optional: how the chat shows structured results
search_issues:
view: table
items: issues
columns: [{ field: key, link: url }, summary, status]
```
```go
p.Tool("tools", pluginapi.Tool{
Name: "search_issues",
Description: "Search issues",
InputSchema: json.RawMessage(`{"type":"object","properties":{"text":{"type":"string"}}}`),
}, func(ctx context.Context, call *pluginsdk.Call, args json.RawMessage) (*pluginapi.ToolResult, error) {
// call.Config.Tenant is the workspace's plugin configuration.
return pluginapi.StructuredResult(map[string]any{"issues": rows}, "3 issues ..."), nil
})
```
How it works:
- **Discovery.** Every workspace that switches the plugin on gets the server
as a read-only MCP service. Agents use it like any MCP service, with the
same per-tool approval settings.
- **Transport.** Messages reach the plugin at `/v1/mcp/{id}` inside the usual
envelope. Each call therefore carries the workspace's context and
configuration (OAuth fields already swapped for tokens), and the server
keeps no session.
- **Results.** The model reads the text content. Return failures the model
should see as `pluginapi.ToolError` (or any error); they become results with
`isError`.
- **`toolViews`** map a tool's `structuredContent` to a view, so the chat can
show it without plugin code:
| View | Shows | Fields |
| --- | --- | --- |
| `table` | A list as rows | `items`, `columns` (field, title, link) |
| `cards` | A list as cards | `items`, `title`, `subtitle`, `body`, `link` |
| `kv` | One object's fields | `items`, `title`, `link`, `fields` |
| `markdown` | Rendered Markdown | `field` (the text content when empty) |
| `json` | The data, formatted | - |
| `page` | A page of the plugin (`entry` under `ui/`) | Gets `{tool, arguments, result}` as its context |
Paths are dotted field paths into the structured content. A page's
requests use the mount `mcpServers/<id>`.
A plugin can also point `mcp.url` at an MCP server it runs elsewhere; then
headers carry the configuration (`${config.<key>}`), and `toolViews` apply
too.
## Dynamic choices and OAuth
Two schema keywords let a form ask the plugin while someone fills it in.
**`x-options`** loads a field's choices from the plugin, for lists that
depend on the account (projects, spaces, channels). It works on a string or
an array of strings:
```yaml
project:
type: string
x-options: { name: projects, dependsOn: [site, token], search: true }
```
```go
p.Options("projects", func(ctx context.Context, call *pluginsdk.Call, in pluginapi.OptionsInput) ([]pluginapi.Option, error) {
// call.Config holds what the form holds now, secrets included.
if call.Config.Tenant["token"] == nil {
return nil, pluginapi.InvalidConfig("enter the token first", map[string]string{"token": "required"})
}
return []pluginapi.Option{{Value: "p1", Label: "Project " + in.Query}}, nil
})
```
- `in.Scope` is `system`, `tenant` or `instance`; the values are in the
matching part of `call.Config`.
- Secrets the form shows redacted are filled in from the stored
configuration.
- The form reloads the list when a `dependsOn` field changes. With
`search`, `in.Query` is what the user typed.
- WeKnora does not check a saved value against the list; the plugin
should.
**`x-oauth`** turns a string field into an account connection. WeKnora runs
the OAuth 2.0 authorization code flow and keeps the tokens:
```yaml
config:
system: schemas/system.yaml # client_id, client_secret (x-secret)
# schemas/tenant.yaml
account:
type: string
x-oauth:
authorizeUrl: https://auth.example.com/oauth/authorize
tokenUrl: https://auth.example.com/oauth/token
scopes: [read]
clientId: ${system.client_id}
clientSecret: ${system.client_secret}
pkce: true
params: { audience: api.example.com } # extra authorize parameters
```
- The form shows a connect button that opens the provider's consent page.
- The field stores `oauth:<connection>`. At call time the plugin receives a
fresh access token in its place; WeKnora refreshes it when it is about to
expire. A connection that can no longer be refreshed arrives as `""`, so
answer `unauthorized` then.
- The platform registers the OAuth app with the provider, with the
redirect URI `<WeKnora address>/api/v1/plugin-oauth/callback`, and enters
its credentials in the plugin's platform configuration.
- `tokenUrl` must be https. WeKnora calls it itself, so it needs no
`permissions.egress` entry.
## Calling back into WeKnora (Host API)
A plugin that declares `permissions.hostApi` gets a short-lived token with
each call; `call.Host()` returns a client for it, or nil without a grant.
Scopes:
- `kv`: a key-value store of the plugin, separate per workspace, so the
plugin needs no database.
- `KVGet`, `KVPut` (optional TTL), `KVDelete` and `KVList`.
- Keys up to 256 bytes, JSON values up to 64 KB, 10,000 keys per
workspace.
The token is valid for a few minutes: use `call.Host()` within the call and
don't keep it.
## Packaging
A package is a zip (`.wkp`) with `plugin.yaml` at its root:
```yaml
schemaVersion: 1
id: acme.notes
version: 1.0.0
apiVersion: weknora.plugin/v1
name: { en-US: ACME Notes }
publisher: { id: acme }
runtime: { type: host, kind: binary, entry: "bin/{os}-{arch}/notes" }
permissions:
egress: [api.acme.example] # hosts the plugin may reach; "*" = any public host
contributes:
connectors:
- id: notes
name: { en-US: ACME Notes }
instanceSchema: schemas/notes.yaml # fields with x-group: settings are settings, the rest credentials
```
A few rules the manifest and host enforce:
- **Binaries.** Build one per platform into `bin/<os>-<arch>/`; WeKnora
runs the build for its own OS and architecture.
- **Python.** A plugin can instead be Python source:
- Declare `runtime: { type: host, kind: python, entry: main.py }`.
- WeKnora runs the entry with its own `python3` (or
`WEKNORA_PLUGIN_PYTHON`).
- `vendor/` and the package root are on `PYTHONPATH`.
- **Environment.** The process gets no environment from WeKnora beyond the
`WEKNORA_PLUGIN_*` variables.
- **Outbound traffic.** It goes through the host's egress proxy, which
forwards only to the hosts in `permissions.egress` and never to private
addresses.
- **Resources.** `runtime.resources: { cpu: "500m", memory: 512Mi }`
caps the process (Linux). Memory is always capped. CPU is capped only
when the platform delegates a cgroup to WeKnora
(`WEKNORA_PLUGIN_CGROUP`).
- **IDs.** Contribution IDs are qualified with the plugin ID
(`acme.notes/notes`); for connectors and web search that must stay within
50 characters.
Complete plugins with their `package.sh`:
- `examples/plugins/rss`: a connector.
- `examples/plugins/subtitles`: a parser using the Host API.
- `examples/plugins/notebooks`: a parser written in Python.
- `examples/plugins/links`: pages (toolbox, settings, knowledge base tab), in Python.
- `examples/plugins/activity`: events, a webhook and a page, in Python.
- `examples/plugins/jira`: a Jira Cloud connector with an OAuth field, dynamic
options, incremental sync with deletions; agent tools with result views; and
a skill.
## Testing
```bash
go run github.com/Tencent/WeKnora/pluginsdk/cmd/weknora-plugin-conformance -url http://localhost:8080 -secret $SECRET
```
In Go tests, run `conformance.Run` against `httptest.NewServer(p.Handler())`.
## Installing
In WeKnora: **System administration → Plugin management → Install plugin**.
Upload the `.wkp` or give its URL, review what it adds and reaches, and
install. Each workspace then enables it under **Settings → Plugins**.
## Running as a remote service
A plugin can also run as a service you deploy yourself, for instance in its
own container or on another team's cluster. Its package then carries only
`plugin.yaml` (plus schemas):
```yaml
runtime: { type: remote }
```
1. **Install.** Give the service URL when installing. WeKnora checks the
service's `/v1/manifest` against the package: same ID, version and
contributions.
2. **Keep the secret.** Installing shows a signing secret once. Start the
service with it as `WEKNORA_PLUGIN_SECRET` (and `WEKNORA_PLUGIN_ADDR`,
default `:8080`). The SDK rejects requests without a valid signature.
3. **Private hosts.** A service on a private network must be listed in
WeKnora's `SSRF_WHITELIST`.
4. **Host API.** Remote plugins get a Host API token only when
`WEKNORA_PLUGIN_HOST_API_URL` tells WeKnora its address as the service
sees it.
The plugin detail page changes the URL and rotates the secret. After a
rotation, calls fail until the service has the new secret.
Upgrade the service and the package together. While the service reports a
version other than the active package, calls are refused and the plugin
shows as degraded.