187 lines
6.8 KiB
Markdown
187 lines
6.8 KiB
Markdown
# WeKnora plugin SDK (Go)
|
|
|
|
Build code plugins for WeKnora: web search providers, data source
|
|
connectors and document parsers that run as their own process. The module has no dependencies.
|
|
|
|
| Package | What it is |
|
|
| --- | --- |
|
|
| `pluginapi` | The extension protocol v1: envelope, errors, stream events, messages. `openapi.yaml` describes it for other languages. |
|
|
| `pluginsdk` | Write a plugin: register contributions, then `Serve`. |
|
|
| `client` | Call a plugin, as WeKnora does. |
|
|
| `conformance`, `cmd/weknora-plugin-conformance` | Check that a plugin speaks the protocol. |
|
|
|
|
Writing Python? [`python/`](python/README.md) is the same SDK for Python
|
|
3.9+, with only the standard library.
|
|
|
|
## A connector in 30 lines
|
|
|
|
```go
|
|
package main
|
|
|
|
import (
|
|
"context"
|
|
"log"
|
|
|
|
"github.com/Tencent/WeKnora/pluginsdk"
|
|
"github.com/Tencent/WeKnora/pluginsdk/pluginapi"
|
|
)
|
|
|
|
type notes struct{}
|
|
|
|
func (notes) Validate(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig) error {
|
|
if cfg.Credentials["token"] == nil {
|
|
return pluginapi.InvalidConfig("token is required", map[string]string{"credentials.token": "required"})
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func (notes) ListResources(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig, parent string) ([]pluginapi.Resource, error) {
|
|
return []pluginapi.Resource{{ExternalID: "inbox", Name: "Inbox"}}, nil
|
|
}
|
|
|
|
func (notes) Fetch(ctx context.Context, call *pluginsdk.Call, cfg pluginsdk.ConnectorConfig, in pluginapi.FetchInput, s *pluginsdk.Stream) (*pluginapi.Cursor, error) {
|
|
if err := s.Item(pluginapi.FetchedItem{ExternalID: "n1", Title: "First note", Content: []byte("# Hello")}); err != nil {
|
|
return nil, err
|
|
}
|
|
return &pluginapi.Cursor{State: map[string]any{"after": "n1"}}, nil
|
|
}
|
|
|
|
func main() {
|
|
p := pluginsdk.New(pluginsdk.Info{ID: "acme.notes", Version: "1.0.0"})
|
|
p.Connector("notes", notes{})
|
|
log.Fatal(p.Serve())
|
|
}
|
|
```
|
|
|
|
What the SDK does for you:
|
|
|
|
- **Handshake.** Started by WeKnora's plugin host, the plugin listens on the
|
|
socket it is given and accepts only the host's token.
|
|
- **Remote mode.** Run on its own, it listens on `WEKNORA_PLUGIN_ADDR` and
|
|
verifies signed requests with `WEKNORA_PLUGIN_SECRET`.
|
|
- **Per call.** Each call's deadline reaches your context. Returned
|
|
`pluginapi` errors keep their code; anything else, and any panic, becomes
|
|
`internal`.
|
|
|
|
Every call carries the tenant and the configuration, secrets already
|
|
decrypted, in three separate scopes:
|
|
|
|
- `call.Config.System`: set by the system administrator.
|
|
- `call.Config.Tenant`: set per workspace.
|
|
- the instance: `cfg`, or `call.DecodeInstance`.
|
|
|
|
Keep plugins stateless and never cache a tenant's credentials across calls.
|
|
|
|
## Parsers
|
|
|
|
`p.Parser(id, ...)` turns a document (`in.Content`, base64 on the wire) into
|
|
Markdown. WeKnora chunks the text and stores the images the Markdown
|
|
references: return them in `Images` with an `OriginalRef` matching the
|
|
`` target. Declare the extensions it handles in `plugin.yaml`
|
|
(`fileTypes: [srt, vtt]`). Return a retryable error (`unavailable`,
|
|
`rate_limited`) to have the document retried later; any other error fails
|
|
it for good.
|
|
|
|
## Calling back into WeKnora (Host API)
|
|
|
|
A plugin that declares `permissions.hostApi` gets a short-lived token with
|
|
each call; `call.Host()` returns a client for it, or nil without a grant.
|
|
Scopes:
|
|
|
|
- `kv`: a key-value store of the plugin, separate per workspace, so the
|
|
plugin needs no database.
|
|
- `KVGet`, `KVPut` (optional TTL), `KVDelete` and `KVList`.
|
|
- Keys up to 256 bytes, JSON values up to 64 KB, 10,000 keys per
|
|
workspace.
|
|
|
|
The token is valid for a few minutes: use `call.Host()` within the call and
|
|
don't keep it.
|
|
|
|
## Packaging
|
|
|
|
A package is a zip (`.wkp`) with `plugin.yaml` at its root:
|
|
|
|
```yaml
|
|
schemaVersion: 1
|
|
id: acme.notes
|
|
version: 1.0.0
|
|
apiVersion: weknora.plugin/v1
|
|
name: { en-US: ACME Notes }
|
|
publisher: { id: acme }
|
|
runtime: { type: host, kind: binary, entry: "bin/{os}-{arch}/notes" }
|
|
permissions:
|
|
egress: [api.acme.example] # hosts the plugin may reach; "*" = any public host
|
|
contributes:
|
|
connectors:
|
|
- id: notes
|
|
name: { en-US: ACME Notes }
|
|
instanceSchema: schemas/notes.yaml # fields with x-group: settings are settings, the rest credentials
|
|
```
|
|
|
|
A few rules the manifest and host enforce:
|
|
|
|
- **Binaries.** Build one per platform into `bin/<os>-<arch>/`; WeKnora
|
|
runs the build for its own OS and architecture.
|
|
- **Python.** A plugin can instead be Python source:
|
|
- Declare `runtime: { type: host, kind: python, entry: main.py }`.
|
|
- WeKnora runs the entry with its own `python3` (or
|
|
`WEKNORA_PLUGIN_PYTHON`).
|
|
- `vendor/` and the package root are on `PYTHONPATH`.
|
|
- **Environment.** The process gets no environment from WeKnora beyond the
|
|
`WEKNORA_PLUGIN_*` variables.
|
|
- **Outbound traffic.** It goes through the host's egress proxy, which
|
|
forwards only to the hosts in `permissions.egress` and never to private
|
|
addresses.
|
|
- **IDs.** Contribution IDs are qualified with the plugin ID
|
|
(`acme.notes/notes`); for connectors and web search that must stay within
|
|
50 characters.
|
|
|
|
Complete plugins with their `package.sh`:
|
|
|
|
- `examples/plugins/rss`: a connector.
|
|
- `examples/plugins/subtitles`: a parser using the Host API.
|
|
- `examples/plugins/notebooks`: a parser written in Python.
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
go run github.com/Tencent/WeKnora/pluginsdk/cmd/weknora-plugin-conformance -url http://localhost:8080 -secret $SECRET
|
|
```
|
|
|
|
In Go tests, run `conformance.Run` against `httptest.NewServer(p.Handler())`.
|
|
|
|
## Installing
|
|
|
|
In WeKnora: **System administration → Plugin management → Install plugin**.
|
|
Upload the `.wkp` or give its URL, review what it adds and reaches, and
|
|
install. Each workspace then enables it under **Settings → Plugins**.
|
|
|
|
## Running as a remote service
|
|
|
|
A plugin can also run as a service you deploy yourself, for instance in its
|
|
own container or on another team's cluster. Its package then carries only
|
|
`plugin.yaml` (plus schemas):
|
|
|
|
```yaml
|
|
runtime: { type: remote }
|
|
```
|
|
|
|
1. **Install.** Give the service URL when installing. WeKnora checks the
|
|
service's `/v1/manifest` against the package: same ID, version and
|
|
contributions.
|
|
2. **Keep the secret.** Installing shows a signing secret once. Start the
|
|
service with it as `WEKNORA_PLUGIN_SECRET` (and `WEKNORA_PLUGIN_ADDR`,
|
|
default `:8080`). The SDK rejects requests without a valid signature.
|
|
3. **Private hosts.** A service on a private network must be listed in
|
|
WeKnora's `SSRF_WHITELIST`.
|
|
4. **Host API.** Remote plugins get a Host API token only when
|
|
`WEKNORA_PLUGIN_HOST_API_URL` tells WeKnora its address as the service
|
|
sees it.
|
|
|
|
The plugin detail page changes the URL and rotates the secret. After a
|
|
rotation, calls fail until the service has the new secret.
|
|
|
|
Upgrade the service and the package together. While the service reports a
|
|
version other than the active package, calls are refused and the plugin
|
|
shows as degraded.
|