From 68e6c03dc44e7faa2f807f70384ea5fc5534b0d7 Mon Sep 17 00:00:00 2001 From: lyingbug Date: Fri, 18 Sep 2026 20:27:43 +0800 Subject: [PATCH] fix(skills): upgrade outdated sandbox skills without rolling back the catalog or taking them offline (#3406) * fix(skills): keep retries and catalog installs from rewriting the skill definition Every install path wrote its archive back to the workspace catalog. After a skill was re-registered as v2, sandboxes still on v1 are pinned to v1's archive, so retrying one of them (a failed retry, or a ready skill retried with guidance) reinstalled v1 and also moved the catalog definition back to v1 and deleted v2's archive. Installs now carry where their archive came from. Only an upload or a source pull is a new definition; a retry or a catalog install uses bytes the workspace already stores and never rewrites the definition. A retry of a pinned install keeps its pin and replays its own version, and bytes that neither the definition nor the row holds are refused with a conflict. The catalog listing also reports each installation's version so the UI can show what an upgrade moves from and to. * feat(frontend): upgrade outdated skill installs from the catalog, sandbox drawer and agent editor An install whose archive differs from the catalog was flagged as outdated, but nothing could move it to the catalog version: the install drawer hid ready installs, and the manage drawer only offered a retry, which replays the sandbox's own archive. - Skill settings: cards with outdated installs show an upgrade button that opens the install drawer with every outdated sandbox preselected. The add wizard and install drawer also let outdated ready installs be picked, labelled with the version they move from and to. - Sandbox skill drawer: an outdated skill shows a "new version" row whose upgrade button installs the catalog onto that one sandbox. - Agent editor: skills on the chosen sandbox that are outdated show the version change and an upgrade button, both for admins only, matching the admin-only catalog install. The install progress entry, which opens a drawer backed by admin-only endpoints, is now admin-only too. - A skill that is being upgraded is no longer pruned from the agent's selected skills while it is briefly not ready. The outdated check is shared in utils/skillUpgrade.ts. * feat(skills): keep the previous version serving while an upgrade runs or after it fails An install rewrote the row to the new version and marked it installing as soon as it started, and the agent is only offered ready rows. So an upgrade took the skill away from every sandbox for its whole run, and a failed upgrade kept it away, even though the image pointer only moves on success and the previous version was in the image the whole time. The row now records what it described when it was last ready (served: version, description, instructions, digest, the archive its files come from, and the snapshot that installed it). The agent's usable set, member env acceptance and the @-mention list read TenantSkillEntity.ServedView, which is that version marked ready while the row is installing or failed. A successful install clears it and releases its archive; a failure or a stop keeps it. Keeping that consistent: - The served archive counts as a reader when a bundle is released or replaced by a re-registered catalog, so v1's files stay readable while v2 installs. - The reaper and a failed removal put the served version back when the image never moved past it, instead of labelling v1's files with the upgrade's metadata. A stuck upgrade whose chain cannot be read is left alone, since v1 keeps serving meanwhile. - The env declaration is stored after the pointer moves rather than before the snapshot, so an upgrade that fails late cannot drop the admin value of a variable the previous version still reads. Migration 000104 adds tenant_skills.served. The console says "upgrading, still running 1.1.0" or "upgrade failed, still running 1.1.0", the agent editor keeps such skills selectable, and the upgrade texts no longer warn that the skill goes offline. * fix(skills): offer only the upgrade to installs the catalog has moved past Review follow-up on the upgrade flow. - An install that differs from the catalog, ready or failed, is now offered the upgrade and not the retries. The retry icon and the retry-with-guidance box both replay the archive the install is pinned to, so next to an Upgrade button they reinstalled the version the operator came to leave behind. A failed first install of v1 whose catalog has since moved to v2 used to retry v1 from the drawer while the agent editor's Install went to v2; every entry now goes to v2, and the agent editor labels it "Upgrade". The drawer explains that the failed install is replaced by the catalog version. - A catalog install or a retry reuses the definition that already holds its bytes instead of re-running the catalog upsert, which rewrote the row and bumped its updated_at every time the skill was installed somewhere. The upsert still runs for a definition that does not exist yet or whose object was lost. --- frontend/src/api/skill/index.ts | 4 + frontend/src/api/system/index.ts | 3 + .../components/SandboxSkillsPanel.test.mjs | 24 ++ .../src/components/SandboxSkillsPanel.vue | 105 ++++- .../useConfigSkillInstallProgress.test.ts | 51 ++- frontend/src/i18n/locales/en-US.ts | 18 + frontend/src/i18n/locales/ja-JP.ts | 18 + frontend/src/i18n/locales/ko-KR.ts | 18 + frontend/src/i18n/locales/ru-RU.ts | 18 + frontend/src/i18n/locales/zh-CN.ts | 18 + frontend/src/utils/skillUpgrade.test.mjs | 51 +++ frontend/src/utils/skillUpgrade.ts | 67 ++++ .../src/views/agent/AgentEditorModal.test.mjs | 20 + frontend/src/views/agent/AgentEditorModal.vue | 81 +++- .../src/views/settings/SkillSettings.test.mjs | 23 ++ frontend/src/views/settings/SkillSettings.vue | 126 +++++- .../application/repository/tenant_skill.go | 1 + .../service/tenant_skill_catalog.go | 86 +++- .../service/tenant_skill_catalog_test.go | 173 +++++++++ .../service/tenant_skill_effective.go | 14 +- .../service/tenant_skill_install.go | 279 ++++++++++--- .../service/tenant_skill_reaper.go | 70 +++- .../service/tenant_skill_remove.go | 44 ++- .../service/tenant_skill_served_test.go | 367 ++++++++++++++++++ .../service/tenant_skill_source.go | 2 +- internal/application/service/user_env.go | 5 +- internal/handler/sandbox_skill.go | 5 + internal/types/tenant_skill.go | 86 ++++ .../000104_skill_served_version.down.sql | 3 + .../000104_skill_served_version.up.sql | 12 + 30 files changed, 1688 insertions(+), 104 deletions(-) create mode 100644 frontend/src/utils/skillUpgrade.test.mjs create mode 100644 frontend/src/utils/skillUpgrade.ts create mode 100644 internal/application/service/tenant_skill_served_test.go create mode 100644 migrations/versioned/000104_skill_served_version.down.sql create mode 100644 migrations/versioned/000104_skill_served_version.up.sql diff --git a/frontend/src/api/skill/index.ts b/frontend/src/api/skill/index.ts index 5610fc66..f4a24fd8 100644 --- a/frontend/src/api/skill/index.ts +++ b/frontend/src/api/skill/index.ts @@ -15,7 +15,11 @@ export interface SkillCatalogInstall { status: string; enabled: boolean; error?: string; + version?: string; bundle_sha256?: string; + // Present while a newer install is in flight or has failed and the sandbox + // still runs the previous version. + served?: { version?: string }; updated_at: string; } diff --git a/frontend/src/api/system/index.ts b/frontend/src/api/system/index.ts index 66f82ff4..c28dd5e6 100644 --- a/frontend/src/api/system/index.ts +++ b/frontend/src/api/system/index.ts @@ -1098,6 +1098,9 @@ export interface ConfigSkill { // decides whether to offer the "view install" entry point. install_session_id?: string install_message_id?: string + // Present while a newer install is in flight or has failed and the sandbox + // still runs the previous version. + served?: { version?: string } created_at: string updated_at: string // Absent for a skill whose installer declared nothing, which is how the diff --git a/frontend/src/components/SandboxSkillsPanel.test.mjs b/frontend/src/components/SandboxSkillsPanel.test.mjs index 6be978bd..192f7dc7 100644 --- a/frontend/src/components/SandboxSkillsPanel.test.mjs +++ b/frontend/src/components/SandboxSkillsPanel.test.mjs @@ -36,3 +36,27 @@ test('focused skill management expands env vars and transcript without a mid-pag assert.doesNotMatch(manageBlock, /skill-manage__footer/) assert.doesNotMatch(manageBlock, /settings\.sandbox\.skillEnv\.save/) }) + +test('the focused skill offers an upgrade when the catalog has moved on', () => { + assert.match(source, /catalogItem\?: SkillCatalogItem \| null/) + assert.match(source, /installUpgradable\(catalog, skill\)/) + assert.match(manageBlock, /v-if="managedUpgradeHint"/) + assert.match(manageBlock, /upgradeSkill\(managedSkill\)/) + assert.match(manageBlock, /settings\.skills\.upgradeRowTitle/) + // The upgrade is the catalog install; the retry replays this sandbox's own archive. + const upgrade = source.slice(source.indexOf('async function upgradeSkill('), source.indexOf('async function stopSkill(')) + assert.match(upgrade, /installSkillCatalog\(catalog\.id, \[configId\]\)/) + assert.doesNotMatch(upgrade, /reinstallConfigSkill/) +}) + +test('a skill mid-upgrade or after a failed upgrade says the previous version still runs', () => { + assert.match(source, /servedPreviousText\(t, managedSkill\.value\)/) + assert.match(manageBlock, /v-if="managedServedNote"/) + assert.match(source, /\.skill-manage__served \{/) +}) + +test('an install the catalog has moved past is offered the upgrade, not the retries', () => { + assert.match(source, /v-if="managedSkill\.status === 'failed' && !managedUpgradable"/) + assert.match(manageBlock, /:can-retry="\(managedSkill\.status === 'ready' \|\| managedSkill\.status === 'failed'\) && !managedUpgradable"/) + assert.match(source, /settings\.skills\.upgradeRowHintFailedVersions/) +}) diff --git a/frontend/src/components/SandboxSkillsPanel.vue b/frontend/src/components/SandboxSkillsPanel.vue index 82ab73cf..347afd4d 100644 --- a/frontend/src/components/SandboxSkillsPanel.vue +++ b/frontend/src/components/SandboxSkillsPanel.vue @@ -134,6 +134,23 @@ + @@ -228,7 +234,7 @@ {{ row.cfg.name }} - {{ sandboxMetaLine(row.cfg) }} + {{ pickRowMeta(row) }} @@ -264,11 +270,11 @@ - -

{{ $t('settings.skills.installToSandboxDesc') }}

+

{{ $t('settings.skills.installToSandboxDesc') }}

{{ row.cfg.name }} - {{ sandboxMetaLine(row.cfg) }} + {{ pickRowMeta(row) }} @@ -304,7 +310,9 @@
-

{{ $t('settings.skills.noSandboxToInstall') }}

+

+ {{ installMode === 'upgrade' ? $t('settings.skills.noSandboxToUpgrade') : $t('settings.skills.noSandboxToInstall') }} +

{{ $t('settings.sandbox.skillInstallerModel') }}

{{ $t('settings.sandbox.skillInstallerModelHint') }}

@@ -317,7 +325,8 @@ width="680px" :min-width="560" :max-width="920" storage-key="setting-drawer:width:skill-catalog-manage" :hide-footer="true" :z-index="2600"> + :focus-skill-id="manageSkillId" :catalog-item="catalogItemById(manageCatalogId)" @updated="onPanelUpdated" + @skills-changed="loadCatalog" /> ([]) const installTargetIds = ref([]) const installSessionIds = ref([]) const installCatalog = ref(null) +const installMode = ref<'install' | 'upgrade'>('install') const manageRecord = ref(null) const manageSkillId = ref('') +const manageCatalogId = ref('') const manageTitle = ref('') const filesDrawerVisible = ref(false) const filesCatalogId = ref('') @@ -445,13 +457,13 @@ const addPrimaryDisabled = computed(() => { const addPrimaryText = computed(() => { if (addStep.value === 0) return t('common.next') - if (addTargetIds.value.length > 0) return t('settings.skills.installToSandbox') + if (addTargetIds.value.length > 0) return pickConfirmText(addPickRows.value, addTargetIds.value) return t('settings.skills.addFinish') }) const installConfirmText = computed(() => installTargetIds.value.length > 0 - ? t('settings.skills.installToSandbox') + ? pickConfirmText(installPickRows.value, installTargetIds.value) : t('settings.skills.addFinish'), ) @@ -460,16 +472,27 @@ const installConfirmDisabled = computed(() => ) const installPickRows = computed(() => - sandboxPickRows(catalogItemById(installCatalog.value?.id), 'remaining', installSessionIds.value), + sandboxPickRows( + catalogItemById(installCatalog.value?.id), + installMode.value === 'upgrade' ? 'upgrade' : 'remaining', + installSessionIds.value, + ), ) const addPickRows = computed(() => sandboxPickRows(catalogItemById(registeredCatalog.value?.id), 'all', addSessionIds.value), ) +const installDrawerTitle = computed(() => + installMode.value === 'upgrade' + ? t('settings.skills.upgradeTitle') + : t('settings.skills.installToSandbox'), +) + const installDrawerDesc = computed(() => { const item = installCatalog.value if (!item) return t('settings.skills.installToSandboxDesc') + if (installMode.value === 'upgrade') return t('settings.skills.upgradeDrawerDesc', { name: item.name }) return t('settings.skills.installDrawerDesc', { name: item.name }) }) @@ -530,6 +553,10 @@ type SandboxPickRow = { selectable: boolean busy: boolean ready: boolean + // A ready install still on an older archive than the catalog. Picking it + // installs the catalog over it, which is the upgrade. + upgradable: boolean + upgradeNote: string } function catalogItemById(id: string | undefined | null): SkillCatalogItem | null { @@ -540,7 +567,7 @@ function catalogItemById(id: string | undefined | null): SkillCatalogItem | null function sandboxPickRows( item: SkillCatalogItem | null, - mode: 'remaining' | 'all', + mode: 'remaining' | 'upgrade' | 'all', sessionIds: string[], ): SandboxPickRow[] { const byId = new Map( @@ -553,6 +580,8 @@ function sandboxPickRows( const inst = byId.get(cfg.id) if (session.has(cfg.id)) return true if (inst && isInstallBusy(inst)) return true + if (inst && item && installUpgradable(item, inst)) return true + if (mode === 'upgrade') return false if (!inst || inst.status === 'failed') return true return false }) @@ -560,16 +589,35 @@ function sandboxPickRows( const install = byId.get(cfg.id) const busy = Boolean(install && isInstallBusy(install)) const ready = install?.status === 'ready' + const upgradable = Boolean(item && install && installUpgradable(item, install)) return { cfg, install, - selectable: !busy && !ready, + selectable: !busy && (!ready || upgradable), busy, ready, + upgradable, + upgradeNote: item && install && upgradable ? upgradeVersionText(item, install) : '', } }) } +function pickRowMeta(row: SandboxPickRow): string { + const meta = sandboxMetaLine(row.cfg) + return row.upgradeNote ? `${row.upgradeNote} · ${meta}` : meta +} + +function pickedOnlyUpgrades(rows: SandboxPickRow[], ids: string[]): boolean { + const byId = new Map(rows.map((row) => [row.cfg.id, row])) + return ids.length > 0 && ids.every((id) => byId.get(id)?.upgradable) +} + +function pickConfirmText(rows: SandboxPickRow[], ids: string[]): string { + return pickedOnlyUpgrades(rows, ids) + ? t('settings.skills.upgrade') + : t('settings.skills.installToSandbox') +} + function sandboxPickPercent(row: SandboxPickRow): number | null { if (!row.busy || !row.install) return null return installEventPercent(row.cfg.id, row.install.skill_id) @@ -635,15 +683,16 @@ function compactText(value: string): string { return value.replace(/\s+/g, ' ').trim() } -function installOutdated(item: SkillCatalogItem, inst: SkillCatalogInstall): boolean { - return Boolean( - item.bundle_sha256 - && inst.bundle_sha256 - && item.bundle_sha256 !== inst.bundle_sha256, - ) +function upgradeVersionText(item: SkillCatalogItem, inst: SkillCatalogInstall): string { + const versions = upgradeVersions(item, inst) + return versions + ? t('settings.skills.upgradeFromTo', versions) + : t('settings.skills.upgradeAvailable') } function installStatusText(inst: SkillCatalogInstall): string { + const served = servedPreviousText(t, inst) + if (served) return served if (inst.status === 'installing') return t('settings.sandbox.skillStatusInstalling') if (inst.status === 'removing') return t('settings.sandbox.skillStatusRemoving') if (inst.status === 'failed') return t('settings.sandbox.skillStatusFailed') @@ -696,11 +745,23 @@ function installsView(item: SkillCatalogItem) { return { installs, available, + upgradable: installs.filter((inst) => installUpgradable(item, inst)), canAdd: available.length > 0, needsPanel: installs.length + available.length > 1, } } +function upgradeLabel(view: ReturnType): string { + const count = view.upgradable.length + return count > 1 ? t('settings.skills.upgradeCount', { count }) : t('settings.skills.upgrade') +} + +function upgradeTooltip(item: SkillCatalogItem, view: ReturnType): string { + return view.upgradable + .map((inst) => `${installName(inst)} · ${upgradeVersionText(item, inst)}`) + .join('\n') +} + function installSummary(item: SkillCatalogItem, view: ReturnType): string { if (view.installs.length === 0) return t('settings.skills.installToSandbox') if (view.installs.length === 1) { @@ -753,6 +814,7 @@ function openManageFromPanel(item: SkillCatalogItem, inst: SkillCatalogInstall) function openInstallTo(item: SkillCatalogItem, cfg: SandboxConfigRecord) { openPanelId.value = '' + installMode.value = 'install' installCatalog.value = item installSessionIds.value = [] installTargetIds.value = [cfg.id] @@ -853,6 +915,7 @@ function addPreviousStep() { } function openInstall(item: SkillCatalogItem) { + installMode.value = 'install' installCatalog.value = item installSessionIds.value = [] const remaining = targetsFor(item) @@ -861,11 +924,25 @@ function openInstall(item: SkillCatalogItem) { showInstall.value = true } +// Upgrading is the catalog install onto sandboxes that already carry an older +// archive; the server treats a same-name install as an in-place upgrade. Every +// outdated sandbox starts checked, so one confirm upgrades them all. +function openUpgrade(item: SkillCatalogItem) { + openPanelId.value = '' + installMode.value = 'upgrade' + installCatalog.value = item + installSessionIds.value = [] + installTargetIds.value = installsView(item).upgradable.map((inst) => inst.sandbox_config_id) + void loadInstallerModel() + showInstall.value = true +} + function openManage(item: SkillCatalogItem, inst: SkillCatalogInstall) { const record = recordFor(inst.sandbox_config_id) if (!record) return manageRecord.value = record manageSkillId.value = inst.skill_id + manageCatalogId.value = item.id manageTitle.value = item.name showManage.value = true } @@ -983,6 +1060,7 @@ async function handleAddPrimary() { revealCatalog(catalogId) return } + const upgrading = pickedOnlyUpgrades(addPickRows.value, targets) installing.value = true try { await ensureInstallerModelIfNeeded(targets) @@ -991,7 +1069,7 @@ async function handleAddPrimary() { if (failed > 0) { MessagePlugin.warning(t('settings.skills.installPartial', { failed })) } else { - MessagePlugin.success(t('settings.skills.installAccepted')) + MessagePlugin.success(t(upgrading ? 'settings.skills.upgradeAccepted' : 'settings.skills.installAccepted')) } rememberSession(addSessionIds, targets) await loadCatalog() @@ -1027,6 +1105,7 @@ async function confirmInstall() { const item = installCatalog.value const targets = [...installTargetIds.value] if (!item || targets.length === 0) return + const upgrading = pickedOnlyUpgrades(installPickRows.value, targets) installing.value = true try { await ensureInstallerModelIfNeeded(targets) @@ -1035,7 +1114,7 @@ async function confirmInstall() { if (failed > 0) { MessagePlugin.warning(t('settings.skills.installPartial', { failed })) } else { - MessagePlugin.success(t('settings.skills.installAccepted')) + MessagePlugin.success(t(upgrading ? 'settings.skills.upgradeAccepted' : 'settings.skills.installAccepted')) } rememberSession(installSessionIds, targets) await loadCatalog() @@ -1115,6 +1194,7 @@ watch(showManage, (open) => { if (!open) { void loadCatalog() manageSkillId.value = '' + manageCatalogId.value = '' manageRecord.value = null } else { openPanelId.value = '' @@ -1445,6 +1525,7 @@ onUnmounted(() => { .skill-card__installs { display: flex; align-items: center; + gap: 6px; min-width: 0; margin-top: auto; } @@ -1483,7 +1564,12 @@ onUnmounted(() => { opacity: 0.6; } - &--idle { + &--upgrade { + flex-shrink: 0; + } + + &--idle, + &--upgrade { color: var(--td-brand-color); background: color-mix(in srgb, var(--td-brand-color) 10%, transparent); diff --git a/internal/application/repository/tenant_skill.go b/internal/application/repository/tenant_skill.go index 547a8605..62c348df 100644 --- a/internal/application/repository/tenant_skill.go +++ b/internal/application/repository/tenant_skill.go @@ -194,6 +194,7 @@ func (r *tenantSkillRepository) UpdateSkill(ctx context.Context, e *types.Tenant "install_session_id": e.InstallSessionID, "install_message_id": e.InstallMessageID, "catalog_id": e.CatalogID, + "served": e.Served, "status": e.Status, "error": e.Error, "installing_since": e.InstallingSince, diff --git a/internal/application/service/tenant_skill_catalog.go b/internal/application/service/tenant_skill_catalog.go index e92e349a..ae49b190 100644 --- a/internal/application/service/tenant_skill_catalog.go +++ b/internal/application/service/tenant_skill_catalog.go @@ -16,15 +16,38 @@ import ( // SkillCatalogInstallView is one installation of a catalog skill onto a sandbox. type SkillCatalogInstallView struct { - SkillID string `json:"skill_id"` - SandboxConfigID string `json:"sandbox_config_id"` - SandboxConfigName string `json:"sandbox_config_name,omitempty"` - SandboxType string `json:"sandbox_type,omitempty"` - Status string `json:"status"` - Enabled bool `json:"enabled"` - Error string `json:"error,omitempty"` - BundleSHA256 string `json:"bundle_sha256,omitempty"` - UpdatedAt time.Time `json:"updated_at"` + SkillID string `json:"skill_id"` + SandboxConfigID string `json:"sandbox_config_id"` + SandboxConfigName string `json:"sandbox_config_name,omitempty"` + SandboxType string `json:"sandbox_type,omitempty"` + Status string `json:"status"` + Enabled bool `json:"enabled"` + Error string `json:"error,omitempty"` + Version string `json:"version,omitempty"` + BundleSHA256 string `json:"bundle_sha256,omitempty"` + // Served is set while a newer install of this skill is in flight or has + // failed and the image still runs the previous version. + Served *SkillServedInfo `json:"served,omitempty"` + UpdatedAt time.Time `json:"updated_at"` +} + +// SkillServedInfo tells the console that a previous version of a skill keeps +// running while the row describes an install that has not replaced it. +type SkillServedInfo struct { + Version string `json:"version,omitempty"` +} + +// ServedInfoOf reports the previous version a row is still serving, or nil +// when the row itself is what runs or nothing does. +func ServedInfoOf(row *types.TenantSkillEntity) *SkillServedInfo { + if row == nil || row.Status == types.SkillStatusReady { + return nil + } + view := row.ServedView() + if view == nil { + return nil + } + return &SkillServedInfo{Version: view.Version} } // SkillCatalogView is a tenant skill definition plus its sandbox installations. @@ -154,7 +177,9 @@ func installView( Status: row.Status, Enabled: row.Enabled, Error: row.Error, + Version: row.Version, BundleSHA256: row.BundleSHA256, + Served: ServedInfoOf(row), UpdatedAt: row.UpdatedAt, } if cfg := configByID[row.SandboxConfigID]; cfg != nil { @@ -221,7 +246,7 @@ func (s *TenantSkillService) InstallCatalogToConfigs( } var firstErr error for _, configID := range ids { - skillID, installErr := s.InstallSkill(ctx, tenantID, configID, archive) + skillID, installErr := s.installSkillArchive(ctx, tenantID, configID, archive, skillArchiveStored) if installErr != nil { logger.Warnf(ctx, "[skill] install catalog %s onto config %s failed: %v", catalogID, configID, installErr) @@ -456,7 +481,15 @@ func (s *TenantSkillService) pinInstallsToReplacedBundle( } keep := false for _, row := range installs { - if row == nil || strings.TrimSpace(row.BundleRef) != "" { + if row == nil { + continue + } + pinnedServed, err := s.pinServedVersionToReplacedBundle(ctx, catalog, row, oldRef, oldSHA) + if err != nil { + return true, err + } + keep = keep || pinnedServed + if strings.TrimSpace(row.BundleRef) != "" { continue } if oldSHA == "" { @@ -478,6 +511,37 @@ func (s *TenantSkillService) pinInstallsToReplacedBundle( return keep, nil } +// pinServedVersionToReplacedBundle keeps the archive a mid-upgrade install is +// still serving from. That version is recorded with the object it was read +// from, but one recorded before this was possible may only name its digest, +// and is pinned here the same way a row following the catalog is. +func (s *TenantSkillService) pinServedVersionToReplacedBundle( + ctx context.Context, catalog *types.TenantSkillCatalogEntity, row *types.TenantSkillEntity, oldRef, oldSHA string, +) (bool, error) { + served := row.Served + if served == nil { + return false, nil + } + if ref := strings.TrimSpace(served.BundleRef); ref != "" { + return ref == oldRef, nil + } + if oldSHA == "" || strings.TrimSpace(served.BundleSHA256) != oldSHA { + return false, nil + } + if err := s.updateSkillFields(ctx, row.TenantID, row.SandboxConfigID, row.ID, + func(e *types.TenantSkillEntity) { + if e.Served != nil && strings.TrimSpace(e.Served.BundleRef) == "" { + pinned := *e.Served + pinned.BundleRef = oldRef + e.Served = &pinned + } + }); err != nil { + return true, fmt.Errorf("pin the served version of install %s to the replaced archive of catalog %s: %w", + row.ID, catalog.ID, err) + } + return true, nil +} + func skillUserErrorMessage(err error) string { if err == nil { return "" diff --git a/internal/application/service/tenant_skill_catalog_test.go b/internal/application/service/tenant_skill_catalog_test.go index 1158b690..d2a8035a 100644 --- a/internal/application/service/tenant_skill_catalog_test.go +++ b/internal/application/service/tenant_skill_catalog_test.go @@ -5,6 +5,7 @@ import ( "errors" "strings" "testing" + "time" "github.com/Tencent/WeKnora/internal/application/repository" apperrors "github.com/Tencent/WeKnora/internal/errors" @@ -459,6 +460,178 @@ func TestRemovingThePinnedInstallReclaimsTheReplacedZip(t *testing.T) { "the pinned archive is reclaimed with its last reader, the definition's is not") } +// A sandbox still on v1 is pinned to v1's archive once the catalog moves to +// v2. Retrying it replays v1 for that sandbox alone: writing v1 back to the +// definition would turn every other sandbox's upgrade target into v1 and delete +// the only copy of v2. +func TestReinstallOfAnInstallStillOnTheReplacedArchiveLeavesTheCatalogAlone(t *testing.T) { + for _, tc := range []struct { + name string + status string + guidance []string + }{ + {name: "failed retry", status: types.SkillStatusFailed}, + {name: "ready with guidance", status: types.SkillStatusReady, guidance: []string{"Install and verify the CLI"}}, + } { + t.Run(tc.name, func(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + first := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v1')\n", + }) + second := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v2')\n", + }) + firstBundle, err := ParseSkillBundle(first) + require.NoError(t, err) + secondBundle, err := ParseSkillBundle(second) + require.NoError(t, err) + + skillID, err := fx.svc.InstallSkill(ctx, 7, "cfg-1", first) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + installed, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + catalogID := installed.CatalogID + firstRef := fx.catalogRefFor(t, catalogID) + + _, err = fx.svc.RegisterCatalogFromArchive(ctx, 7, second) + require.NoError(t, err) + secondRef := fx.catalogRefFor(t, catalogID) + + installed, err = fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + installed.Status = tc.status + require.NoError(t, fx.skillRepo.UpdateSkill(ctx, installed)) + + _, err = fx.svc.ReinstallSkill(ctx, 7, "cfg-1", skillID, tc.guidance...) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + + cat, err := fx.skillRepo.GetCatalog(ctx, 7, catalogID) + require.NoError(t, err) + require.Equal(t, secondBundle.SHA256, cat.BundleSHA256, "the definition stays on v2") + require.Equal(t, secondRef, cat.BundleRef) + require.NotContains(t, fx.deletedBundles, secondRef, "v2's archive must survive a v1 retry") + require.NotContains(t, fx.deletedBundles, firstRef, "the retried install still reads v1's archive") + + installed, err = fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + require.Equal(t, firstRef, installed.BundleRef, "the retry keeps its pin to v1") + require.Equal(t, firstBundle.SHA256, installed.BundleSHA256) + }) + } +} + +// Upgrading is installing the catalog onto a sandbox that is still on the +// replaced archive: the install moves to v2 and v1's archive, with no reader +// left, is reclaimed. +func TestInstallCatalogToConfigsUpgradesAnInstallStillOnTheReplacedArchive(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + first := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v1')\n", + }) + second := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v2')\n", + }) + secondBundle, err := ParseSkillBundle(second) + require.NoError(t, err) + + skillID, err := fx.svc.InstallSkill(ctx, 7, "cfg-1", first) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + installed, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + firstRef := fx.catalogRefFor(t, installed.CatalogID) + + cat, err := fx.svc.RegisterCatalogFromArchive(ctx, 7, second) + require.NoError(t, err) + result, err := fx.svc.InstallCatalogToConfigs(ctx, 7, cat.ID, []string{"cfg-1"}) + require.NoError(t, err) + require.Equal(t, skillID, result.Installs["cfg-1"], "an upgrade reuses the sandbox's row") + fx.awaitSkillSettled(t, skillID) + + installed, err = fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + require.Equal(t, types.SkillStatusReady, installed.Status) + require.Equal(t, secondBundle.SHA256, installed.BundleSHA256) + require.Empty(t, installed.BundleRef, "the upgraded install follows the definition again") + require.Equal(t, secondBundle.SHA256, fx.catalogSHAFor(t, cat.ID)) + require.Contains(t, fx.deletedBundles, firstRef, "v1's archive lost its last reader") +} + +// Installing a definition onto a sandbox reads it; it does not re-register it. +// Rewriting the row would move its updated_at every time someone installs it. +func TestInstallCatalogToConfigsLeavesTheDefinitionUntouched(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + cat, err := fx.svc.RegisterCatalogFromArchive(ctx, 7, zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v1')\n", + })) + require.NoError(t, err) + registered, err := fx.skillRepo.GetCatalog(ctx, 7, cat.ID) + require.NoError(t, err) + savedBefore := fx.savedBundles + fx.svc.now = func() time.Time { return registered.UpdatedAt.Add(time.Hour) } + + result, err := fx.svc.InstallCatalogToConfigs(ctx, 7, cat.ID, []string{"cfg-1"}) + require.NoError(t, err) + fx.awaitSkillSettled(t, result.Installs["cfg-1"]) + + after, err := fx.skillRepo.GetCatalog(ctx, 7, cat.ID) + require.NoError(t, err) + require.Equal(t, registered.UpdatedAt, after.UpdatedAt) + require.Equal(t, registered.BundleRef, after.BundleRef) + require.Equal(t, savedBefore, fx.savedBundles, "the definition's bytes are not stored again") +} + +// Bytes neither the definition nor the row holds mean the catalog moved after +// its archive was read. Installing them would write that older archive back. +func TestStoredArchiveThatNoLongerMatchesTheCatalogIsRefused(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + first := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v1')\n", + }) + second := zipBundle(t, map[string]string{ + "SKILL.md": validSkillMD, + "scripts/extract.py": "print('v2')\n", + }) + firstBundle, err := ParseSkillBundle(first) + require.NoError(t, err) + _, err = fx.svc.RegisterCatalogFromArchive(ctx, 7, second) + require.NoError(t, err) + + _, err = fx.svc.storedArchiveKeepsItsPin(ctx, 7, nil, firstBundle) + + var appErr *apperrors.AppError + require.ErrorAs(t, err, &appErr) + require.Equal(t, apperrors.ErrConflict, appErr.Code) +} + +func (f *installFixture) awaitSkillSettled(t *testing.T, skillID string) { + t.Helper() + require.Eventually(t, func() bool { + row, err := f.skillRepo.GetSkill(context.Background(), 7, "cfg-1", skillID) + return err == nil && row != nil && row.Status != types.SkillStatusInstalling + }, 5*time.Second, 10*time.Millisecond) +} + +func (f *installFixture) catalogSHAFor(t *testing.T, catalogID string) string { + t.Helper() + cat, err := f.skillRepo.GetCatalog(context.Background(), 7, catalogID) + require.NoError(t, err) + require.NotNil(t, cat) + return cat.BundleSHA256 +} + func (f *installFixture) catalogRefFor(t *testing.T, catalogID string) string { t.Helper() cat, err := f.skillRepo.GetCatalog(context.Background(), 7, catalogID) diff --git a/internal/application/service/tenant_skill_effective.go b/internal/application/service/tenant_skill_effective.go index 6fa9f683..03d7d012 100644 --- a/internal/application/service/tenant_skill_effective.go +++ b/internal/application/service/tenant_skill_effective.go @@ -65,9 +65,11 @@ func skillsForRun( // snapshot, sessions boot the BASE template, which carries none of these // skills. Announcing them then costs the model several turns on calls that // can only fail, so ZERO skills are returned - not a degraded subset. -// - The row must be ready and enabled. Anything else is either not in the -// image yet, not in it any more, or deliberately hidden by an -// administrator. +// - The row must be enabled and the image must serve a version of it: the +// row itself when ready, or the version kept while a newer install is in +// flight or has failed (see TenantSkillEntity.ServedView). Anything else is +// either not in the image yet, not in it any more, or deliberately hidden +// by an administrator. // // Every failure returns nothing: a chat turn must not fail because a skill // lookup did, and a workspace with no installed skills is the common case. @@ -108,10 +110,12 @@ func effectiveTenantSkills( } usable := make([]*types.TenantSkillEntity, 0, len(rows)) for _, row := range rows { - if row == nil || !row.Enabled || row.Status != types.SkillStatusReady { + if row == nil || !row.Enabled { continue } - usable = append(usable, row) + if served := row.ServedView(); served != nil { + usable = append(usable, served) + } } if len(usable) == 0 { return nil diff --git a/internal/application/service/tenant_skill_install.go b/internal/application/service/tenant_skill_install.go index 6c0898c1..812f69f8 100644 --- a/internal/application/service/tenant_skill_install.go +++ b/internal/application/service/tenant_skill_install.go @@ -58,17 +58,32 @@ const ( skillInstallVerifyRounds = 2 ) +// skillArchiveOrigin says what an install's archive is to the workspace. +// +// Only an upload or a source pull is a new definition of the skill. A retry +// replays the archive its row already names, and a catalog install copies the +// definition's own bytes. Letting either write the definition back is how +// retrying a sandbox that was still on v1 moved the workspace from v2 back to +// v1 and deleted v2's archive. +type skillArchiveOrigin int + +const ( + skillArchiveUploaded skillArchiveOrigin = iota + skillArchiveStored +) + // InstallSkill validates an uploaded archive, records it, and kicks off the // install in the background. It returns the skill ID so the caller can answer // 202 and let the UI subscribe to progress. func (s *TenantSkillService) InstallSkill( ctx context.Context, tenantID uint64, configID string, archive []byte, ) (string, error) { - return s.installSkillArchive(ctx, tenantID, configID, archive) + return s.installSkillArchive(ctx, tenantID, configID, archive, skillArchiveUploaded) } func (s *TenantSkillService) installSkillArchive( - ctx context.Context, tenantID uint64, configID string, archive []byte, instructions ...string, + ctx context.Context, tenantID uint64, configID string, archive []byte, + origin skillArchiveOrigin, instructions ...string, ) (string, error) { cfgEntity, err := s.configs.GetByID(ctx, tenantID, configID) if err != nil { @@ -82,11 +97,12 @@ func (s *TenantSkillService) installSkillArchive( if err != nil { return "", err } - return s.installParsedSkill(ctx, tenantID, configID, bundle, archive, instructions...) + return s.installParsedSkill(ctx, tenantID, configID, bundle, archive, origin, instructions...) } func (s *TenantSkillService) installParsedSkill( - ctx context.Context, tenantID uint64, configID string, bundle *SkillBundle, archive []byte, instructions ...string, + ctx context.Context, tenantID uint64, configID string, bundle *SkillBundle, archive []byte, + origin skillArchiveOrigin, instructions ...string, ) (string, error) { if bundle == nil { return "", fmt.Errorf("skill bundle is required") @@ -109,8 +125,18 @@ func (s *TenantSkillService) installParsedSkill( "skill is busy; send guidance to the active install or wait for it to finish", ) } + keepPin := false + if origin == skillArchiveStored { + keepPin, err = s.storedArchiveKeepsItsPin(ctx, tenantID, existing, bundle) + if err != nil { + return "", err + } + } if guidance == "" && s.canSkipInstall(ctx, existing, bundle) { - catalog, catalogErr := s.upsertCatalogFromBundle(ctx, tenantID, bundle, archive, true) + if keepPin { + return existing.ID, nil + } + catalog, catalogErr := s.catalogForArchive(ctx, tenantID, bundle, archive, origin) if catalogErr != nil { return "", fmt.Errorf("store bundle for skill %s: %w", existing.ID, catalogErr) } @@ -124,6 +150,7 @@ func (s *TenantSkillService) installParsedSkill( now := s.now() if existing != nil { skillID = existing.ID + existing.Served = s.servedVersionOf(ctx, existing) takeSkillRowForInstall(existing, bundle, now) if err := s.skills.UpdateSkill(ctx, existing); err != nil { return "", err @@ -149,6 +176,7 @@ func (s *TenantSkillService) installParsedSkill( return "", err } skillID = winner.ID + winner.Served = s.servedVersionOf(ctx, winner) takeSkillRowForInstall(winner, bundle, now) if err := s.skills.UpdateSkill(ctx, winner); err != nil { return "", err @@ -159,26 +187,11 @@ func (s *TenantSkillService) installParsedSkill( // The zip lives on the catalog, not on this sandbox: uninstalling from // the last config must not take the definition's files with it. The // install row only stores CatalogID; readers follow that to the zip. - catalog, err := s.upsertCatalogFromBundle(ctx, tenantID, bundle, archive, true) - if err != nil { - failCtx, cancelFail := s.cleanupContext(ctx) - defer cancelFail() - storeErr := fmt.Errorf("store bundle: %w", err) - logger.Errorf(ctx, "[skill] store bundle failed tenant=%d config=%s skill=%s name=%s: %v", - tenantID, configID, skillID, bundle.Name, err) - s.failSkill(failCtx, tenantID, configID, skillID, bundle, storeErr) - return "", fmt.Errorf("store bundle for skill %s: %w", skillID, err) - } - if err := s.pointInstallAtCatalog(ctx, &types.TenantSkillEntity{ - ID: skillID, TenantID: tenantID, SandboxConfigID: configID, - }, catalog); err != nil { - failCtx, cancelFail := s.cleanupContext(ctx) - defer cancelFail() - storeErr := fmt.Errorf("store bundle: %w", err) - logger.Errorf(ctx, "[skill] store bundle failed tenant=%d config=%s skill=%s name=%s: %v", - tenantID, configID, skillID, bundle.Name, err) - s.failSkill(failCtx, tenantID, configID, skillID, bundle, storeErr) - return "", fmt.Errorf("store bundle for skill %s: %w", skillID, err) + // A pinned retry is the exception: its row already names its own archive. + if !keepPin { + if err := s.bindInstallToCatalog(ctx, tenantID, configID, skillID, bundle, archive, origin); err != nil { + return "", err + } } s.publishProgress(ctx, tenantID, configID, skillID, SkillProgress{ @@ -200,6 +213,153 @@ func (s *TenantSkillService) installParsedSkill( return skillID, nil } +// catalogForArchive returns the definition an install of these bytes follows. +// An upload or a source pull becomes the definition. Stored bytes reuse the +// definition that already holds them, untouched: storedArchiveKeepsItsPin has +// refused any other version by now, so what is left to write is only a +// definition that does not exist yet or an object that was lost. +func (s *TenantSkillService) catalogForArchive( + ctx context.Context, tenantID uint64, bundle *SkillBundle, archive []byte, origin skillArchiveOrigin, +) (*types.TenantSkillCatalogEntity, error) { + if origin == skillArchiveStored { + catalog, err := s.skills.GetCatalogByName(ctx, tenantID, bundle.Name) + if err != nil { + return nil, err + } + if catalog != nil && catalog.BundleSHA256 == bundle.SHA256 && + s.catalogBundleStillHeld(ctx, tenantID, catalog) { + return catalog, nil + } + } + return s.upsertCatalogFromBundle(ctx, tenantID, bundle, archive, true) +} + +// bindInstallToCatalog records the archive on the definition and points the +// install at it. A failure marks the row failed, because the row has already +// been handed to this run. +func (s *TenantSkillService) bindInstallToCatalog( + ctx context.Context, tenantID uint64, configID, skillID string, bundle *SkillBundle, archive []byte, + origin skillArchiveOrigin, +) error { + catalog, err := s.catalogForArchive(ctx, tenantID, bundle, archive, origin) + if err == nil { + err = s.pointInstallAtCatalog(ctx, &types.TenantSkillEntity{ + ID: skillID, TenantID: tenantID, SandboxConfigID: configID, + }, catalog) + } + if err == nil { + return nil + } + failCtx, cancelFail := s.cleanupContext(ctx) + defer cancelFail() + logger.Errorf(ctx, "[skill] store bundle failed tenant=%d config=%s skill=%s name=%s: %v", + tenantID, configID, skillID, bundle.Name, err) + s.failSkill(failCtx, tenantID, configID, skillID, bundle, fmt.Errorf("store bundle: %w", err)) + return fmt.Errorf("store bundle for skill %s: %w", skillID, err) +} + +// storedArchiveKeepsItsPin decides whether an install of bytes the workspace +// already holds has to leave the definition alone. +// +// It answers true when the definition has moved on and the install's own row +// still pins these bytes: a retry of a sandbox that is still on v1. Writing v1 +// to the definition there is a rollback, so the row keeps its pin and the +// definition stays on v2. Bytes that neither the definition nor the row holds +// are a catalog that moved between reading its archive and installing it, +// which is refused rather than written back. +func (s *TenantSkillService) storedArchiveKeepsItsPin( + ctx context.Context, tenantID uint64, existing *types.TenantSkillEntity, bundle *SkillBundle, +) (bool, error) { + catalog, err := s.skills.GetCatalogByName(ctx, tenantID, bundle.Name) + if err != nil { + return false, err + } + // No definition yet is a row older than the catalog, and the same digest is + // the definition's own bytes: storing either is a creation or a repair. + if catalog == nil || catalog.BundleSHA256 == bundle.SHA256 { + return false, nil + } + if existing != nil && strings.TrimSpace(existing.BundleRef) != "" && + existing.BundleSHA256 == bundle.SHA256 { + return true, nil + } + // A definition that never recorded its digest cannot be told apart from + // these bytes, and they are the only ones anyone can name. + if strings.TrimSpace(catalog.BundleSHA256) == "" { + return false, nil + } + return false, apperrors.NewConflictError( + "this skill was updated in the catalog while the install was starting; try again") +} + +// servedVersionOf is what the image goes on serving once row is handed to a new +// install. A ready row describes the image, so it becomes the served version. +// A row that is already mid-install or failed keeps the version it recorded +// when it last stopped being ready: this attempt replaces the previous attempt, +// not what the image carries. Any other row has nothing in the image. +func (s *TenantSkillService) servedVersionOf( + ctx context.Context, row *types.TenantSkillEntity, +) *types.SkillServedVersion { + switch row.Status { + case types.SkillStatusReady: + return &types.SkillServedVersion{ + Version: row.Version, + Description: row.Description, + Instructions: row.Instructions, + BundleSHA256: row.BundleSHA256, + BundleRef: s.servedBundleRef(ctx, row), + SnapshotID: row.InstalledSnapshotID, + } + case types.SkillStatusInstalling, types.SkillStatusFailed: + return row.Served + default: + return nil + } +} + +// servedBundleRef names the object a ready row's files are read from. A row +// that follows the catalog names none itself, and the catalog is exactly what +// an upload is about to replace, so the definition's object is named instead +// while it still holds these bytes. Recording it is what makes the served +// version a reader that the replacement cannot reclaim from under it. +func (s *TenantSkillService) servedBundleRef(ctx context.Context, row *types.TenantSkillEntity) string { + if ref := strings.TrimSpace(row.BundleRef); ref != "" { + return ref + } + cid := strings.TrimSpace(row.CatalogID) + sha := strings.TrimSpace(row.BundleSHA256) + if cid == "" || sha == "" { + return "" + } + catalog, err := s.skills.GetCatalog(ctx, row.TenantID, cid) + if err != nil || catalog == nil || strings.TrimSpace(catalog.BundleSHA256) != sha { + return "" + } + return strings.TrimSpace(catalog.BundleRef) +} + +// restoreServedVersion makes the served version the row's own again, for the +// paths that conclude the image never moved past it. It returns the archive +// the abandoned attempt had pinned, which nothing names once the row is back. +func restoreServedVersion(e *types.TenantSkillEntity) string { + served := e.Served + if served == nil { + return "" + } + abandoned := strings.TrimSpace(e.BundleRef) + e.Version = served.Version + e.Description = served.Description + e.Instructions = served.Instructions + e.BundleSHA256 = served.BundleSHA256 + e.BundleRef = served.BundleRef + e.InstalledSnapshotID = served.SnapshotID + e.Served = nil + if abandoned == strings.TrimSpace(served.BundleRef) { + return "" + } + return abandoned +} + // takeSkillRowForInstall hands an existing row to the run about to start. // // The transcript locators are cleared along with the error. They name the @@ -253,7 +413,7 @@ func (s *TenantSkillService) ReinstallSkill( "the archive of this skill is no longer stored; install it again from the original bundle", ) } - return s.installSkillArchive(ctx, tenantID, configID, archive, instructions...) + return s.installSkillArchive(ctx, tenantID, configID, archive, skillArchiveStored, instructions...) } func (s *TenantSkillService) runInstall( @@ -421,8 +581,11 @@ func (s *TenantSkillService) runInstall( } // Read before the scratch wipe. requirements.json lives under skillDir and // is not scratch, but reading it first removes an implicit dependency on - // what cleanImageScratch happens to delete. - s.recordEnvDeclaration(ctx, mgr, sess.ID, tenantID, configID, skillID, bundle) + // what cleanImageScratch happens to delete. It is stored only once the + // pointer has moved: the declaration replaces the previous one and drops + // values of variables the new version no longer reads, and until the + // pointer moves it is the previous version that runs and reads them. + declaredEnvs, envsDeclared := s.readEnvDeclaration(ctx, mgr, sess.ID, skillID, bundle) s.publishProgress(ctx, tenantID, configID, skillID, SkillProgress{Percent: 90, Stage: "verified"}) // 6. Wipe the scratch state. It must happen BEFORE the snapshot, or the @@ -512,6 +675,9 @@ func (s *TenantSkillService) runInstall( if err := s.writeReadySkillState(readyCtx, tenantID, configID, skillID, ref.ID, bundle); err != nil { return err } + if envsDeclared { + s.storeEnvDeclaration(readyCtx, tenantID, configID, skillID, bundle, declaredEnvs) + } s.markConfigSandboxesStale(ctx, tenantID, configID) s.publishProgress(ctx, tenantID, configID, skillID, SkillProgress{ Percent: 100, Stage: "done", Status: types.SkillStatusReady, @@ -1244,6 +1410,7 @@ func (s *TenantSkillService) updateSkillFields( func (s *TenantSkillService) writeReadySkillState( ctx context.Context, tenantID uint64, configID, skillID, snapshotID string, bundle *SkillBundle, ) error { + retired := "" if err := s.retrySkillBookkeeping(ctx, func() error { owned, err := s.installStillOwnsTheRow(ctx, tenantID, configID, skillID, bundle) if err != nil { @@ -1254,6 +1421,12 @@ func (s *TenantSkillService) writeReadySkillState( } return s.updateSkillFields(ctx, tenantID, configID, skillID, func(e *types.TenantSkillEntity) { + // The pointer has moved, so the version kept for the length + // of the upgrade is no longer in any image new sessions boot. + if e.Served != nil { + retired = strings.TrimSpace(e.Served.BundleRef) + } + e.Served = nil e.Status = types.SkillStatusReady e.Error = "" e.InstalledSnapshotID = snapshotID @@ -1263,6 +1436,7 @@ func (s *TenantSkillService) writeReadySkillState( return fmt.Errorf("skill %s is installed and serving but could not be marked ready: %w", skillID, err) } + s.releaseInstallBundle(ctx, tenantID, retired) return nil } @@ -2126,32 +2300,32 @@ func (s *TenantSkillService) writeManifestEntry( return store.WriteSessionFile(ctx, sessionID, sandbox.SkillsManifestPath, payload) } -// recordEnvDeclaration reads the environment variables the installer agent -// declared and stores the ones that survive validation. +// readEnvDeclaration reads the environment variables the installer agent +// declared and keeps the ones that survive validation. ok is false when there +// is nothing to store. // -// It returns nothing because none of its failures is a failed install. The -// agent may not have written the file, may have written prose, or may have -// listed nothing that exists in the bundle; in every case the skill is -// installed and working, and the missing declaration costs an admin one manual -// entry in the settings page. Failing the install over it would throw away the -// minutes of dependency installation that already succeeded. -func (s *TenantSkillService) recordEnvDeclaration( - ctx context.Context, mgr sandbox.Manager, sessionID string, - tenantID uint64, configID, skillID string, bundle *SkillBundle, -) { +// None of its failures is a failed install. The agent may not have written the +// file, may have written prose, or may have listed nothing that exists in the +// bundle; in every case the skill is installed and working, and the missing +// declaration costs an admin one manual entry in the settings page. Failing the +// install over it would throw away the minutes of dependency installation that +// already succeeded. +func (s *TenantSkillService) readEnvDeclaration( + ctx context.Context, mgr sandbox.Manager, sessionID, skillID string, bundle *SkillBundle, +) (types.SkillEnvVars, bool) { if bundle == nil { - return + return nil, false } reader, ok := mgr.(sandbox.SessionFileReader) if !ok { logger.Warnf(ctx, "[skill] sandbox backend cannot read files back; skill %s keeps no env declaration", skillID) - return + return nil, false } requirementsPath := sandbox.SkillRequirementsPath(bundle.Name) if requirementsPath == "" { - return + return nil, false } raw, err := reader.ReadSessionFile(ctx, sessionID, requirementsPath) if err != nil { @@ -2161,16 +2335,16 @@ func (s *TenantSkillService) recordEnvDeclaration( if sandbox.IsRemoteNotFound(err) { logger.Infof(ctx, "[skill] %s declared no environment variables (no %s)", skillID, requirementsPath) - return + return nil, false } logger.Warnf(ctx, "[skill] %s: reading its env declaration at %s failed: %v", skillID, requirementsPath, err) - return + return nil, false } declared, err := parseEnvDeclaration(raw) if err != nil { logger.Warnf(ctx, "[skill] %s wrote an unreadable env declaration: %v", skillID, err) - return + return nil, false } envs := validateEnvDeclarations(declared, bundle) if len(envs) == 0 && len(declared) > 0 { @@ -2181,14 +2355,25 @@ func (s *TenantSkillService) recordEnvDeclaration( "[skill] all %d environment variable(s) declared for %s were rejected "+ "(bad name, not mentioned anywhere in the bundle, or reserved)", len(declared), skillID) - return + return nil, false } + return envs, true +} +// storeEnvDeclaration folds a declaration read by readEnvDeclaration onto the +// row. It writes only onto the row this run just made ready: a newer upload or +// a queued remove that took the row over owns its declaration too. +func (s *TenantSkillService) storeEnvDeclaration( + ctx context.Context, tenantID uint64, configID, skillID string, bundle *SkillBundle, envs types.SkillEnvVars, +) { skill, err := s.skills.GetSkill(ctx, tenantID, configID, skillID) if err != nil || skill == nil { logger.Warnf(ctx, "[skill] load %s to store its env declaration failed: %v", skillID, err) return } + if skill.Status != types.SkillStatusReady || skill.BundleSHA256 != bundle.SHA256 { + return + } merged := mergeEnvDeclaration(skill.Envs, envs) if err := s.skills.UpdateSkillEnvs(ctx, tenantID, configID, skillID, merged); err != nil { logger.Warnf(ctx, "[skill] store the env declaration of %s failed: %v", skillID, err) diff --git a/internal/application/service/tenant_skill_reaper.go b/internal/application/service/tenant_skill_reaper.go index ee36767a..33d6697a 100644 --- a/internal/application/service/tenant_skill_reaper.go +++ b/internal/application/service/tenant_skill_reaper.go @@ -104,6 +104,12 @@ func (s *TenantSkillService) ReapStuckRuns(ctx context.Context) (int, error) { snapshotID, serving, known := s.skillFilesInLiveImage(ctx, row) switch row.Status { case types.SkillStatusInstalling: + if row.Served != nil { + if s.reapStuckUpgrade(ctx, row, snapshotID, serving, known) { + reaped++ + } + continue + } // An unreadable config or ledger must not be treated as a failed // install: the image may still be serving this skill. if serving || !known { @@ -144,8 +150,12 @@ func (s *TenantSkillService) ReapStuckRuns(ctx context.Context) (int, error) { continue } if serving { + // A row whose upgrade never landed goes back to the version + // the image has, exactly as a failed removal restores it. + abandoned := "" if err := s.updateSkillFields(ctx, row.TenantID, row.SandboxConfigID, row.ID, func(e *types.TenantSkillEntity) { + abandoned = restoreServedVersion(e) e.Status = types.SkillStatusReady e.Error = "" e.InstallingSince = nil @@ -156,10 +166,11 @@ func (s *TenantSkillService) ReapStuckRuns(ctx context.Context) (int, error) { logger.Warnf(ctx, "[skill] restore abandoned removal %s failed: %v", row.ID, err) continue } + s.releaseInstallBundle(ctx, row.TenantID, abandoned) reaped++ continue } - pinned := strings.TrimSpace(row.BundleRef) + pinned := installBundleRefs(row) if err := s.skills.DeleteSkill(ctx, row.TenantID, row.SandboxConfigID, row.ID); err != nil { logger.Warnf(ctx, "[skill] drop abandoned removal %s failed: %v", row.ID, err) continue @@ -167,7 +178,9 @@ func (s *TenantSkillService) ReapStuckRuns(ctx context.Context) (int, error) { // The row was the last thing naming an archive it owned outright, // so the sweep that drops it is what makes those bytes reachable // by nothing. A definition's own object has other names and stays. - s.releaseInstallBundle(ctx, row.TenantID, pinned) + for _, ref := range pinned { + s.releaseInstallBundle(ctx, row.TenantID, ref) + } reaped++ } } @@ -177,6 +190,59 @@ func (s *TenantSkillService) ReapStuckRuns(ctx context.Context) (int, error) { return reaped, nil } +// reapStuckUpgrade settles an install that died while a previous version was +// being served. That version keeps the skill available for as long as the row +// stays installing, so nothing here is urgent enough to guess: a chain the +// reaper cannot read is left for the next sweep. +// +// The nearest install generation of this skill on the live chain says what +// happened. The served version's own generation means the pointer never moved, +// so the row goes back to that version. A newer one means the upgrade landed and +// only its terminal write was lost, so the row is ready as it stands. None means +// the image carries nothing of the skill, and there is nothing left to serve. +func (s *TenantSkillService) reapStuckUpgrade( + ctx context.Context, row *types.TenantSkillEntity, snapshotID string, serving, known bool, +) bool { + if !known { + return false + } + var released []string + err := s.updateSkillFields(ctx, row.TenantID, row.SandboxConfigID, row.ID, + func(e *types.TenantSkillEntity) { + e.InstallingSince = nil + if e.Served == nil { + return + } + switch { + case serving && snapshotID != "" && snapshotID != e.Served.SnapshotID: + released = append(released, strings.TrimSpace(e.Served.BundleRef)) + e.Served = nil + e.Status = types.SkillStatusReady + e.Error = "" + e.InstalledSnapshotID = snapshotID + case serving: + // The catalog still differs from the restored row, so the + // console offers the upgrade again. + released = append(released, restoreServedVersion(e)) + e.Status = types.SkillStatusReady + e.Error = "" + default: + released = append(released, strings.TrimSpace(e.Served.BundleRef)) + e.Served = nil + e.Status = types.SkillStatusFailed + e.Error = skillInstallInterruptedMessage + } + }) + if err != nil { + logger.Warnf(ctx, "[skill] settle abandoned upgrade %s failed: %v", row.ID, err) + return false + } + for _, ref := range released { + s.releaseInstallBundle(ctx, row.TenantID, ref) + } + return true +} + // skillFilesInLiveImage reports whether the image every new session boots // still carries this skill, and which snapshot of the chain put it there. // known is false when the config or the ledger cannot be read, so a caller diff --git a/internal/application/service/tenant_skill_remove.go b/internal/application/service/tenant_skill_remove.go index 053c7fb1..949438de 100644 --- a/internal/application/service/tenant_skill_remove.go +++ b/internal/application/service/tenant_skill_remove.go @@ -409,9 +409,9 @@ func (s *TenantSkillService) finishRemoval( // Read before the delete: past it the only record of what this row named is // gone, and a pinned archive would be left behind with nothing to reclaim it. - pinned := "" + var pinned []string if row, err := s.skills.GetSkill(ctx, tenantID, configID, skillID); err == nil && row != nil { - pinned = strings.TrimSpace(row.BundleRef) + pinned = installBundleRefs(row) } if err := s.retrySkillBookkeeping(ctx, func() error { return s.skills.DeleteSkill(ctx, tenantID, configID, skillID) @@ -421,7 +421,9 @@ func (s *TenantSkillService) finishRemoval( } // The definition's archive survives this; only an object no catalog and no // other install names is reclaimed. - s.releaseInstallBundle(ctx, tenantID, pinned) + for _, ref := range pinned { + s.releaseInstallBundle(ctx, tenantID, ref) + } // Only an image that actually changed can leave a bound sandbox out of // date. Marking after a removal that moved no pointer would destroy and // rebuild every live sandbox of the config - throwing away each session's @@ -444,12 +446,15 @@ func (s *TenantSkillService) restoreSkillAfterFailedRemoval( return } status := types.SkillStatusFailed + abandoned := "" _ = s.updateSkillFields(ctx, tenantID, configID, skillID, func(e *types.TenantSkillEntity) { // A skill that reached an image is still in it, so it goes back to // ready and the operator can retry. One that never did has nothing // to be ready for, and calling it ready would point the agent at - // files no image carries. + // files no image carries. A row whose upgrade never landed goes + // back to the version the image actually has. + abandoned = restoreServedVersion(e) if e.InstalledSnapshotID != "" { status = types.SkillStatusReady } @@ -457,11 +462,40 @@ func (s *TenantSkillService) restoreSkillAfterFailedRemoval( e.Error = cause.Error() e.InstallingSince = nil }) + s.releaseInstallBundle(ctx, tenantID, abandoned) s.publishProgress(ctx, tenantID, configID, skillID, SkillProgress{ Percent: 100, Stage: "failed", Status: status, Log: cause.Error(), }) } +// installReadsBundle reports whether an install still needs the object ref: +// the archive its row names, or the one its served version was built from. +func installReadsBundle(row *types.TenantSkillEntity, ref string) bool { + if row == nil || ref == "" { + return false + } + for _, named := range installBundleRefs(row) { + if named == ref { + return true + } + } + return false +} + +// installBundleRefs lists the objects an install names. +func installBundleRefs(row *types.TenantSkillEntity) []string { + var refs []string + if ref := strings.TrimSpace(row.BundleRef); ref != "" { + refs = append(refs, ref) + } + if row.Served != nil { + if ref := strings.TrimSpace(row.Served.BundleRef); ref != "" && (len(refs) == 0 || refs[0] != ref) { + refs = append(refs, ref) + } + } + return refs +} + // removeStillOwnsTheRow is the lock-side counterpart of RemoveSkill's // optimistic status flip. A newer upload of the same name already set the // row back to installing; deleting it (or restoring it to ready/failed) @@ -511,7 +545,7 @@ func (s *TenantSkillService) releaseInstallBundle( return } for _, row := range installs { - if row != nil && strings.TrimSpace(row.BundleRef) == ref { + if installReadsBundle(row, ref) { return } } diff --git a/internal/application/service/tenant_skill_served_test.go b/internal/application/service/tenant_skill_served_test.go new file mode 100644 index 00000000..90f1bc63 --- /dev/null +++ b/internal/application/service/tenant_skill_served_test.go @@ -0,0 +1,367 @@ +package service + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/Tencent/WeKnora/internal/types" +) + +func servedSkillMD(version, body string) string { + return "---\nname: pdf-tools\ndescription: Extract text from PDF files\n" + + "version: " + version + "\n---\n\n" + body + "\n" +} + +func servedSkillZip(t *testing.T, version, body, script string) []byte { + t.Helper() + return zipBundle(t, map[string]string{ + "SKILL.md": servedSkillMD(version, body), + "scripts/extract.py": script, + }) +} + +func (f *installFixture) usableSkills(t *testing.T) []*types.TenantSkillEntity { + t.Helper() + return effectiveTenantSkills(context.Background(), f.configRepo, f.skillRepo, 7, "cfg-1") +} + +func (f *installFixture) installReady(t *testing.T, archive []byte) string { + t.Helper() + skillID, err := f.svc.InstallSkill(context.Background(), 7, "cfg-1", archive) + require.NoError(t, err) + f.awaitSkillSettled(t, skillID) + row, err := f.skillRepo.GetSkill(context.Background(), 7, "cfg-1", skillID) + require.NoError(t, err) + require.Equal(t, types.SkillStatusReady, row.Status) + return skillID +} + +// The agent is told about a skill for as long as the image carries it. An +// upgrade only replaces the image when it succeeds, so while it runs the +// previous version is what every sandbox executes, and the agent must keep +// seeing exactly that version rather than losing the skill for minutes. +func TestUpgradeKeepsThePreviousVersionServingWhileItRuns(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + first := servedSkillZip(t, "1.0.0", "Use v1.", "print('v1')\n") + firstBundle, err := ParseSkillBundle(first) + require.NoError(t, err) + skillID := fx.installReady(t, first) + installed, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + firstRef := fx.catalogRefFor(t, installed.CatalogID) + + cat, err := fx.svc.RegisterCatalogFromArchive(ctx, 7, servedSkillZip(t, "2.0.0", "Use v2.", "print('v2')\n")) + require.NoError(t, err) + + var during []*types.TenantSkillEntity + var rowDuring *types.TenantSkillEntity + var deletedDuring []string + fx.beforeExecute = func() { + during = fx.usableSkills(t) + rowDuring, _ = fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + deletedDuring = append([]string(nil), fx.deletedBundles...) + } + _, err = fx.svc.InstallCatalogToConfigs(ctx, 7, cat.ID, []string{"cfg-1"}) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + + require.Equal(t, types.SkillStatusInstalling, rowDuring.Status) + require.Equal(t, "2.0.0", rowDuring.Version, "the row describes the install in flight") + require.Len(t, during, 1, "the skill must not drop out of the agent's set mid-upgrade") + require.Equal(t, "1.0.0", during[0].Version) + require.Contains(t, during[0].Instructions, "Use v1.") + require.Equal(t, firstBundle.SHA256, during[0].BundleSHA256) + require.Equal(t, firstRef, during[0].BundleRef, "resource files are read from the archive v1 was built from") + // Pointing the install at v2 releases the v1 archive its row was pinned + // to; the served version still reads it, so it must survive that. + require.NotContains(t, deletedDuring, firstRef) + require.Equal(t, types.SkillStatusReady, during[0].Status) + + after := fx.usableSkills(t) + require.Len(t, after, 1) + require.Equal(t, "2.0.0", after[0].Version) + require.Contains(t, after[0].Instructions, "Use v2.") + final, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + require.Nil(t, final.Served, "once the pointer moves nothing older is served") + require.Contains(t, fx.deletedBundles, firstRef, "v1's archive lost its last reader with the upgrade") +} + +// An upgrade that fails never moved the pointer, so v1 is still what runs. +// Dropping the skill until someone fixes the upgrade would turn a failed +// attempt at an improvement into an outage. +func TestFailedUpgradeLeavesThePreviousVersionServing(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + skillID := fx.installReady(t, servedSkillZip(t, "1.0.0", "Use v1.", "print('v1')\n")) + installed, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + firstRef := fx.catalogRefFor(t, installed.CatalogID) + + fx.agentErr = errAgentBoom + // An upload is the harder path: the catalog object v1 was read from is + // replaced by v2's before the install even starts. + _, err = fx.svc.InstallSkill(ctx, 7, "cfg-1", servedSkillZip(t, "2.0.0", "Use v2.", "print('v2')\n")) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + + failed, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + require.Equal(t, types.SkillStatusFailed, failed.Status) + require.Equal(t, &SkillServedInfo{Version: "1.0.0"}, ServedInfoOf(failed)) + require.NotContains(t, fx.deletedBundles, firstRef, + "the served version still reads its files from v1's archive") + + usable := fx.usableSkills(t) + require.Len(t, usable, 1) + require.Equal(t, "1.0.0", usable[0].Version) + require.Equal(t, firstRef, usable[0].BundleRef) + + // A retry of the failed upgrade retries v2, and a success retires v1. + fx.agentErr = nil + _, err = fx.svc.ReinstallSkill(ctx, 7, "cfg-1", skillID) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + usable = fx.usableSkills(t) + require.Len(t, usable, 1) + require.Equal(t, "2.0.0", usable[0].Version) + require.Contains(t, fx.deletedBundles, firstRef) +} + +// A first install has nothing in the image yet, so there is nothing to serve +// while it runs or after it fails. +func TestFirstInstallServesNothingUntilItIsReady(t *testing.T) { + fx := newInstallFixture(t) + fx.agentErr = errAgentBoom + + skillID, err := fx.svc.InstallSkill(context.Background(), 7, "cfg-1", + servedSkillZip(t, "1.0.0", "Use v1.", "print('v1')\n")) + require.NoError(t, err) + fx.awaitSkillSettled(t, skillID) + + row, err := fx.skillRepo.GetSkill(context.Background(), 7, "cfg-1", skillID) + require.NoError(t, err) + require.Equal(t, types.SkillStatusFailed, row.Status) + require.Nil(t, row.Served) + require.Nil(t, ServedInfoOf(row)) + require.Empty(t, fx.usableSkills(t)) +} + +// The declaration replaces the previous one and drops the values of variables +// the new version stops reading. Written before the pointer moves, an upgrade +// that then fails would leave v1 running without a credential an admin typed. +func TestUpgradeFailingAfterVerificationKeepsTheServedVersionsEnvValues(t *testing.T) { + fx := newInstallFixture(t) + ctx := context.Background() + fx.seedReadySkillWithSHA(fx.bundle.SHA256, "snap-v1") + require.NoError(t, fx.skillRepo.UpdateSkillEnvs(ctx, 7, "cfg-1", "sk-1", types.SkillEnvVars{ + {Name: "LEGACY_KEY", Required: true, Value: "admin-typed"}, + })) + row, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", "sk-1") + require.NoError(t, err) + row.Served = fx.svc.servedVersionOf(ctx, row) + row.Status = types.SkillStatusInstalling + require.NoError(t, fx.skillRepo.UpdateSkill(ctx, row)) + + fx.readsTavilyKey() + fx.declareEnvFile(`{"env":[{"name":"TAVILY_API_KEY","required":true}]}`) + runCtx, cancel := context.WithCancel(ctx) + defer cancel() + fx.cancelDuringSnapshot = cancel + + require.Error(t, fx.svc.runInstall(runCtx, 7, "cfg-1", "sk-1", fx.bundle)) + + after, err := fx.skillRepo.GetSkill(ctx, 7, "cfg-1", "sk-1") + require.NoError(t, err) + require.Equal(t, types.SkillEnvVars{{Name: "LEGACY_KEY", Required: true, Value: "admin-typed"}}, after.Envs) + require.NotNil(t, after.ServedView(), "v1 still runs, and still reads its credential") +} + +func TestServedViewReadsThePreviousVersionOnlyWhileItIsServed(t *testing.T) { + served := &types.SkillServedVersion{ + Version: "1.0.0", Description: "v1", Instructions: "Use v1.", + BundleSHA256: "sha-v1", BundleRef: "file://v1.zip", SnapshotID: "snap-v1", + } + base := types.TenantSkillEntity{ + ID: "sk-1", Name: "pdf-tools", Version: "2.0.0", Instructions: "Use v2.", + BundleSHA256: "sha-v2", Enabled: true, Served: served, Error: "boom", + } + + for _, status := range []string{types.SkillStatusInstalling, types.SkillStatusFailed} { + row := base + row.Status = status + view := row.ServedView() + require.NotNil(t, view, status) + require.Equal(t, types.SkillStatusReady, view.Status) + require.Equal(t, "1.0.0", view.Version) + require.Equal(t, "Use v1.", view.Instructions) + require.Equal(t, "file://v1.zip", view.BundleRef) + require.Equal(t, "snap-v1", view.InstalledSnapshotID) + require.Empty(t, view.Error) + require.Nil(t, view.Served) + require.Equal(t, "2.0.0", row.Version, "the row itself is not rewritten") + } + + removing := base + removing.Status = types.SkillStatusRemoving + require.Nil(t, removing.ServedView(), "a removal takes the skill away on purpose") + + ready := base + ready.Status = types.SkillStatusReady + require.Same(t, &ready, ready.ServedView()) + + firstInstall := base + firstInstall.Status = types.SkillStatusInstalling + firstInstall.Served = nil + require.Nil(t, firstInstall.ServedView()) +} + +func TestServedVersionRoundTripsThroughTheRepository(t *testing.T) { + repo := catalogTestRepo(t, "file:"+t.Name()+"?mode=memory&cache=shared") + ctx := context.Background() + now := time.Now() + require.NoError(t, repo.CreateSkill(ctx, &types.TenantSkillEntity{ + ID: "sk-1", TenantID: 7, SandboxConfigID: "cfg-1", Name: "pdf-tools", + Status: types.SkillStatusReady, CreatedAt: now, UpdatedAt: now, + })) + row, err := repo.GetSkill(ctx, 7, "cfg-1", "sk-1") + require.NoError(t, err) + require.Nil(t, row.Served) + + row.Status = types.SkillStatusInstalling + row.Served = &types.SkillServedVersion{Version: "1.0.0", BundleRef: "file://v1.zip", SnapshotID: "snap-v1"} + require.NoError(t, repo.UpdateSkill(ctx, row)) + row, err = repo.GetSkill(ctx, 7, "cfg-1", "sk-1") + require.NoError(t, err) + require.Equal(t, &types.SkillServedVersion{Version: "1.0.0", BundleRef: "file://v1.zip", SnapshotID: "snap-v1"}, + row.Served) + + row.Served = nil + require.NoError(t, repo.UpdateSkill(ctx, row)) + row, err = repo.GetSkill(ctx, 7, "cfg-1", "sk-1") + require.NoError(t, err) + require.Nil(t, row.Served) +} + +func (f *reaperFixture) stuckUpgrade() { + staleSince := f.now.Add(-skillInstallStuckTTL - time.Minute) + f.skills.put(&types.TenantSkillEntity{ + ID: "sk-1", TenantID: 7, SandboxConfigID: "cfg-1", Name: "pdf", Enabled: true, + Version: "2.0.0", Instructions: "Use v2.", BundleSHA256: "sha-v2", BundleRef: "file://v2.zip", + InstalledSnapshotID: "snap-v1", Status: types.SkillStatusInstalling, InstallingSince: &staleSince, + Served: &types.SkillServedVersion{ + Version: "1.0.0", Instructions: "Use v1.", BundleSHA256: "sha-v1", + BundleRef: "file://v1.zip", SnapshotID: "snap-v1", + }, + }) + f.installed("sk-1", "snap-v1", "") +} + +// An upgrade whose process died before the pointer moved left v1 in the image. +// Healing the row to ready as it stands would label v1's files as v2. +func TestReapStuckRunsRestoresTheServedVersionOfAnUpgradeThatNeverLanded(t *testing.T) { + fx := newReaperFixture(t) + fx.stuckUpgrade() + fx.live("snap-v1") + + n, err := fx.svc.ReapStuckRuns(context.Background()) + + require.NoError(t, err) + require.Equal(t, 1, n) + got := fx.skills.mustGet("sk-1") + require.Equal(t, types.SkillStatusReady, got.Status) + require.Equal(t, "1.0.0", got.Version) + require.Equal(t, "Use v1.", got.Instructions) + require.Equal(t, "sha-v1", got.BundleSHA256) + require.Equal(t, "file://v1.zip", got.BundleRef) + require.Equal(t, "snap-v1", got.InstalledSnapshotID) + require.Nil(t, got.Served) + require.Nil(t, got.InstallingSince) +} + +// The upgrade's own generation is on the live chain: only its terminal write +// was lost, so the row already says what the image serves. +func TestReapStuckRunsKeepsAnUpgradeThatLanded(t *testing.T) { + fx := newReaperFixture(t) + fx.stuckUpgrade() + fx.installed("sk-1", "snap-v2", "snap-v1") + fx.live("snap-v2") + + n, err := fx.svc.ReapStuckRuns(context.Background()) + + require.NoError(t, err) + require.Equal(t, 1, n) + got := fx.skills.mustGet("sk-1") + require.Equal(t, types.SkillStatusReady, got.Status) + require.Equal(t, "2.0.0", got.Version) + require.Equal(t, "snap-v2", got.InstalledSnapshotID) + require.Nil(t, got.Served) +} + +// v1 keeps the skill available while the row stays installing, so a chain the +// reaper cannot read is no reason to guess which version the image has. +func TestReapStuckRunsLeavesAnUpgradeAloneWhenTheChainCannotBeFollowed(t *testing.T) { + fx := newReaperFixture(t) + fx.stuckUpgrade() + fx.live("snap-missing") + + n, err := fx.svc.ReapStuckRuns(context.Background()) + + require.NoError(t, err) + require.Zero(t, n) + got := fx.skills.mustGet("sk-1") + require.Equal(t, types.SkillStatusInstalling, got.Status) + require.NotNil(t, got.Served) + require.NotNil(t, got.ServedView()) +} + +// Removing a skill whose upgrade failed, and failing at that, puts the row back +// as the version the image still has rather than the upgrade that never landed. +func TestFailedRemovalOfAFailedUpgradeRestoresTheServedVersion(t *testing.T) { + fx := newInstallFixture(t) + fx.seedInstalledSkill("sk-1", "snap-old", 2) + fx.seedInstalledSkill("sk-2", "snap-old", 2) + require.NoError(t, fx.svc.updateSkillFields(context.Background(), 7, "cfg-1", "sk-1", + func(e *types.TenantSkillEntity) { + e.Served = &types.SkillServedVersion{ + Version: e.Version, Instructions: "Use v1.", BundleRef: e.BundleRef, SnapshotID: "snap-old", + } + e.Version = "2.0.0" + e.Instructions = "Use v2." + e.BundleRef = "" + })) + fx.rmExitCode = 1 + + require.Error(t, fx.svc.runRemove(context.Background(), 7, "cfg-1", "sk-1")) + + skill, err := fx.skillRepo.GetSkill(context.Background(), 7, "cfg-1", "sk-1") + require.NoError(t, err) + require.Equal(t, types.SkillStatusReady, skill.Status) + require.Equal(t, "1.0.0", skill.Version) + require.Equal(t, "Use v1.", skill.Instructions) + require.Equal(t, "file://sk-1.zip", skill.BundleRef) + require.Nil(t, skill.Served) +} + +func TestReapStuckRunsRestoresTheServedVersionOfAnAbandonedRemoval(t *testing.T) { + fx := newReaperFixture(t) + fx.stuckUpgrade() + row := fx.skills.mustGet("sk-1") + row.Status = types.SkillStatusRemoving + fx.skills.put(row) + fx.live("snap-v1") + + n, err := fx.svc.ReapStuckRuns(context.Background()) + + require.NoError(t, err) + require.Equal(t, 1, n) + got := fx.skills.mustGet("sk-1") + require.Equal(t, types.SkillStatusReady, got.Status) + require.Equal(t, "1.0.0", got.Version) + require.Equal(t, "file://v1.zip", got.BundleRef) + require.Nil(t, got.Served) +} diff --git a/internal/application/service/tenant_skill_source.go b/internal/application/service/tenant_skill_source.go index f1b998d9..2d371802 100644 --- a/internal/application/service/tenant_skill_source.go +++ b/internal/application/service/tenant_skill_source.go @@ -115,7 +115,7 @@ func (s *TenantSkillService) InstallSkillFromSource( if err != nil { return "", err } - return s.installParsedSkill(ctx, tenantID, configID, bundle, archive) + return s.installParsedSkill(ctx, tenantID, configID, bundle, archive, skillArchiveUploaded) } func skillSourceHTTPClient(override *http.Client) *http.Client { diff --git a/internal/application/service/user_env.go b/internal/application/service/user_env.go index 27a021f7..e53b6362 100644 --- a/internal/application/service/user_env.go +++ b/internal/application/service/user_env.go @@ -469,7 +469,8 @@ func (s *UserEnvService) findVisibleSkill( } // skillAcceptsUserEnvs reports whether a skill is one the agent can actually -// run, which is the only kind worth asking a member for a credential for. +// run, which is the only kind worth asking a member for a credential for. A +// skill mid-upgrade still runs its previous version, so it still counts. func skillAcceptsUserEnvs(row *types.TenantSkillEntity) bool { - return row != nil && row.Enabled && row.Status == types.SkillStatusReady + return row != nil && row.Enabled && row.ServedView() != nil } diff --git a/internal/handler/sandbox_skill.go b/internal/handler/sandbox_skill.go index 5ad5b508..ac601fff 100644 --- a/internal/handler/sandbox_skill.go +++ b/internal/handler/sandbox_skill.go @@ -117,6 +117,10 @@ type skillResponse struct { CreatedAt time.Time `json:"created_at"` UpdatedAt time.Time `json:"updated_at"` + // Served is the previous version that keeps running while this install + // is in flight or after it failed. + Served *service.SkillServedInfo `json:"served,omitempty"` + Envs []skillEnvResponse `json:"envs,omitempty"` } @@ -164,6 +168,7 @@ func toSkillResponse(e *types.TenantSkillEntity) skillResponse { InstallMessageID: e.InstallMessageID, CreatedAt: e.CreatedAt, UpdatedAt: e.UpdatedAt, + Served: service.ServedInfoOf(e), Envs: toSkillEnvResponses(e.Envs), } } diff --git a/internal/types/tenant_skill.go b/internal/types/tenant_skill.go index b0cc9084..17e852e9 100644 --- a/internal/types/tenant_skill.go +++ b/internal/types/tenant_skill.go @@ -1,6 +1,8 @@ package types import ( + "database/sql/driver" + "encoding/json" "time" "gorm.io/gorm" @@ -89,6 +91,15 @@ type TenantSkillEntity struct { // this skill needs, each optionally carrying a workspace-wide admin value. Envs SkillEnvVars `json:"envs,omitempty" gorm:"type:jsonb"` + // Served is the version the live image still carries while a newer install + // of this skill is in flight or has failed. The fields above describe the + // latest install attempt from the moment it starts, but the image pointer + // only moves when that attempt succeeds, so without this the agent would + // lose a skill whose previous version is sitting in the image. Nil when + // the row is ready (the fields above are what is served) or when nothing + // of this skill ever reached the image. + Served *SkillServedVersion `json:"served,omitempty" gorm:"type:jsonb"` + Status string `gorm:"type:varchar(32);not null"` Error string `gorm:"type:text"` // InstallingSince drives the stuck-run reaper for both install and remove. @@ -102,6 +113,81 @@ type TenantSkillEntity struct { // TableName pins the table so GORM's pluralizer cannot drift. func (e *TenantSkillEntity) TableName() string { return "tenant_skills" } +// ServedView is the row as the agent should see it: the row itself when it is +// ready, the version the image still carries while a newer install is in +// flight or has failed, and nil when the image serves nothing of this skill. +// +// The view is a copy marked ready, so every reader downstream of the usable +// set - discovery, instructions, resource files - answers from the served +// version without knowing an upgrade exists. Enabled and Envs stay the row's +// own: an administrator's switch and stored values apply to whichever version +// is running. +func (e *TenantSkillEntity) ServedView() *TenantSkillEntity { + if e == nil { + return nil + } + if e.Status == SkillStatusReady { + return e + } + if e.Served == nil || + (e.Status != SkillStatusInstalling && e.Status != SkillStatusFailed) { + return nil + } + view := *e + view.Served = nil + view.Version = e.Served.Version + view.Description = e.Served.Description + view.Instructions = e.Served.Instructions + view.BundleSHA256 = e.Served.BundleSHA256 + view.BundleRef = e.Served.BundleRef + view.InstalledSnapshotID = e.Served.SnapshotID + view.Status = SkillStatusReady + view.Error = "" + return &view +} + +// SkillServedVersion is what a row described when it was last ready, kept +// while a newer install of the same skill has not replaced it in the image. +type SkillServedVersion struct { + Version string `json:"version,omitempty"` + Description string `json:"description,omitempty"` + Instructions string `json:"instructions,omitempty"` + BundleSHA256 string `json:"bundle_sha256,omitempty"` + // BundleRef names the archive the served files came from. It is resolved + // when the version is recorded, including for a row that was following the + // catalog, because the catalog may be re-registered while the upgrade + // runs and the served files must stay readable after it is. + BundleRef string `json:"bundle_ref,omitempty"` + // SnapshotID is the image generation that put this version in. The reaper + // compares it with the live chain to tell an upgrade that never moved the + // pointer from one whose terminal write was lost. + SnapshotID string `json:"snapshot_id,omitempty"` +} + +// Value stores the served version as one JSON column. +func (v SkillServedVersion) Value() (driver.Value, error) { + return json.Marshal(v) +} + +// Scan reads the served version back. +func (v *SkillServedVersion) Scan(value interface{}) error { + var b []byte + switch raw := value.(type) { + case nil: + return nil + case []byte: + b = raw + case string: + b = []byte(raw) + default: + return nil + } + if len(b) == 0 { + return nil + } + return json.Unmarshal(b, v) +} + // TenantSkillSnapshotEntity is the image-chain ledger. It exists because // snapshots are billable provider resources whose IDs we hand out: we must be // able to say which generation came from which parent, and never leave a diff --git a/migrations/versioned/000104_skill_served_version.down.sql b/migrations/versioned/000104_skill_served_version.down.sql new file mode 100644 index 00000000..0a6c2d39 --- /dev/null +++ b/migrations/versioned/000104_skill_served_version.down.sql @@ -0,0 +1,3 @@ +DO $$ BEGIN RAISE NOTICE '[Migration 000104 down] Dropping tenant_skills.served'; END $$; + +ALTER TABLE tenant_skills DROP COLUMN IF EXISTS served; diff --git a/migrations/versioned/000104_skill_served_version.up.sql b/migrations/versioned/000104_skill_served_version.up.sql new file mode 100644 index 00000000..4fb7c774 --- /dev/null +++ b/migrations/versioned/000104_skill_served_version.up.sql @@ -0,0 +1,12 @@ +-- Description: The version of a skill the live image still carries while a +-- newer install of it is in flight or has failed. An install rewrites the row +-- to describe the version being installed as soon as it starts, but the image +-- pointer only moves when that install succeeds, so the previous version keeps +-- running in every sandbox. This column is what lets the agent go on using it +-- instead of losing the skill for the length of the upgrade. +DO $$ BEGIN RAISE NOTICE '[Migration 000104] Adding tenant_skills.served'; END $$; + +ALTER TABLE tenant_skills ADD COLUMN IF NOT EXISTS served JSONB; + +COMMENT ON COLUMN tenant_skills.served IS + 'Version still in the image while a newer install is pending or failed: {version,description,instructions,bundle_sha256,bundle_ref,snapshot_id}. NULL when the row itself is what is served.';