diff --git a/.agents/backend-signing.md b/.agents/backend-signing.md index 6d388dbbaf2a..f52be93f0984 100644 --- a/.agents/backend-signing.md +++ b/.agents/backend-signing.md @@ -150,3 +150,10 @@ compromised and we've shipped malicious signed images: page if/when added. - **Tarball/HTTP backend signing.** Cosign can sign arbitrary blobs, but for now non-OCI backends keep using the `sha256:` field in YAML. + +Embedded consumers with host-provisioned private Sigstore trust can use +`cosignverify.NewVerifierWithTrustedMaterial`. It bypasses public TUF fetching, +not signature, identity, SCT, or Rekor checks. Never construct this input from a +registry attachment or catalog response. `NewVerifier` retains its public TUF +default. Hermetic cryptographic regressions live in `offline_internal_test.go`; +the optional live registry tests remain separate and are not acceptance evidence. diff --git a/.agents/localai-assistant-mcp.md b/.agents/localai-assistant-mcp.md index bd391164c781..deeb0283757a 100644 --- a/.agents/localai-assistant-mcp.md +++ b/.agents/localai-assistant-mcp.md @@ -92,6 +92,32 @@ The in-process MCP server runs inside the same LocalAI binary that serves chat. The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.md` rule 1) that requires the LLM to summarise the action and wait for explicit user confirmation before calling it. There is no `plan_*`/`apply_*` two-step in code. If you add a mutating tool, do **not** add per-tool confirmation logic in Go — instead, list the new tool name in `prompts/10_safety.md` so the LLM knows it falls under the confirmation rule. +## Scoped exception: add-on tools are not assistant tools + +The prompt-only confirmation above applies to the admin catalog in +`pkg/mcp/localaitools/` and to nothing else. Tools of installed add-ons (for +example Tables) follow a different, code-enforced rule, documented in +[docs/content/features/tables.md](../docs/content/features/tables.md): + +- They are never registered in this catalog and no `/api/addons` route appears + in `toolToHTTPRoute`. `TestToolHTTPRouteMappingComplete` stays unchanged, and + the consumer matrix test (`core/services/addons/tools`) fails if a tool name + or route of the add-on surface appears here. +- A change proposed by an add-on tool is a stored proposal. Only the proposer's + own browser session confirms it, with the digest of what the host card showed. + A model's "yes" or an argument named `approved` is never authority. +- REST-to-MCP parity is intentionally partial for add-ons: the lifecycle + mutations (install, enable, disable, upgrade, uninstall, library changes) are + deliberately **not** exposed as assistant tools, because the assistant is one + process-wide server with no per-call principal and would inherit prompt-only + authorization. Do not add them without a per-call actor and a host approval + card. +- New code that discovers or executes tools must be classified in + `tools.Consumers` (`core/services/addons/tools/consumers.go`) as route or + reject. The matrix test scans the sources and fails for an unclassified file. + +Do not rewrite the unrelated assistant tools to match this rule. + ## Distributed mode The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period. diff --git a/.github/workflows/addons-package.yml b/.github/workflows/addons-package.yml new file mode 100644 index 000000000000..043bb97b055d --- /dev/null +++ b/.github/workflows/addons-package.yml @@ -0,0 +1,426 @@ +--- +name: 'add-on packages' + +# Builds, checks, signs and publishes the first-party add-on OCI packages +# (Tables, Voice, Face, Fine-tuning and Quantization). This workflow is +# independent of the backend image workflows: it has its own registry input, +# its own signer identity and its own jobs. +# +# Event policy (the rules live in scripts/addons/channel.sh): +# pull_request validate and build legs, publish nothing, no secrets +# push to master publish the development channel +# push to other branch publish nothing +# push of a v tag prepare the immutable version, move no alias +# release published promote latest, for a stable non-draft release only +# +# Every add-on has its own platform file (addons//package/platforms.yaml), +# its own package legs and its own repository. Tables publishes to +# ADDON_OCI_REPOSITORY, which is the repository /tables. Each other +# add-on publishes to /, where the namespace is +# ADDON_OCI_REPOSITORY without its last path element. A publishing event fails +# for an add-on until one of its platforms is enabled. +# +# Aliases move only in the promote job, after every platform leg passed, the +# index was pushed by digest, signed and verified. + +on: + push: + branches: ['**'] + tags: ['v*'] + paths: + - 'addons/**' + - 'pkg/addons/**' + - 'scripts/addons/**' + - 'gallery/addons/**' + - '.github/workflows/addons-package.yml' + pull_request: + paths: + - 'addons/**' + - 'pkg/addons/**' + - 'scripts/addons/**' + - 'gallery/addons/**' + - '.github/workflows/addons-package.yml' + release: + types: [published] + +permissions: + contents: read + +jobs: + validate: + name: validate package rules + runs-on: ubuntu-24.04 + timeout-minutes: 20 + permissions: + contents: read + strategy: + fail-fast: true + matrix: + addon: [tables, voice, face, fine-tuning, quantization, studio, talk] + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + - uses: actions/setup-node@v6 + with: + node-version: '22' + - name: Install the host UI build tools + # The package checks build the add-on UI with the bundler the host UI + # already installs. + run: npm ci --prefix core/http/react-ui + - name: Generate the gRPC stubs the packaging checks compile + # The generated stubs are not in the tree and the Go packages of the + # host import them. Same pinned tools as the test workflow. + if: matrix.addon == 'tables' + run: | + curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip + sudo unzip -j -d /usr/local/bin protoc.zip bin/protoc + rm protoc.zip + go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2 + go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af + PATH="$PATH:$HOME/go/bin" make protogen-go + - name: Build the UI of this add-on + env: + ADDON: ${{ matrix.addon }} + run: node "addons/$ADDON/ui/build.mjs" + - name: Fill the module cache the packaging checks read + # package_test.sh runs offline (GOPROXY=off) against its own cache. + # The suite covers every add-on, so one leg runs it. + if: matrix.addon == 'tables' + run: | + GOPATH="$PWD/.cache/addons/go" GOMODCACHE="$PWD/.cache/addons/go/pkg/mod" \ + GOCACHE="$PWD/.cache/addons/go-build" go mod download + - name: Run the packaging checks + if: matrix.addon == 'tables' + run: sh scripts/addons/package_test.sh + + plan: + name: plan the event + runs-on: ubuntu-24.04 + timeout-minutes: 10 + permissions: + contents: read + env: + ADDON_OCI_REPOSITORY: ${{ vars.ADDON_OCI_REPOSITORY }} + outputs: + publish: ${{ steps.plan.outputs.publish }} + immutable: ${{ steps.plan.outputs.immutable }} + aliases: ${{ steps.plan.outputs.aliases }} + channel: ${{ steps.plan.outputs.channel }} + version: ${{ steps.plan.outputs.version }} + matrix: ${{ steps.plan.outputs.matrix }} + addons: ${{ steps.plan.outputs.addons }} + has_platforms: ${{ steps.plan.outputs.has_platforms }} + registry: ${{ steps.plan.outputs.registry }} + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + - name: Decide what this event may publish + id: plan + env: + EVENT: ${{ github.event_name }} + REF: ${{ github.ref }} + SHA: ${{ github.sha }} + DRAFT: ${{ github.event.release.draft || false }} + PRERELEASE: ${{ github.event.release.prerelease || false }} + run: | + set -eu + flags="" + [ "$DRAFT" = "true" ] && flags="$flags --draft" + [ "$PRERELEASE" = "true" ] && flags="$flags --prerelease" + # shellcheck disable=SC2086 + sh scripts/addons/channel.sh --event "$EVENT" --ref "$REF" --sha "$SHA" $flags > decision.txt + cat decision.txt + version=$(sed -n 's/^version=//p' decision.txt) + publish=$(sed -n 's/^publish=//p' decision.txt) + [ -n "$version" ] || version="0.0.0-dev.${SHA%"${SHA#????????????}"}" + if [ "$publish" = "true" ]; then + # Fail closed: a publishing event needs the namespace input and a + # non-empty release platform matrix. There is no default registry. + if [ -z "$ADDON_OCI_REPOSITORY" ]; then + echo "ADDON_OCI_REPOSITORY is not configured; refusing to publish" >&2 + exit 1 + fi + fi + # One set of legs for each add-on, from its own platform file. Only + # the "no enabled platform" answer is tolerated, and only on an event + # that publishes nothing: that add-on gets no legs. On a publishing + # event it fails the plan. A missing or malformed platform file, or + # any other error, fails the plan on every event. + include='[]' + for id in tables voice face fine-tuning quantization studio talk; do + cfg="addons/$id/package/platforms.yaml" + err="matrix-$id.err" + if rows=$(go run ./scripts/addons/cmd/addonpkg matrix --config "$cfg" 2>"$err"); then + : + elif grep -q 'no enabled platform' "$err"; then + if [ "$publish" = "true" ]; then + echo "add-on $id has no enabled platform; refusing to publish" >&2 + exit 1 + fi + continue + else + echo "cannot read the platform matrix of add-on $id" >&2 + cat "$err" >&2 + exit 1 + fi + include=$(printf '%s' "$rows" | jq -c --argjson all "$include" --arg addon "$id" '$all + [.include[] | . + {addon: $addon}]') + done + matrix=$(printf '%s' "$include" | jq -c '{include: .}') + addons=$(printf '%s' "$include" | jq -c 'map(.addon) | unique') + has=$(printf '%s' "$include" | jq 'length > 0') + { + cat decision.txt + echo "version=$version" + echo "matrix=$matrix" + echo "addons=$addons" + echo "has_platforms=$has" + echo "registry=${ADDON_OCI_REPOSITORY%%/*}" + } >> "$GITHUB_OUTPUT" + + package: + name: package ${{ matrix.addon }} ${{ matrix.platform }} + needs: [plan, validate] + if: needs.plan.outputs.has_platforms == 'true' + runs-on: ${{ matrix.runner }} + timeout-minutes: 60 + permissions: + contents: read + strategy: + fail-fast: true + matrix: ${{ fromJSON(needs.plan.outputs.matrix) }} + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + - uses: actions/setup-node@v6 + with: + node-version: '22' + - name: Build the UI + env: + ADDON: ${{ matrix.addon }} + run: | + npm ci --prefix core/http/react-ui + node "addons/$ADDON/ui/build.mjs" + - name: Package this platform + env: + ADDON: ${{ matrix.addon }} + VERSION: ${{ needs.plan.outputs.version }} + PLATFORM: ${{ matrix.platform }} + SHA: ${{ github.sha }} + run: | + sh scripts/addons/package.sh --config "addons/$ADDON/package/platforms.yaml" --layout "layout" --version "$VERSION" --platform "$PLATFORM" --revision "$SHA" + - name: Native runtime check + env: + ADDON: ${{ matrix.addon }} + VERSION: ${{ needs.plan.outputs.version }} + PLATFORM: ${{ matrix.platform }} + run: | + go run ./scripts/addons/cmd/addonpkg native-check --config "addons/$ADDON/package/platforms.yaml" --layout layout --platform "$PLATFORM" --version "$VERSION" + - uses: actions/upload-artifact@v7 + with: + name: layout-${{ matrix.addon }}-${{ matrix.slug }} + path: layout + if-no-files-found: error + retention-days: 7 + + publish: + name: publish and sign ${{ matrix.addon }} + needs: [plan, validate, package] + if: needs.plan.outputs.publish == 'true' + runs-on: ubuntu-24.04 + timeout-minutes: 60 + environment: addons-release + concurrency: + group: addons-publish-${{ matrix.addon }}-${{ needs.plan.outputs.immutable }} + cancel-in-progress: false + permissions: + contents: read + id-token: write + strategy: + # A leg that is pushing must finish, so one failure never cancels another. + fail-fast: false + matrix: + addon: ${{ fromJSON(needs.plan.outputs.addons) }} + env: + ADDON_OCI_BASE: ${{ vars.ADDON_OCI_REPOSITORY }} + ADDON: ${{ matrix.addon }} + # cosign gates --registry-referrers-mode=oci-1-1 behind this flag. + COSIGN_EXPERIMENTAL: '1' + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + - name: Choose the repository of this add-on + # Tables publishes to the configured repository. Every other add-on + # publishes to /. The later steps read + # ADDON_OCI_REPOSITORY. + run: | + set -eu + case "$ADDON_OCI_BASE" in + */*) ;; + *) echo "ADDON_OCI_REPOSITORY needs a namespace and a name" >&2; exit 1 ;; + esac + if [ "$ADDON" = tables ]; then + repo="$ADDON_OCI_BASE" + else + repo="${ADDON_OCI_BASE%/*}/$ADDON" + fi + echo "ADDON_OCI_REPOSITORY=$repo" >> "$GITHUB_ENV" + - uses: actions/download-artifact@v8 + with: + pattern: layout-${{ matrix.addon }}-* + path: legs + - name: Join the platform legs into one index + id: merge + env: + VERSION: ${{ needs.plan.outputs.version }} + SHA: ${{ github.sha }} + run: | + set -eu + go run ./scripts/addons/cmd/addonpkg merge --config "addons/$ADDON/package/platforms.yaml" \ + --out merged --version "$VERSION" --revision "$SHA" legs/* + - name: Registry login + uses: docker/login-action@v3 + with: + registry: ${{ needs.plan.outputs.registry }} + username: ${{ secrets.ADDON_OCI_USERNAME }} + password: ${{ secrets.ADDON_OCI_PASSWORD }} + - name: Push the platform digests and the index + id: push + env: + IMMUTABLE: ${{ needs.plan.outputs.immutable }} + run: | + set -eu + go run ./scripts/addons/cmd/addonpkg publish --config "addons/$ADDON/package/platforms.yaml" \ + --layout merged --repository "$ADDON_OCI_REPOSITORY" --tag "$IMMUTABLE" --adopt-existing | tee push.txt + echo "digest=$(sed -n 's/^digest=//p' push.txt)" >> "$GITHUB_OUTPUT" + echo "children=$(sed -n 's/^children=//p' push.txt)" >> "$GITHUB_OUTPUT" + - uses: sigstore/cosign-installer@v3 + with: + cosign-release: 'v2.6.5' + - name: Sign by digest + id: sign + env: + DIGEST: ${{ steps.push.outputs.digest }} + run: | + set -eu + cosign sign --yes --recursive \ + --new-bundle-format \ + --registry-referrers-mode=oci-1-1 \ + "${ADDON_OCI_REPOSITORY}@${DIGEST}" + - name: Verify the signature with the exact add-on workflow identity + id: verify + env: + DIGEST: ${{ steps.push.outputs.digest }} + CHILDREN: ${{ steps.push.outputs.children }} + IDENTITY: https://github.com/${{ github.repository }}/.github/workflows/addons-package.yml@${{ github.ref }} + ISSUER: https://token.actions.githubusercontent.com + run: | + set -eu + for d in "$DIGEST" $(echo "$CHILDREN" | tr ',' ' '); do + cosign verify --new-bundle-format \ + --certificate-identity "$IDENTITY" \ + --certificate-oidc-issuer "$ISSUER" \ + "${ADDON_OCI_REPOSITORY}@${d}" > /dev/null + done + - name: Keep the digest for the promote job + # A matrix job cannot hand one output to the next job, so the digest of + # each add-on travels in an artifact of its own. + env: + DIGEST: ${{ steps.push.outputs.digest }} + run: printf '%s\n' "$DIGEST" > digest.txt + - uses: actions/upload-artifact@v7 + with: + name: digest-${{ matrix.addon }} + path: digest.txt + if-no-files-found: error + retention-days: 7 + + promote: + name: promote channel aliases ${{ matrix.addon }} + needs: [plan, validate, publish] + if: needs.plan.outputs.publish == 'true' && needs.plan.outputs.aliases != '' + runs-on: ubuntu-24.04 + timeout-minutes: 20 + environment: addons-release + # One promotion at a time per channel, and a queued run never cancels a + # running one. The freshness check below makes a slow, older run a no-op. + concurrency: + group: addons-promote-${{ matrix.addon }}-${{ needs.plan.outputs.channel }} + cancel-in-progress: false + permissions: + contents: read + strategy: + fail-fast: false + matrix: + addon: ${{ fromJSON(needs.plan.outputs.addons) }} + env: + ADDON_OCI_BASE: ${{ vars.ADDON_OCI_REPOSITORY }} + ADDON: ${{ matrix.addon }} + steps: + - uses: actions/checkout@v7 + with: + fetch-depth: 0 + persist-credentials: false + - uses: actions/setup-go@v6 + with: + go-version-file: go.mod + - name: Choose the repository of this add-on + # The same rule as in publish: Tables keeps the configured repository, + # every other add-on uses /. + run: | + set -eu + case "$ADDON_OCI_BASE" in + */*) ;; + *) echo "ADDON_OCI_REPOSITORY needs a namespace and a name" >&2; exit 1 ;; + esac + if [ "$ADDON" = tables ]; then + repo="$ADDON_OCI_BASE" + else + repo="${ADDON_OCI_BASE%/*}/$ADDON" + fi + echo "ADDON_OCI_REPOSITORY=$repo" >> "$GITHUB_ENV" + - uses: actions/download-artifact@v8 + with: + name: digest-${{ matrix.addon }} + path: digest + - name: Registry login + uses: docker/login-action@v3 + with: + registry: ${{ needs.plan.outputs.registry }} + username: ${{ secrets.ADDON_OCI_USERNAME }} + password: ${{ secrets.ADDON_OCI_PASSWORD }} + - name: Move the aliases + env: + ALIASES: ${{ needs.plan.outputs.aliases }} + VERSION: ${{ needs.plan.outputs.version }} + SHA: ${{ github.sha }} + REF: ${{ github.ref }} + EVENT: ${{ github.event_name }} + run: | + set -eu + DIGEST=$(tr -d '[:space:]' < digest/digest.txt) + case "$DIGEST" in + sha256:????????????????????????????????????????????????????????????????) ;; + *) echo "the digest of $ADDON is missing or malformed" >&2; exit 1 ;; + esac + for alias in $(echo "$ALIASES" | tr ',' ' '); do + go run ./scripts/addons/cmd/addonpkg promote \ + --repository "$ADDON_OCI_REPOSITORY" --alias "$alias" \ + --digest "$DIGEST" --version "$VERSION" --revision "$SHA" \ + --ref "$REF" --event "$EVENT" --git-dir .git + done diff --git a/.gitignore b/.gitignore index 3572eaaf8bbb..48ebf2eee318 100644 --- a/.gitignore +++ b/.gitignore @@ -140,3 +140,4 @@ formal-verification/out/ /backend/go/gemxcpp/build-*/ /backend/go/gemxcpp/package/ /backend/go/gemxcpp/gemxcpp +addons/internal/uibuild/test/*/dist diff --git a/addons/face/cmd/face/main.go b/addons/face/cmd/face/main.go new file mode 100644 index 000000000000..5531849e53d4 --- /dev/null +++ b/addons/face/cmd/face/main.go @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: MIT + +// Command face is the executable of the Face add-on. It serves no tools: +// the add-on works in the browser, through the SDK and the core API. +package main + +import ( + "os" + + "github.com/mudler/LocalAI/addons/internal/idle" +) + +func main() { + os.Exit(idle.Run(os.Args)) +} diff --git a/addons/face/manifest.json b/addons/face/manifest.json new file mode 100644 index 000000000000..9aba07e9c38f --- /dev/null +++ b/addons/face/manifest.json @@ -0,0 +1,25 @@ +{ + "schemaVersion": 1, + "id": "face", + "version": "0.1.0", + "executable": "face", + "os": "linux", + "arch": "amd64", + "host": { "min": 1, "max": 1 }, + "api": { "min": 1, "max": 1 }, + "protocol": { "min": 1, "max": 1 }, + "dataFormat": { "min": 1, "max": 1 }, + "capabilities": { + "ui": { + "entry": "ui/face.js", + "css": ["ui/face.css"], + "sdk": { "min": 2, "max": 2 }, + "pages": ["recognition"], + "navigation": [], + "mainPage": "recognition", + "icon": "ui/icon.png", + "title": "Face", + "description": "Find out whose face is in a photo, or whether two photos show the same person." + } + } +} diff --git a/addons/face/package/platforms.yaml b/addons/face/package/platforms.yaml new file mode 100644 index 000000000000..23666df9dbff --- /dev/null +++ b/addons/face/package/platforms.yaml @@ -0,0 +1,38 @@ +# Release platform matrix of the face add-on package. +# +# A platform ships only while "enabled" is true. Enable a platform in the same +# change that shows its native check passing on a runner of that platform; +# until then it stays listed and disabled. The release workflow fails closed +# when no platform is enabled, and it never builds a platform that is not +# enabled here. +# +# runner: the GitHub runner label that builds and checks the platform +# natively. Required for an enabled platform. +# native_check: "executable-start": the packaged executable is extracted on +# the native runner and started. Required for an enabled platform. +# runtime: files besides the executable that the platform needs at run +# time, relative to the package root. They are taken from the +# runtime directory of the build and must exist. +schema: 1 +addon: face +executable: face +manifest: addons/face/manifest.json +# The directory whose content is laid at the package root (ui/face.js ...). +ui: addons/face/ui/dist +platforms: + - platform: linux/amd64 + enabled: true + runner: ubuntu-24.04 + native_check: executable-start + - platform: linux/arm64 + enabled: true + runner: ubuntu-24.04-arm + native_check: executable-start + - platform: darwin/arm64 + enabled: true + runner: macos-14 + native_check: executable-start + - platform: darwin/amd64 + enabled: true + runner: macos-15-intel + native_check: executable-start diff --git a/addons/face/ui/.gitignore b/addons/face/ui/.gitignore new file mode 100644 index 000000000000..1225aa2fae58 --- /dev/null +++ b/addons/face/ui/.gitignore @@ -0,0 +1,2 @@ +# Written by build.mjs; the package build makes it. +/dist/ diff --git a/addons/face/ui/build.mjs b/addons/face/ui/build.mjs new file mode 100644 index 000000000000..99db94154c44 --- /dev/null +++ b/addons/face/ui/build.mjs @@ -0,0 +1,13 @@ +import { fileURLToPath } from 'node:url' +import { dirname } from 'node:path' +import { buildAddonUI } from '../../internal/uibuild/uibuild.mjs' + +const here = dirname(fileURLToPath(import.meta.url)) + +export async function build() { + return buildAddonUI({ root: here, name: 'face', rgb: [20, 100, 90] }) +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + build().then((dir) => console.log(`built ${dir}`), (e) => { console.error(e); process.exit(1) }) +} diff --git a/addons/face/ui/icon.png b/addons/face/ui/icon.png new file mode 100644 index 000000000000..a72b97f95f37 Binary files /dev/null and b/addons/face/ui/icon.png differ diff --git a/addons/face/ui/package.json b/addons/face/ui/package.json new file mode 100644 index 000000000000..f3a51c012802 --- /dev/null +++ b/addons/face/ui/package.json @@ -0,0 +1,8 @@ +{ + "name": "localai-face-ui", + "private": true, + "version": "0.1.0", + "description": "Native UI of the Face add-on: a standalone ES module that the host loads after its own build.", + "type": "module", + "scripts": { "build": "node build.mjs", "test": "node test/run.mjs" } +} diff --git a/addons/face/ui/src/FaceRecognition.jsx b/addons/face/ui/src/FaceRecognition.jsx new file mode 100644 index 000000000000..bd5689cbc5de --- /dev/null +++ b/addons/face/ui/src/FaceRecognition.jsx @@ -0,0 +1,116 @@ +// The Faces page: who is this, same person, the people this browser knows and +// what is stored. Ported from pages/FaceRecognition.jsx. The known-faces list +// and the enrol sheet are the shared views inside the shared Workbench. +// Detect and analyze, and the raw faceprint, sit behind a disclosure. +import { createClipInput } from '../../../internal/identity-ui/src/ClipInput.jsx' +import { createWorkbench } from '../../../internal/identity-ui/src/Workbench.jsx' +import { createKnownList } from '../../../internal/identity-ui/src/KnownList.jsx' +import { createEnrolSheet } from '../../../internal/identity-ui/src/EnrolSheet.jsx' +import { createVerdict } from '../../../internal/identity-ui/src/Verdict.jsx' +import { createDistanceScale } from '../../../internal/identity-ui/src/DistanceScale.jsx' +import { createErrorNote } from '../../../internal/identity-ui/src/ErrorNote.jsx' +import { createModelNeeded } from '../../../internal/identity-ui/src/ModelNeeded.jsx' +import { createBoundingBoxCanvas } from '../../../internal/identity-ui/src/BoundingBoxCanvas.jsx' +import { createIdentityRegistry } from '../../../internal/identity-ui/src/registry.js' +import { createMoreFaceTools } from './MoreFaceTools.jsx' + +const CAP_FACE_RECOGNITION = 'FLAG_FACE_RECOGNITION' + +// `identity` is the shared helper module (identity.js), passed as in the Voice +// add-on. The Faces page has no tab counts, so it reads nothing from it. +// eslint-disable-next-line no-unused-vars +export function createFaceRecognition(ctx, api, identity) { + const { h, t, Icon, ui, viewer } = ctx + const { useCallback, useEffect, useMemo, useState } = ctx.React + + const { ClipInput } = createClipInput(ctx) + const { KnownList } = createKnownList(ctx) + const { Verdict } = createVerdict(ctx) + const { DistanceScale } = createDistanceScale(ctx) + const { ErrorNote } = createErrorNote(ctx) + const { useRegistry } = createIdentityRegistry(ctx) + const { BoundingBoxCanvas } = createBoundingBoxCanvas(ctx) + const { EnrolSheet } = createEnrolSheet(ctx, { ClipInput, ErrorNote, api }) + const { ModelNeeded } = createModelNeeded(ctx, api) + const { MoreFaceTools } = createMoreFaceTools(ctx, api) + const { Workbench } = createWorkbench(ctx, { ClipInput, KnownList, EnrolSheet, Verdict, DistanceScale, ErrorNote, useRegistry, BoundingBoxCanvas, api }) + + // Shown in place of a page whose feature the signed-in user may not use. It + // replaces a silent redirect: the person learns what is off and what turns it on. + function FeatureOff() { + return ( +
+
+ {t('off.badge')} +

{t('off.face.title')}

+

{t('off.face.text')}

+
+ {t('needed.whatTitle')} +

{t('off.face.what')}

+
+
+
+ ) + } + + // The face models that are installed: the capability list of the core, + // narrowed to the face-recognition flag, as the core hook useModels does. + function useFaceModels() { + const [state, setState] = useState({ models: [], loading: true }) + const load = useCallback(async (silent) => { + if (!silent) setState(s => ({ ...s, loading: true })) + try { + const data = await api.models.listCapabilities() + const models = (data?.data || []).filter(m => !m.disabled && m.capabilities?.includes(CAP_FACE_RECOGNITION)) + setState({ models, loading: false }) + } catch { + setState({ models: [], loading: false }) + } + }, []) + useEffect(() => { load(false) }, [load]) + return { ...state, refetch: useCallback(() => load(true), [load]) } + } + + function FacesPage({ path }) { + const isAdmin = viewer.isAdmin() + const [model, setModel] = useState(() => ctx.modelFrom(path) || '') + const { models, loading, refetch } = useFaceModels() + const names = useMemo(() => models.map(m => m.id), [models]) + const needModel = !loading && names.length === 0 + + return ( +
+ + + + ) : null} + /> + : null} + more={needModel ? null : ( +
+ {t('face.moreTitle')} {t('face.moreHint')} + +
+ )} + /> +
+ ) + } + + // The page is for a viewer who holds face_recognition (spec, "Classification": + // the gate of the page is the gate of the core route, and the library inside + // it has no stricter gate). The server checks every /v1/face/* call again. A + // viewer without the feature gets the notice and no call. + function FaceRecognition(props) { + if (!viewer.hasFeature('face_recognition')) return + return + } + + return { FaceRecognition } +} diff --git a/addons/face/ui/src/MoreFaceTools.jsx b/addons/face/ui/src/MoreFaceTools.jsx new file mode 100644 index 000000000000..df70f8c6a8c0 --- /dev/null +++ b/addons/face/ui/src/MoreFaceTools.jsx @@ -0,0 +1,246 @@ +// The two developer tools behind the face workbench: detect faces and guess +// attributes (guesses, often wrong), and the raw faceprint. Ported from +// components/identity/MoreFaceTools.jsx; the shared widgets come from the SDK. +import { createBoundingBoxCanvas } from '../../../internal/identity-ui/src/BoundingBoxCanvas.jsx' + +const TABS = [ + { id: 'analyze', icon: 'chart-bar', label: 'Detect and analyze' }, + { id: 'embed', icon: 'code', label: 'Embedding' }, +] + +export function createMoreFaceTools(ctx, api) { + const { h, Fragment, Icon, ui } = ctx + const { useMemo, useState } = ctx.React + const { BoundingBoxCanvas } = createBoundingBoxCanvas(ctx) + + function MoreFaceTools({ model }) { + const [tab, setTab] = useState('analyze') + return ( +
+ +
+ {tab === 'analyze' && } + {tab === 'embed' && } +
+
+ ) + } + + function AnalyzeTab({ model }) { + const [img, setImg] = useState(null) + const [actions, setActions] = useState({ age: false, gender: false, emotion: false, race: false }) + const [antiSpoofing, setAntiSpoofing] = useState(false) + const [loading, setLoading] = useState(false) + const [error, setError] = useState(null) + const [result, setResult] = useState(null) + const [focusIdx, setFocusIdx] = useState(0) + + const submit = async (e) => { + e.preventDefault() + if (!model) { ctx.toast('Select a face model first', 'warning'); return } + if (!img) { ctx.toast('Add an image to analyze', 'warning'); return } + setLoading(true); setError(null); setResult(null); setFocusIdx(0) + try { + const data = await api.face.analyze({ + model, + img: img.dataUrl, + actions: Object.entries(actions).filter(([, v]) => v).map(([k]) => k), + anti_spoofing: antiSpoofing, + }) + setResult(data) + if (!data?.faces?.length) ctx.toast('No face detected in the image', 'warning') + } catch (err) { + setError(err.message) + } finally { + setLoading(false) + } + } + + const boxes = useMemo(() => (result?.faces || []).map((f, i) => ({ + x: f.region.x, y: f.region.y, w: f.region.w, h: f.region.h, + label: f.dominant_emotion || f.dominant_gender || `Face ${i + 1}`, + sublabel: f.age ? `~${Math.round(f.age)}y` : null, + tone: i === focusIdx ? 'accent' : 'default', + })), [result, focusIdx]) + + const faces = result?.faces || [] + const focus = faces[focusIdx] + + return ( +
+ + +
+ {loading &&
} + {error && } + {!loading && !error && !result && ( + + )} + {result && img && ( + <> +
+
+ + {faces.length > 1 && ( +
+ {faces.map((_, i) => ( + + ))} +
+ )} +
+
+ {focus && ( + <> +
+
+

Face {focusIdx + 1}

+ {antiSpoofing && } +
+
+ {focus.age != null && <>
Age
~{Math.round(focus.age)}
} + {focus.dominant_gender && <>
Gender
{focus.dominant_gender}
} + {focus.dominant_emotion && <>
Emotion
{focus.dominant_emotion}
} + {focus.dominant_race && <>
Race
{focus.dominant_race}
} + {focus.face_confidence != null && <>
Detection
{(focus.face_confidence * 100).toFixed(1)}%
} +
+
+ + + + + )} +
+
+ + + )} +
+
+ ) + } + + function EmbedTab({ model }) { + const [img, setImg] = useState(null) + const [loading, setLoading] = useState(false) + const [error, setError] = useState(null) + const [result, setResult] = useState(null) + const [elapsedMs, setElapsedMs] = useState(null) + + const submit = async (e) => { + e.preventDefault() + if (!model) { ctx.toast('Select a face model first', 'warning'); return } + if (!img) { ctx.toast('Add an image', 'warning'); return } + setLoading(true); setError(null); setResult(null) + const started = performance.now() + try { + const data = await api.face.embed({ model, img: img.dataUrl }) + setElapsedMs(performance.now() - started) + setResult(data) + } catch (err) { + setError(err.message) + } finally { + setLoading(false) + } + } + + return ( +
+ +
+ {loading &&
} + {error && } + {!loading && !error && !result && ( + + )} + {result && ( + + )} +
+
+ ) + } + + function LivenessPill({ isReal, score }) { + if (isReal == null) { + return Not checked + } + return ( + + + {isReal ? 'Real' : 'Spoof'} + {score != null && {(score * 100).toFixed(0)}%} + + ) + } + + // Not ui.EmptyState: this is the biometrics empty card of the core page. + function EmptyState({ icon, title, body }) { + return ( +
+
+ ) + } + + function ResponseDetails({ data }) { + return ( +
+ Raw response + 
{JSON.stringify(data, null, 2)}
+
+ ) + } + + return { MoreFaceTools } +} diff --git a/addons/face/ui/src/api.js b/addons/face/ui/src/api.js new file mode 100644 index 000000000000..52ca36227b4d --- /dev/null +++ b/addons/face/ui/src/api.js @@ -0,0 +1,35 @@ +// The core routes the Face add-on uses, reached with the signed-in user's own +// session (sdk.core.fetch). Paths and bodies are the ones of faceApi in +// core/http/react-ui/src/utils/api.js; only the transport changed. register +// takes { model, name, img, labels }, identify { model, img, ... }. +import { createIdentityApi } from '../../../internal/identity-ui/src/registry.js' + +export function createApi(ctx) { + const { json } = ctx + const query = (params = {}) => { + const q = new URLSearchParams() + for (const [k, v] of Object.entries(params)) if (v !== undefined && v !== null && v !== '') q.set(k, String(v)) + const s = q.toString() + return s ? `?${s}` : '' + } + // /v1/voice/* and /v1/face/*: the shared views read both kinds; the page only + // uses the face one. A failed call keeps its status (a 404 on forget reads as + // already gone). + const identity = createIdentityApi(ctx) + return { + identity, + face: identity.face, + models: { + listCapabilities: () => json('/api/models/capabilities'), + // The gallery list, as modelsApi.list builds it: empty parameters are left out. + list: (params) => json(`/api/models${query(params)}`), + // Starts a gallery install (modelsApi.install). + install: (id) => json(`/api/models/install/${encodeURIComponent(id)}`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: '{}' }), + }, + // The running installs, polled while one is in progress. + operations: { + list: () => json('/api/operations'), + }, + raw: ctx.core, + } +} diff --git a/addons/face/ui/src/ctx.jsx b/addons/face/ui/src/ctx.jsx new file mode 100644 index 000000000000..92c318f7ffa4 --- /dev/null +++ b/addons/face/ui/src/ctx.jsx @@ -0,0 +1,67 @@ +import en from './i18n/en.js' +import { modelFrom as pickModel } from './lib/model.js' + +// The one object every page factory of the Face add-on receives. It binds the +// SDK once: React, the shared components, the session-bound core fetch, the +// translation and the small adapters that replace the host router and toasts. +export function createContext(sdk) { + const { React } = sdk + const h = React.createElement + const Fragment = React.Fragment + sdk.locale.define('en', Object.fromEntries(Object.entries(en).filter(([, v]) => typeof v === 'string'))) + const lists = Object.fromEntries(Object.entries(en).filter(([, v]) => Array.isArray(v))) + + const t = (key, vars) => sdk.locale.t(key, key, vars) + // A list of strings or objects (the "why" and "stored" notes). + const list = (key) => lists[key] || [] + + // A link to a page of this add-on, as an anchor the shell navigates. + function Link({ page, query, className, children, ...rest }) { + const href = sdk.navigation.href(query ? `${page}?${query}` : page) + return h('a', { + href, className, + onClick: (e) => { if (e.metaKey || e.ctrlKey || e.shiftKey || e.button !== 0) return; e.preventDefault(); sdk.navigation.navigate(query ? `${page}?${query}` : page) }, + ...rest, + }, children) + } + + const useSearch = () => new URLSearchParams(sdk.navigation.current().search) + // The model a page works on: ?model= first (a moved link carries it + // there, because a model id may hold "/" or ":" that a path segment cannot), + // then the first path segment of a link typed by hand. + const modelFrom = (path) => pickModel(sdk.navigation.current().search, path) + + const toast = (message, kind = 'info') => (sdk.notify[kind] || sdk.notify.info)(message) + + // The path prefix LocalAI is served under, read the way the host reads it + // (utils/basePath.js: the element). document.baseURI would be the + // address of the current page when there is no such element. + const basePath = (() => { + if (typeof document === 'undefined') return '' + const el = document.querySelector('base[href]') + if (!el) return '' + try { return new URL(el.getAttribute('href')).pathname.replace(/\/+$/, '') } catch { return '' } + })() + + const json = async (path, init) => { + const response = await sdk.core.fetch(path, init) + if (!response.ok) { + const body = await response.json().catch(() => ({})) + throw new Error(body?.error?.message || `HTTP ${response.status}`) + } + return response.json() + } + + // The undo bar of a delayed action: text and an Undo button. Replaces the + // host's toast with an undo, which the SDK does not expose. + function UndoBar({ text, undoLabel, onUndo }) { + return h('div', { className: 'vc-undo', role: 'status' }, h('span', null, text), h(sdk.ui.Button, { variant: 'ghost', size: 'sm', onClick: onUndo }, undoLabel)) + } + + return { + sdk, React, h, Fragment, t, list, Icon: sdk.Icon, ui: sdk.ui, hooks: sdk.hooks, viewer: sdk.viewer, core: sdk.core, + Link, useSearch, modelFrom, toast, json, UndoBar, basePath, + openPanel: (title, render) => sdk.layers.openPanel({ title, render }), + markDirty: (flag) => sdk.markDirty(flag), + } +} diff --git a/addons/face/ui/src/i18n/en.js b/addons/face/ui/src/i18n/en.js new file mode 100644 index 000000000000..efa468ebb761 --- /dev/null +++ b/addons/face/ui/src/i18n/en.js @@ -0,0 +1,198 @@ +// Generated once from the host locale files; edit by hand from here on. +export default { + "faces.title": "Faces", + "faces.lede": "Find out whose face is in a photo, or whether two photos show the same person. Photos go only to the model running on this server.", + "tool.aria": "Recognition tool", + "tool.sameTab": "Same person?", + "tool.clip": "Clip", + "tool.clipA": "Clip A", + "tool.clipB": "Clip B", + "tool.photo": "Photo", + "tool.photoA": "Photo A", + "tool.photoB": "Photo B", + "tool.compare": "Compare", + "tool.comparing": "Comparing", + "tool.workingAria": "Working on it", + "face.person": "person", + "face.print": "faceprint", + "face.whoTab": "Who is this", + "face.whoLede": "Matches a photo against the people you enrolled.", + "face.sameLede": "Two photos, one question. Nobody has to be enrolled.", + "face.whoGo": "Find who this is", + "face.working": "Searching", + "face.whoNote": "Searches the people enrolled on this server. The photo you test with is not stored.", + "face.sameNote": "The answer comes from one faceprint per photo. Nothing is stored.", + "face.workingText": "Comparing the photo with the faceprints the server holds.", + "face.probably": "Probably {{name}}", + "face.matchText": "{{name}}'s faceprint is the closest of {{count}} compared, {{distance}} away. Anything under {{cutoff}} counts as the same person.", + "face.nobody": "Nobody enrolled looks like this", + "face.missText": "The closest was {{name}} at {{distance}}, past the cut-off of {{cutoff}}. That usually means a new person, or a photo that is blurred, turned away or in poor light.", + "face.noneEnrolled": "Nobody is enrolled to compare with", + "face.noneEnrolledText": "Enrol a person first, then look up a photo against them.", + "face.noneReturned": "The server returned no people", + "face.noneReturnedText": "It holds no faceprints made with this model. If the server restarted, enrolled faces are gone: enrol them again.", + "face.enrolThis": "Enrol this face", + "face.same": "Probably the same person", + "face.different": "Probably different people", + "face.knownTitle": "Known people", + "face.enrol": "Enrol a person", + "face.enrolTitle": "Enrol a person", + "face.enrolIntro": "The server keeps a faceprint and a name. It does not keep the photo.", + "face.emptyTitle": "Nobody enrolled in this browser yet", + "face.emptyText": "Enrol a person with one clear photo, then test a photo against them.", + "face.agreedHint": "I have their permission, or another lawful basis, to store a faceprint of them.", + "face.keepHint": "Lets you see the photo and enrol again after a server restart. Off by default. It is stored in this browser only.", + "face.antiSpoof": "Check for photos of photos", + "face.antiSpoofHint": "Needs a model that supports it.", + "face.real": "Real", + "face.spoof": "Spoof", + "face.found": "Face in photo {{n}}", + "face.neededTitle": "Faces need a face model", + "face.neededText": "Face recognition turns a photo into a faceprint and compares faceprints. It runs on a face-recognition model, and none is installed on this server.", + "face.neededWhat": "Install a model that supports face recognition. Once one is installed, this page turns on by itself. Users also need the Face recognition permission.", + "face.moreTitle": "More tools", + "face.moreHint": "detect faces, guess traits, get the raw faceprint", + "face.why": [ + "The server matches one face per photo. A group photo is matched on a single face.", + "Light, angle, glasses and blur change a face a lot.", + "This is a similarity score, not a security check. Do not use it to unlock anything." + ], + "face.stored": [ + { + "where": "On the server", + "what": "A {{print}} (numbers that describe the face, not the picture), a name and your labels. They are kept in memory and are lost when the server restarts. The photo is not kept." + }, + { + "where": "In this browser", + "what": "The name, labels and time of each person you enrolled. A copy of the photo only if you tick Keep a copy when enrolling." + }, + { + "where": "Photos you test with", + "what": "They go to the model on this server to be compared and are not kept." + } + ], + "same.text": "The two {{print}}s are {{distance}} apart. Anything under {{cutoff}} counts as the same person.", + "same.distance": "distance", + "word.strong": "Strong match", + "word.likely": "Likely match", + "word.close": "Close call", + "word.near": "Just past the cut-off", + "scale.aria": "Distance scale with the cut-off at {{cutoff}}. Lower means more alike. {{dots}}", + "scale.cutoff": "cut-off {{value}}", + "scale.axis": "distance between {{print}}s, lower means more alike", + "scale.oneDot": "one dot per enrolled {{person}}", + "scale.onePair": "one dot for the pair", + "ranks.aria": "Closest people, nearest first", + "ranks.within": "within cut-off", + "ranks.far": "too far", + "cutoff.label": "Cut-off", + "cutoff.default": "model default", + "cutoff.reset": "Reset to {{value}}", + "cutoff.stricter": "Stricter", + "cutoff.looser": "Looser", + "cutoff.explain": "Stricter means fewer wrong matches and more misses. The answer above updates as you move it. The server is not asked again.", + "how.title": "How to read this.", + "how.body": "Distance is how far apart two {{print}}s are, 0 meaning identical. The cut-off is the line between same and different. The word Strong or Likely says how far inside the line a result sits. It is not a probability.", + "why.title": "Why this might be wrong", + "why.short": "The clip is short: {{s}} s. Three seconds or more of speech works better.", + "clip.mic": "microphone", + "clip.camera": "camera", + "clip.record": "Record", + "clip.takePhoto": "Take a photo", + "clip.chooseAudio": "Choose a file", + "clip.choosePhoto": "Choose a photo", + "clip.readAloud": "Read this aloud, normal pace", + "clip.remove": "Remove", + "clip.blocked": "The {{thing}} is blocked. Allow it in the browser's site settings and try again, or choose a file instead.", + "clip.insecure": "A browser only opens the {{thing}} on https or on localhost. You are on {{origin}}. Open this page through localhost or add TLS, or choose a file.", + "clip.unsupported": "This browser cannot open the {{thing}}. Choose a file instead.", + "clip.missing": "No {{thing}} was found on this device. Choose a file instead.", + "clip.failed": "The {{thing}} could not be opened. Choose a file instead.", + "clip.allowed": "Microphone allowed.", + "clip.capture": "Capture", + "clip.cancel": "Cancel", + "clip.recording": "Recording, {{s}} s", + "clip.micReady": "Microphone ready", + "clip.stop": "Stop", + "clip.start": "Start recording", + "clip.recorded": "Recording", + "clip.photographed": "Photo", + "clip.uploaded": "Uploaded file", + "clip.goodLength": "good length", + "clip.short": "short", + "clip.checkLevel": "check level", + "clip.preview": "Preview", + "clip.photoAlt": "The photo you chose", + "known.caption": "People saved in this browser, with their {{noun}} state", + "known.name": "Name", + "known.added": "Added", + "known.actions": "Actions", + "known.best": "best match", + "known.notOnServer": "not on the server", + "known.notInBrowser": "not in this browser's list", + "known.removing": "removing", + "known.goneText": "The last search did not return this person. The server may have restarted, so the {{print}} is gone, or it was enrolled with another model.", + "known.extraText": "The server returned this person but this browser has no record of them.", + "known.noLabels": "no labels", + "known.undo": "Undo", + "known.reenrolSaved": "Re-enrol from saved copy", + "known.enrolAgain": "Enrol again", + "known.play": "Play the saved sample of {{name}}", + "known.forget": "Remove {{name}}", + "known.note": "This list is kept in this browser. The server has no list call, so it cannot be checked until you run a search.", + "known.noteSearched": "This list is kept in this browser. The last search checked it against the server; a person it did not return is marked.", + "stored.title": "What is stored, and where", + "enrol.close": "Close", + "enrol.cancel": "Cancel", + "enrol.submit": "Enrol", + "enrol.working": "Enrolling", + "enrol.ready": "Ready to enrol.", + "enrol.stillNeeded": "Still needed: {{list}}.", + "enrol.needModel": "a model", + "enrol.needRecording": "a recording", + "enrol.needPhoto": "a photo", + "enrol.needName": "a name", + "enrol.needTranscript": "a transcript", + "enrol.needPermission": "permission", + "enrol.recordingStep": "Recording", + "enrol.photoStep": "Photo", + "enrol.recordingLabel": "Recording", + "enrol.photoLabel": "Photo", + "enrol.whoStep": "Who is it", + "enrol.name": "Name", + "enrol.namePlaceholder": "e.g. Marta Rossi", + "enrol.labels": "Labels", + "enrol.labelsHint": "optional, one per line as key: value", + "enrol.asSpeech": "Also keep it as a speech voice", + "enrol.asSpeechHint": "Text-to-speech can then speak in this voice. Needs the exact transcript.", + "enrol.transcript": "Exact transcript", + "enrol.permissionStep": "Permission", + "enrol.agreed": "This person agreed", + "enrol.keep": "Keep a copy in this browser", + "toast.enrolled": "Enrolled {{name}}", + "toast.enrolledBoth": "Enrolled {{name}} and saved a speech voice", + "toast.tooBig": "The copy was too large for browser storage, so it was not kept.", + "toast.speechFailed": "Enrolled, but the speech voice could not be saved: {{message}}", + "toast.removing": "{{name}} will be removed.", + "toast.removed": "Removed {{name}}", + "toast.alreadyGone": "{{name}} was already gone from the server.", + "error.retry": "Try again", + "error.traces": "Open traces", + "needed.badge": "Not enabled on this server", + "needed.backend": "backend {{backend}}", + "needed.install": "Install", + "needed.installing": "Installing", + "needed.started": "Installing {{name}}. Progress is in Operations.", + "needed.loadFailed": "Models could not be listed.", + "needed.noneListed": "No matching model is listed in the galleries.", + "needed.browse": "Browse all models", + "needed.askAdmin": "Ask an administrator to install one.", + "needed.whatTitle": "What turns this on", + "off.badge": "Not enabled for your account", + "off.voice.title": "Voice recognition is off for your account", + "off.voice.text": "An administrator has not allowed you to use voice recognition.", + "off.voice.what": "An administrator can turn on the Voice recognition permission for your user in Users and keys.", + "off.face.title": "Face recognition is off for your account", + "off.face.text": "An administrator has not allowed you to use face recognition.", + "off.face.what": "An administrator can turn on the Face recognition permission for your user in Users and keys." +} diff --git a/addons/face/ui/src/index.jsx b/addons/face/ui/src/index.jsx new file mode 100644 index 000000000000..c6cd5f677d3b --- /dev/null +++ b/addons/face/ui/src/index.jsx @@ -0,0 +1,27 @@ +import { createContext } from './ctx.jsx' +import { createApi } from './api.js' +import { createFaceRecognition } from './FaceRecognition.jsx' +import * as identity from '../../../internal/identity-ui/src/identity.js' + +// The second argument is for tests: the host passes the SDK only. +export function createAddon(sdk, options = {}) { + const ctx = createContext(sdk) + // The JSX of every factory compiles to calls of `h`; ctx binds it once and + // each factory reads it from ctx. This `h` is for the JSX in this file. + // eslint-disable-next-line no-unused-vars + const { h } = ctx + const api = createApi(ctx) + const pages = { + recognition: createFaceRecognition(ctx, api, identity).FaceRecognition, + ...options.pages, + } + + function Component({ page, path }) { + const Page = pages[page] + return Page ? : null + } + return { Component, components: {}, chatActions: [], dispose() {} } +} + +// For tests: the context of an SDK, built the way createAddon builds it. +export const __ctxFor = (sdk) => createContext(sdk) diff --git a/addons/face/ui/src/lib/model.js b/addons/face/ui/src/lib/model.js new file mode 100644 index 000000000000..1482007d9ec8 --- /dev/null +++ b/addons/face/ui/src/lib/model.js @@ -0,0 +1,8 @@ +// The model a page works on. A moved link carries it as ?model=, because +// a model id may hold "/" or ":" that an add-on path segment refuses; a link +// typed by hand may still carry it as the first path segment. +export function modelFrom(search, path) { + const fromQuery = new URLSearchParams(search || '').get('model') + if (fromQuery) return fromQuery + return String(path || '').split('/')[0] +} diff --git a/addons/face/ui/src/style.css b/addons/face/ui/src/style.css new file mode 100644 index 000000000000..b5c764bb33a2 --- /dev/null +++ b/addons/face/ui/src/style.css @@ -0,0 +1,242 @@ +.vc-undo { display: flex; gap: var(--dk-space-2, 8px); align-items: center; } +.vc-field { display: flex; flex-direction: column; gap: var(--dk-space-1, 4px); margin-bottom: var(--dk-space-3, 12px); } +.vc-note { color: var(--dk-text-muted, var(--dk-text)); } + +/* ========================================================================== + Voices and Faces. Built on the kit's table, sheet, hub tab, range, field, + chip, badge and toast classes. What is here is what the kit does not say: + the distance scale, the verdict, the clip card, the registry's person cell + and the speech voice rows. Colours, radii and spacing are kit roles (--dk-*). + Surfaces are flat and nothing sits on a coloured rail at the edge. Motion + is switched off for reduced motion at the end of this file. + ========================================================================== */ + +.idn-page { display: flex; flex-direction: column; gap: var(--dk-space-5); min-width: 0; } +.idn-page :is(a, button, summary, select, input, textarea, [tabindex]):focus-visible { outline: 2px solid var(--dk-accent-text); outline-offset: 2px; } +.idn-h2 { margin: 0; font: var(--dk-weight-semibold) var(--dk-text-17) / 1.3 var(--dk-font-display); color: var(--dk-text); } +.idn-page > .page-header { margin-bottom: 0; } +.idn-modelpick { min-width: 240px; max-width: 100%; } +.idn-model { display: inline-flex; align-items: center; gap: var(--dk-space-2); width: 100%; height: 36px; padding: 0 var(--dk-space-3); border: 1px solid var(--dk-control-edge); border-radius: var(--dk-radius-2); background: var(--dk-card); color: var(--dk-text); font: var(--dk-weight-semibold) var(--dk-text-13) / 1 var(--dk-font-mono); cursor: pointer; } +.idn-model:hover:not(:disabled) { background: var(--dk-hover); } +.idn-model:disabled { opacity: .5; cursor: not-allowed; } +.idn-model .searchable-select__label { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; text-align: left; } +.idn-model .searchable-select__chevron { color: var(--dk-muted); font-size: 10px; } +.idn-tabs { align-self: stretch; max-width: 100%; } +.idn-tabs .dk-tab { text-decoration: none; display: inline-flex; align-items: center; gap: var(--dk-space-2); } +.idn-tabs .dk-tab[aria-selected="true"]::after { transform: scaleX(1); } + +/* ---- The tool card -------------------------------------------------------- */ +.idn-bench { display: flex; flex-direction: column; gap: var(--dk-space-5); min-width: 0; } +.idn-tool { padding: var(--dk-space-4) var(--dk-space-5) var(--dk-space-5); display: flex; flex-direction: column; gap: var(--dk-space-4); min-width: 0; } +.idn-tool__bar { display: flex; align-items: center; gap: var(--dk-space-3); flex-wrap: wrap; } +.idn-tool__lede { margin: 0; } +.idn-tool__body { display: flex; flex-direction: column; gap: var(--dk-space-4); min-width: 0; } +.idn-tool__go { display: flex; align-items: center; gap: var(--dk-space-3); flex-wrap: wrap; } +.idn-tool__option { align-items: flex-start; } +.idn-pair { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: var(--dk-space-3); } +.idn-working { display: flex; flex-direction: column; gap: var(--dk-space-2); } +.idn-working .dk-hint { margin: 0; } + +/* ---- Clip input ----------------------------------------------------------- */ +.idn-clip { display: flex; flex-direction: column; gap: var(--dk-space-3); padding: var(--dk-space-3) var(--dk-space-4); border-radius: var(--dk-radius-3); background: var(--dk-inset); border: 1px solid var(--dk-edge); min-width: 0; } +.idn-clip__head { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-2); min-height: 28px; } +.idn-clip__choices { display: flex; flex-direction: column; gap: var(--dk-space-3); align-items: flex-start; } +.idn-clip__buttons { display: flex; align-items: center; gap: var(--dk-space-2); flex-wrap: wrap; } +.idn-clip__prompt { margin: 0; padding: var(--dk-space-3); border-radius: var(--dk-radius-2); background: var(--dk-card); border: 1px solid var(--dk-edge); font-size: var(--dk-text-15); line-height: 1.5; } +.idn-clip__prompt .dk-eyebrow { display: block; margin-bottom: 4px; } +.idn-clip__hint, .idn-clip__notice { display: flex; align-items: flex-start; gap: var(--dk-space-2); margin: 0; font-size: var(--dk-text-13); color: var(--dk-muted); } +.idn-clip__hint .lai-icon, .idn-clip__notice .lai-icon { flex: none; margin-top: 2px; } +.idn-clip__notice { padding: var(--dk-space-2) var(--dk-space-3); border-radius: var(--dk-radius-2); background: var(--dk-card); border: 1px solid var(--dk-edge); color: var(--dk-text); } +.idn-clip__notice[role="alert"] { background: var(--dk-warn-wash); border-color: transparent; color: var(--dk-warn-text); } +.idn-clip__live { display: flex; flex-direction: column; gap: var(--dk-space-3); } +.idn-clip__video { width: 100%; max-width: 420px; border-radius: var(--dk-radius-2); background: var(--dk-canvas); } +.idn-clip__rec { margin: 0; display: flex; align-items: center; gap: var(--dk-space-2); font-size: var(--dk-text-15); } +.idn-clip__card { display: flex; flex-direction: column; gap: var(--dk-space-3); min-width: 0; } +.idn-clip__meta { display: flex; align-items: center; gap: var(--dk-space-2); flex-wrap: wrap; min-width: 0; } +.idn-clip__name { min-width: 0; overflow-wrap: anywhere; font-size: var(--dk-text-15); } +.idn-clip__img { max-width: 100%; max-height: 260px; width: auto; border-radius: var(--dk-radius-2); object-fit: contain; align-self: flex-start; background: var(--dk-canvas); } +.idn-clip__notes { margin: 0; padding-left: 18px; font-size: var(--dk-text-13); color: var(--dk-warn-text); } +.idn-clip .voice-profile-player, .idn-clip .waveform-player { min-width: 0; } + +/* ---- Result: verdict, scale, ranks, cut-off ------------------------------- */ +.idn-result { display: flex; flex-direction: column; gap: var(--dk-space-4); padding-top: var(--dk-space-4); border-top: 1px solid var(--dk-edge); min-width: 0; } +.idn-verdict { display: grid; grid-template-columns: 44px minmax(0, 1fr); gap: var(--dk-space-3); align-items: start; } +.idn-verdict__icon { display: grid; place-items: center; width: 44px; height: 44px; border-radius: 50%; background: var(--dk-inset); color: var(--dk-muted); font-size: 20px; } +.idn-verdict[data-tone="match"] .idn-verdict__icon { background: var(--dk-ok-wash); color: var(--dk-ok-text); } +.idn-verdict__title { margin: 0; display: flex; align-items: center; flex-wrap: wrap; gap: var(--dk-space-2); font: var(--dk-weight-bold) var(--dk-text-21) / 1.25 var(--dk-font-display); overflow-wrap: anywhere; } +.idn-verdict__title .dk-badge { font-family: var(--dk-font-body); } +.idn-verdict__text { margin: 4px 0 0; max-width: 70ch; font-size: var(--dk-text-15); line-height: 1.5; } + +.idn-scale { margin: 0; padding: 0 36px; } +.idn-scale__plot { position: relative; height: calc(108px + var(--rows, 0) * 18px); } +.idn-scale__track { position: absolute; left: 0; right: 0; top: 38px; height: 8px; border-radius: var(--dk-radius-pill); background: var(--dk-inset); overflow: hidden; border: 1px solid var(--dk-edge); box-sizing: border-box; } +.idn-scale__inside { position: absolute; left: 0; top: 0; bottom: 0; width: var(--at); background: var(--dk-accent-wash); } +.idn-scale__cut { position: absolute; left: var(--at); top: 22px; height: 32px; width: 2px; margin-left: -1px; background: var(--dk-text); z-index: 1; } +.idn-scale__cut-label { position: absolute; bottom: 100%; left: 50%; transform: translateX(-50%); white-space: nowrap; padding-bottom: 2px; font-size: 11px; font-weight: 600; } +.idn-scale__dot { position: absolute; left: var(--at); top: 0; width: 0; height: 0; z-index: 2; } +.idn-scale__mark { position: absolute; left: -8px; top: 34px; width: 16px; height: 16px; box-sizing: border-box; border-radius: 50%; background: var(--dk-card); border: 2px solid var(--dk-control-edge); } +.idn-scale__dot[data-within="true"] .idn-scale__mark { border-color: var(--dk-accent-text); } +.idn-scale__dot[data-best="true"] .idn-scale__mark { background: var(--dk-accent); border-color: var(--dk-accent); box-shadow: 0 0 0 3px var(--dk-card); } +.idn-scale__name { position: absolute; left: 0; transform: translateX(-50%); top: 58px; max-width: 120px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; font-size: 12px; } +.idn-scale__dot[data-row="1"] .idn-scale__name { top: 76px; } +.idn-scale__dot[data-row="2"] .idn-scale__name { top: 94px; } +.idn-scale__ticks { position: absolute; left: 0; right: 0; top: calc(86px + var(--rows, 0) * 18px); height: 16px; } +.idn-scale__tick { position: absolute; left: var(--at); transform: translateX(-50%); font-size: 11px; color: var(--dk-muted); } +.idn-scale__tick::before { content: ""; position: absolute; left: 50%; bottom: 100%; height: 5px; width: 1px; background: var(--dk-control-edge); margin-bottom: 1px; } +.idn-scale__caption { display: flex; justify-content: space-between; gap: var(--dk-space-3); margin: var(--dk-space-2) -36px 0; padding: 0 4px; font-size: 12px; color: var(--dk-muted); } + +.idn-ranks { margin: 0; padding: 0; list-style: none; } +.idn-ranks li { display: grid; grid-template-columns: 32px minmax(0, 1fr) auto auto; align-items: center; gap: var(--dk-space-3); min-height: 40px; border-bottom: 1px solid var(--dk-edge); font-size: var(--dk-text-15); } +.idn-ranks li:last-child { border-bottom: 0; } +.idn-ranks li[data-within="true"] .idn-ranks__name { font-weight: var(--dk-weight-semibold); } +.idn-ranks__n { color: var(--dk-muted); font-size: 12px; } +.idn-ranks__name { min-width: 0; overflow-wrap: anywhere; } + +.idn-result__tune { display: grid; grid-template-columns: minmax(0, 1fr) minmax(0, 1.2fr); gap: var(--dk-space-5); align-items: start; } +.idn-cutoff { display: flex; flex-direction: column; gap: var(--dk-space-2); min-width: 0; } +.idn-cutoff__head { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-2); flex-wrap: wrap; } +.idn-cutoff__row { display: grid; grid-template-columns: auto minmax(0, 1fr) auto; align-items: center; gap: var(--dk-space-3); } +.idn-cutoff .dk-hint, .idn-how { margin: 0; font-size: var(--dk-text-13); color: var(--dk-muted); line-height: 1.5; } +.idn-how strong { color: var(--dk-text); } +.idn-result__enrol { align-self: flex-start; } +.idn-face { margin: 0; display: flex; flex-direction: column; gap: var(--dk-space-2); } + +/* ---- Disclosures ---------------------------------------------------------- */ +.idn-disclosure > summary { display: flex; align-items: center; gap: var(--dk-space-2); min-height: 40px; cursor: pointer; font-weight: var(--dk-weight-semibold); font-size: var(--dk-text-15); list-style: none; } +.idn-disclosure > summary::-webkit-details-marker { display: none; } +.idn-disclosure > summary .lai-icon { color: var(--dk-muted); transition: transform var(--dk-dur-1) var(--dk-ease-out); } +.idn-disclosure[open] > summary .lai-icon:first-child { transform: rotate(90deg); } +.idn-disclosure > summary .dk-hint { font-weight: 400; } +.idn-disclosure__list { margin: 0 0 var(--dk-space-3); padding-left: 38px; font-size: var(--dk-text-13); color: var(--dk-muted); line-height: 1.6; } +.idn-disclosure__text { margin: 0 0 var(--dk-space-3); padding-left: 24px; font-size: var(--dk-text-13); color: var(--dk-muted); max-width: 70ch; } +.idn-disclosure--page { border-top: 1px solid var(--dk-edge); } +.idn-stored { margin: 0 0 var(--dk-space-3); padding-left: 24px; display: grid; gap: var(--dk-space-3); } +.idn-stored div { display: grid; grid-template-columns: 160px minmax(0, 1fr); gap: var(--dk-space-3); } +.idn-stored dt { font-weight: var(--dk-weight-semibold); font-size: var(--dk-text-13); } +.idn-stored dd { margin: 0; font-size: var(--dk-text-13); color: var(--dk-muted); line-height: 1.55; max-width: 70ch; } +.idn-more__body { padding: 0 0 var(--dk-space-4) 24px; } + +/* ---- Known people --------------------------------------------------------- */ +.idn-known { display: flex; flex-direction: column; gap: var(--dk-space-2); min-width: 0; } +.idn-known__head { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-3); flex-wrap: wrap; } +.idn-known__note { margin: 0; } +.idn-known .dk-table-wrap { border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-3); background: var(--dk-card); } +.idn-known__empty { padding: var(--dk-space-6) var(--dk-space-4); border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-3); background: var(--dk-card); } +.idn-who { display: flex; align-items: center; gap: var(--dk-space-3); min-width: 0; } +.idn-who__text { display: flex; flex-direction: column; min-width: 0; } +.idn-who .dk-table-name { display: flex; align-items: center; flex-wrap: wrap; gap: var(--dk-space-2); } +.idn-avatar { display: inline-grid; place-items: center; flex: none; width: 36px; height: 36px; border-radius: 50%; overflow: hidden; background: var(--dk-inset); border: 1px solid var(--dk-edge); font: var(--dk-weight-bold) 12px var(--dk-font-mono); color: var(--dk-text); } +.idn-avatar img { width: 100%; height: 100%; object-fit: cover; } +.idn-avatar[data-tone="warn"] { background: var(--dk-warn-wash); color: var(--dk-warn-text); border-color: transparent; } +.idn-when { white-space: nowrap; font-size: 12px; color: var(--dk-muted); } +.idn-actions { text-align: right; white-space: nowrap; } +.idn-actions > * { vertical-align: middle; } +.idn-known tr[data-pending="true"] .idn-who { opacity: .55; } + +/* ---- Enrol sheet ---------------------------------------------------------- */ +.idn-sheet { display: flex; flex-direction: column; gap: var(--dk-space-5); } +.idn-sheet__step { display: flex; flex-direction: column; gap: var(--dk-space-3); } +.idn-sheet__h { margin: 0; display: flex; align-items: center; gap: var(--dk-space-2); font: var(--dk-weight-semibold) var(--dk-text-15) / 1.3 var(--dk-font-display); } +.idn-sheet__n { display: inline-grid; place-items: center; width: 24px; height: 24px; border-radius: 50%; border: 1.5px solid var(--dk-accent-text); color: var(--dk-accent-text); font: 600 12px var(--dk-font-mono); } +.idn-sheet__choice { align-items: flex-start; gap: var(--dk-space-3); } +.idn-sheet__choice > span { display: flex; flex-direction: column; gap: 2px; } +.idn-sheet__choice .dk-hint { margin: 0; } +.idn-sheet__need { margin-right: auto; font-size: var(--dk-text-13); color: var(--dk-muted); } + +/* ---- Errors, disabled state, recording tab -------------------------------- */ +.idn-error { display: grid; grid-template-columns: 32px minmax(0, 1fr); gap: var(--dk-space-3); padding: var(--dk-space-3) var(--dk-space-4); border-radius: var(--dk-radius-2); background: var(--dk-error-wash); color: var(--dk-error-text); } +.idn-error__icon { font-size: 22px; padding-top: 2px; } +.idn-error__title { margin: 0; font: var(--dk-weight-semibold) var(--dk-text-15) / 1.3 var(--dk-font-display); } +.idn-error__text, .idn-error__raw { margin: 4px 0 0; font-size: var(--dk-text-13); } +.idn-error__raw { opacity: .85; overflow-wrap: anywhere; } +.idn-error__acts { display: flex; gap: var(--dk-space-2); flex-wrap: wrap; margin-top: var(--dk-space-3); } +.idn-error .dk-btn { color: inherit; } +.idn-needed { padding: var(--dk-space-5); display: flex; flex-direction: column; align-items: flex-start; gap: var(--dk-space-3); } +.idn-needed__text { margin: 0; max-width: 62ch; font-size: var(--dk-text-15); line-height: 1.5; } +.idn-needed__list { margin: 0; padding: 0; list-style: none; width: min(100%, 560px); border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-3); } +.idn-needed__list li { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-3); padding: var(--dk-space-3) var(--dk-space-4); border-bottom: 1px solid var(--dk-edge); } +.idn-needed__list li:last-child { border-bottom: 0; } +.idn-needed__list li > span { display: flex; flex-direction: column; min-width: 0; } +.idn-needed__list .dk-hint { margin: 0; } +.idn-off { min-height: 40vh; } +.idn-recording { padding: var(--dk-space-5); display: flex; flex-direction: column; align-items: flex-start; gap: var(--dk-space-3); } +.idn-recording__text { margin: 0; max-width: 62ch; line-height: 1.5; } +.idn-recording__steps { margin: 0; padding-left: 20px; display: grid; gap: 4px; color: var(--dk-muted); font-size: var(--dk-text-15); } +.idn-recording .dk-hint { margin: 0; max-width: 62ch; } + +/* ---- Speech voices -------------------------------------------------------- */ +.voice-library-page { display: flex; flex-direction: column; gap: var(--dk-space-4); min-width: 0; } +.idn-speech__head { display: flex; align-items: flex-start; justify-content: space-between; gap: var(--dk-space-4); flex-wrap: wrap; } +.idn-speech__lede { margin: 4px 0 0; max-width: 56ch; } +.idn-speech__acts { display: flex; align-items: center; gap: var(--dk-space-3); flex-wrap: wrap; } +.voice-library-toolbar { display: flex; gap: var(--dk-space-3); flex-wrap: wrap; } +.voice-library-search { flex: 1 1 260px; min-width: 0; } +.voice-library-language { min-width: 160px; } +.voice-library-loading { display: flex; align-items: center; gap: var(--dk-space-3); padding: var(--dk-space-5); color: var(--dk-muted); } +.idn-speech__list { border: 1px solid var(--dk-edge); border-radius: var(--dk-radius-3); background: var(--dk-card); overflow: hidden; } +.idn-speech__list .dk-row { text-align: left; } +.voice-row__end { display: flex; align-items: center; gap: var(--dk-space-3); } +.voice-row__meta { font-size: 12px; color: var(--dk-muted); white-space: nowrap; } +.voice-row__waveform { display: inline-flex; align-items: center; gap: 2px; width: 120px; height: 22px; } +.voice-row__waveform > span { flex: 1; min-width: 2px; border-radius: 2px; background: var(--dk-control-edge); } +.voice-row__chevron { color: var(--dk-muted); } +.voice-library-empty { min-height: 0; } +.voice-library-detail { display: flex; flex-direction: column; gap: var(--dk-space-4); min-width: 0; } +.voice-detail__player { min-width: 0; } +.voice-detail__section { display: flex; flex-direction: column; gap: var(--dk-space-2); } +.voice-detail__section h3 { margin: 0; } +.voice-detail__section blockquote { margin: 0; padding: var(--dk-space-3) var(--dk-space-4); border-radius: var(--dk-radius-2); background: var(--dk-inset); border: 1px solid var(--dk-edge); font-size: var(--dk-text-15); line-height: 1.55; overflow-wrap: anywhere; } +.voice-detail__metadata { margin: 0; } +.voice-detail__model-setup { display: flex; flex-direction: column; gap: var(--dk-space-3); padding: var(--dk-space-4); border-radius: var(--dk-radius-3); background: var(--dk-inset); border: 1px solid var(--dk-edge); } +.voice-detail__model-setup-heading { display: flex; gap: var(--dk-space-3); align-items: flex-start; } +.voice-detail__model-setup-heading h3 { margin: 0; font: var(--dk-weight-semibold) var(--dk-text-15) / 1.3 var(--dk-font-display); } +.voice-detail__model-setup-heading p, .voice-detail__model-fallback, .voice-detail__capability-note { margin: 4px 0 0; font-size: var(--dk-text-13); color: var(--dk-muted); } +.voice-detail__model-list { margin: 0; padding: 0; list-style: none; display: grid; gap: var(--dk-space-2); } +.voice-detail__model-list li { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-3); } +.voice-detail__model-list li > span { display: flex; flex-direction: column; min-width: 0; } +.voice-detail__model-list small { color: var(--dk-muted); } +.voice-detail__model-loading { display: flex; align-items: center; gap: var(--dk-space-2); font-size: var(--dk-text-13); color: var(--dk-muted); } +.voice-detail__model-fallback a { color: var(--dk-accent-text); } +.voice-detail__api { border-top: 1px solid var(--dk-edge); } +.voice-detail__api-body { display: flex; flex-direction: column; gap: var(--dk-space-3); padding: 0 0 var(--dk-space-3) 24px; font-size: var(--dk-text-13); color: var(--dk-muted); } +.voice-detail__api-body > p { margin: 0; } +.voice-detail__compatible-models { display: flex; flex-direction: column; gap: var(--dk-space-2); } +.voice-detail__compatible-models ul { display: flex; flex-wrap: wrap; gap: var(--dk-space-2); margin: 0; padding: 0; list-style: none; } +.voice-detail__compatible-models li { display: inline-flex; align-items: center; gap: var(--dk-space-2); padding: 2px 10px; border-radius: var(--dk-radius-pill); background: var(--dk-ok-wash); color: var(--dk-ok-text); } +.voice-detail__code-heading { display: flex; align-items: center; justify-content: space-between; gap: var(--dk-space-2); } +.voice-detail__api pre { margin: 0; padding: var(--dk-space-3); overflow-x: auto; border-radius: var(--dk-radius-2); background: var(--dk-inset); border: 1px solid var(--dk-edge); color: var(--dk-text); font-size: 12px; line-height: 1.55; } +.voice-delete { margin-right: auto; } + +/* ---- Phone ---------------------------------------------------------------- */ +@media (max-width: 720px) { + .idn-tool { padding: var(--dk-space-3); } + .idn-pair { grid-template-columns: minmax(0, 1fr); } + .idn-result__tune { grid-template-columns: minmax(0, 1fr); gap: var(--dk-space-4); } + .idn-scale { padding: 0 26px; } + .idn-scale__caption { margin: var(--dk-space-2) -26px 0; flex-direction: column; gap: 2px; } + .idn-stored div { grid-template-columns: minmax(0, 1fr); gap: 2px; } + .idn-verdict { grid-template-columns: 36px minmax(0, 1fr); } + .idn-verdict__icon { width: 36px; height: 36px; font-size: 17px; } + .idn-verdict__title { font-size: var(--dk-text-17); } + .idn-modelpick { min-width: 0; width: 100%; } + .voice-row__waveform { display: none; } + .idn-ranks li { grid-template-columns: 20px minmax(0, 1fr) auto auto; gap: var(--dk-space-2); font-size: var(--dk-text-13); } + .idn-known .dk-table td { padding-inline: var(--dk-space-2); } + .idn-actions { white-space: normal; } + .idn-actions .dk-btn--secondary { margin-bottom: 4px; } + .idn-more__body { padding-left: 0; } + .idn-more > summary .dk-hint { display: none; } + .idn-needed__list li { flex-direction: column; align-items: flex-start; } +} + +@media (prefers-reduced-motion: no-preference) { + .idn-clip__rec .dk-dot[data-live="true"] { animation: idn-pulse 1.1s ease-in-out infinite; } +} +@keyframes idn-pulse { 50% { opacity: .35; } } +@media (prefers-reduced-motion: reduce) { + .idn-disclosure > summary .lai-icon { transition: none; } + /* The kit's indeterminate bar keeps sliding; here it sits still at its width. */ + .idn-working .dk-progress[data-indeterminate] .dk-progress-bar { animation: none; } +} + +.idn-who .dk-table-sub { white-space: normal; overflow: visible; text-overflow: clip; max-width: 62ch; } diff --git a/addons/face/ui/test/model.test.mjs b/addons/face/ui/test/model.test.mjs new file mode 100644 index 000000000000..3d863ec38680 --- /dev/null +++ b/addons/face/ui/test/model.test.mjs @@ -0,0 +1,17 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { modelFrom } from '../src/lib/model.js' + +test('the model comes from ?model first and keeps slashes, colons and percent signs', () => { + assert.equal(modelFrom('model=org%2Fname%3Av1', 'ignored'), 'org/name:v1') +}) + +test('without ?model the model is the first path segment', () => { + assert.equal(modelFrom('', 'speechbrain/extra'), 'speechbrain') + assert.equal(modelFrom('x=1', 'speechbrain'), 'speechbrain') +}) + +test('no model gives an empty string', () => { + assert.equal(modelFrom('', ''), '') + assert.equal(modelFrom(undefined, undefined), '') +}) diff --git a/addons/face/ui/test/recognition.test.mjs b/addons/face/ui/test/recognition.test.mjs new file mode 100644 index 000000000000..701e8d6b19bd --- /dev/null +++ b/addons/face/ui/test/recognition.test.mjs @@ -0,0 +1,86 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { createRequire } from 'node:module' +import { fileURLToPath } from 'node:url' +import { dirname, join, resolve } from 'node:path' +import { load } from './render.test.mjs' + +const here = dirname(fileURLToPath(import.meta.url)) +const require = createRequire(join(resolve(here, '../../../../core/http/react-ui'), 'package.json')) +const React = require('react') +const { renderToString } = require('react-dom/server') + +test('recognition: the first paint has the faces page test id', async () => { + const { mod, sdk } = await load() + const addon = await mod.createAddon(sdk) + const html = renderToString(React.createElement(addon.Component, { page: 'recognition', path: '' })) + assert.match(html, /data-testid="faces-page"/) +}) + +test('recognition: a viewer without face_recognition gets the off notice', async () => { + const { mod, sdk } = await load({ viewer: { isAdmin: () => false, hasFeature: (f) => f !== 'face_recognition' } }) + const addon = await mod.createAddon(sdk) + const html = renderToString(React.createElement(addon.Component, { page: 'recognition', path: '' })) + assert.match(html, /data-testid="face-off"/) + assert.doesNotMatch(html, /idn-bench/) +}) + +test('recognition: register, identify and forget use the core face routes', async () => { + const { createApi } = await import('../src/api.js') + const seen = [] + const ctx = { core: { fetch: async (p, i) => { seen.push([p, i?.method]); return { ok: true, json: async () => ({}) } } }, json: async (p, i) => { seen.push([p, i?.method]); return {} } } + const api = createApi(ctx) + await api.face.register({ model: 'm', img: 'x', name: 'a' }) + await api.face.identify({ model: 'm', img: 'x' }) + await api.face.forget({ model: 'm', id: 'a' }) + assert.deepEqual(seen.map(s => s[0]), ['/v1/face/register', '/v1/face/identify', '/v1/face/forget']) + assert.deepEqual(seen.map(s => s[1]), ['POST', 'POST', 'POST']) +}) + +test('recognition: any viewer who holds face_recognition gets the library, administrator or not', async () => { + const { mod, sdk } = await load({ viewer: { isAdmin: () => false, hasFeature: () => true } }) + const addon = await mod.createAddon(sdk) + const html = renderToString(React.createElement(addon.Component, { page: 'recognition', path: '' })) + assert.match(html, /idn-bench/) + assert.doesNotMatch(html, /data-testid="face-off"/) +}) + +test('recognition: the model of the path reaches the model selection', async () => { + const seen = [] + const names = ['Button', 'Spinner', 'EmptyState', 'Badge', 'Stack', 'Text', 'PageHeader', 'ConfirmDialog', 'ErrorWithTraceLink', 'MediaInput', 'WaveformPlayer', 'DistributionBars', 'EmbeddingInspector', 'TabSwitch'] + const components = Object.fromEntries(names.map(n => [n, (p) => React.createElement('div', null, p.title || p.label || p.children || null)])) + components.PageHeader = (p) => React.createElement('header', null, p.title, p.actions) + components.ModelSelector = (p) => { seen.push(p.value); return React.createElement('div') } + const { mod, sdk } = await load({ components }) + const addon = await mod.createAddon(sdk) + renderToString(React.createElement(addon.Component, { page: 'recognition', path: 'insightface' })) + assert.deepEqual(seen, ['insightface']) +}) + +test('recognition: the bundle has the face canvas, the face strings and no host import', async () => { + const { readFileSync } = await import('node:fs') + const { build } = await import('../build.mjs') + const dir = await build() + const js = readFileSync(join(dir, 'ui/face.js'), 'utf8') + assert.match(js, /biometrics-bbox/) + assert.doesNotMatch(js, /from ['"]react(-dom)?/) + assert.match(js, /faces\.title/) +}) + +test('recognition: every translation key the sources read is in the Face bundle', async () => { + const { readFileSync, readdirSync, statSync } = await import('node:fs') + const { default: en } = await import('../src/i18n/en.js') + const roots = [join(here, '../src'), join(here, '../../../internal/identity-ui/src')] + const files = [] + const walk = (d) => { for (const f of readdirSync(d)) { const p = join(d, f); if (statSync(p).isDirectory()) walk(p); else if (/\.jsx?$/.test(p) && !p.includes('i18n')) files.push(p) } } + roots.forEach(walk) + const missing = [] + for (const f of files) { + for (const m of readFileSync(f, 'utf8').matchAll(/\bt\(\s*'([a-zA-Z]+\.[a-zA-Z0-9_.]+)'/g)) { + // The voice branch of a shared view is never reached on this page. + if (m[1].startsWith('voice.')) continue + if (!(m[1] in en)) missing.push(`${f.split('/').pop()}: ${m[1]}`) + } + } + assert.deepEqual(missing, []) +}) diff --git a/addons/face/ui/test/render.test.mjs b/addons/face/ui/test/render.test.mjs new file mode 100644 index 000000000000..30eacd508332 --- /dev/null +++ b/addons/face/ui/test/render.test.mjs @@ -0,0 +1,75 @@ +import test from 'node:test' +import assert from 'node:assert/strict' +import { createRequire } from 'node:module' +import { fileURLToPath, pathToFileURL } from 'node:url' +import { dirname, join, resolve } from 'node:path' +import { build } from '../build.mjs' + +const here = dirname(fileURLToPath(import.meta.url)) +const hostUi = resolve(here, '../../../../core/http/react-ui') +const require = createRequire(join(hostUi, 'package.json')) +const React = require('react') +const { renderToString } = require('react-dom/server') + +export async function load(extra = {}) { + const dir = await build() + const mod = await import(pathToFileURL(join(dir, 'ui/face.js')).href + `?t=${Date.now()}`) + const { createHostSDK } = await import(pathToFileURL(join(hostUi, 'src/addons/sdk.js')).href) + const stub = (tag) => (p) => React.createElement(tag, { 'data-stub': tag }, p.title || p.label || p.children || null) + const calls = [] + const controller = new AbortController() + const sdk = createHostSDK({ + addonId: 'face', generation: 1, signal: controller.signal, pages: ['recognition'], + host: { + React, + components: Object.fromEntries(['Button', 'Spinner', 'EmptyState', 'Badge', 'Stack', 'Text', 'PageHeader', 'ConfirmDialog', 'ErrorWithTraceLink', 'ModelSelector', 'MediaInput', 'WaveformPlayer', 'DistributionBars', 'EmbeddingInspector', 'TabSwitch'].map(n => [n, n === 'PageHeader' ? (p => React.createElement('header', { 'data-stub': 'header' }, p.title ? React.createElement('h1', null, p.title) : null, p.supporting)) : stub('div')])), + hooks: { useMediaCapture: () => ({}), useObjectUrl: () => null, useDelayedAction: () => ({ pending: null, schedule() {}, undo() {} }) }, + viewer: { isAdmin: () => true, hasFeature: () => true }, + basePath: '', + fetch: async (url, init) => { calls.push([url, init]); return { ok: true, json: async () => ({ data: [] }), blob: async () => new Blob([]) } }, + Icon: () => null, theme: { mode: () => 'light', subscribe: () => () => {} }, language: () => 'en', subscribeLanguage: () => () => {}, api: {}, + ...extra, + }, + bindings: { current: () => ({ page: 'recognition', path: '', search: '' }), openLayer: () => ({ close() {} }), notify() {} }, + }) + return { mod, sdk, controller, calls } +} + +test('render: createAddon returns a page component and dispose', async () => { + const { mod, sdk, controller } = await load() + const addon = await mod.createAddon(sdk) + assert.equal(typeof addon.Component, 'function') + assert.equal(typeof addon.dispose, 'function') + addon.dispose() + controller.abort('done') +}) + +test('render: an unknown page draws nothing', async () => { + const { mod, sdk } = await load() + const addon = await mod.createAddon(sdk) + assert.equal(renderToString(React.createElement(addon.Component, { page: 'nope', path: '' })), '') +}) + +test('render: the manifest declares the recognition page, the main page and SDK 2', async () => { + const { readFileSync } = await import('node:fs') + const m = JSON.parse(readFileSync(join(here, '../../manifest.json'), 'utf8')) + assert.equal(m.id, 'face') + assert.deepEqual(m.capabilities.ui.pages, ['recognition']) + assert.deepEqual(m.capabilities.ui.navigation, []) + assert.equal(m.capabilities.ui.mainPage, 'recognition') + assert.deepEqual(m.capabilities.ui.sdk, { min: 2, max: 2 }) + assert.deepEqual(m.capabilities.tools ?? [], []) +}) + +test('render: a registered page is drawn with its path', async () => { + const { mod, sdk } = await load() + const addon = await mod.createAddon(sdk, { pages: { recognition: ({ path }) => React.createElement('p', null, `page:${path}`) } }) + assert.equal(renderToString(React.createElement(addon.Component, { page: 'recognition', path: 'abc' })), '

page:abc

') +}) + +test('render: the recognition page draws exactly one h1, its own title', async () => { + const { mod, sdk } = await load() + const addon = await mod.createAddon(sdk) + const html = renderToString(React.createElement(addon.Component, { page: 'recognition', path: '' })) + assert.deepEqual([...html.matchAll(/]*>(.*?)<\/h1>/g)].map(m => m[1]), ['Faces']) +}) diff --git a/addons/face/ui/test/run.mjs b/addons/face/ui/test/run.mjs new file mode 100644 index 000000000000..f95b03d477b5 --- /dev/null +++ b/addons/face/ui/test/run.mjs @@ -0,0 +1,23 @@ +// Runs the node unit tests of the Voice UI package and prints one JSON line +// per test, the same shape as the host's runner (core/http/react-ui/tests/unit): +// {"case":"unit: ","status":"passed"|"failed"|"skipped"} +import { run } from 'node:test' +import { readdirSync } from 'node:fs' +import { dirname, join } from 'node:path' +import { fileURLToPath } from 'node:url' + +const here = dirname(fileURLToPath(import.meta.url)) +const names = readdirSync(here).filter(f => f.endsWith('.test.mjs')).sort() +let failed = false +const stream = run({ files: names.map(f => join(here, f)), concurrency: 1 }) +stream.on('test:pass', ({ name, skip, todo, nesting }) => { + if (nesting > 0) return + console.log(JSON.stringify({ case: `unit: ${name}`, status: skip || todo ? 'skipped' : 'passed' })) +}) +stream.on('test:fail', ({ name, nesting, details }) => { + if (nesting > 0) return + failed = true + console.log(JSON.stringify({ case: `unit: ${name}`, status: 'failed', error: String(details && details.error && details.error.message || 'failed').slice(0, 400) })) +}) +stream.on('end', () => { process.exit(failed ? 1 : 0) }) +stream.resume() diff --git a/addons/fine-tuning/cmd/fine-tuning/main.go b/addons/fine-tuning/cmd/fine-tuning/main.go new file mode 100644 index 000000000000..782e810ba62e --- /dev/null +++ b/addons/fine-tuning/cmd/fine-tuning/main.go @@ -0,0 +1,21 @@ +// SPDX-License-Identifier: MIT + +// Command fine-tuning is the executable of the fine-tuning add-on. It serves +// the typed tools of the add-on and reaches the job engine through the host +// operations; the add-on UI works in the browser through the SDK. +package main + +import ( + "os" + + "github.com/mudler/LocalAI/addons/fine-tuning/internal/tools" + "github.com/mudler/LocalAI/addons/internal/hostclient" + "github.com/mudler/LocalAI/addons/internal/protocol" + "github.com/mudler/LocalAI/pkg/addons/contract" +) + +func main() { + os.Exit(protocol.Run(os.Args, func(s protocol.Startup) contract.Runtime { + return tools.New(hostclient.New(s), s.DataDir) + })) +} diff --git a/addons/fine-tuning/internal/tools/journal.go b/addons/fine-tuning/internal/tools/journal.go new file mode 100644 index 000000000000..cb9897b7aa44 --- /dev/null +++ b/addons/fine-tuning/internal/tools/journal.go @@ -0,0 +1,142 @@ +// SPDX-License-Identifier: MIT +package tools + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "sync" + + "github.com/mudler/LocalAI/pkg/addons/contract" +) + +// maxReceipts bounds the receipts the journal keeps. A repeat of an older +// operation is not answered from memory any more; the host asks for a receipt +// only for a recent, unsettled operation. +const maxReceipts = 1024 + +// job is what the add-on knows of one resource a tool created: a job it +// started or a dataset it sent. The engine job ID +// never leaves the add-on: a model names the job by its resource. +type job struct { + JobID string `json:"jobId,omitempty"` + Name string `json:"name"` + // Dataset marks a resource that stands for an inline dataset and not for a + // job; Ref is the reference the host issued for it. + Dataset bool `json:"dataset,omitempty"` + Ref string `json:"ref,omitempty"` + // Revision counts the changes a tool made to the job. It is the data + // revision of the resource, so a stale approval is noticed. + Revision uint64 `json:"revision"` +} + +type journalFile struct { + Jobs map[string]job `json:"jobs"` + Receipts map[string]contract.Receipt `json:"receipts"` + Order []string `json:"order"` +} + +// journal maps resource IDs to engine job IDs and keeps the receipts of the +// operations that changed something. It is one JSON file in the data +// directory, written through a temporary file and renamed over the old one. +type journal struct { + mu sync.Mutex + path string + data journalFile +} + +func openJournal(dir string) *journal { + j := &journal{path: filepath.Join(dir, "journal.json")} + j.data = journalFile{Jobs: map[string]job{}, Receipts: map[string]contract.Receipt{}} + if b, err := os.ReadFile(j.path); err == nil { + var f journalFile + if json.Unmarshal(b, &f) == nil { + if f.Jobs != nil { + j.data.Jobs = f.Jobs + } + if f.Receipts != nil { + j.data.Receipts = f.Receipts + } + j.data.Order = f.Order + } + } + return j +} + +func (j *journal) job(resource string) (job, bool) { + j.mu.Lock() + defer j.mu.Unlock() + v, ok := j.data.Jobs[resource] + return v, ok +} + +func (j *journal) receipt(op string) (contract.Receipt, bool) { + j.mu.Lock() + defer j.mu.Unlock() + r, ok := j.data.Receipts[op] + return r, ok +} + +// commit stores the receipt of an operation and, when upd is not nil, the new +// state of a job in one write. A failed write leaves the memory as it was, so +// the caller reports the failure and nothing claims a state that is not on disk. +func (j *journal) commit(resource string, upd *job, r contract.Receipt) error { + j.mu.Lock() + defer j.mu.Unlock() + next := journalFile{Jobs: map[string]job{}, Receipts: map[string]contract.Receipt{}, Order: append([]string(nil), j.data.Order...)} + for k, v := range j.data.Jobs { + next.Jobs[k] = v + } + for k, v := range j.data.Receipts { + next.Receipts[k] = v + } + if upd != nil { + next.Jobs[resource] = *upd + } + if _, seen := next.Receipts[r.OperationID]; !seen { + next.Order = append(next.Order, r.OperationID) + } + next.Receipts[r.OperationID] = r + for len(next.Order) > maxReceipts { + delete(next.Receipts, next.Order[0]) + next.Order = next.Order[1:] + } + if err := writeAtomic(j.path, next); err != nil { + return err + } + j.data = next + return nil +} + +func writeAtomic(path string, v any) error { + b, err := json.Marshal(v) + if err != nil { + return err + } + tmp, err := os.CreateTemp(filepath.Dir(path), "journal-*.tmp") + if err != nil { + return err + } + name := tmp.Name() + fail := func(e error) error { _ = tmp.Close(); _ = os.Remove(name); return e } + if _, err := tmp.Write(b); err != nil { + return fail(err) + } + if err := tmp.Sync(); err != nil { + return fail(err) + } + if err := tmp.Close(); err != nil { + _ = os.Remove(name) + return err + } + if err := os.Chmod(name, 0o600); err != nil { + _ = os.Remove(name) + return errors.Join(err) + } + if err := os.Rename(name, path); err != nil { + _ = os.Remove(name) + return err + } + return nil +} diff --git a/addons/fine-tuning/internal/tools/suite_test.go b/addons/fine-tuning/internal/tools/suite_test.go new file mode 100644 index 000000000000..ab3dabb08619 --- /dev/null +++ b/addons/fine-tuning/internal/tools/suite_test.go @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: MIT +package tools + +import ( + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +func TestTools(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Fine-tuning tools suite") +} diff --git a/addons/fine-tuning/internal/tools/tools.go b/addons/fine-tuning/internal/tools/tools.go new file mode 100644 index 000000000000..f83573ca4991 --- /dev/null +++ b/addons/fine-tuning/internal/tools/tools.go @@ -0,0 +1,557 @@ +// SPDX-License-Identifier: MIT + +// Package tools is the runtime of the fine-tuning add-on executable. A job that +// a tool starts is a resource of the host: the host reserves the resource ID, +// and this package remembers which engine job it stands for. A tool can only +// control a job that a tool started, because only those have a resource. +package tools + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "regexp" + "strings" + "time" + "unicode/utf8" + + "github.com/mudler/LocalAI/addons/internal/hostclient" + "github.com/mudler/LocalAI/addons/internal/protocol" + "github.com/mudler/LocalAI/pkg/addons/contract" +) + +// Tool names. They are the entries of capabilities.tools in the manifest and +// the exact values of arguments.tool. +const ( + ToolResourceExists = "resource-exists" + ToolList = "fine-tune-list" + ToolStart = "fine-tune-start" + ToolDataset = "fine-tune-dataset" + ToolStatus = "fine-tune-status" + ToolCheckpoints = "fine-tune-checkpoints" + ToolStop = "fine-tune-stop" + ToolExport = "fine-tune-export" +) + +// Tools is the declared tool set of this add-on, in manifest order. +// +// fine-tune-list is a read tool that names no resource: it lists the jobs of the +// calling user, the ones started in the UI included. The host runs it as that +// user and the host operation scopes the answer to the same user. Every other +// tool works on a resource that a tool created. +var Tools = []string{ToolResourceExists, ToolList, ToolStart, ToolDataset, ToolStatus, ToolCheckpoints, ToolStop, ToolExport} + +// Engine is the host operation channel. *hostclient.Client implements it, and +// a nil client answers hostclient.ErrNoHostOps. +type Engine interface { + Do(ctx context.Context, operationID, op string, args, out any) error +} + +var _ Engine = (*hostclient.Client)(nil) + +// schema is the schema revision of every job. The data revision counts the +// changes a tool made. +const schema uint64 = 1 + +var ( + modelName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`) + exportFormats = map[string]bool{"lora": true, "merged_16bit": true, "merged_4bit": true, "gguf": true} +) + +const maxNameRunes = 80 + +// maxDatasetBytes is the bound of an inline dataset, the one of the host +// operation. +const maxDatasetBytes = 900 << 10 + +var datasetExt = []string{".jsonl", ".json", ".csv", ".txt"} + +type runtime struct { + eng Engine + j *journal + now func() time.Time +} + +// New returns the runtime of the executable. dataDir holds the journal. +func New(eng Engine, dataDir string) contract.Runtime { + return &runtime{eng: eng, j: openJournal(dataDir), now: time.Now} +} + +// startArgs are the arguments of fine-tune-start. The host adds "tool"; the +// name is the label the host shows in the approval card and the resource list. +type startArgs struct { + Tool string `json:"tool"` + Name string `json:"name"` + Model string `json:"model"` + Backend string `json:"backend"` + TrainingType string `json:"trainingType"` + TrainingMethod string `json:"trainingMethod"` + DatasetSource string `json:"datasetSource"` + DatasetSplit string `json:"datasetSplit"` + LearningRate float64 `json:"learningRate"` + NumEpochs int64 `json:"numEpochs"` + BatchSize int64 `json:"batchSize"` + MaxSteps int64 `json:"maxSteps"` + AdapterRank int64 `json:"adapterRank"` + AdapterAlpha int64 `json:"adapterAlpha"` +} + +// engineStart is the argument object of finetune.start. It has exactly the +// fields the host operation accepts and none of the add-on's own. +type engineStart struct { + Model string `json:"model"` + Backend string `json:"backend,omitempty"` + TrainingType string `json:"trainingType,omitempty"` + TrainingMethod string `json:"trainingMethod,omitempty"` + DatasetSource string `json:"datasetSource"` + DatasetSplit string `json:"datasetSplit,omitempty"` + LearningRate float64 `json:"learningRate,omitempty"` + NumEpochs int64 `json:"numEpochs,omitempty"` + BatchSize int64 `json:"batchSize,omitempty"` + MaxSteps int64 `json:"maxSteps,omitempty"` + AdapterRank int64 `json:"adapterRank,omitempty"` + AdapterAlpha int64 `json:"adapterAlpha,omitempty"` +} + +// datasetArgs are the arguments of fine-tune-dataset. The name is the file +// name of the dataset and the label of the resource. +type datasetArgs struct { + Tool string `json:"tool"` + Name string `json:"name"` + Content string `json:"content"` +} + +type stopArgs struct { + Tool string `json:"tool"` + SaveCheckpoint bool `json:"saveCheckpoint"` +} + +type exportArgs struct { + Tool string `json:"tool"` + ExportFormat string `json:"exportFormat"` + Name string `json:"name"` +} + +// decode reads the arguments into a closed struct: an unknown field or +// trailing data is a bad request. +func decode(raw json.RawMessage, v any) error { + if len(raw) == 0 { + raw = json.RawMessage(`{}`) + } + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.DisallowUnknownFields() + if err := dec.Decode(v); err != nil { + return protocol.ErrBadRequest + } + if _, err := dec.Token(); err != io.EOF { + return protocol.ErrBadRequest + } + return nil +} + +func toolOf(env contract.Envelope) (string, error) { + var a struct { + Tool string `json:"tool"` + } + if json.Unmarshal(env.Request.Arguments, &a) != nil || a.Tool == "" { + return "", protocol.ErrBadRequest + } + return a.Tool, nil +} + +func payload(v any) json.RawMessage { b, _ := json.Marshal(v); return b } + +// mapErr turns the refusal of a host operation into the protocol error that +// tells the owner what happened. A failure that does not say the request was +// wrong stays as it is, and the owner reads it as uncertain. +func mapErr(err error) error { + var he *contract.HostError + if !errors.As(err, &he) { + return err + } + switch he.Code { + case contract.HostBadRequest, contract.HostTooLarge: + return fmt.Errorf("%w: %s", protocol.ErrBadRequest, he.Message) + case contract.HostNotFound: + return fmt.Errorf("%w: %s", protocol.ErrNotFound, he.Message) + case contract.HostConflict: + return fmt.Errorf("%w: %s", protocol.ErrConflict, he.Message) + case contract.HostFeature, contract.HostScope: + return fmt.Errorf("%w: %s", protocol.ErrForbidden, he.Message) + } + return err +} + +// jobOf is the job of the resource. A dataset resource is not a job. +func (r *runtime) jobOf(env contract.Envelope) (job, error) { + j, ok := r.entryOf(env) + if !ok || j.Dataset { + return job{}, protocol.ErrNotFound + } + return j, nil +} + +func (r *runtime) entryOf(env contract.Envelope) (job, bool) { + if env.Request.ResourceID == "" { + return job{}, false + } + return r.j.job(env.Request.ResourceID) +} + +func (r *runtime) Read(ctx context.Context, env contract.Envelope) (contract.Result, error) { + tool, err := toolOf(env) + if err != nil { + return contract.Result{}, err + } + op := env.Request.OperationID + switch tool { + case ToolResourceExists: + j, ok := r.j.job(env.Request.ResourceID) + if !ok { + return contract.Result{Payload: payload(map[string]bool{"exists": false})}, nil + } + if j.Dataset { + // The reference lives in the host and the journal; there is no + // engine job to ask. + return contract.Result{Payload: payload(map[string]bool{"exists": true})}, nil + } + err := r.eng.Do(ctx, op, "finetune.get", map[string]string{"jobId": j.JobID}, nil) + var he *contract.HostError + switch { + case err == nil: + return contract.Result{Payload: payload(map[string]bool{"exists": true})}, nil + case errors.As(err, &he) && he.Code == contract.HostNotFound: + return contract.Result{Payload: payload(map[string]bool{"exists": false})}, nil + } + // The engine could not answer. The host settles a creation only on an + // explicit answer, so this is an error and never "does not exist". + return contract.Result{}, err + case ToolList: + // No resource: the host runs this read as the calling user and the + // host operation lists only that user's jobs. + var a struct { + Tool string `json:"tool"` + } + if err := decode(env.Request.Arguments, &a); err != nil { + return contract.Result{}, err + } + var got json.RawMessage + if err := r.eng.Do(ctx, op, "finetune.list", nil, &got); err != nil { + return contract.Result{}, mapErr(err) + } + return contract.Result{Payload: got}, nil + case ToolStatus: + j, ok := r.entryOf(env) + if !ok { + return contract.Result{}, protocol.ErrNotFound + } + if j.Dataset { + return contract.Result{Payload: payload(map[string]any{ + "name": j.Name, "revision": contract.Revision{Schema: schema, Data: j.Revision}, "kind": "dataset", "datasetRef": j.Ref, + })}, nil + } + var got map[string]any + if err := r.eng.Do(ctx, op, "finetune.get", map[string]string{"jobId": j.JobID}, &got); err != nil { + return contract.Result{}, mapErr(err) + } + // The engine job ID stays in the add-on. The name and the revision + // are what the host reads to label and track the resource. + delete(got, "id") + got["name"] = j.Name + got["revision"] = contract.Revision{Schema: schema, Data: j.Revision} + return contract.Result{Payload: payload(got)}, nil + case ToolCheckpoints: + j, err := r.jobOf(env) + if err != nil { + return contract.Result{}, err + } + var got json.RawMessage + if err := r.eng.Do(ctx, op, "finetune.checkpoints", map[string]string{"jobId": j.JobID}, &got); err != nil { + return contract.Result{}, mapErr(err) + } + return contract.Result{Payload: got}, nil + } + return contract.Result{}, protocol.ErrBadRequest +} + +func (r *runtime) checkStart(a startArgs) error { + if a.Model == "" || a.DatasetSource == "" || len([]rune(a.Name)) > maxNameRunes || + a.LearningRate < 0 || a.NumEpochs < 0 || a.BatchSize < 0 || a.MaxSteps < 0 || a.AdapterRank < 0 || a.AdapterAlpha < 0 { + return protocol.ErrBadRequest + } + return nil +} + +func checkExport(a exportArgs) error { + if a.ExportFormat != "" && !exportFormats[a.ExportFormat] { + return protocol.ErrBadRequest + } + if a.Name != "" && !modelName.MatchString(a.Name) { + return protocol.ErrBadRequest + } + return nil +} + +func (r *runtime) Preview(_ context.Context, env contract.Envelope) (contract.Preview, error) { + tool, err := toolOf(env) + if err != nil { + return contract.Preview{}, err + } + out := contract.Preview{OperationID: env.Request.OperationID, ExpiresAt: r.now().Add(time.Minute)} + switch tool { + case ToolStart: + var a startArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return out, err + } + if err := r.checkStart(a); err != nil { + return out, err + } + // The resource does not exist yet; an operation that already + // committed is replayed by Apply, so a repeat is not a clash. + if _, taken := r.j.job(env.Request.ResourceID); taken { + if _, done := r.j.receipt(env.Request.OperationID); !done { + return out, protocol.ErrConflict + } + } + out.Revision = contract.Revision{Schema: schema, Data: 0} + out.Summary = payload(map[string]any{"count": 1, "sample": map[string]string{"action": "start fine-tuning job", "name": nameOf(a), "model": a.Model}}) + return out, nil + case ToolDataset: + var a datasetArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return out, err + } + if err := checkDataset(a); err != nil { + return out, err + } + if _, taken := r.j.job(env.Request.ResourceID); taken { + if _, done := r.j.receipt(env.Request.OperationID); !done { + return out, protocol.ErrConflict + } + } + out.Revision = contract.Revision{Schema: schema, Data: 0} + // The card shows the name and the size, never the content. + out.Summary = payload(map[string]any{"count": 1, "sample": map[string]any{"action": "send dataset", "name": a.Name, "bytes": len(a.Content)}}) + return out, nil + case ToolStop, ToolExport: + j, err := r.jobOf(env) + if err != nil { + return out, err + } + sample := map[string]any{"name": j.Name} + if tool == ToolStop { + var a stopArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return out, err + } + sample["action"], sample["saveCheckpoint"] = "stop fine-tuning job", a.SaveCheckpoint + } else { + var a exportArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return out, err + } + if err := checkExport(a); err != nil { + return out, err + } + sample["action"], sample["exportFormat"] = "export fine-tuned model", a.ExportFormat + } + out.Revision = contract.Revision{Schema: schema, Data: j.Revision} + out.Summary = payload(map[string]any{"count": 1, "sample": sample}) + return out, nil + } + return out, protocol.ErrBadRequest +} + +// checkDataset applies the rules of the host operation early, so a card is not +// shown for a dataset the host would refuse. +func checkDataset(a datasetArgs) error { + if a.Content == "" || len(a.Content) > maxDatasetBytes || !utf8.ValidString(a.Content) || len(a.Name) == 0 || len(a.Name) > 255 || strings.ContainsAny(a.Name, `/\`) || a.Name == "." || a.Name == ".." { + return protocol.ErrBadRequest + } + for _, ext := range datasetExt { + if strings.HasSuffix(a.Name, ext) && len(a.Name) > len(ext) { + return nil + } + } + return protocol.ErrBadRequest +} + +func nameOf(a startArgs) string { + if a.Name != "" { + return a.Name + } + return a.Model +} + +func (r *runtime) Apply(ctx context.Context, env contract.Envelope, rev contract.Revision) (contract.Receipt, error) { + tool, err := toolOf(env) + if err != nil { + return contract.Receipt{}, err + } + // An operation that already committed is answered from its receipt: the + // engine ran once and runs no more. + if rec, ok := r.j.receipt(env.Request.OperationID); ok { + return rec, nil + } + switch tool { + case ToolStart: + return r.applyStart(ctx, env) + case ToolDataset: + return r.applyDataset(ctx, env) + case ToolStop: + var a stopArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return contract.Receipt{}, err + } + return r.mutate(ctx, env, rev, "finetune.stop", func(id string) any { + return map[string]any{"jobId": id, "saveCheckpoint": a.SaveCheckpoint} + }) + case ToolExport: + var a exportArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return contract.Receipt{}, err + } + if err := checkExport(a); err != nil { + return contract.Receipt{}, err + } + return r.mutate(ctx, env, rev, "finetune.export", func(id string) any { + args := map[string]any{"jobId": id} + if a.ExportFormat != "" { + args["exportFormat"] = a.ExportFormat + } + if a.Name != "" { + args["name"] = a.Name + } + return args + }) + } + return contract.Receipt{}, protocol.ErrBadRequest +} + +func (r *runtime) applyStart(ctx context.Context, env contract.Envelope) (contract.Receipt, error) { + var a startArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return contract.Receipt{}, err + } + if err := r.checkStart(a); err != nil { + return contract.Receipt{}, err + } + res, op := env.Request.ResourceID, env.Request.OperationID + if res == "" { + return contract.Receipt{}, protocol.ErrBadRequest + } + if _, taken := r.j.job(res); taken { + return contract.Receipt{}, protocol.ErrConflict + } + var started struct { + ID string `json:"id"` + Status string `json:"status"` + Message string `json:"message"` + } + err := r.eng.Do(ctx, op, "finetune.start", engineStart{ + Model: a.Model, Backend: a.Backend, TrainingType: a.TrainingType, TrainingMethod: a.TrainingMethod, + DatasetSource: a.DatasetSource, DatasetSplit: a.DatasetSplit, LearningRate: a.LearningRate, + NumEpochs: a.NumEpochs, BatchSize: a.BatchSize, MaxSteps: a.MaxSteps, AdapterRank: a.AdapterRank, AdapterAlpha: a.AdapterAlpha, + }, &started) + if err != nil { + return contract.Receipt{}, mapErr(err) + } + if started.ID == "" { + return contract.Receipt{}, errors.New("the engine started a job and returned no ID") + } + rec := contract.Receipt{ + OperationID: op, State: contract.ReceiptCommitted, Revision: contract.Revision{Schema: schema, Data: 1}, + Result: contract.Result{Payload: payload(map[string]string{"status": started.Status, "message": started.Message})}, + } + // If this write fails the job runs without a mapping. The error is + // uncertain, so the host asks again with the same operation ID and the + // engine answers it with the same job. + if err := r.j.commit(res, &job{JobID: started.ID, Name: nameOf(a), Revision: 1}, rec); err != nil { + return contract.Receipt{}, err + } + return rec, nil +} + +// applyDataset sends the inline dataset through the host operation, which +// returns a reference only the calling user can use, and remembers it for the +// resource. The reference is in the receipt the person sees and in the status +// of the resource, which only the owner of the resource can read. +func (r *runtime) applyDataset(ctx context.Context, env contract.Envelope) (contract.Receipt, error) { + var a datasetArgs + if err := decode(env.Request.Arguments, &a); err != nil { + return contract.Receipt{}, err + } + if err := checkDataset(a); err != nil { + return contract.Receipt{}, err + } + res, op := env.Request.ResourceID, env.Request.OperationID + if res == "" { + return contract.Receipt{}, protocol.ErrBadRequest + } + if _, taken := r.j.job(res); taken { + return contract.Receipt{}, protocol.ErrConflict + } + var stored struct { + DatasetRef string `json:"datasetRef"` + } + if err := r.eng.Do(ctx, op, "finetune.dataset", map[string]string{"name": a.Name, "content": a.Content}, &stored); err != nil { + return contract.Receipt{}, mapErr(err) + } + if stored.DatasetRef == "" { + return contract.Receipt{}, errors.New("the engine stored a dataset and returned no reference") + } + rec := contract.Receipt{ + OperationID: op, State: contract.ReceiptCommitted, Revision: contract.Revision{Schema: schema, Data: 1}, + Result: contract.Result{Payload: payload(map[string]string{"datasetRef": stored.DatasetRef})}, + } + if err := r.j.commit(res, &job{Name: a.Name, Dataset: true, Ref: stored.DatasetRef, Revision: 1}, rec); err != nil { + return contract.Receipt{}, err + } + return rec, nil +} + +// mutate runs a host operation on the job of the resource, once, if the +// resource is still at the revision the person approved. +func (r *runtime) mutate(ctx context.Context, env contract.Envelope, rev contract.Revision, op string, args func(jobID string) any) (contract.Receipt, error) { + j, err := r.jobOf(env) + if err != nil { + return contract.Receipt{}, err + } + id := env.Request.OperationID + if rev.Schema != schema || rev.Data != j.Revision { + return contract.Receipt{OperationID: id, State: contract.ReceiptRejected, + Result: contract.Result{Payload: payload(map[string]any{"error": "stale", "revision": contract.Revision{Schema: schema, Data: j.Revision}})}}, nil + } + var out json.RawMessage + if err := r.eng.Do(ctx, id, op, args(j.JobID), &out); err != nil { + return contract.Receipt{}, mapErr(err) + } + next := j + next.Revision++ + rec := contract.Receipt{OperationID: id, State: contract.ReceiptCommitted, Revision: contract.Revision{Schema: schema, Data: next.Revision}} + if len(out) > 0 && json.Valid(out) { + rec.Result.Payload = out + } else { + rec.Result.Payload = payload(map[string]any{}) + } + if err := r.j.commit(env.Request.ResourceID, &next, rec); err != nil { + return contract.Receipt{}, err + } + return rec, nil +} + +func (r *runtime) Receipt(_ context.Context, op string) (contract.Receipt, error) { + if rec, ok := r.j.receipt(op); ok { + return rec, nil + } + return contract.Receipt{}, protocol.ErrNotFound +} + +// OpenOutput refuses: no tool of this add-on produces a file. +func (r *runtime) OpenOutput(context.Context, contract.Envelope, string) (io.ReadCloser, error) { + return nil, protocol.ErrNotFound +} diff --git a/addons/fine-tuning/internal/tools/tools_test.go b/addons/fine-tuning/internal/tools/tools_test.go new file mode 100644 index 000000000000..475ce594a274 --- /dev/null +++ b/addons/fine-tuning/internal/tools/tools_test.go @@ -0,0 +1,527 @@ +// SPDX-License-Identifier: MIT +package tools + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "strings" + "sync" + + "time" + + "github.com/mudler/LocalAI/addons/internal/hostclient" + "github.com/mudler/LocalAI/addons/internal/protocol" + "github.com/mudler/LocalAI/pkg/addons/contract" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +var ctx = context.Background() + +type call struct { + opID, op string + args map[string]any +} + +// fakeEngine records host operations and answers from canned results. A +// result of type error is returned as the failure of that operation. +type fakeEngine struct { + mu sync.Mutex + result map[string]any + calls []call +} + +func (f *fakeEngine) Do(_ context.Context, opID, op string, args, out any) error { + f.mu.Lock() + defer f.mu.Unlock() + var a map[string]any + if args != nil { + b, _ := json.Marshal(args) + _ = json.Unmarshal(b, &a) + } + f.calls = append(f.calls, call{opID, op, a}) + r, ok := f.result[op] + if !ok { + r = map[string]any{} + } + if err, isErr := r.(error); isErr { + return err + } + if out != nil { + b, _ := json.Marshal(r) + return json.Unmarshal(b, out) + } + return nil +} + +func (f *fakeEngine) ops() []string { + f.mu.Lock() + defer f.mu.Unlock() + var out []string + for _, c := range f.calls { + out = append(out, c.op) + } + return out +} + +func (f *fakeEngine) count(op string) int { + n := 0 + for _, o := range f.ops() { + if o == op { + n++ + } + } + return n +} + +func (f *fakeEngine) last(op string) call { + f.mu.Lock() + defer f.mu.Unlock() + for i := len(f.calls) - 1; i >= 0; i-- { + if f.calls[i].op == op { + return f.calls[i] + } + } + return call{} +} + +func envelope(tool, opID, resource, args string) contract.Envelope { + m := map[string]any{} + if args != "" { + _ = json.Unmarshal([]byte(args), &m) + } + m["tool"] = tool + raw, _ := json.Marshal(m) + return contract.Envelope{ + Request: contract.Request{AddonID: "fine-tuning", ResourceID: resource, OperationID: opID, Generation: 1, Action: "apply", Arguments: raw}, + Authority: contract.Authority{Actor: contract.Actor{UserID: "u1", CredentialID: "c1", Mode: "named-user"}, Scope: "apply", ExpiresAt: time.Now().Add(time.Minute)}, + } +} + +const startJSON = `{"name":"my run","model":"org/base","datasetSource":"org/data"}` + +// started returns a runtime that already holds the job job-1 for resource res1. +func started() (contract.Runtime, *fakeEngine) { + eng := &fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1", "status": "queued"}}} + rt := New(eng, GinkgoT().TempDir()) + rec, err := rt.Apply(ctx, envelope("fine-tune-start", "create-res1", "res1", startJSON), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred(), "setup") + Expect(rec.State).To(Equal(contract.ReceiptCommitted), "setup: %+v", rec) + return rt, eng +} + +func exists(rt contract.Runtime, res string) (bool, error) { + r, err := rt.Read(ctx, envelope("resource-exists", "x-"+res, res, "")) + if err != nil { + return false, err + } + var v struct { + Exists bool `json:"exists"` + } + _ = json.Unmarshal(r.Payload, &v) + return v.Exists, nil +} + +func loadManifest() contract.Manifest { + b, err := os.ReadFile("../../manifest.json") + Expect(err).NotTo(HaveOccurred()) + m, err := contract.ParseManifest(b) + Expect(err).NotTo(HaveOccurred(), "the manifest must satisfy the contract") + return m +} + +const datasetJSON = `{"name":"data.jsonl","content":"{\"text\":\"a\"}\n"}` + +var _ = Describe("fine-tuning tools", func() { + It("start calls the engine once and remembers the job", func() { + eng := &fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1", "status": "queued"}}} + rt := New(eng, GinkgoT().TempDir()) + rec, err := rt.Apply(ctx, envelope("fine-tune-start", "op1", "res1", startJSON), contract.Revision{Schema: 1, Data: 0}) + Expect(err).NotTo(HaveOccurred()) + Expect(rec.State).To(Equal(contract.ReceiptCommitted)) + Expect(rec.Revision.Schema).NotTo(BeZero()) + Expect(eng.ops()).To(Equal([]string{"finetune.start"})) + // The same operation again does not start a second job. + again, err := rt.Apply(ctx, envelope("fine-tune-start", "op1", "res1", startJSON), contract.Revision{Schema: 1, Data: 1}) + Expect(err).NotTo(HaveOccurred()) + Expect(again.State).To(Equal(contract.ReceiptCommitted)) + Expect(eng.count("finetune.start")).To(Equal(1)) + }) + + It("start sends only the fields of the engine operation", func() { + eng := &fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1", "status": "queued"}}} + rt := New(eng, GinkgoT().TempDir()) + args := `{"name":"my run","model":"org/base","datasetSource":"org/data","numEpochs":2,"learningRate":0.0002,"adapterRank":8}` + _, err := rt.Apply(ctx, envelope("fine-tune-start", "op1", "res1", args), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred()) + c := eng.last("finetune.start") + Expect(c.opID).To(Equal("op1"), "the call must carry the operation of the request") + for _, banned := range []string{"tool", "name", "resourceId"} { + Expect(c.args).NotTo(HaveKey(banned), "the engine must not receive %q", banned) + } + Expect(c.args["model"]).To(Equal("org/base")) + Expect(c.args["numEpochs"]).To(Equal(float64(2))) + Expect(c.args["adapterRank"]).To(Equal(float64(8))) + }) + + It("previews a start with a nonzero revision and creates nothing", func() { + eng := &fakeEngine{} + rt := New(eng, GinkgoT().TempDir()) + p, err := rt.Preview(ctx, envelope("fine-tune-start", "op1", "res1", startJSON)) + Expect(err).NotTo(HaveOccurred()) + Expect(p.Revision.Schema).NotTo(BeZero()) + Expect(eng.ops()).To(BeEmpty()) + Expect(json.Valid(p.Summary)).To(BeTrue(), "summary %s", p.Summary) + _, err = rt.Preview(ctx, envelope("fine-tune-start", "op1", "res1", `{"name":"x","model":"","datasetSource":"d"}`)) + Expect(err).To(MatchError(protocol.ErrBadRequest), "an empty model must be refused") + }) + + It("status reads the job of the resource", func() { + rt, eng := started() + eng.result["finetune.get"] = map[string]any{"id": "job-1", "status": "training", "model": "org/base"} + res, err := rt.Read(ctx, envelope("fine-tune-status", "r1", "res1", "")) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.last("finetune.get").args["jobId"]).To(Equal("job-1")) + var v map[string]any + Expect(json.Unmarshal(res.Payload, &v)).To(Succeed()) + Expect(v["name"]).To(Equal("my run")) + Expect(v["status"]).To(Equal("training")) + rev, _ := v["revision"].(map[string]any) + Expect(rev["schema"]).NotTo(BeNil(), "inspect needs a revision: %v", v) + Expect(rev["schema"]).NotTo(Equal(float64(0)), "inspect needs a revision: %v", v) + Expect(v).NotTo(HaveKey("id"), "the engine job ID must not reach the model") + Expect(v).NotTo(HaveKey("progressPercent"), "a snapshot percent is unreliable and must not be reported") + }) + + It("checkpoints read the job of the resource", func() { + rt, eng := started() + eng.result["finetune.checkpoints"] = map[string]any{"checkpoints": []any{map[string]any{"step": 10}}} + res, err := rt.Read(ctx, envelope("fine-tune-checkpoints", "r1", "res1", "")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(res.Payload)).To(ContainSubstring(`"step":10`)) + Expect(eng.last("finetune.checkpoints").args["jobId"]).To(Equal("job-1"), "wrong job") + }) + + It("does not let a tool control a job it did not start", func() { + eng := &fakeEngine{} + rt := New(eng, GinkgoT().TempDir()) + _, err := rt.Apply(ctx, envelope("fine-tune-stop", "op2", "nope", `{}`), contract.Revision{Schema: 1}) + Expect(err).To(MatchError(protocol.ErrNotFound), "stop") + _, err = rt.Apply(ctx, envelope("fine-tune-export", "op3", "nope", `{}`), contract.Revision{Schema: 1}) + Expect(err).To(MatchError(protocol.ErrNotFound), "export") + _, err = rt.Read(ctx, envelope("fine-tune-status", "r1", "nope", "")) + Expect(err).To(MatchError(protocol.ErrNotFound), "status") + _, err = rt.Preview(ctx, envelope("fine-tune-stop", "op2", "nope", `{}`)) + Expect(err).To(MatchError(protocol.ErrNotFound), "preview") + Expect(eng.ops()).To(BeEmpty(), "the engine must not be called") + }) + + It("stop and export use the stored job and advance the revision", func() { + rt, eng := started() + eng.result["finetune.stop"] = map[string]any{} + eng.result["finetune.export"] = map[string]any{"model": "my-model"} + p, err := rt.Preview(ctx, envelope("fine-tune-stop", "op2", "res1", `{"saveCheckpoint":true}`)) + Expect(err).NotTo(HaveOccurred()) + Expect(p.Revision.Schema).NotTo(BeZero()) + rec, err := rt.Apply(ctx, envelope("fine-tune-stop", "op2", "res1", `{"saveCheckpoint":true}`), p.Revision) + Expect(err).NotTo(HaveOccurred()) + Expect(rec.State).To(Equal(contract.ReceiptCommitted)) + c := eng.last("finetune.stop") + Expect(c.args["jobId"]).To(Equal("job-1")) + Expect(c.args["saveCheckpoint"]).To(Equal(true)) + Expect(rec.Revision.Data).To(BeNumerically(">", p.Revision.Data), "the revision must advance") + // A repeat of the same operation is answered from the receipt. + _, err = rt.Apply(ctx, envelope("fine-tune-stop", "op2", "res1", `{"saveCheckpoint":true}`), p.Revision) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.count("finetune.stop")).To(Equal(1)) + // Export names the format and the model, and never a path. + ex, err := rt.Apply(ctx, envelope("fine-tune-export", "op3", "res1", `{"exportFormat":"gguf","name":"my-model"}`), rec.Revision) + Expect(err).NotTo(HaveOccurred()) + Expect(ex.State).To(Equal(contract.ReceiptCommitted)) + Expect(string(ex.Result.Payload)).To(ContainSubstring("my-model")) + a := eng.last("finetune.export").args + Expect(a["jobId"]).To(Equal("job-1")) + Expect(a["exportFormat"]).To(Equal("gguf")) + Expect(a["name"]).To(Equal("my-model")) + }) + + It("rejects a stale revision and does not call the engine", func() { + rt, eng := started() + rec, err := rt.Apply(ctx, envelope("fine-tune-stop", "op2", "res1", `{}`), contract.Revision{Schema: 1, Data: 99}) + Expect(err).NotTo(HaveOccurred()) + Expect(rec.State).To(Equal(contract.ReceiptRejected)) + Expect(eng.count("finetune.stop")).To(BeZero(), "the engine was called") + }) + + Describe("host refusals", func() { + for code, want := range map[string]error{ + contract.HostFeature: protocol.ErrForbidden, contract.HostBadRequest: protocol.ErrBadRequest, + contract.HostNotFound: protocol.ErrNotFound, contract.HostConflict: protocol.ErrConflict, + } { + It("maps "+code+" to a protocol error", func() { + rt, eng := started() + eng.result["finetune.export"] = &contract.HostError{Code: code, Message: "m"} + _, err := rt.Apply(ctx, envelope("fine-tune-export", "op-"+code, "res1", `{}`), contract.Revision{Schema: 1, Data: 1}) + Expect(err).To(MatchError(want)) + }) + } + + It("keeps an unavailable engine uncertain", func() { + rt, eng := started() + eng.result["finetune.export"] = &contract.HostError{Code: contract.HostUnavailable, Message: "m"} + _, err := rt.Apply(ctx, envelope("fine-tune-export", "op-u", "res1", `{}`), contract.Revision{Schema: 1, Data: 1}) + Expect(err).To(HaveOccurred()) + for _, e := range []error{protocol.ErrBadRequest, protocol.ErrForbidden, protocol.ErrNotFound, protocol.ErrConflict} { + Expect(err).NotTo(MatchError(e), "an unavailable engine must stay uncertain") + } + }) + }) + + It("answers resource-exists from the journal and the engine", func() { + rt, eng := started() + eng.result["finetune.get"] = map[string]any{"id": "job-1", "status": "training"} + ok, err := exists(rt, "res1") + Expect(err).NotTo(HaveOccurred(), "mapped") + Expect(ok).To(BeTrue(), "mapped") + ok, err = exists(rt, "other") + Expect(err).NotTo(HaveOccurred(), "unmapped") + Expect(ok).To(BeFalse(), "unmapped") + // The job was deleted in the UI: the engine says not found. + eng.result["finetune.get"] = &contract.HostError{Code: contract.HostNotFound, Message: "gone"} + ok, err = exists(rt, "res1") + Expect(err).NotTo(HaveOccurred(), "deleted job") + Expect(ok).To(BeFalse(), "deleted job") + // The engine cannot answer: unknown, never false. + eng.result["finetune.get"] = &contract.HostError{Code: contract.HostUnavailable, Message: "down"} + _, err = exists(rt, "res1") + Expect(err).To(HaveOccurred(), "an unavailable engine must not read as absent") + eng.result["finetune.get"] = hostclient.ErrNoHostOps + _, err = exists(rt, "res1") + Expect(err).To(HaveOccurred(), "no host endpoint must not read as absent") + }) + + It("keeps the journal across a restart", func() { + dir := GinkgoT().TempDir() + eng := &fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1", "status": "queued"}, "finetune.get": map[string]any{"status": "training"}}} + _, err := New(eng, dir).Apply(ctx, envelope("fine-tune-start", "op1", "res1", startJSON), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred()) + again := New(eng, dir) + _, err = again.Read(ctx, envelope("fine-tune-status", "r1", "res1", "")) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.last("finetune.get").args["jobId"]).To(Equal("job-1"), "the restarted runtime lost the job") + // The receipt of the first run answers a repeat after the restart. + _, err = again.Apply(ctx, envelope("fine-tune-start", "op1", "res1", startJSON), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.count("finetune.start")).To(Equal(1)) + r, err := again.Receipt(ctx, "op1") + Expect(err).NotTo(HaveOccurred()) + Expect(r.State).To(Equal(contract.ReceiptCommitted)) + _, err = again.Receipt(ctx, "never") + Expect(err).To(MatchError(protocol.ErrNotFound), "an unknown operation must be not found") + // The journal is a plain file written through a temporary one. + entries, _ := os.ReadDir(dir) + for _, e := range entries { + Expect(e.Name()).NotTo(ContainSubstring(".tmp"), "a temporary file was left") + } + _, err = os.Stat(filepath.Join(dir, "journal.json")) + Expect(err).NotTo(HaveOccurred()) + }) + + It("refuses every tool without a host endpoint", func() { + var nilClient *hostclient.Client + dir := GinkgoT().TempDir() + rt := New(nilClient, dir) + _, err := rt.Apply(ctx, envelope("fine-tune-start", "op1", "res1", startJSON), contract.Revision{Schema: 1}) + Expect(err).To(MatchError(hostclient.ErrNoHostOps), "start") + // Seed a job as an earlier run with an endpoint would have. + seed := New(&fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1"}}}, dir) + _, err = seed.Apply(ctx, envelope("fine-tune-start", "op0", "res1", startJSON), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred()) + rt = New(nilClient, dir) + for _, tool := range []string{"fine-tune-status", "fine-tune-checkpoints", "resource-exists"} { + _, err := rt.Read(ctx, envelope(tool, "r-"+tool, "res1", "")) + Expect(err).To(MatchError(hostclient.ErrNoHostOps), tool) + } + for _, tool := range []string{"fine-tune-stop", "fine-tune-export"} { + _, err := rt.Apply(ctx, envelope(tool, "a-"+tool, "res1", `{}`), contract.Revision{Schema: 1, Data: 1}) + Expect(err).To(MatchError(hostclient.ErrNoHostOps), tool) + } + }) + + It("treats an unknown tool as a bad request", func() { + rt := New(&fakeEngine{}, GinkgoT().TempDir()) + _, err := rt.Read(ctx, envelope("fine-tune-nope", "r1", "res1", "")) + Expect(err).To(MatchError(protocol.ErrBadRequest), "read") + _, err = rt.Apply(ctx, envelope("fine-tune-nope", "a1", "res1", ""), contract.Revision{Schema: 1}) + Expect(err).To(MatchError(protocol.ErrBadRequest), "apply") + _, err = rt.OpenOutput(ctx, envelope("fine-tune-status", "o1", "res1", ""), "h") + Expect(err).To(MatchError(protocol.ErrNotFound), "output") + }) + + Describe("manifest", func() { + It("declares the tools and the resource mapping", func() { + m := loadManifest() + Expect(m.ID).To(Equal("fine-tuning")) + Expect(m.Capabilities.UI.MainPage).To(Equal("jobs")) + Expect(m.Capabilities.UI.Pages).To(Equal([]string{"jobs"})) + Expect(m.Capabilities.UI.SDK.Min).To(BeEquivalentTo(2)) + Expect(m.Capabilities.UI.SDK.Max).To(BeEquivalentTo(2)) + Expect(m.Capabilities.Tools).To(Equal(Tools)) + r := m.Capabilities.Resources + Expect(r).NotTo(BeNil()) + Expect(r.Noun).To(Equal("fine-tuning job")) + Expect(r.Create).To(Equal("fine-tune-start")) + Expect(r.Inspect).To(Equal("fine-tune-status")) + kinds := map[string]string{} + for _, d := range m.Capabilities.ToolDefinitions { + kinds[d.Name] = d.Kind + } + want := map[string]string{"fine-tune-list": "read", "fine-tune-status": "read", "fine-tune-checkpoints": "read", "fine-tune-start": "create", "fine-tune-dataset": "create", "fine-tune-stop": "write", "fine-tune-export": "write"} + Expect(kinds).To(Equal(want)) + for _, d := range m.Capabilities.ToolDefinitions { + if d.Name == "fine-tune-list" { + Expect(contract.IsResourceless(d)).To(BeTrue(), "the list tool must declare no resource") + } else { + Expect(contract.IsResourceless(d)).To(BeFalse(), "%s must not be resource-less", d.Name) + } + Expect(len(d.Description)).To(BeNumerically("<", 400), d.Name) + Expect(d.Description).To(ContainSubstring("started"), "%s: the description must say that only jobs a tool started can be controlled", d.Name) + } + }) + + It("names no user or path in any tool argument", func() { + m := loadManifest() + for _, d := range m.Capabilities.ToolDefinitions { + var s struct { + Properties map[string]json.RawMessage `json:"properties"` + } + Expect(json.Unmarshal(d.InputSchema, &s)).To(Succeed()) + for name := range s.Properties { + lower := strings.ToLower(name) + Expect(contract.ForbiddenToolProperty(name) || strings.Contains(lower, "path") || strings.Contains(lower, "dir")).To(BeFalse(), + "%s: property %q names an identity or a path", d.Name, name) + } + } + }) + + It("gives the start schema the task 20 fields and the name the host needs", func() { + m := loadManifest() + found := false + for _, d := range m.Capabilities.ToolDefinitions { + if d.Name != "fine-tune-start" { + continue + } + found = true + var s struct { + Properties map[string]json.RawMessage `json:"properties"` + Required []string `json:"required"` + } + _ = json.Unmarshal(d.InputSchema, &s) + want := []string{"name", "model", "backend", "trainingType", "trainingMethod", "datasetSource", "datasetSplit", "learningRate", "numEpochs", "batchSize", "maxSteps", "adapterRank", "adapterAlpha"} + Expect(s.Properties).To(HaveLen(len(want))) + for _, w := range want { + Expect(s.Properties).To(HaveKey(w)) + } + break + } + Expect(found).To(BeTrue(), "no start definition") + }) + }) + + It("lists the caller's jobs without a resource", func() { + eng := &fakeEngine{result: map[string]any{"finetune.list": map[string]any{"jobs": []any{map[string]any{"id": "ui-job", "status": "completed"}}}}} + rt := New(eng, GinkgoT().TempDir()) + res, err := rt.Read(ctx, envelope("fine-tune-list", "r1", "", "")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(res.Payload)).To(ContainSubstring("ui-job")) + Expect(eng.ops()).To(Equal([]string{"finetune.list"})) + Expect(eng.last("finetune.list").args).To(BeNil(), "the list takes no argument") + _, err = rt.Read(ctx, envelope("fine-tune-list", "r2", "", `{"userId":"other"}`)) + Expect(err).To(MatchError(protocol.ErrBadRequest), "an argument must be refused") + var nilClient *hostclient.Client + _, err = New(nilClient, GinkgoT().TempDir()).Read(ctx, envelope("fine-tune-list", "r3", "", "")) + Expect(err).To(MatchError(hostclient.ErrNoHostOps), "no endpoint") + }) + + It("sends a dataset once and returns its ref only through the resource", func() { + eng := &fakeEngine{result: map[string]any{"finetune.dataset": map[string]any{"datasetRef": "0000000a-data.jsonl"}}} + rt := New(eng, GinkgoT().TempDir()) + p, err := rt.Preview(ctx, envelope("fine-tune-dataset", "create-d1", "d1", datasetJSON)) + Expect(err).NotTo(HaveOccurred()) + Expect(p.Revision.Schema).NotTo(BeZero()) + Expect(eng.ops()).To(BeEmpty()) + Expect(string(p.Summary)).NotTo(ContainSubstring("text"), "the approval card must not carry the content") + rec, err := rt.Apply(ctx, envelope("fine-tune-dataset", "create-d1", "d1", datasetJSON), p.Revision) + Expect(err).NotTo(HaveOccurred()) + Expect(rec.State).To(Equal(contract.ReceiptCommitted)) + Expect(string(rec.Result.Payload)).To(ContainSubstring("0000000a-data.jsonl")) + _, err = rt.Apply(ctx, envelope("fine-tune-dataset", "create-d1", "d1", datasetJSON), p.Revision) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.count("finetune.dataset")).To(Equal(1)) + a := eng.last("finetune.dataset").args + Expect(a["name"]).To(Equal("data.jsonl")) + Expect(a["content"]).NotTo(BeNil()) + Expect(a).To(HaveLen(2), "host operation args") + // The resource is the dataset: its status gives the reference to the owner of the resource. + st, err := rt.Read(ctx, envelope("fine-tune-status", "r1", "d1", "")) + Expect(err).NotTo(HaveOccurred()) + Expect(string(st.Payload)).To(ContainSubstring(`"datasetRef":"0000000a-data.jsonl"`)) + Expect(string(st.Payload)).To(ContainSubstring(`"kind":"dataset"`)) + Expect(eng.ops()).To(HaveLen(1), "the status of a dataset must not call the engine") + ok, err := exists(rt, "d1") + Expect(err).NotTo(HaveOccurred()) + Expect(ok).To(BeTrue()) + // A dataset is not a job. + for _, tool := range []string{"fine-tune-checkpoints"} { + _, err := rt.Read(ctx, envelope(tool, "r2", "d1", "")) + Expect(err).To(MatchError(protocol.ErrNotFound), tool) + } + _, err = rt.Apply(ctx, envelope("fine-tune-stop", "s1", "d1", `{}`), contract.Revision{Schema: 1, Data: 1}) + Expect(err).To(MatchError(protocol.ErrNotFound), "stop") + }) + + Describe("dataset validation", func() { + big := strings.Repeat("a", 900<<10+1) + for name, args := range map[string]string{ + "empty": `{"name":"d.jsonl","content":""}`, + "too large": `{"name":"d.jsonl","content":"` + big + `"}`, + "a folder": `{"name":"a/d.jsonl","content":"x"}`, + "parent": `{"name":"../d.jsonl","content":"x"}`, + "wrong type": `{"name":"d.exe","content":"x"}`, + "only the type": `{"name":".jsonl","content":"x"}`, + "extra field": `{"name":"d.jsonl","content":"x","path":"/etc"}`, + } { + It("refuses "+name, func() { + rt := New(&fakeEngine{}, GinkgoT().TempDir()) + _, err := rt.Preview(ctx, envelope("fine-tune-dataset", "o-"+name, "d9", args)) + Expect(err).To(MatchError(protocol.ErrBadRequest)) + }) + } + + It("accepts the largest allowed dataset", func() { + rt := New(&fakeEngine{}, GinkgoT().TempDir()) + ok := `{"name":"d.jsonl","content":"` + strings.Repeat("a", 900<<10) + `"}` + _, err := rt.Preview(ctx, envelope("fine-tune-dataset", "o-ok", "d9", ok)) + Expect(err).NotTo(HaveOccurred(), "900 KiB") + }) + }) + + It("passes a dataset ref to start and keeps no job when the host refuses it", func() { + eng := &fakeEngine{result: map[string]any{"finetune.start": map[string]any{"id": "job-1", "status": "queued"}}} + rt := New(eng, GinkgoT().TempDir()) + args := `{"name":"run","model":"org/base","datasetSource":"0000000a-data.jsonl"}` + _, err := rt.Apply(ctx, envelope("fine-tune-start", "create-r1", "r1", args), contract.Revision{Schema: 1}) + Expect(err).NotTo(HaveOccurred()) + Expect(eng.last("finetune.start").args["datasetSource"]).To(Equal("0000000a-data.jsonl")) + // The host refuses the reference (the per-user check is proven in the host tests); the add-on only maps the refusal and keeps nothing. + eng.result["finetune.start"] = &contract.HostError{Code: contract.HostBadRequest, Message: "datasetSource is not a dataset reference of this user"} + _, err = rt.Apply(ctx, envelope("fine-tune-start", "create-r2", "r2", `{"name":"run","model":"org/base","datasetSource":"0000000b-theirs.jsonl"}`), contract.Revision{Schema: 1}) + Expect(err).To(MatchError(protocol.ErrBadRequest)) + _, err = rt.Read(ctx, envelope("fine-tune-status", "r3", "r2", "")) + Expect(err).To(MatchError(protocol.ErrNotFound), "a refused start must not leave a job") + _, err = rt.Receipt(ctx, "create-r2") + Expect(err).To(MatchError(protocol.ErrNotFound), "a refused start must not leave a receipt") + }) +}) diff --git a/addons/fine-tuning/manifest.json b/addons/fine-tuning/manifest.json new file mode 100644 index 000000000000..07d7e5532018 --- /dev/null +++ b/addons/fine-tuning/manifest.json @@ -0,0 +1,285 @@ +{ + "schemaVersion": 1, + "id": "fine-tuning", + "version": "0.1.0", + "executable": "fine-tuning", + "os": "linux", + "arch": "amd64", + "host": { + "min": 1, + "max": 1 + }, + "api": { + "min": 1, + "max": 1 + }, + "protocol": { + "min": 1, + "max": 1 + }, + "dataFormat": { + "min": 1, + "max": 1 + }, + "capabilities": { + "tools": [ + "resource-exists", + "fine-tune-list", + "fine-tune-start", + "fine-tune-dataset", + "fine-tune-status", + "fine-tune-checkpoints", + "fine-tune-stop", + "fine-tune-export" + ], + "toolDefinitions": [ + { + "name": "fine-tune-list", + "kind": "read", + "description": "List the fine-tuning jobs of the person you are talking to, newest first, including jobs started in the web page. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": {}, + "additionalProperties": false + } + }, + { + "name": "fine-tune-start", + "kind": "create", + "description": "Propose a new fine-tuning job. A person confirms it in the approval card; nothing starts before that. datasetSource is a Hugging Face ID or a reference from fine-tune-dataset. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "A label for the job, 1 to 80 characters.", + "maxLength": 80 + }, + "model": { + "type": "string", + "description": "Base model as a gallery or Hugging Face ID. A local path is refused.", + "maxLength": 256 + }, + "backend": { + "type": "string", + "description": "Fine-tuning backend. The default is trl.", + "maxLength": 64 + }, + "trainingType": { + "type": "string", + "description": "Kind of training. The default is lora.", + "enum": [ + "lora", + "loha", + "lokr", + "full" + ] + }, + "trainingMethod": { + "type": "string", + "description": "Training method. The default is sft.", + "enum": [ + "sft", + "dpo", + "grpo", + "rloo", + "reward", + "kto", + "orpo" + ] + }, + "datasetSource": { + "type": "string", + "description": "Dataset as a Hugging Face ID or a reference from fine-tune-dataset. A local path is refused.", + "maxLength": 256 + }, + "datasetSplit": { + "type": "string", + "description": "Dataset split, such as train.", + "maxLength": 64 + }, + "learningRate": { + "type": "number", + "description": "Learning rate. Zero keeps the default.", + "minimum": 0 + }, + "numEpochs": { + "type": "integer", + "description": "Number of epochs. Zero keeps the default.", + "minimum": 0, + "maximum": 1000 + }, + "batchSize": { + "type": "integer", + "description": "Batch size. Zero keeps the default.", + "minimum": 0, + "maximum": 4096 + }, + "maxSteps": { + "type": "integer", + "description": "Stop after this many steps. Zero means no limit.", + "minimum": 0 + }, + "adapterRank": { + "type": "integer", + "description": "Rank of the adapter. Zero keeps the default.", + "minimum": 0, + "maximum": 1024 + }, + "adapterAlpha": { + "type": "integer", + "description": "Alpha of the adapter. Zero keeps the default.", + "minimum": 0, + "maximum": 4096 + } + }, + "required": [ + "name", + "model", + "datasetSource" + ], + "additionalProperties": false + } + }, + { + "name": "fine-tune-dataset", + "kind": "create", + "description": "Propose sending a small text dataset (up to 900 KiB, a .jsonl, .json, .csv or .txt file) to the server. A person confirms it. The reference it returns is the datasetSource of fine-tune-start, works for that person only and is lost when the server restarts. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "File name of the dataset, ending in .jsonl, .json, .csv or .txt. No folder.", + "maxLength": 255 + }, + "content": { + "type": "string", + "description": "The dataset as UTF-8 text, 1 byte to 900 KiB.", + "maxLength": 921600 + } + }, + "required": [ + "name", + "content" + ], + "additionalProperties": false + } + }, + { + "name": "fine-tune-status", + "kind": "read", + "description": "Show the status of a fine-tuning job that a tool started, or the reference of a dataset that fine-tune-dataset sent. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "resourceId": { + "type": "string", + "description": "ID of the fine-tuning job, as returned by a job start. Only a job that a tool started has one." + } + }, + "required": [ + "resourceId" + ], + "additionalProperties": false + } + }, + { + "name": "fine-tune-checkpoints", + "kind": "read", + "description": "List the saved checkpoints of a fine-tuning job that a tool started, with step and time. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "resourceId": { + "type": "string", + "description": "ID of the fine-tuning job, as returned by a job start. Only a job that a tool started has one." + } + }, + "required": [ + "resourceId" + ], + "additionalProperties": false + } + }, + { + "name": "fine-tune-stop", + "kind": "write", + "description": "Propose stopping a fine-tuning job that a tool started. A person confirms it. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "resourceId": { + "type": "string", + "description": "ID of the fine-tuning job, as returned by a job start. Only a job that a tool started has one." + }, + "saveCheckpoint": { + "type": "boolean", + "description": "Keep a checkpoint of the work so far. The default is false." + } + }, + "required": [ + "resourceId" + ], + "additionalProperties": false + } + }, + { + "name": "fine-tune-export", + "kind": "write", + "description": "Propose registering the result of a fine-tuning job that a tool started as a model. Administrators only. A model can only control jobs it started.", + "inputSchema": { + "type": "object", + "properties": { + "resourceId": { + "type": "string", + "description": "ID of the fine-tuning job, as returned by a job start. Only a job that a tool started has one." + }, + "exportFormat": { + "type": "string", + "description": "Export format. The default is the engine default.", + "enum": [ + "lora", + "merged_16bit", + "merged_4bit", + "gguf" + ] + }, + "name": { + "type": "string", + "description": "Name of the model, 1 to 64 letters, digits, dots, dashes or underscores. Leave out to let the engine choose.", + "maxLength": 64 + } + }, + "required": [ + "resourceId" + ], + "additionalProperties": false + } + } + ], + "resources": { + "noun": "fine-tuning job", + "create": "fine-tune-start", + "inspect": "fine-tune-status" + }, + "ui": { + "entry": "ui/fine-tuning.js", + "css": [ + "ui/fine-tuning.css" + ], + "sdk": { + "min": 2, + "max": 2 + }, + "pages": [ + "jobs" + ], + "navigation": [], + "mainPage": "jobs", + "icon": "ui/icon.png", + "title": "Fine-tuning", + "description": "Train an adapter on your own data, then export it as a model you can chat with." + } + } +} diff --git a/addons/fine-tuning/package/platforms.yaml b/addons/fine-tuning/package/platforms.yaml new file mode 100644 index 000000000000..5dec8568b09b --- /dev/null +++ b/addons/fine-tuning/package/platforms.yaml @@ -0,0 +1,38 @@ +# Release platform matrix of the fine-tuning add-on package. +# +# A platform ships only while "enabled" is true. Enable a platform in the same +# change that shows its native check passing on a runner of that platform; +# until then it stays listed and disabled. The release workflow fails closed +# when no platform is enabled, and it never builds a platform that is not +# enabled here. +# +# runner: the GitHub runner label that builds and checks the platform +# natively. Required for an enabled platform. +# native_check: "executable-start": the packaged executable is extracted on +# the native runner and started. Required for an enabled platform. +# runtime: files besides the executable that the platform needs at run +# time, relative to the package root. They are taken from the +# runtime directory of the build and must exist. +schema: 1 +addon: fine-tuning +executable: fine-tuning +manifest: addons/fine-tuning/manifest.json +# The directory whose content is laid at the package root (ui/fine-tuning.js ...). +ui: addons/fine-tuning/ui/dist +platforms: + - platform: linux/amd64 + enabled: true + runner: ubuntu-24.04 + native_check: executable-start + - platform: linux/arm64 + enabled: true + runner: ubuntu-24.04-arm + native_check: executable-start + - platform: darwin/arm64 + enabled: true + runner: macos-14 + native_check: executable-start + - platform: darwin/amd64 + enabled: true + runner: macos-15-intel + native_check: executable-start diff --git a/addons/fine-tuning/ui/.gitignore b/addons/fine-tuning/ui/.gitignore new file mode 100644 index 000000000000..1225aa2fae58 --- /dev/null +++ b/addons/fine-tuning/ui/.gitignore @@ -0,0 +1,2 @@ +# Written by build.mjs; the package build makes it. +/dist/ diff --git a/addons/fine-tuning/ui/build.mjs b/addons/fine-tuning/ui/build.mjs new file mode 100644 index 000000000000..1c980473c36e --- /dev/null +++ b/addons/fine-tuning/ui/build.mjs @@ -0,0 +1,13 @@ +import { fileURLToPath } from 'node:url' +import { dirname } from 'node:path' +import { buildAddonUI } from '../../internal/uibuild/uibuild.mjs' + +const here = dirname(fileURLToPath(import.meta.url)) + +export async function build() { + return buildAddonUI({ root: here, name: 'fine-tuning', rgb: [92, 74, 150] }) +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + build().then((dir) => console.log(`built ${dir}`), (e) => { console.error(e); process.exit(1) }) +} diff --git a/addons/fine-tuning/ui/icon.png b/addons/fine-tuning/ui/icon.png new file mode 100644 index 000000000000..5f76999d12e1 Binary files /dev/null and b/addons/fine-tuning/ui/icon.png differ diff --git a/addons/fine-tuning/ui/package.json b/addons/fine-tuning/ui/package.json new file mode 100644 index 000000000000..45d0c72499d4 --- /dev/null +++ b/addons/fine-tuning/ui/package.json @@ -0,0 +1,8 @@ +{ + "name": "localai-fine-tuning-ui", + "private": true, + "version": "0.1.0", + "description": "Native UI of the fine-tuning add-on: a standalone ES module that the host loads after its own build.", + "type": "module", + "scripts": { "build": "node build.mjs", "test": "node test/run.mjs" } +} diff --git a/addons/fine-tuning/ui/src/FineTune.jsx b/addons/fine-tuning/ui/src/FineTune.jsx new file mode 100644 index 000000000000..8879aa979949 --- /dev/null +++ b/addons/fine-tuning/ui/src/FineTune.jsx @@ -0,0 +1,1350 @@ +/* eslint-disable no-unused-vars -- components used only inside JSX look unused to this config */ +// The fine-tuning page, ported from core/http/react-ui/src/pages/FineTune.jsx +// with the port table: React, the translator, the shared components, the +// session-bound fetch and the viewer come from ctx; the job views are the +// shared ones of addons/internal/tools-ui. Class names, test ids and keys are +// the core page's. +import { percentOf, ACTIVE } from './lib/progress.js' +import { createLossChart } from './LossChart.jsx' + +const TRAINING_METHODS = ['sft', 'dpo', 'grpo', 'rloo', 'reward', 'kto', 'orpo'] +const ADAPTER_KINDS = ['lora', 'loha', 'lokr'] +const FALLBACK_BACKEND = 'trl' +const OPTIMIZERS = ['adamw_torch', 'adamw_8bit', 'sgd', 'adafactor', 'prodigy'] +const MIXED_PRECISION_OPTS = ['', 'fp16', 'bf16', 'no'] +const QUANT_PRESETS = ['q4_k_m', 'q5_k_m', 'q8_0', 'f16', 'q4_0', 'q5_0'] + +const BUILTIN_REWARDS = [ + { name: 'format_reward', params: [] }, + { name: 'reasoning_accuracy_reward', params: [] }, + { name: 'length_reward', params: [{ key: 'target_length', default: '200' }] }, + { name: 'xml_tag_reward', params: [] }, + { name: 'no_repetition_reward', params: [] }, + { name: 'code_execution_reward', params: [] }, +] + +const BADGE = { + queued: '', loading_model: 'dk-badge--warn', loading_dataset: 'dk-badge--warn', training: 'dk-badge--accent', + saving: 'dk-badge--accent', completed: 'dk-badge--ok', failed: 'dk-badge--error', stopped: '', +} +const EXTRA_HANDLED = ['max_seq_length', 'save_total_limit', 'hf_token', 'eval_strategy', 'eval_steps', 'eval_split', 'eval_dataset_source', 'eval_split_ratio', 'voice', 'val_dataset'] + +function etaText(t, seconds) { + if (!seconds || seconds <= 0) return '' + const h = Math.floor(seconds / 3600) + const m = Math.floor((seconds % 3600) / 60) + if (h > 0) return t('run.eta', { time: `${h}h ${m}m` }) + if (m > 0) return t('run.eta', { time: `${m} min` }) + return t('run.etaSeconds', { seconds: Math.max(1, Math.floor(seconds)) }) +} + +const kindOf = (job) => `${job.training_type || 'lora'}, ${job.training_method || 'sft'}` +const startedAt = (job) => { + const d = new Date(job.created_at) + return Number.isNaN(d.getTime()) ? (job.created_at || '') : d.toLocaleString([], { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) +} + +// `shared` is the shared job views: { tools, readSse, ToolSteps, Checks, StageLine, +// JobLog, ToolOff }. Returns the page and, for tests, the parts it is made of. +export function createFineTune(ctx, api, shared) { + const { h, Fragment, t, Icon, ui, viewer } = ctx + const { useState, useEffect, useRef, useCallback, useMemo } = ctx.React + const { FT_STAGES, TERMINAL, appendLog, blocked, fineTuneChecks, logLines, looksLikeMemoryFailure, machineFacts, warned } = shared.tools + const { ToolSteps, Checks, StageLine, JobLog, ToolOff, readSse } = shared + const { LossChart } = createLossChart(ctx) + + function StatusBadge({ status }) { + return {ctx.tOr(`fineTune.status.${status}`, String(status || '').replace(/_/g, ' '))} + } + + function Field({ id, label, hint, className = '', children }) { + return ( +
+ + {children} + {hint &&

{hint}

} +
+ ) + } + + function Select({ id, value, onChange, children }) { + return ( + + + + ) + } + + function Section({ done, title, children }) { + return ( +
+

+ + {title} +

+
{children}
+
+ ) + } + + function KeyValueEditor({ entries, onChange }) { + const update = (i, field, val) => onChange(entries.map((e, idx) => (idx === i ? { ...e, [field]: val } : e))) + return ( +
+ {entries.map((entry, i) => ( +
+ update(i, 'key', e.target.value)} placeholder={t('form.key')} aria-label={`Extra option ${i + 1} key`} /> + update(i, 'value', e.target.value)} placeholder={t('form.value')} aria-label={`Extra option ${i + 1} value`} /> + +
+ ))} + +
+ ) + } + + function CopyButton({ text }) { + const [copied, setCopied] = useState(false) + const copy = (e) => { + e.stopPropagation() + navigator.clipboard?.writeText(text).then(() => { setCopied(true); setTimeout(() => setCopied(false), 1500) }).catch(() => {}) + } + return ( + + ) + } + + // The foot of the stop layer. The host draws the dialog around it. + function StopDialogBody({ onChoose, close }) { + return ( +
+ + + +
+ ) + } + + // The foot of the delete layer. + function DeleteDialogBody({ onConfirm, close }) { + return ( +
+ + +
+ ) + } + + // A running, failed or finished job: progress, the loss chart, the log, the + // checkpoints and what to do with the result. The server streams progress + // events while the job is active; a finished job is read from the job record. + // `seed` starts the view with an event and checkpoints already read; tests use + // it to draw a state the stream would reach. + function JobView({ job, onStop, onReuse, onResume, onTerminal, exportCheckpoint, seed }) { + const isAdmin = viewer.isAdmin() + const [events, setEvents] = useState([]) + const [latest, setLatest] = useState(seed?.latest ?? null) + const [log, setLog] = useState([]) + const [connecting, setConnecting] = useState(!seed) + const [checkpoints, setCheckpoints] = useState(seed?.checkpoints || []) + const previousRef = useRef(null) + const terminalRef = useRef(onTerminal) + terminalRef.current = onTerminal + + useEffect(() => { + previousRef.current = null + setEvents([]) + setLatest(null) + setLog([]) + if (!job || !ACTIVE.includes(job.status)) { setConnecting(false); return undefined } + setConnecting(true) + // The progress stream is a fetch read with readSse: it carries the + // session and the base path of sdk.core.fetch. Unmounting aborts it. + const controller = new AbortController() + const onEvent = (data) => { + setConnecting(false) + setLatest(data) + setLog(prev => appendLog(prev, logLines(data, previousRef.current))) + previousRef.current = data + if (data.loss > 0 || data.eval_loss > 0 || data.learning_rate > 0 || data.grad_norm > 0) { + setEvents(prev => { + const last = prev[prev.length - 1] + return last && last.current_step === data.current_step ? [...prev.slice(0, -1), data] : [...prev, data] + }) + } + if (TERMINAL.includes(data.status)) { controller.abort(); terminalRef.current?.() } + } + ;(async () => { + try { + const response = await api.fineTune.progress(job.id, controller.signal) + if (response.ok) await readSse(response, onEvent, controller.signal) + } catch (_) { /* the stream ended; the job list poll shows the state */ } + if (!controller.signal.aborted) setConnecting(false) + })() + return () => controller.abort() + // eslint-disable-next-line react-hooks/exhaustive-deps -- keyed on the job id: the effect restarts per job, not per poll refresh + }, [job?.id]) + + const status = latest?.status || job.status + const active = ACTIVE.includes(status) + const failed = status === 'failed' + // A running job can read 0% before it has reported anything. That 0 is + // unknown, not a measurement, so the page draws no figure for it. + const progress = percentOf(latest ? { ...latest, status } : status === 'completed' ? { status } : null) + const known = progress != null + + useEffect(() => { + if (!job || active) return + api.fineTune.listCheckpoints(job.id).then(r => setCheckpoints(r.checkpoints || [])).catch(() => {}) + // eslint-disable-next-line react-hooks/exhaustive-deps -- keyed on the job id: the effect restarts per job, not per poll refresh + }, [job?.id, active]) + + const message = latest?.message || job.message || '' + const lastCheckpoint = checkpoints.length ? checkpoints[checkpoints.length - 1] : null + const [fixes, setFixes] = useState({ batch: true, checkpointing: true }) + + const stageLabels = Object.fromEntries(FT_STAGES.map(s => [s, t(`fineTune.status.${s}`)])) + const openStop = () => ctx.openDialog({ + title: t('stop.title'), + description: t('stop.text'), + render: ({ close }) => onStop(job.id, keep)} />, + }) + + return ( +
+ {failed && ( +
+ +
+

+ {latest?.current_step > 0 + ? t('fineTune.failedAt', { step: latest.current_step, total: latest.total_steps }) + : t('fineTune.failed')} +

+

{message || t('fineTune.noMessage')}

+
+
+ )} + +
+
+
+

{job.model}

+

+ {t('fineTune.jobMeta', { backend: job.backend || FALLBACK_BACKEND, method: job.training_method || 'sft', kind: job.training_type || 'lora', started: startedAt(job) })} + · {job.id?.slice(0, 8)} +

+
+
+ + {active && ( + + )} +
+
+ + {connecting && !latest && active ? ( +

{t('run.connecting')}

+ ) : ( + <> + {(latest || status === 'completed') && ( +
+ {known + ? {progress.toFixed(0)}% + : {t('run.unknownPercent')}} + {latest && latest.total_steps > 0 && {t('run.step')} {latest.current_step} {t('run.of')} {latest.total_steps}} + {latest && latest.total_epochs > 0 && {t('run.epoch')} {latest.current_epoch?.toFixed(1)} {t('run.of')} {latest.total_epochs?.toFixed(0)}} + {active && etaText(t, latest?.eta_seconds) && {etaText(t, latest.eta_seconds)}} + {latest?.extra_metrics?.tokens_per_second > 0 && {latest.extra_metrics.tokens_per_second.toFixed(0)} {t('run.tokens')}} +
+ )} + {(active || latest) && ( +
+ +
+ )} + {active && ( + + )} + {!failed && message &&

{message}

} + {status === 'completed' && ( +

+ {events.length && events[events.length - 1].loss > 0 + ? t('fineTune.finishedLoss', { loss: events[events.length - 1].loss.toFixed(4) }) + : t('fineTune.finished')} +

+ )} + {status === 'stopped' &&

{t('fineTune.stopped')}

} + + )} +
+ + {failed && ( +
+

{t('fineTune.tryTitle')}

+ {looksLikeMemoryFailure(message) ? ( + <> +

{t('fineTune.tryText')}

+ + + + ) : ( +

{t('fineTune.tryPlain')}

+ )} +
+ + {lastCheckpoint && ( + + )} +
+
+ )} + + {(active || events.length > 0) && } + {(active || log.length > 0) && } + + {!active && checkpoints.length > 0 && ( +
+

{t('checkpoints.title')}

+
+ + + + + {checkpoints.map(cp => ( + + + + + + + + ))} + +
{t('checkpoints.title')}
{t('checkpoints.step')}{t('checkpoints.epoch')}{t('checkpoints.loss')}{t('checkpoints.path')}{t('checkpoints.actions')}
{cp.step}{cp.created_at}{cp.epoch?.toFixed(2)}{cp.loss?.toFixed(4)}{cp.path} +
+ + {isAdmin && ( + + )} +
+
+
+
+ )} +
+ ) + } + + // What to do with a finished job: export it as a model, then use it. + function ExportForm({ job, prefilledCheckpoint, seed }) { + const [checkpoints, setCheckpoints] = useState(seed?.checkpoints || []) + const [exportFormat, setExportFormat] = useState('lora') + const [quantMethod, setQuantMethod] = useState('q4_k_m') + const [modelName, setModelName] = useState('') + const [selectedCheckpoint, setSelectedCheckpoint] = useState('') + const [exporting, setExporting] = useState(false) + const [message, setMessage] = useState('') + const [exportedModelName, setExportedModelName] = useState(job?.export_status === 'completed' ? job.export_model_name || '' : '') + const [exportFailed, setExportFailed] = useState(false) + const pollRef = useRef(null) + + useEffect(() => { + if (!job) return + api.fineTune.listCheckpoints(job.id).then(r => setCheckpoints(r.checkpoints || [])).catch(() => {}) + // eslint-disable-next-line react-hooks/exhaustive-deps -- keyed on the job id: the effect restarts per job, not per poll refresh + }, [job?.id]) + + useEffect(() => { if (prefilledCheckpoint) setSelectedCheckpoint(prefilledCheckpoint.path || '') }, [prefilledCheckpoint]) + + // Sync export state from the job record (first load, list refresh). + useEffect(() => { + if (!job) return + if (job.export_status === 'exporting') { + setExporting(true); setExportFailed(false) + setMessage(job.export_message || t('export.working')) + } else if (job.export_status === 'completed' && job.export_model_name) { + setExporting(false); setExportFailed(false) + setExportedModelName(job.export_model_name) + setMessage(t('export.done', { name: job.export_model_name })) + } else if (job.export_status === 'failed') { + setExporting(false); setExportFailed(true) + setMessage(t('export.failed', { message: job.export_message || t('export.unknown') })) + } + // eslint-disable-next-line react-hooks/exhaustive-deps -- keyed on the job id: the effect restarts per job, not per poll refresh + }, [job?.export_status, job?.export_model_name, job?.export_message]) + + useEffect(() => { + if (!exporting || !job) return undefined + pollRef.current = setInterval(async () => { + try { + const updated = await api.fineTune.getJob(job.id) + if (updated.export_status === 'completed') { + setExporting(false); setExportFailed(false) + const name = updated.export_model_name || modelName || 'exported model' + setExportedModelName(name) + setMessage(t('export.done', { name })) + clearInterval(pollRef.current) + } else if (updated.export_status === 'failed') { + setExporting(false); setExportFailed(true) + setMessage(t('export.failed', { message: updated.export_message || t('export.unknown') })) + clearInterval(pollRef.current) + } else if (updated.export_status === 'exporting' && updated.export_message) { + setMessage(updated.export_message) + } + } catch (_) { /* try again at the next tick */ } + }, 3000) + return () => clearInterval(pollRef.current) + // eslint-disable-next-line react-hooks/exhaustive-deps -- keyed on the job id: the effect restarts per job, not per poll refresh + }, [exporting, job?.id]) + + const handleExport = async () => { + setExporting(true); setExportFailed(false) + setMessage(t('export.working')) + setExportedModelName('') + try { + await api.fineTune.exportModel(job.id, { + name: modelName || undefined, + checkpoint_path: selectedCheckpoint || job.output_dir, + export_format: exportFormat, + quantization_method: exportFormat === 'gguf' ? quantMethod : '', + model: job.model, + }) + } catch (e) { + setMessage(t('export.failed', { message: e.message })) + setExportFailed(true); setExporting(false) + } + } + + if (!job || !TERMINAL.includes(job.status)) return null + + return ( + <> + {exportedModelName && !exportFailed && ( +
+ +
+

{t('export.nextTitle', { name: exportedModelName })}

+

{t('export.nextText')}

+ +
+
+ )} +
+

{t('export.title')}

+

{t('export.lede')}

+
+ {checkpoints.length > 0 && ( + + + + )} +
+ + + + {exportFormat === 'gguf' && ( + + setQuantMethod(e.target.value)} placeholder="q4_k_m, bf16, f32" /> + {QUANT_PRESETS.map(q => + + )} +
+ + setModelName(e.target.value)} placeholder="my-finetuned-model" /> + +
+ +
+ {message && (!exportedModelName || exportFailed) && ( +

+ {exporting && } {message} +

+ )} +
+
+ + ) + } + + // The wrapper of the export controls. Registering a model is for + // administrators; anyone else sees why the controls are missing. + function ExportPanel({ job, prefilledCheckpoint, seed }) { + if (!job || !TERMINAL.includes(job.status)) return null + if (!viewer.isAdmin()) { + return ( +
+

{t('export.adminOnly.title')}

+

{t('export.adminOnly.text')}

+
+ ) + } + return + } + + // The machine facts for the checks: the one request useResources makes, as + // a poll that waits while the tab is hidden and catches up on return. + function useResources(interval) { + const [resources, setResources] = useState(null) + useEffect(() => { + let off = false + const read = async () => { + if (typeof document !== 'undefined' && document.hidden) return + try { + const data = await api.resources.get() + if (!off) setResources(data) + } catch (_) { /* the next tick tries again */ } + } + read() + const timer = setInterval(read, interval) + document.addEventListener('visibilitychange', read) + return () => { off = true; clearInterval(timer); document.removeEventListener('visibilitychange', read) } + }, [interval]) + return resources + } + + function FineTuneBody({ path }) { + const isAdmin = viewer.isAdmin() + const resources = useResources(10000) + const facts = useMemo(() => machineFacts(resources), [resources]) + + const [jobs, setJobs] = useState([]) + const [jobsLoaded, setJobsLoaded] = useState(false) + const [selectedJob, setSelectedJob] = useState(null) + const [loading, setLoading] = useState(false) + const [error, setError] = useState('') + const [backends, setBackends] = useState(null) + const [exportCheckpoint, setExportCheckpoint] = useState(null) + const initialConfigRef = useRef(null) + + // Form state + // A deep link names the base model: ?model= or the first path segment. + const [model, setModel] = useState(() => ctx.modelFrom(path) || '') + const [backend, setBackend] = useState(FALLBACK_BACKEND) + const [trainingMethod, setTrainingMethod] = useState('sft') + const [trainingType, setTrainingType] = useState('lora') + const [datasetSource, setDatasetSource] = useState('') + const [datasetFile, setDatasetFile] = useState(null) + const [datasetSplit, setDatasetSplit] = useState('') + const [numEpochs, setNumEpochs] = useState(3) + const [batchSize, setBatchSize] = useState(2) + const [learningRate, setLearningRate] = useState(0.0002) + const [learningRateText, setLearningRateText] = useState('0.0002') + const [adapterRank, setAdapterRank] = useState(16) + const [adapterAlpha, setAdapterAlpha] = useState(16) + const [adapterDropout, setAdapterDropout] = useState(0) + const [targetModules, setTargetModules] = useState('') + const [gradAccum, setGradAccum] = useState(4) + const [warmupSteps, setWarmupSteps] = useState(5) + const [maxSteps, setMaxSteps] = useState(0) + const [saveSteps, setSaveSteps] = useState(500) + const [weightDecay, setWeightDecay] = useState(0) + const [maxSeqLength, setMaxSeqLength] = useState(2048) + const [optimizer, setOptimizer] = useState('adamw_torch') + const [gradCheckpointing, setGradCheckpointing] = useState(false) + const [seed, setSeed] = useState(0) + const [mixedPrecision, setMixedPrecision] = useState('') + const [extraOptions, setExtraOptions] = useState([]) + const [liquidAudioVoice, setLiquidAudioVoice] = useState('') + const [liquidAudioValDataset, setLiquidAudioValDataset] = useState('') + const [hfToken, setHfToken] = useState('') + const [showAdvanced, setShowAdvanced] = useState(false) + const [resumeFromCheckpoint, setResumeFromCheckpoint] = useState('') + const [saveTotalLimit, setSaveTotalLimit] = useState(0) + const [evalEnabled, setEvalEnabled] = useState(false) + const [evalStrategy, setEvalStrategy] = useState('steps') + const [evalSteps, setEvalSteps] = useState(0) + const [evalSplit, setEvalSplit] = useState('') + const [evalDatasetSource, setEvalDatasetSource] = useState('') + const [evalSplitRatio, setEvalSplitRatio] = useState(0.1) + const [rewardFunctions, setRewardFunctions] = useState([]) + const [showAddCustomReward, setShowAddCustomReward] = useState(false) + const [customRewardName, setCustomRewardName] = useState('') + const [customRewardCode, setCustomRewardCode] = useState('') + + const loadJobs = useCallback(async () => { + try { + const data = await api.fineTune.listJobs() + const list = data || [] + setJobs(list) + setSelectedJob(prev => { + if (!prev) return prev + const fresh = list.find(j => j.id === prev.id) + return fresh ? { ...prev, ...fresh } : prev + }) + } catch (_) { /* the next poll tries again */ } finally { + setJobsLoaded(true) + } + }, []) + + useEffect(() => { + loadJobs() + const interval = setInterval(loadJobs, 10000) + return () => clearInterval(interval) + }, [loadJobs]) + + useEffect(() => { + api.fineTune.listBackends() + .then(data => { + const list = Array.isArray(data) ? data : [] + setBackends(list) + if (list.length > 0) setBackend(prev => (list.some(b => b.name === prev) ? prev : list[0].name)) + }) + .catch(() => setBackends(null)) + }, []) + + const backendNames = backends && backends.length > 0 ? backends.map(b => b.name) : [FALLBACK_BACKEND] + const isAdapter = ADAPTER_KINDS.includes(trainingType) + + const handleSubmit = async (e) => { + e.preventDefault() + setLoading(true) + setError('') + try { + let dsSource = datasetSource + if (datasetFile) { + const result = await api.fineTune.uploadDataset(datasetFile) + dsSource = result.path + } + const extra = {} + if (maxSeqLength) extra.max_seq_length = String(maxSeqLength) + if (hfToken.trim()) extra.hf_token = hfToken.trim() + if (saveTotalLimit > 0) extra.save_total_limit = String(saveTotalLimit) + if (evalEnabled) { + extra.eval_strategy = evalStrategy || 'steps' + if (evalSteps > 0) extra.eval_steps = String(evalSteps) + if (evalSplit.trim()) extra.eval_split = evalSplit.trim() + if (evalDatasetSource.trim()) extra.eval_dataset_source = evalDatasetSource.trim() + if (evalSplitRatio > 0 && evalSplitRatio !== 0.1) extra.eval_split_ratio = String(evalSplitRatio) + } else { + extra.eval_strategy = 'no' + } + for (const { key, value } of extraOptions) if (key.trim()) extra[key.trim()] = value + // The Python backend reads `voice` and `val_dataset` from extra_options. + if (backend === 'liquid-audio') { + if (liquidAudioVoice) extra.voice = liquidAudioVoice + if (liquidAudioValDataset.trim()) extra.val_dataset = liquidAudioValDataset.trim() + } + + const req = { + model, + backend, + training_method: trainingMethod, + training_type: trainingType, + dataset_source: dsSource, + dataset_split: datasetSplit || undefined, + num_epochs: numEpochs, + batch_size: batchSize, + learning_rate: learningRate, + adapter_rank: isAdapter ? adapterRank : 0, + adapter_alpha: isAdapter ? adapterAlpha : 0, + adapter_dropout: isAdapter && adapterDropout > 0 ? adapterDropout : undefined, + target_modules: isAdapter && targetModules.trim() ? targetModules.split(',').map(s => s.trim()) : undefined, + gradient_accumulation_steps: gradAccum, + warmup_steps: warmupSteps, + max_steps: maxSteps > 0 ? maxSteps : undefined, + save_steps: saveSteps > 0 ? saveSteps : undefined, + weight_decay: weightDecay > 0 ? weightDecay : undefined, + gradient_checkpointing: gradCheckpointing, + optimizer, + seed: seed > 0 ? seed : undefined, + mixed_precision: mixedPrecision || undefined, + resume_from_checkpoint: resumeFromCheckpoint || undefined, + extra_options: Object.keys(extra).length > 0 ? extra : undefined, + reward_functions: trainingMethod === 'grpo' && rewardFunctions.length > 0 ? rewardFunctions : undefined, + } + + const resp = await api.fineTune.startJob(req) + setResumeFromCheckpoint('') + // Job submitted: rebaseline so leaving the page no longer warns. + initialConfigRef.current = JSON.stringify(getFormConfig()) + await loadJobs() + setSelectedJob({ ...req, id: resp.id, status: 'queued', created_at: new Date().toISOString() }) + } catch (err) { + setError(err.message) + } + setLoading(false) + } + + const handleStop = async (jobId, saveCheckpoint = true) => { + try { + await api.fineTune.stopJob(jobId, saveCheckpoint) + await loadJobs() + } catch (err) { + setError(err.message) + } + } + + const confirmDelete = async (jobId) => { + try { + await api.fineTune.deleteJob(jobId) + if (selectedJob?.id === jobId) setSelectedJob(null) + await loadJobs() + } catch (err) { + setError(err.message) + } + } + + const askDelete = (jobId) => ctx.openDialog({ + title: t('jobs.deleteDialogTitle'), + description: t('jobs.deleteDialogText'), + render: ({ close }) => confirmDelete(jobId)} />, + }) + + const getFormConfig = () => { + const extra = {} + for (const { key, value } of extraOptions) if (key.trim()) extra[key.trim()] = value + if (backend === 'liquid-audio') { + if (liquidAudioVoice) extra.voice = liquidAudioVoice + if (liquidAudioValDataset.trim()) extra.val_dataset = liquidAudioValDataset.trim() + } + return { + model, backend, + training_method: trainingMethod, + training_type: trainingType, + adapter_rank: adapterRank, + adapter_alpha: adapterAlpha, + adapter_dropout: adapterDropout, + target_modules: targetModules.trim() ? targetModules.split(',').map(s => s.trim()) : [], + dataset_source: datasetSource, + dataset_split: datasetSplit, + num_epochs: numEpochs, + batch_size: batchSize, + learning_rate: learningRate, + gradient_accumulation_steps: gradAccum, + warmup_steps: warmupSteps, + max_steps: maxSteps, + save_steps: saveSteps, + weight_decay: weightDecay, + gradient_checkpointing: gradCheckpointing, + optimizer, seed, + mixed_precision: mixedPrecision, + max_seq_length: maxSeqLength, + eval_strategy: evalEnabled ? (evalStrategy || 'steps') : 'no', + eval_steps: evalSteps, + eval_split: evalSplit, + eval_dataset_source: evalDatasetSource, + eval_split_ratio: evalSplitRatio, + extra_options: Object.keys(extra).length > 0 ? extra : {}, + reward_functions: rewardFunctions.length > 0 ? rewardFunctions : undefined, + } + } + + const applyFormConfig = (config) => { + if (config.model != null) setModel(config.model) + if (config.backend != null) setBackend(config.backend) + if (config.training_method != null) setTrainingMethod(config.training_method) + if (config.training_type != null) setTrainingType(config.training_type) + if (config.adapter_rank != null) setAdapterRank(Number(config.adapter_rank)) + if (config.adapter_alpha != null) setAdapterAlpha(Number(config.adapter_alpha)) + if (config.adapter_dropout != null) setAdapterDropout(Number(config.adapter_dropout)) + if (config.target_modules != null) { + setTargetModules(Array.isArray(config.target_modules) ? config.target_modules.join(', ') : String(config.target_modules)) + } + if (config.dataset_source != null) setDatasetSource(config.dataset_source) + if (config.dataset_split != null) setDatasetSplit(config.dataset_split) + if (config.num_epochs != null) setNumEpochs(Number(config.num_epochs)) + if (config.batch_size != null) setBatchSize(Number(config.batch_size)) + if (config.learning_rate != null) { setLearningRate(Number(config.learning_rate)); setLearningRateText(String(config.learning_rate)) } + if (config.gradient_accumulation_steps != null) setGradAccum(Number(config.gradient_accumulation_steps)) + if (config.warmup_steps != null) setWarmupSteps(Number(config.warmup_steps)) + if (config.max_steps != null) setMaxSteps(Number(config.max_steps)) + if (config.save_steps != null) setSaveSteps(Number(config.save_steps)) + if (config.weight_decay != null) setWeightDecay(Number(config.weight_decay)) + if (config.gradient_checkpointing != null) setGradCheckpointing(Boolean(config.gradient_checkpointing)) + if (config.optimizer != null) setOptimizer(config.optimizer) + if (config.seed != null) setSeed(Number(config.seed)) + if (config.mixed_precision != null) setMixedPrecision(config.mixed_precision) + + // max_seq_length is a top-level field or sits inside extra_options. + if (config.max_seq_length != null) { + setMaxSeqLength(Number(config.max_seq_length)) + } else if (config.extra_options?.max_seq_length != null) { + setMaxSeqLength(Number(config.extra_options.max_seq_length)) + } + + // Eval options: the strategy tells whether evaluation was on. + const restoreEval = (strategy, steps, split, src, ratio) => { + if (strategy != null && strategy !== 'no') { + setEvalEnabled(true) + setEvalStrategy(strategy) + } else if (strategy === 'no') { + setEvalEnabled(false) + } + if (steps != null) setEvalSteps(Number(steps)) + if (split != null) setEvalSplit(split) + if (src != null) setEvalDatasetSource(src) + if (ratio != null) setEvalSplitRatio(Number(ratio)) + } + restoreEval(config.eval_strategy, config.eval_steps, config.eval_split, config.eval_dataset_source, config.eval_split_ratio) + // extra_options overrides the top level when it carries them. + const eo = config.extra_options + if (eo) restoreEval(eo.eval_strategy, eo.eval_steps, eo.eval_split, eo.eval_dataset_source, eo.eval_split_ratio) + if (eo?.save_total_limit != null) setSaveTotalLimit(Number(eo.save_total_limit)) + + // liquid-audio extras; they are also kept out of the free-form list below. + if (eo?.voice != null) setLiquidAudioVoice(String(eo.voice)) + if (eo?.val_dataset != null) setLiquidAudioValDataset(String(eo.val_dataset)) + + if (config.extra_options && typeof config.extra_options === 'object') { + setExtraOptions(Object.entries(config.extra_options) + .filter(([k]) => !EXTRA_HANDLED.includes(k)) + .map(([key, value]) => ({ key, value: String(value) }))) + } + setRewardFunctions(Array.isArray(config.reward_functions) ? config.reward_functions : []) + } + + const handleExportConfig = () => { + const url = URL.createObjectURL(new Blob([JSON.stringify(getFormConfig(), null, 2)], { type: 'application/json' })) + const a = document.createElement('a') + a.href = url + a.download = 'finetune-config.json' + document.body.appendChild(a) + a.click() + document.body.removeChild(a) + URL.revokeObjectURL(url) + } + + const handleImportConfig = () => { + const input = document.createElement('input') + input.type = 'file' + input.accept = '.json' + input.onchange = (e) => { + const file = e.target.files[0] + if (!file) return + const reader = new FileReader() + reader.onload = (ev) => { + try { + applyFormConfig(JSON.parse(ev.target.result)) + setSelectedJob(null) + setError('') + } catch { + setError(t('fineTune.badConfig')) + } + } + reader.readAsText(file) + } + input.click() + } + + // Put a job's setup back in the form. `fixes` are the changes ticked on a + // memory failure: they are applied to this copy and nothing starts until the + // person presses Start. + const handleUseConfig = (job, fixes) => { + applyFormConfig(job.config || job) + setResumeFromCheckpoint('') + if (fixes?.batch) setBatchSize(1) + if (fixes?.checkpointing) setGradCheckpointing(true) + setSelectedJob(null) + } + + const handleResumeFromCheckpoint = (checkpoint) => { + if (!selectedJob) return + applyFormConfig(selectedJob.config || selectedJob) + setResumeFromCheckpoint(checkpoint.path) + setShowAdvanced(true) + setSelectedJob(null) + } + + // Baseline for the unsaved-changes guard: lazily taken on first render. + if (initialConfigRef.current === null) initialConfigRef.current = JSON.stringify(getFormConfig()) + const dirty = JSON.stringify(getFormConfig()) !== initialConfigRef.current + // Unsaved edits: the shell asks before it replaces or leaves the page. + const unsaved = dirty && !selectedJob && !loading + useEffect(() => { ctx.markDirty(unsaved) }, [unsaved]) + useEffect(() => () => ctx.markDirty(false), []) + + const checks = useMemo(() => fineTuneChecks({ + form: { + model, datasetSource, datasetFileName: datasetFile?.name || '', backend, batchSize, maxSeqLength, gradAccum, + gradCheckpointing, trainingType, trainingMethod, rewardCount: rewardFunctions.length, + resumeFrom: resumeFromCheckpoint, hfToken: hfToken.trim(), + }, + facts, + backends: { fineTune: backends }, + }), [model, datasetSource, datasetFile, backend, batchSize, maxSeqLength, gradAccum, gradCheckpointing, trainingType, trainingMethod, rewardFunctions.length, resumeFromCheckpoint, hfToken, facts, backends]) + const stop = blocked(checks) + const warnings = warned(checks) + const firstMissing = checks.find(c => c.tone === 'fail') + + const steps = [ + { key: 'setup', label: t('steps.setup') }, + { key: 'check', label: t('steps.check') }, + { key: 'run', label: t('steps.run') }, + { key: 'result', label: t('steps.result') }, + ] + let current = stop ? 0 : 1 + let failed = false + if (selectedJob) { + if (selectedJob.status === 'failed') { current = 2; failed = true } else if (TERMINAL.includes(selectedJob.status)) current = 3 + else current = 2 + } + + const kindWords = trainingType === 'full' ? t('form.summaryFull') : t(`form.summary_${trainingType}`) + const modelShort = model.split('/').pop() || model + + return ( +
+ {t('fineTune.title')} {t('landing.experimental')}} + supporting={t('fineTune.lede')} + actions={( +
+ {selectedJob && ( + + )} + +
+ )} + /> + + + {error && ( +
+ +

{error}

+
+ )} + + {selectedJob ? ( + <> + + + + ) : ( +
+ {resumeFromCheckpoint && ( +
+ + {t('fineTune.resuming')} {resumeFromCheckpoint} + +
+ )} + +
+ + setModel(e.target.value)} placeholder="TinyLlama/TinyLlama-1.1B-Chat-v1.0" required /> + +
+ +
+
+ + setDatasetSource(e.target.value)} placeholder="tatsu-lab/alpaca" /> + + + setDatasetSplit(e.target.value)} placeholder="train" /> + +
+
+ setDatasetFile(e.target.files[0] || null)} accept=".json,.jsonl,.csv" /> + + {datasetFile ? ( + + {datasetFile.name} + + + ) : ( + {t('form.uploadHint')} + )} +
+
+ +
+
+ {t('form.kind')} + {[['adapter', isAdapter], ['full', trainingType === 'full']].map(([id, on]) => ( + + ))} +
+
+ + setNumEpochs(Number(e.target.value))} min={1} /> + + + setBatchSize(Number(e.target.value))} min={1} /> + + + { + setLearningRateText(e.target.value) + const parsed = Number(e.target.value) + if (!isNaN(parsed) && parsed > 0) setLearningRate(parsed) + }} + placeholder="5e-5" + /> + +
+ +
+ + {showAdvanced && ( +
+
+ + + + + + + {isAdapter && ( + + + + )} + {isAdapter && ( + <> + + setAdapterRank(Number(e.target.value))} min={1} /> + + + setAdapterAlpha(Number(e.target.value))} min={1} /> + + + setAdapterDropout(Number(e.target.value))} min={0} max={1} step={0.05} /> + + + )} + + setGradAccum(Number(e.target.value))} min={1} /> + + + setWarmupSteps(Number(e.target.value))} min={0} /> + + + setMaxSeqLength(Number(e.target.value))} min={64} /> + + + + + + setMaxSteps(Number(e.target.value))} min={0} /> + + + setSaveSteps(Number(e.target.value))} min={0} /> + + + setSaveTotalLimit(Number(e.target.value))} min={0} /> + + + setWeightDecay(Number(e.target.value))} min={0} step={0.01} /> + + + setSeed(Number(e.target.value))} min={0} /> + + + + +
+ {isAdapter && ( + + setTargetModules(e.target.value)} placeholder="q_proj, v_proj, k_proj, o_proj" /> + + )} + + + setHfToken(e.target.value)} placeholder="hf_..." autoComplete="off" /> + + +
+
+
+ {evalEnabled && ( +
+ + + + + setEvalSteps(Number(e.target.value))} min={0} /> + + + setEvalSplit(e.target.value)} placeholder="validation" /> + + + setEvalDatasetSource(e.target.value)} placeholder={t('form.evalDatasetPlaceholder')} /> + + + setEvalSplitRatio(Number(e.target.value))} min={0.01} max={0.5} step={0.01} /> + +
+ )} +
+ + {trainingMethod === 'grpo' && ( +
+

{t('form.rewards')}

+

{t('form.rewardsHint')}

+ {BUILTIN_REWARDS.map(builtin => { + const selected = rewardFunctions.find(rf => rf.type === 'builtin' && rf.name === builtin.name) + return ( +
+ + {selected && builtin.params.map(param => ( + + setRewardFunctions(prev => prev.map(rf => (rf.type === 'builtin' && rf.name === builtin.name ? { ...rf, params: { ...(rf.params || {}), [param.key]: e.target.value } } : rf)))} + /> + + ))} +
+ ) + })} + {rewardFunctions.filter(rf => rf.type === 'inline').map((rf, idx) => ( +
+
+ {rf.name} + +
+ 
{rf.code}
+
+ ))} + {showAddCustomReward ? ( +
+ + setCustomRewardName(e.target.value)} placeholder="my_custom_reward" /> + + +