fix: describe baseline sample scans instead of reporting them as failed

SECURITY.md

@@ -471,7 +471,7 @@ or of the platform is still locked, whichever unit `--tenant` selects, and
 reports that count as `deployment_locked`, never a URL. The console
 notification test covers only the unit's own destinations, so a unit's
 administrators learn nothing about another unit's or the platform's. A
-database upgraded to schema 64 must not be opened by an older EdgeWatch
+database upgraded to schema 65 must not be opened by an older EdgeWatch
 binary; downgrade by restoring the complete pre-upgrade `./data` backup
 before starting the old version. The
 daemon and the host commands that write to the database, including `backup`,
@@ -480,7 +480,8 @@ Schema 63 also marks legacy unsent deliveries with at least eight attempts
 and a retry scheduled before v0.22.1 as terminal; deliveries still retrying
 under the newer fifteen-attempt policy remain eligible. Schema 64 adds an
 index for pruning old restore-quarantine records without rewriting delivery
-history.
+history. Schema 65 records how each new scan was compared with its job's
+baseline; existing scans and their results are not changed.
 Only the daemon migrates. The host commands that act on business units or
 accounts (`admin`, `scan`, `status`, `history`, `baseline`, and `notify
 test`) refuse a schema that the daemon has not upgraded yet, such as a

docs/src/content/docs/getting-started/first-scan.md

@@ -40,7 +40,10 @@ still apply to the job.
 ## Establish the baseline
 
 Run the job and inspect its results. Approve a successful scan as the baseline
-once you have verified that it represents the surface you expect.
+once you have verified that it represents the surface you expect. Until the
+job has collected its baseline samples, the scan detail describes each scan as
+a baseline sample instead of a comparison; see
+[Scan comparison](/user-guide/jobs-baselines-incidents/#scan-comparison).
 
 :::note[Incomplete observations do not change expectations]
 Failed, canceled, timed-out, or incomplete scans remain available for

docs/src/content/docs/reference/api-compatibility.md

@@ -38,6 +38,26 @@ are a positive port state or `not-open`, and `key` has the form
 `address`. Clients that handle change kinds individually should treat an
 unknown kind as a generic change.
 
+## Scan comparison states
+
+`GET /api/v1/jobs/{jobID}/scans/{scanID}` and
+`GET /api/v1/jobs/{jobID}/scans/{scanID}/changes` describe how the scan was
+compared with the job's baseline:
+
+| `comparison_state` | `comparison_source` | Meaning |
+| --- | --- | --- |
+| `compared` | `scan_time` | Compared with the baseline that existed when the scan finished. `changes` is the diff stored with the scan, and `baseline_scan_id` names that baseline. |
+| `compared` | `current_baseline_legacy` | A scan recorded before v0.26.0 without a scan-time comparison, compared with the current baseline on each request. |
+| `baseline_sample` | `none` | The job had no baseline when the scan finished, so nothing was compared. |
+| `baseline_established` | `none` | The scan was the sample that completed the baseline; `baseline_scan_id` is the scan's own ID. |
+| `not_compared` | `none` | The scan failed, timed out, or was canceled, or its result was kept without a comparison. |
+
+v0.26.0 adds `baseline_sample` and `baseline_established`. Earlier releases
+reported those scans as `not_compared` while the job had no baseline, and
+compared them with the current baseline once it had one. The scan objects of
+the scan and job scan endpoints also carry the recorded `comparison`; it is
+omitted for scans recorded before v0.26.0.
+
 ## Business units
 
 v0.20.0 adds business units to every installation. The routes and response

docs/src/content/docs/reference/database-compatibility.md

@@ -5,19 +5,21 @@ description: Understand the current SQLite schema, forward-only migrations, and
 
 ## Current schema and rollback
 
-The current schema is version 64. Schema 31 records terminal notification
+The current schema is version 65. Schema 31 records terminal notification
 deliveries and marks rows that had already exhausted the original eight
 attempts. Schema 63 repairs databases that had already passed schema 31 by
 marking still-unsent rows with at least eight attempts and a scheduled retry
 before v0.22.1, when the retry budget increased to fifteen. Retries scheduled
 on or after that release remain eligible, so upgrading does not replay
 deliveries that were still retrying. Schema 64 adds an index for pruning old
-restore-quarantine records; it does not rewrite delivery history. Database
-migrations are forward-only. An older image must not be pointed at a database
-already upgraded by a newer image; restore the matching pre-upgrade `./data`
-backup if a rollback is required. The daemon and the commands that write to the
-database (admin, scan, notify test, baseline approve and reset, and backup)
-refuse versions above their supported schema version with the error
+restore-quarantine records; it does not rewrite delivery history. Schema 65
+records how each new scan was compared with its job's baseline; it does not
+change existing scans. Database migrations are forward-only. An older image
+must not be pointed at a database already upgraded by a newer image; restore
+the matching pre-upgrade `./data` backup if a rollback is required. The daemon
+and the commands that write to the database (admin, scan, notify test,
+baseline approve and reset, and backup) refuse versions above their supported
+schema version with the error
 `database schema version N is newer than supported version M`. Back up such a
 database with the release that upgraded it, or copy `./data` while
 EdgeWatch is stopped. A daemon that finds another daemon's live lease exits
@@ -33,6 +35,23 @@ baseline evidence in restartable batches. Service names and products are
 prioritized so services on late ports remain searchable even when a host has
 many positive ports. The rebuild does not change scan results or baselines.
 
+## Schema 65
+
+Schema 65, introduced in v0.26.0, adds a `comparison` column to the scans
+table. When a scan of a job finishes, it records whether the scan was compared
+with the baseline, was a baseline sample, established the baseline, or was not
+compared. The scan detail reports that outcome, so a successful baseline sample
+is no longer described as a scan that did not complete, and its result no
+longer changes after the baseline is established, an incident is accepted, or
+the baseline is reset; see
+[Scan comparison](/user-guide/jobs-baselines-incidents/#scan-comparison).
+Existing scans keep an empty value, because the migration cannot tell an
+earlier baseline sample from a scan recorded before scan-time comparisons. They
+behave as before: a scan without changes recorded at scan time is compared with
+the current baseline. It is a quick in-place change with no background phase.
+An older release refuses the upgraded database, so a rollback means restoring
+the pre-upgrade `./data` backup.
+
 ## Schema 62
 
 Schema 62, introduced in v0.25.14, adds the work queues that erase a

docs/src/content/docs/user-guide/jobs-baselines-incidents.md

@@ -99,3 +99,30 @@ Host searches cover partial IP addresses, DNS names, targets, job names, and
 service names or products. Enter at least 3 and no more than 256 characters;
 searches stay on the indexed path and service names/products are prioritized
 within the bounded search document.
+
+## Scan comparison
+
+Each scan's detail says how the scan was compared with the job's baseline when
+it finished. That description is recorded with the scan, so it stays the same
+after the baseline is established, an incident is accepted, or the baseline is
+reset.
+
+- **Baseline sample:** the job was still learning its baseline, so there was
+  nothing to compare the scan with. A successful scan counts as one of the
+  job's baseline samples; an incomplete scan does not. Jobs created in the
+  console learn from two samples by default, so their first scan is a baseline
+  sample. Each scope change or baseline reset starts a new set of samples.
+- **Established the baseline:** the scan was the sample that completed the
+  baseline.
+- **Changes recorded at scan time:** the scan was compared with the baseline
+  that existed when it finished. Its changes are kept with it and are not
+  recalculated later.
+- **Not compared because it did not complete successfully:** the scan failed,
+  timed out, or was canceled.
+- **Not compared with the baseline:** the scan completed, but EdgeWatch kept
+  its result without a comparison, for example because the job's security
+  settings changed while it ran.
+
+Scans recorded before v0.26.0 have no recorded comparison. When such a scan has
+no changes recorded at scan time, its detail compares it with the current
+baseline and reports the changes against the current baseline.

internal/app/app.go

@@ -1081,7 +1081,9 @@ func (a *App) runJobWithQueueMarker(ctx context.Context, scope store.TenantScope
 	if destinationErr != nil {
 		// Preserve the completed scan even when notification configuration
 		// cannot be read. Runtime state is deliberately left unchanged,
-		// matching the pre-transaction behavior.
+		// matching the pre-transaction behavior, so the scan is recorded as
+		// not compared.
+		scan.Comparison = model.ScanComparisonNotCompared
 		saveCtx, saveCancel := context.WithTimeout(persistCtx, scanPersistenceWriterWaitTimeout+persistTimeout)
 		defer saveCancel()
 		if saveErr := system.SaveScan(saveCtx, scan); saveErr != nil {

internal/engine/comparison_outcome_test.go

@@ -0,0 +1,99 @@
+package engine
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/crypt0rr/edgewatch/internal/config"
+	"github.com/crypt0rr/edgewatch/internal/model"
+	"github.com/crypt0rr/edgewatch/internal/store"
+	"github.com/crypt0rr/edgewatch/internal/store/storetest"
+)
+
+// Finalizing a managed scan records what the comparison did: the samples
+// that a job learns its baseline from, the sample that completes it, a
+// comparison with an existing baseline, and a scan that did not complete.
+// The recorded outcome does not change when the baseline does.
+func TestFinalizeManagedScanRecordsComparisonOutcome(t *testing.T) {
+	ctx := context.Background()
+	db, err := store.Open(storetest.FreshPath(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	record, err := defaultTenant(db).CreateJob(ctx, config.NormalizeJob(config.Job{
+		Name: "comparison-outcome", Schedule: "0 * * * *", Timezone: "UTC", Targets: []string{"192.0.2.1"},
+		TCP: &config.Protocol{Ports: "80,443", Mode: "connect"}, Timing: "balanced", Timeout: config.Duration(time.Minute),
+		Baseline: config.Baseline{Samples: 2}, Change: config.Change{Confirmations: 1},
+	}))
+	if err != nil {
+		t.Fatal(err)
+	}
+	e := Engine{Store: db}
+	finished := time.Now().UTC().Add(-time.Hour)
+	finalize := func(id, status string, snap model.Snapshot) model.Scan {
+		t.Helper()
+		finished = finished.Add(time.Minute)
+		current := scan(id, snap)
+		current.Status, current.StartedAt, current.FinishedAt = status, finished, finished
+		if status == "failed" {
+			current.Error = "scanner stopped"
+		}
+		current.JobID, current.JobRevision, current.Job = record.ID, record.Revision, record.Job.Name
+		current.ConfigHash = record.Job.SecurityHash()
+		if _, err := e.FinalizeManagedScan(ctx, record.ID, record.Job, &current, nil); err != nil {
+			t.Fatalf("finalize %s: %v", id, err)
+		}
+		stored, err := defaultTenant(db).GetScan(ctx, id)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if stored.Comparison != current.Comparison {
+			t.Fatalf("stored %s comparison = %q, finalized %q", id, stored.Comparison, current.Comparison)
+		}
+		return stored
+	}
+	expect := func(id, comparison, baselineScanID string, changes int) {
+		t.Helper()
+		stored, err := defaultTenant(db).GetScan(ctx, id)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if stored.Comparison != comparison || stored.BaselineScanID != baselineScanID || len(stored.Changes) != changes {
+			t.Fatalf("scan %s = comparison %q, baseline %q, %d changes; want %q, %q, %d", id, stored.Comparison, stored.BaselineScanID, len(stored.Changes), comparison, baselineScanID, changes)
+		}
+	}
+
+	finalize("sample-1", "success", snapshot("open"))
+	expect("sample-1", model.ScanComparisonBaselineSample, "", 0)
+	finalize("sample-2", "success", snapshot("open"))
+	expect("sample-2", model.ScanComparisonBaselineEstablished, "sample-2", 0)
+	expect("sample-1", model.ScanComparisonBaselineSample, "", 0)
+
+	opened := snapshot("open")
+	opened.Units[0].Ports = append(opened.Units[0].Ports, model.PortState{Port: 80, State: "open"})
+	opened.Normalize()
+	finalize("compared", "success", opened)
+	expect("compared", model.ScanComparisonCompared, "sample-2", 1)
+	if stored := finalize("failed", "failed", model.Snapshot{}); stored.Comparison != model.ScanComparisonNotCompared {
+		t.Fatalf("failed scan comparison = %q", stored.Comparison)
+	}
+
+	if _, err := defaultTenant(db).ResetRuntime(ctx, record.ID, record.Job.Name); err != nil {
+		t.Fatal(err)
+	}
+	incomplete := snapshot("open")
+	incomplete.TargetFailures = []model.TargetCoverageFailure{{Target: "192.0.2.1", Reason: "coverage did not complete"}}
+	if stored := finalize("incomplete-sample", "success", incomplete); stored.Status != "incomplete" {
+		t.Fatalf("incomplete sample status = %q", stored.Status)
+	}
+	expect("incomplete-sample", model.ScanComparisonBaselineSample, "", 0)
+	finalize("relearn-1", "success", opened)
+	expect("relearn-1", model.ScanComparisonBaselineSample, "", 0)
+
+	// The reset changed none of the earlier outcomes.
+	expect("sample-1", model.ScanComparisonBaselineSample, "", 0)
+	expect("sample-2", model.ScanComparisonBaselineEstablished, "sample-2", 0)
+	expect("compared", model.ScanComparisonCompared, "sample-2", 1)
+}

internal/engine/engine.go

@@ -64,7 +64,8 @@ func (e *Engine) FinalizeManagedScanWithOptions(ctx context.Context, jobID strin
 			MarkIncompleteScan(current)
 		}
 		if current.Status == "success" || current.Status == "incomplete" {
-			if state.Baseline != nil {
+			hadBaseline := state.Baseline != nil
+			if hadBaseline {
 				current.BaselineScanID = state.BaselineScanID
 				current.BaselineConfigHash = state.BaselineConfigHash
 			}
@@ -86,6 +87,7 @@ func (e *Engine) FinalizeManagedScanWithOptions(ctx context.Context, jobID strin
 				current.BaselineScanID = state.BaselineScanID
 				current.BaselineConfigHash = state.BaselineConfigHash
 			}
+			current.Comparison = comparisonOutcome(hadBaseline, state, current)
 			if current.Resumable && current.CycleAttempt > 1 {
 				return append(events, model.Event{Type: "scan-recovered", Job: scan.Job, ScanID: scan.ID, Message: "Resumable scan cycle completed after earlier paused attempts", CreatedAt: scan.FinishedAt}), nil
 			}
@@ -94,10 +96,28 @@ func (e *Engine) FinalizeManagedScanWithOptions(ctx context.Context, jobID strin
 		// Failed, timed-out, and canceled scans never enter comparison state,
 		// but every terminal non-success outcome is retained as an event so the
 		// configured notification destinations can alert the operator.
+		current.Comparison = model.ScanComparisonNotCompared
 		return processFailure(state, job.Name, *current)
 	})
 }
 
+// comparisonOutcome names what finalizing a successful or incomplete scan
+// did. A scan that finished while the job had a baseline was compared with
+// it, even when the comparison found nothing. Without one, the scan was a
+// baseline sample, or the sample that completed the baseline. Recording the
+// outcome keeps a sample's history from being compared later with a
+// baseline that did not exist when it ran.
+func comparisonOutcome(hadBaseline bool, state *model.JobState, scan *model.Scan) string {
+	switch {
+	case hadBaseline:
+		return model.ScanComparisonCompared
+	case scan.Status == "success" && state.Baseline != nil && state.BaselineScanID == scan.ID:
+		return model.ScanComparisonBaselineEstablished
+	default:
+		return model.ScanComparisonBaselineSample
+	}
+}
+
 func processSuccess(state *model.JobState, job config.Job, scan model.Scan) ([]model.Event, error) {
 	events, _, err := processSuccessWithChanges(state, job, scan)
 	return events, err

internal/engine/engine_test.go

@@ -1586,6 +1586,9 @@ func TestFinalizeManagedScanRecordsInitialBaselineScanMetadata(t *testing.T) {
 	if stored.BaselineScanID != current.ID || stored.BaselineConfigHash != current.ConfigHash {
 		t.Fatalf("initial baseline metadata = %#v, want scan=%s hash=%s", stored, current.ID, current.ConfigHash)
 	}
+	if stored.Comparison != model.ScanComparisonBaselineEstablished {
+		t.Fatalf("initial baseline comparison = %q, want %q", stored.Comparison, model.ScanComparisonBaselineEstablished)
+	}
 	if exists, err := defaultTenant(db).BaselineHostProjectionExists(ctx, record.ID); err != nil || !exists {
 		t.Fatalf("automatic baseline did not maintain host projection: exists=%v err=%v", exists, err)
 	}

internal/model/model.go

@@ -153,6 +153,29 @@ type Snapshot struct {
 	TargetFailures []TargetCoverageFailure `json:"target_failures,omitempty"`
 }
 
+// Scan comparison outcomes. A managed scan records one when it is finalized,
+// so its history keeps the meaning it had when it ran after the baseline is
+// established, changed, or reset.
+const (
+	// ScanComparisonLegacy is the empty value of a scan recorded before the
+	// outcome was stored. Only such a scan may be compared with the current
+	// baseline when it has no scan-time comparison.
+	ScanComparisonLegacy = ""
+	// ScanComparisonCompared marks a scan compared with the baseline that
+	// existed when it finished; its changes are the scan-time diff.
+	ScanComparisonCompared = "compared"
+	// ScanComparisonBaselineSample marks a scan that finished while the job
+	// had no baseline, so there was nothing to compare it with.
+	ScanComparisonBaselineSample = "baseline_sample"
+	// ScanComparisonBaselineEstablished marks the sample that completed the
+	// job's baseline.
+	ScanComparisonBaselineEstablished = "baseline_established"
+	// ScanComparisonNotCompared marks a scan that did not complete
+	// successfully, or whose result was kept without being compared, such as
+	// a result for a security scope that changed while it ran.
+	ScanComparisonNotCompared = "not_compared"
+)
+
 type Scan struct {
 	ID                     string    `json:"id"`
 	JobID                  string    `json:"job_id,omitempty"`
@@ -186,10 +209,13 @@ type Scan struct {
 	// BaselineScanID and BaselineConfigHash identify the comparison used when
 	// this scan was processed. Changes is the immutable scan-time diff; list
 	// endpoints intentionally use ScanSummary instead of loading it.
-	BaselineScanID     string   `json:"baseline_scan_id,omitempty"`
-	BaselineConfigHash string   `json:"baseline_config_hash,omitempty"`
-	Changes            []Change `json:"changes,omitempty"`
-	Snapshot           Snapshot `json:"snapshot"`
+	BaselineScanID     string `json:"baseline_scan_id,omitempty"`
+	BaselineConfigHash string `json:"baseline_config_hash,omitempty"`
+	// Comparison is the ScanComparison outcome recorded at finalization.
+	// It is empty for a scan recorded before the outcome was stored.
+	Comparison string   `json:"comparison,omitempty"`
+	Changes    []Change `json:"changes,omitempty"`
+	Snapshot   Snapshot `json:"snapshot"`
 	// Interrupted marks a canceled scan that stopped because the daemon
 	// stopped, not because someone canceled it. It decides the scan's event
 	// and is not stored.
@@ -229,6 +255,7 @@ type ScanSummary struct {
 	NoProgressTries        int       `json:"no_progress_attempts,omitempty"`
 	BaselineScanID         string    `json:"baseline_scan_id,omitempty"`
 	BaselineConfigHash     string    `json:"baseline_config_hash,omitempty"`
+	Comparison             string    `json:"comparison,omitempty"`
 	// TenantID is the tenant that owns the scan. It is set by the store's
 	// tenant-scoped reads and is never part of an API response.
 	TenantID string `json:"-"`