Feat/p0 usefulness first

docs/ARCHIVE.md

@@ -3,6 +3,7 @@
 These documents are historical — they describe plans that have been executed, reviews whose findings have been addressed, or assessments of a prior state of the codebase. They are retained for context and decision history.
 
 **For current documentation, see:**
+- `strategy/2026-09-24-usefulness-first-strategy.md` — Living forward plan (useful first, unique second)
 - `MAJOR_CHANGES.md` — Running log of substantive changes
 - `DEPLOY.md` — Production deployment guide
 - `ENV.md` — Environment variable reference

docs/formualizer-gaps.md

@@ -1,26 +1,27 @@
 # Formualizer Gaps & Issues for SmartSht
 
 > Compiled from: GitHub issues, source analysis, SmartSht integration behavior, and formualizer.dev docs.
-> SmartSht uses `@ocean8219/formualizer@0.7.2` — latest is `0.8.4`.
+> SmartSht uses `@ocean8219/formualizer@^0.9.3` (aligned with upstream 0.9.3 as of 2026-09-11).
+> Last doc refresh: 2026-09-25.
 
 ---
 
-## 1. Version Gap (0.7.2 → 0.8.4)
+## 1. Version Status (current: 0.9.3)
 
-SmartSht is 3 minor versions behind. The following was fixed/added in 0.7.3–0.8.4 that SmartSht is missing:
+SmartSht is on **0.9.3**. The earlier 0.7.2 → 0.8.4 upgrade path is complete.
 
-- Multiple structural edit undo bugs fixed
-- Dependency graph edge-drop fixes (silent stale values)
-- Database function empty-text handling fix
-- SheetPort GIL deadlock fix (Python, but Rust core changes)
-- FormulaPlane span evaluation improvements
-- Semantic reference consolidation hardening
+Notable 0.8.x–0.9.x gains already in tree:
 
-**Recommendation:** Upgrade to `@ocean8219/formualizer@^0.8.4`
+- Structural edit / dependency-graph hardening from 0.7.3–0.8.4
+- 0.9.0 circular-cell fixed-point retention (large recalc wins for converged cycles)
+- 0.9.2 cache-only XLSX recalc APIs (opt-in; SmartSht does not use these yet)
+- 0.9.3 broader date/time text parsing (partial #290), COUNTIF/COUNTBLANK blank arithmetic (partial #285), bulk-ingest dependency fixes, formula-assignment failure reporting
+
+**Recommendation:** Stay on `^0.9.3` for now. Do not blind-upgrade further until golden-set tests and import-honesty warnings are green in CI. Re-evaluate when upstream closes remaining P0 issues below.
 
 ---
 
-## 2. Known Open Bugs (Affecting SmartSht Import/Eval)
+## 2. Remaining Risks — Open Upstream Issues (Affecting SmartSht Import/Eval)
 
 These are open issues on GitHub that directly impact SmartSht's uploaded worksheet behavior:
 
@@ -89,19 +90,14 @@ Based on the documented categories and what's commonly used in Excel but NOT lis
 
 ## 4. Date/Time Specific Issues
 
-Dates are the most problematic area for SmartSht imports:
+Dates remain a high-risk area for SmartSht imports, with partial relief in 0.9.3:
 
 1. **Serial number interpretation** — Formualizer handles Excel serial dates (1900 system) but:
-   - Issue #312: Date arithmetic (+/-) preserves a Date type Excel doesn't have
-   - Issue #290: Many text-to-date conversions fail that Excel accepts
-   - Issue #291: Date text comparisons in COUNTIF/SUMIF diverge
+   - Issue #312: Date arithmetic (+/-) preserves a Date type Excel doesn't have (**still open**)
+   - Issue #290: Partially addressed in 0.9.3 (#398) for single-digit years, `24:00`, truncated fractional seconds, month-year forms; remaining cases tracked in #416
+   - Issue #291: Date text comparisons in COUNTIF/SUMIF diverge (**still open**)
 
-2. **Missing date text formats** that Excel parses but Formualizer rejects:
-   - Single-digit year: `1/2/5` (January 2, 2005)
-   - 24:00 as midnight
-   - Fractional seconds: `12:30:45.5`
-   - Month-year only: `Jan 2024`
-   - DATEVALUE/TIMEVALUE with non-standard inputs
+2. **Date text forms** — 0.9.3 accepts more Excel-compatible text; still verify DATEVALUE/TIMEVALUE routing and `"24:00"`-as-full-day arithmetic (#416).
 
 3. **Time functions with dates** — When a cell contains a datetime serial like `45488.75`, functions like HOUR/MINUTE/SECOND should extract the fractional part, but the Date type preservation bug (#312) may cause incorrect extraction.
 
@@ -141,14 +137,14 @@ All functions using criteria matching (wildcards, comparisons) are affected by #
 ### P0 — Blocking SmartSht core functionality
 
 1. **Date arithmetic type preservation** (#312) — Causes formula results to be Date objects instead of numbers
-2. **VLOOKUP/HLOOKUP approximate mode sortedness** (#283) — Returns garbage data  
-3. **Date text parsing gaps** (#290) — Common date formats rejected
+2. **VLOOKUP/HLOOKUP approximate mode sortedness** (#283) — Returns garbage data
+3. **Date text parsing residual gaps** (#290 / #416) — Partial fix in 0.9.3; remaining DATEVALUE/TIMEVALUE / full-day `24:00` cases
 4. **Wildcard/criteria matching** (#295) — COUNTIF/SUMIF wrong results and perf issues
 
 ### P1 — Causes incorrect data after user edits
 
 5. **MATCH blank→0 coercion** (#319)
-6. **COUNTBLANK on sparse ranges** (#285)
+6. **COUNTBLANK on sparse ranges** (#285) — Partial arithmetic blank counting in 0.9.3; not all semantics
 7. **Date text in COUNTIF/SUMIF criteria** (#291)
 8. **Row insert invalidation for MATCH/INDEX** (#313)
 
@@ -180,13 +176,17 @@ This means imported worksheets display correctly, but editing an imported formul
 
 ## 9. Upgrade Path
 
+Current pin:
+
 ```bash
-npm install @ocean8219/formualizer@^0.8.4
+# already in package.json
+"@ocean8219/formualizer": "^0.9.3"
 ```
 
-After upgrading, re-test:
-- Date arithmetic formulas
-- VLOOKUP with approximate matching
-- COUNTIF/SUMIF with wildcard criteria
-- Named range references across sheets
-- Row/column insert operations with dependent formulas
+Before any further upgrade:
+
+1. Run `npm run test:realengine` (golden set in `formualizer.realengine.test.ts`)
+2. Re-check import honesty warnings on a styled workbook with formulas
+3. Spot-check: date arithmetic (#312), approx VLOOKUP (#283), COUNTIF wildcards (#295)
+
+Do **not** upgrade past 0.9.3 solely because a newer tag exists — require golden-set green + known-risk review.

docs/planning/README.md

@@ -2,6 +2,9 @@
 
 These documents describe the technical architecture of smartsh!t's internal systems.
 
+> **Living forward plan:** [`docs/strategy/2026-09-24-usefulness-first-strategy.md`](../strategy/2026-09-24-usefulness-first-strategy.md)  
+> Useful first, unique second. P0 complete; formatting sandbox is P1.
+
 ## Core Focus
 
 smartsh!t is a **spreadsheet understanding tool**. The AI and tooling exist to help users

docs/strategy/2026-09-24-usefulness-first-strategy.md

@@ -0,0 +1,133 @@
+# Strategy: Useful First, Unique Second
+
+**Status:** Living — current forward plan  
+**Date:** 2026-09-24 (updated 2026-09-25)  
+**Principle:** Ship a spreadsheet people can trust and finish work in. Differentiation (auditor, privacy, formatting polish) comes after the core job works.
+
+> **Trust code over reviews.** Several docs in `docs/` are historical snapshots. Prefer this file + `package.json` + current `src/` over stale review claims.
+
+---
+
+## 1. North star
+
+**Core job (must work):**
+
+```
+Import → Trust numbers → Understand structure → Ask grounded questions → Safe edit → Format polish
+```
+
+**Not the job (v1):**
+
+- Unbounded autonomous agent (plan → tool → observe → replan forever)
+- Replace Excel / Google Sheets for teams
+- Uniqueness theater (stub NLP, unused ONNX upload, collab before usefulness)
+
+**Agent posture:** Bounded tools with preview/undo. Formatting and tedious styling are in-scope. Open-ended workbook rewriting is not.
+
+---
+
+## 2. P0 Backlog — Useful First
+
+| ID | Item | Status | Notes |
+|----|------|--------|-------|
+| **P0.1** | Formula engine parity | **DONE** | On `@ocean8219/formualizer@^0.9.3`. Gaps doc updated; golden-set realengine tests extended. |
+| **P0.2** | Import honesty | **DONE** | Warn when formulas use Excel cached values / styles appear dropped. Surfaced via import meta → chat/toast. |
+| **P0.3** | Act-path safety | **DONE** | `apply_formula` gap detection + Apply/Reject preview; `confirmGaps` override. |
+| **P0.4** | Activation UI | **DONE** | Overlay waits for audit/grace; never auto-dismisses on critical/high; lists finding titles. |
+| **P0.5** | AI quality loop (foundation) | **DONE** | Thumbs + optional detail on thumbs-down for failover analysis. |
+
+### Implementation paths
+
+| ID | Key files |
+|----|-----------|
+| P0.1 | `docs/formualizer-gaps.md`, `src/engine/formualizer.realengine.test.ts` |
+| P0.2 | `src/io/xlsx.ts`, `src/store/importOrchestration.ts`, `src/components/Toolbar.tsx` |
+| P0.3 | `src/agent/toolHandlers/columnOps.ts`, `src/lib/previewBuilders.ts`, `src/store/slices/chatSlice.ts` |
+| P0.4 | `src/components/ImportInsightsOverlay.tsx` |
+| P0.5 | `src/ai/chatFeedback.ts`, `src/components/ChatPanel.tsx` |
+
+---
+
+## 3. Priority order going forward
+
+### P0 — Useful — **COMPLETE (2026-09-25)**
+
+Gate before uniqueness spend (smoke-test after sync):
+
+1. Import a representative budget (.xlsx)  
+2. Key totals match Excel (or user is warned)  
+3. Insights + critical audit findings visible without chat  
+4. ≥5 grounded Q&A turns without nonsense  
+5. Safe edits with preview / undo  
+6. No marketing claims for stub surfaces  
+
+### P1 — Competitive polish (including formatting sandbox) — **NEXT**
+
+| ID | Work | Outcome |
+|----|------|---------|
+| P1.1 | Extend `format_cells` | Expose `CellFormat`: underline, strikethrough, fontFamily, align, wrap, borders |
+| P1.2 | Layout tools | Agent + sandbox: column width / row height |
+| P1.3 | Parser phrases | NL coverage for borders, auto-fit, row height, fonts |
+| P1.4 | Style recipes | Bounded macros: `header`, `total_row`, `table_polish` (preview once) |
+| P1.5 | Free-tier alignment | Gate auto-fix depth, not first understanding |
+
+**In scope:** cell styles, borders, fonts, sizes, alignment, wrap, row/col sizing, preset recipes.  
+**Out of scope:** Excel group/outline hierarchy (unless UI supports it), unbounded “make it pretty”, open autonomy loops.
+
+### P2 — Unique (last)
+
+| ID | Work |
+|----|------|
+| P2.1 | Auditor as brand / free viral audit funnel |
+| P2.2 | Honest privacy/BYOK narrative (no stub NLP claims) |
+| P2.3 | Kill or ship façades (NLP MiniLM / ONNX upload) |
+
+---
+
+## 4. Formatting sandbox strategy (P1 detail)
+
+```
+NL → Pipeline (regex / macro / LLM)
+   → format_cells | layout tools | style recipes
+   → Apply/Reject (LLM) or single undo (safe regex)
+   → Store mutations (setCellFormat / setColumnWidth / setRowHeight)
+```
+
+Same executor spine. No new agent runtime.
+
+1. Wire `buildFormatPatch` + tool schema to full `CellFormat`  
+2. Add layout tools + ExecutionContext hooks  
+3. Sandbox API parity  
+4. Parser + false-positive corpus tests  
+5. Preset recipes via macro/template — preview bulk format  
+
+---
+
+## 5. Doc hierarchy
+
+| Layer | Role | Canonical files |
+|-------|------|-----------------|
+| **Living strategy** | What we build next | **This file** |
+| **Product truth** | Public claims | `README.md`, landing — must match code |
+| **Ops** | Deploy / env | `DEPLOY.md`, `ENV.md` |
+| **Change log** | What landed | `MAJOR_CHANGES.md` |
+| **Historical** | Context only | `docs/ARCHIVE.md` |
+| **Planning specs** | Architecture intent | `docs/planning/*` — verify against `src/` |
+
+---
+
+## 6. Explicit non-goals
+
+- Cursor-like unbounded agent loop  
+- More LLM backends before trust is green  
+- Collaboration / marketplace before core trust  
+- Expanding uniqueness narrative while core job fails  
+
+---
+
+## Related
+
+- Planning priorities: [`docs/planning/README.md`](../planning/README.md)  
+- Formatter sketch: [`docs/planning/11-tools-formatter.md`](../planning/11-tools-formatter.md)  
+- Engine gaps: [`docs/formualizer-gaps.md`](../formualizer-gaps.md)  
+- Archived reviews: [`docs/ARCHIVE.md`](../ARCHIVE.md)

src/agent/toolHandlers/columnOps.test.ts

@@ -0,0 +1,94 @@
+import { describe, it, expect, vi } from 'vitest'
+import { createEmptySheet, refToCell } from '@/engine/spreadsheet'
+import { detectApplyFormulaRangeGapRisk, handleApplyFormula } from './columnOps'
+import type { ExecutionContext } from '../executor'
+
+function makeCtx(sheet: ReturnType<typeof createEmptySheet>): ExecutionContext {
+  return {
+    getActiveSheet: () => sheet,
+    getSheets: () => [sheet],
+    getComputedValue: (row, col) => {
+      const cell = sheet.cells[refToCell(row, col)]
+      if (cell?.value == null) return ''
+      return String(cell.value)
+    },
+    setCellValue: (cellId, value, formula) => {
+      sheet.cells[cellId] = {
+        value: formula ? null : value,
+        formula,
+      }
+    },
+    setCellFormat: () => {},
+    bulkSetCells: () => {},
+    applySortPatch: () => {},
+    setFilters: () => {},
+    deleteRow: () => {},
+    insertRow: () => {},
+    addSheet: () => {},
+    renameSheet: () => {},
+    pushHistory: vi.fn(),
+  }
+}
+
+describe('detectApplyFormulaRangeGapRisk', () => {
+  it('flags SUM ranges that exclude an adjacent numeric cell', () => {
+    const sheet = createEmptySheet('S')
+    sheet.cells = {
+      A1: { value: 10 },
+      A2: { value: 20 },
+      A3: { value: 30 },
+      A4: { value: 40 },
+    }
+    const risk = detectApplyFormulaRangeGapRisk(
+      '=SUM(A1:A3)',
+      sheet,
+      (row, col) => String(sheet.cells[refToCell(row, col)]?.value ?? ''),
+      'A5',
+    )
+    expect(risk).toContain('A4')
+  })
+
+  it('returns null when the range covers adjacent data', () => {
+    const sheet = createEmptySheet('S')
+    sheet.cells = {
+      A1: { value: 10 },
+      A2: { value: 20 },
+      A3: { value: 30 },
+    }
+    const risk = detectApplyFormulaRangeGapRisk(
+      '=SUM(A1:A3)',
+      sheet,
+      () => '',
+      'A4',
+    )
+    expect(risk).toBeNull()
+  })
+})
+
+describe('handleApplyFormula range-gap safety', () => {
+  it('blocks gap-risk formulas unless confirmGaps is set', () => {
+    const sheet = createEmptySheet('S')
+    sheet.cells = {
+      A1: { value: 10 },
+      A2: { value: 20 },
+      A3: { value: 30 },
+      A4: { value: 40 },
+    }
+    const ctx = makeCtx(sheet)
+    const blocked = handleApplyFormula(
+      { cell: 'A5', formula: '=SUM(A1:A3)' },
+      ctx,
+      sheet,
+    )
+    expect(blocked.success).toBe(false)
+    expect(blocked.message).toMatch(/range gap risk/i)
+
+    const forced = handleApplyFormula(
+      { cell: 'A5', formula: '=SUM(A1:A3)', confirmGaps: true },
+      ctx,
+      sheet,
+    )
+    expect(forced.success).toBe(true)
+    expect(sheet.cells.A5?.formula).toBe('=SUM(A1:A3)')
+  })
+})